> For the complete documentation index, see [llms.txt](https://panav.gitbook.io/robotics-handbook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://panav.gitbook.io/robotics-handbook/perception-and-computer-vision/ml-and-perception.md). # ML and Perception How classical computer vision and learned perception fit together in a modern robotics stack, and how to navigate this section - sensing, OpenCV, detection and tracking, vision foundation models, 3D p Modern robot perception spans **classical computer vision** (OpenCV, feature detection, multi-view geometry), **deep learning for vision** (object detection, segmentation, tracking), **3D perception** (NeRF, Gaussian Splatting, monocular depth), and **vision foundation models** (SAM 2, Grounding DINO, Depth Anything, DINOv2). This section covers all of them. These are not competing approaches so much as different layers of the same stack. Multi-view geometry, calibration, and epipolar constraints still underwrite anything that needs a verifiable, non-learned guarantee - camera calibration, structure-from-motion, the geometric back end of a SLAM system. Learned models take over wherever the input is too unstructured for a hand-built model to cover: detecting arbitrary objects, segmenting arbitrary scenes, estimating depth from a single image with no baseline at all. Vision foundation models are the newest layer on top of both - promptable, zero-shot models trained on internet-scale data that generalize to objects and scenes they were never explicitly trained on. Most production perception pipelines in 2026 blend all three: a learned or foundation-model front end feeding a classical geometric back end (a Kalman filter, a bundle adjuster, an occupancy grid) that knows how to fuse noisy estimates into something a planner can actually trust. For policy learning - behavior cloning, RL, foundation-model VLAs that turn perception into action - see the dedicated [Robot Learning](/robotics-handbook/robot-learning/robot-learning.md) section. This section stops at *perceiving* the world; that one starts at *acting* on it. ### In this section Start with sensing hardware if the topic is new to you, then classical CV, then the learned methods roughly in order of how unstructured the problem gets - detection and tracking, foundation vision models, 3D perception, and event cameras. [Example of a Vision Pipeline](/robotics-handbook/perception-and-computer-vision/example-of-a-vision-pipeline.md) ties several of these together into one worked system.
Cameras, Depth Sensors and LiDARCameras, Depth Sensors and LiDARgiphy-9.webp
Image Processing (OpenCV)Image Processing Basics (OpenCV)ball-tracking-animated-02.gif
Object Detection and TrackingObject Detection and Tracking216924389-9979e1e8-58b1-4b6c-99bf-c32d5f913c1a.gif
Foundation Vision Models (SAM, DINO, Depth Anything)Foundation Vision Models (SAM, DINO, Depth Anything)216924389-9979e1e8-58b1-4b6c-99bf-c32d5f913c1a.gif
3D Perception (NeRF, Gaussian Splatting)3D Perception (NeRF, Gaussian Splatting)giphy-9.webp
Event CamerasEvent Cameras0_MQtVJIoAqO9Txqc4.gif
Vision Pipeline (example)Example of a Vision Pipelinetim-and-eric-mind-blown-2-1.webp
Reinforcement Learning (basics) →Reinforcement Learning (classical foundations)0_MQtVJIoAqO9Txqc4.gif