Computer Vision: Convolutions, Feature Pyramids, Object Detection & Vision Transformers
Computer Vision transforms raw discrete pixel matrices into high-level spatial and semantic understandings of physical reality. Explore the mathematical mechanics of 2D convolutions, receptive fields, real-time object detectors, and Vision Transformers (ViT).
1. The Mathematical Foundation: 2D Spatial Discrete Convolutions
Unlike fully connected layers that discard spatial geometry, a 2D convolution slides a parameterized kernel matrixK ∈ ℝ^(k_h × k_w) over input image tensor I to extract localized features:
Translation Equivariance
Shifting an object in the input image shifts the representation by the exact same amount in the feature map: f(g(x)) = g(f(x)).
Weight Sharing
The identical kernel weights are reused across the entire image space, reducing parameter counts from billions to thousands.
Hierarchical Receptive Field
Early layers detect low-level edges and Gabor-like gradients; deeper layers compose these into textures, object parts, and semantic categories.
2. Classification, Detection & Segmentation Architectures
| Task Level | Granularity | Core Benchmark Architectures | Evaluation Metric |
|---|---|---|---|
| Image Classification | Entire image assigned a single label | ResNet-50, EfficientNet, ConvNeXt | Top-1 / Top-5 Accuracy |
| Object Detection | Category label + Spatial 2D/3D Bounding Box | YOLOv8, Faster R-CNN, DETR (Transformer) | mAP@0.5, mAP@0.5:0.95 |
| Semantic Segmentation | Per-pixel classification without object instance identity | FCN, DeepLabV3+, SegFormer | Mean Intersection over Union (mIoU) |
| Instance Segmentation | Per-pixel mask delineating individual distinct objects | Mask R-CNN, Segment Anything Model (SAM) | Mask AP |
3. The Vision Transformer (ViT) Paradigm Shift
Introduced by Dosovitskiy et al. (2020), Vision Transformers apply pure standard Transformer encoders directly to visual data:
- An image
H × W × Cis partitioned into a sequence of non-overlapping 2D patches (e.g. 16×16 pixels). - Each patch is flattened into a 1D vector and linearly projected into a D-dimensional embedding space.
- A learnable
[CLS]token and 1D positional embeddings are prepended to the sequence. - Standard multi-head self-attention computes global pairwise relationships across all visual patches without inductive convolutional bias, outperforming CNNs at internet-scale pretraining.
