Skip to content

DL_W10.md

Latest commit

 

History

History
409 lines (275 loc) · 20.3 KB

DL_W10.md

File metadata and controls

409 lines (275 loc) · 20.3 KB

Object Detection Algorithm: YOLO (You Only Look Once)

YOLO, short for “You Only Look Once,” is one of the most revolutionary and widely adopted object detection algorithms in modern computer vision. It was introduced by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi in 2016. Unlike older object detection frameworks (like R-CNN and Fast R-CNN), which required multiple stages for region proposal and classification, YOLO treats detection as a single regression problem, predicting both object classes and bounding box coordinates in one forward pass through the network.

1. Background and Motivation

Before YOLO, object detection pipelines (like R-CNN, Fast R-CNN, and Faster R-CNN) followed a multi-stage process:

  1. Region Proposal – Using algorithms like Selective Search to identify potential object regions.
  2. Feature Extraction – Feeding each region into a CNN (like VGG or ResNet) to extract features.
  3. Classification + Regression – Using fully connected layers or SVMs to classify objects and refine bounding boxes.

While accurate, this approach was computationally expensive, since each image required hundreds or thousands of CNN passes, making real-time detection nearly impossible.

YOLO was designed to solve this:

  • It performs detection in a single step, without separate proposal or refinement stages.
  • It achieves real-time detection while maintaining strong accuracy.
  • It reframes detection as a single regression problem from image pixels directly to bounding box coordinates and class probabilities.

Benefits:

  • Extremely fast inference (real-time, high fps).
  • Simple end-to-end training.
  • Global reasoning (each prediction sees the entire image context).

Tradeoff: earlier YOLOs traded some localization accuracy (especially small objects) for speed. Later versions closed the gap.

2. YOLO’s Core Idea: Detection as Regression

The YOLO algorithm divides an image into an S × S grid (for example, 7×7 in YOLOv1). Each grid cell is responsible for detecting objects whose centers fall inside that cell.

Each grid cell predicts:

  • B bounding boxes (e.g., 2 or 3 boxes per cell)

  • For each box:

    • The (x, y) coordinates of the box center (relative to the grid cell)
    • The width (w) and height (h) of the box (relative to the image size)
    • The objectness(confidence) score, representing the probability that an object is present in the box

  • Class probabilities for all object categories (e.g., person, dog, car, etc.)

The final confidence score for each box = P(object) × P(class | object) This tells both what the object is and how sure the model is that it exists.

3. Core architecture (backbone, neck, head) — what each part does

A typical YOLO-style model has three logical blocks:

  1. Backbone (feature extractor)

    • A convolutional classification network (Darknet/ResNet/CSP/EfficientNet) pretrained on ImageNet for feature extraction.
    • Produces hierarchical feature maps at different resolutions.

  2. Neck (feature aggregation)

    • FPN / PANet / SPP modules that fuse multi-scale features so the detector can see both high-resolution localization cues and deep semantics.
    • SPP (spatial pyramid pooling) pools at multiple scales to increase receptive field and add global context.

  3. Head (detection layers)

    • Dense prediction layers applied to several scales (feature maps) that output, at each spatial position, predictions for:

      • Box regression (dx, dy, dw, dh) relative to anchor/prior
      • Objectness score (confidence)
      • Class probabilities

    • Predictions per grid cell: typically B boxes × (5 + C) values, where 5 = (tx, ty, tw, th, obj).

Modern YOLO heads predict at three scales (small, medium, large) to cover small→large objects.

4. YOLO Architecture Overview

The original YOLOv1 architecture was inspired by GoogLeNet (Inception architecture). It has:

  • 24 convolutional layers for feature extraction
  • 2 fully connected layers for prediction

Input:

  • Image of size 448×448×3

Output:

  • A tensor of size S × S × (B×5 + C)

    • S = grid size (7×7)
    • B = number of bounding boxes per cell (usually 2)
    • 5 = (x, y, w, h, confidence)
    • C = number of classes (e.g., 20 for Pascal VOC dataset)

So for S=7, B=2, and C=20, the output tensor is: 7 × 7 × (2×5 + 20) = 7 × 7 × 30

5. Mathematical Formulation

Each grid cell predicts:

  • $P_{\text{obj}}$: probability of object presence
  • $(x, y)$: coordinates of the box center relative to the grid cell
  • $(w, h)$: width and height relative to the full image
  • $P(\text{class}_i \mid \text{object})$: conditional class probabilities

The confidence score is defined as:

$$ \text{Confidence} = P_{\text{obj}} \times \text{IoU}_{\text{pred}}^{\text{truth}} $$

Where IoU (Intersection over Union) measures how closely the predicted box matches the ground truth box.

The loss function is a sum-squared error that combines:

  1. Localization loss: difference between predicted and true bounding box coordinates
  2. Confidence loss: difference between predicted and true objectness (confidence) score
  3. Classification loss: difference between predicted and true class probabilities

A simplified representation:

$$ \text{Loss} = \lambda_{\text{coord}} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbf{1}_{ij}^{\text{obj}} \big[(x_i - \hat{x}_i)^2 + (y_i - \hat{y}_i)^2\big] + \ldots $$

Where $\mathbf{1}_{ij}^{\text{obj}} = 1$ if an object appears in cell i and box j is responsible for its detection.

6. Loss Function — Multi-Task Design

YOLO uses a multi-part loss combining localization, confidence/objectness, and classification terms. A canonical form (from YOLOv1) is:

$$ \begin{aligned} L = & \ \lambda_{\text{coord}} \sum_{i=1}^{S^2}\sum_{j=1}^{B} \mathbf{1}_{ij}^{\text{obj}} \Big[(x_i - \hat{x}_i)^2 + (y_i - \hat{y}_i)^2\Big] [6pt] & + \lambda_{\text{coord}} \sum_{i=1}^{S^2}\sum_{j=1}^{B} \mathbf{1}_{ij}^{\text{obj}} \Big[(\sqrt{w_i} - \sqrt{\hat{w}_i})^2 + (\sqrt{h_i} - \sqrt{\hat{h}_i})^2\Big] [6pt] & + \sum_{i=1}^{S^2}\sum_{j=1}^{B} \mathbf{1}_{ij}^{\text{obj}} (C_i - \hat{C}_i)^2 [6pt] & + \lambda_{\text{noobj}} \sum_{i=1}^{S^2}\sum_{j=1}^{B} \mathbf{1}_{ij}^{\text{noobj}} (C_i - \hat{C}_i)^2 [6pt] & + \sum_{i=1}^{S^2} \mathbf{1}_{i}^{\text{obj}} \sum_{c \in C} (p_i(c) - \hat{p}_i(c))^2 \end{aligned} $$

Where:

  • $\mathbf{1}_{ij}^{\text{obj}}$ indicates the j-th box predictor responsible for an object in grid cell i.
  • $\lambda_{\text{coord}}$ and $\lambda_{\text{noobj}}$ balance localization vs. confidence/no-object penalties.
  • $C_i$ and $\hat{C}_i$ are predicted and target confidence scores.
  • $p_i(c)$ and $\hat{p}_i(c)$ are predicted and true class probabilities.

Modern YOLO versions replace MSE-based terms with improved alternatives:

  • Localization: CIoU / GIoU / DIoU loss (IoU-optimized)
  • Classification: Binary cross-entropy (BCE) or focal loss for class imbalance
  • Objectness: BCE or focal loss

7. Matching strategy and label assignment

Key training detail: which predicted anchor is responsible for which GT object?

  • IoU-based assignment: For each ground truth, the anchor with highest IoU (after scaling to grid cell) is matched as positive. Others are negatives unless IoU > some threshold.
  • Ignore or neutral anchors: Some anchors with medium IoU are ignored (no gradient) to reduce confusion.
  • Auto-anchor: modern implementations compute k-means anchor clustering on training boxes to create anchors that match dataset scale/aspect ratio distribution.

Anchor-free variants (some later YOLO variants) remove anchors and instead predict center/size directly.

8. Postprocessing — Non-Maximum Suppression (NMS) and variants

After prediction, multiple boxes may overlap the same object. To remove redundant detections, Non-Maximum Suppression (NMS) is applied:

  1. Sort boxes by confidence score.
  2. Select the box with the highest score.
  3. Remove boxes with IOU > threshold (e.g., 0.5).
  4. Repeat until no boxes remain.

NMS ensures that only the most confident bounding box per object remains.

In other words, After decoding boxes and scores:

  1. Filter by objectness(confidence) threshold.

  2. Apply per-class NMS: sort by score, remove boxes with IoU > threshold against kept boxes.

  3. Variants:

    • Soft-NMS: decay scores instead of hard removing — preserves detections in crowded scenarios.
    • DIoU-NMS / CIoU-NMS: incorporate IoU variants for suppression.
    • Weighted box fusion: combine overlapping boxes from multiple models or augmentation.

Proper NMS tuning (IoU threshold) is critical for crowded scenes.

9. Training recipes, data augmentation and stabilization tricks

YOLOs are known for a practical set of augmentations and tricks that boost accuracy while preserving speed:

  • Mosaic augmentation: Combine 4 images into one during training (random scaling/cropping/arrangement). Improves small object detection, diversity, batch variance.
  • MixUp: Blend two images and labels.
  • Multi-scale training: Randomly resize input (e.g., 320–608) every few batches to make detector robust to resolution.
  • Label smoothing: Avoids overconfidence by softening class labels.
  • Warmup LR: Start with very small LR, ramp up to stabilize early training.
  • Cosine annealing / step decay: common LR schedules.
  • Batch normalization and weight initialization: critical for stability.
  • Mish / SiLU activations: smoother activations used in some versions to improve gradient flow.
  • Class balancing and focal loss: to handle heavy class imbalance.

10. Practical hyperparameters & tips

  • Input size: influences both speed and accuracy. 416×416 or 640×640 are common defaults.
  • Anchors: choose by k-means on labels; often 3 anchors per scale (3 scales → 9 anchors).
  • Batch size & SyncBN: small per-GPU batch sizes may require SyncBatchNorm for stable BN stats.
  • IoU threshold for positive: typical values ~0.5; two-stage networks use higher thresholds for positives in later stages.
  • λ terms in loss: tune to balance localization vs classification; defaults in original YOLO were set experimentally.
  • Gradient clipping: sometimes used to avoid divergence in early training.

11. Strengths, limitations & failure modes

Strengths of YOLO

  • Real-Time Speed: YOLO is exceptionally fast, capable of processing video streams in real-time (e.g., over 100 FPS on GPUs), which makes it ideal for applications in robotics, autonomous vehicles, and live video analytics.
  • Unified and Simple Architecture: It treats object detection as a single regression problem, using one network in a single forward pass. This unified architecture eliminates separate proposal stages and simplifies the end-to-end training process.
  • Global Context Awareness: Unlike methods that examine only proposed regions, YOLO processes the entire image at once. This global reasoning helps the model learn contextual information, reducing background false positives.
  • Scalability and Strong Ecosystem: The YOLO family includes models of various sizes (e.g., "tiny" or "nano" versions) that work efficiently on mobile and embedded systems It also benefits from a strong engineering ecosystem with many pre-trained models and easy-to-use export options for deployment.

Limitations and Failure Modes of YOLO

  • Difficulty with Small or Crowded Objects: YOLO historically struggles with accurate localization, especially for very small objects or objects that are close together and overlapping. While techniques like multi-scale prediction in later versions have mitigated this, it can still be a challenge. Non-Maximum Suppression (NMS) can also incorrectly suppress true positives in crowded scenes.
  • Anchor Box Sensitivity: The performance can be sensitive to the choice of anchor boxes, which may require specific tuning for datasets with unusual object shapes or aspect ratios. Anchor-free variants have been developed to address this limitation.
  • Grid-Based Detection Limits: The core mechanism relies on a grid, where each cell is responsible for detecting objects whose centers fall within it. This can lead to missed detections if multiple small objects have their centers in the same grid cell.
  • Speed vs. Accuracy Tradeoff: While YOLO is known for speed, there is an inherent tradeoff. Larger, more complex versions of the model (e.g., YOLOv8-x) achieve higher accuracy but are slower, while the smaller, "nano" versions are extremely fast but less accurate.

12. YOLO Versions and Evolution

YOLO has evolved through multiple versions, each improving accuracy, speed, and architecture design.

YOLOv1 (2016):

  • Introduced the one-shot detection paradigm.
  • Used fully connected layers for bounding box prediction.
  • Struggled with small objects and localization precision.

YOLOv2 / YOLO9000 (2017):

  • Introduced Anchor Boxes for better localization.
  • Added Batch Normalization, improving convergence and stability.
  • Trained on combined COCO + ImageNet, detecting over 9000 classes.
  • Increased input size to 416×416.

YOLOv3 (2018):

  • Used Darknet-53 backbone (deeper and residual connections like ResNet).
  • Multi-scale detection using Feature Pyramid Networks (FPN).
  • Predicted boxes at three different scales, helping detect both large and small objects.
  • Used logistic regression for objectness(confidence) and binary cross-entropy loss for class prediction.

YOLOv4 (2020):

  • Introduced CSPDarknet53 backbone for higher efficiency.
  • Used new techniques: Mish activation, DropBlock, Cross Stage Partial connections, Mosaic data augmentation.
  • Enhanced inference speed on GPUs.

YOLOv5 (2020–2021):

  • Implemented in PyTorch (unlike previous Darknet-based versions).
  • Highly modular and optimized for production.
  • Supports mixed precision training, auto-learning anchors, and advanced data augmentation (like CutMix and Mosaic).

YOLOv6–YOLOv9 (2022–2025):

  • Continuous improvement focusing on real-time edge performance, transformer-based attention layers, and lightweight architectures.
  • YOLOv8 and YOLOv9 integrate Vision Transformers (ViTs) and Decoupled Heads for better localization and classification separation.
  • Achieve state-of-the-art mAP with real-time inference speeds on GPUs and edge devices.

13. YOLO vs other detectors (conceptual comparison)

  • Vs Two-stage (Faster R-CNN): YOLO is faster, simpler; two-stage often more accurate for heavy localization, small objects, and scientific accuracy needs.
  • Vs other one-stage (SSD / RetinaNet): YOLO emphasizes speed and practical augmentations (mosaic); RetinaNet introduced focal loss to address imbalance — YOLO variants usually adopt similar losses.
  • Vs Anchor-free detectors: Anchor-free reduces hyperparameter tuning; many modern YOLO implementations support anchor-free heads.
Aspect YOLO R-CNN/Faster R-CNN SSD
Speed Very high (real-time) Slow High
Accuracy Slightly less than two-stage Very high High
Architecture Single-stage, end-to-end Two-stage (proposal + classify) Single-stage
Localization Moderate Excellent Good
Best For Real-time apps Accuracy-critical tasks Mobile efficiency

14. Practical deployment & optimization

YOLO models are widely deployed on edge devices and servers. Typical deployment considerations:

  • Model scaling: choose size (nano, small, medium, large) depending on latency vs accuracy needs.
  • Quantization & pruning: INT8 quantization and pruning reduce memory/compute with small accuracy drop.
  • TensorRT / ONNX / OpenVINO: convert to optimized runtime for CPU/GPU/embedded devices.
  • Batching and NMS on GPU: speed up inference pipelines; GPU NMS or batched NMS implementations exist.
  • TTA & ensembling: improves accuracy but increases latency — used for competitions/benchmarks.

15. Advanced variants & research directions

  • CSP (Cross Stage Partial) networks: reduce computation while preserving accuracy.
  • BiFPN (Weighted feature fusion): efficient multi-scale fusion (in EfficientDet).
  • CenterNet & keypoint-based YOLOs: detect centers & sizes anchor-free.
  • Transformer-based detectors (DETR-like or hybrid): use attention for global reasoning; inspire new YOLO hybrids.
  • Self-supervised pretraining / semi-supervised detection: reduce labeled data needs.
  • Losses improving IoU: GIoU / DIoU / CIoU improve localization performance.

16. Example numeric flow — one grid cell prediction (illustrative)

Suppose input 416×416, S=13 (feature map 13×13), anchor pw=116, ph=90 on this scale. For grid cell (i=5,j=7) network outputs:

  • t_x= -0.4, t_y = 0.8 → sigmoid(t_x)=0.401, sigmoid(t_y)=0.689.
  • b_x = (c_x + 0.401)/S, b_y = (c_y + 0.689)/S → normalized center.
  • t_w=0.2 → b_w = pw * exp(0.2)/416.
  • Objectness(confidence) p_obj = sigmoid(t_obj) gives probability of object in box.
  • Class probabilities p_c = softmax (or sigmoid for multi-label).
  • Final score = p_obj * p_c[class].

After decoding all boxes, apply NMS per class.

17. Evaluation metrics for YOLO

  • mAP@0.50 (VOC style) and mAP@[.50:.95] (COCO style) are standard.
  • Report AP by object size (small/medium/large) to understand scale sensitivity.
  • Also measure fps, model size (MB), GFLOPs, and latency on target device for deployment tradeoffs.

18. Checklist for training a YOLO model on a new dataset

  1. Prepare dataset in COCO/PASCAL format; generate anchors via k-means.
  2. Choose model scale (s/m/l) balancing speed and accuracy.
  3. Pretrain backbone if possible; use transfer learning.
  4. Enable augmentations: mosaic, flip, color jitter, random scale.
  5. Use warmup LR, good optimizer (SGD+momentum or AdamW), and LR schedule.
  6. Monitor mAP and per-class AP; tweak anchors, augmentations, and loss terms as needed.
  7. Export to optimized runtime (ONNX/TensorRT) and run quantization for deployment.

19. Real-World Applications

  • Autonomous Driving: Pedestrian and vehicle detection.
  • Security Systems: Real-time surveillance and anomaly detection.
  • Retail Analytics: Customer movement tracking and product recognition.
  • Medical Imaging: Tumor or organ detection.
  • Robotics and Drones: Real-time object navigation and obstacle detection.

20. Final remarks

YOLO’s impact is more than a single algorithm — it’s an engineering philosophy: fast, simple, end-to-end, and practical. Over years the family has steadily improved accuracy without abandoning speed. For real-world systems (drones, robotics, video analytics, mobile apps), YOLO variants remain a top choice because they strike an excellent balance between accuracy, latency, and implementation simplicity.

YOLO redefined object detection by introducing the “single-shot” detection paradigm, making real-time detection practical for real-world applications. Over time, with innovations like anchor boxes, multi-scale detection, advanced loss functions, and transformer-based enhancements, YOLO evolved into a family of models balancing speed, accuracy, and efficiency.

Modern YOLO variants (v8, v9) now power applications from autonomous vehicles to industrial vision systems, solidifying YOLO’s role as one of the most impactful advancements in computer vision and deep learning.