Object detection (OD) is a subfield of computer vision that aims to identify and locate objects in a given image. Object detection goes through the classic story of many tasks where ML has been applied, that is to say, OD is separated into pre-neural network based approaches (which were, generally speaking, included pretty cumbersome pipelines and manual feature creation) and convolutional neural network-based approaches, the latter of which was facilitated by the hardware advancements over the years.
In this topic, we take a look at the historical foundations of object detection, review the details of some OD models, discuss the common terminology of the field, and look at the ways to evaluate an object detection model.
Intro to the object detection task
Object detection includes two main tasks: identifying an object of specific pre-determined class and finding the specific spatial location of the object.
Object detection models typically output a vector that contains a probability that there is an object present, the associated class probabilities, and the spatial data of the bounding box that contains the object (e.g., a four element tuple where and are the height and the width of the bounding box in some format, and and are the coordinates for the center of the bounding box). It's rather clear where OD differs from regular classification, the outputs of which only contain the probabilities that an image corresponds to some specific class.
One of the first and straigh-forward approaches to object detection is to use sliding windows and go through all possible locations and scales of an image to see if a specific window a) contains an object, and, in case there is an object b) which object is it specifically (which is what one of the early approaches, the Viola-Jones algorithm, essentially does). This might look great on paper, but the task becomes too computationally heavy and inefficient.
Next, HOG (histogram of oriented gradients) detector feature descriptor was introduced. The main idea of HOG is that the appearance and the shape of the object can be inferred from the distribution of edge directions (intensity gradients). HOG mainly improved scale invariance (different sizes of objects shouldn't have impact on the detection results) and shape contexts (identifying objects in an image regardless of their location, scale, or rotation). DPM (Deformable Part-based model) is an extension of HOG that considers training as a way to decompose an object, and inference can be though of as an ensemble of detections on different object parts.
The timeline for object detection methods is outlined below:
In the next section, we will take a closer look at the deep learning-based methods for OD.
DL-based object detection
Starting from RCNN in 2014, convolutional networks have been widely applied to OD. Initially, RCNN draws out a number of object proposals (object candidate boxes) using selective search. Every proposal is then resized to an image with fixed dimensions and is input into a CNN model that has been pre-trained on ImageNet (e.g., AlexNet) to derive features. Linear SVM classifiers are then employed to estimate the existence of an object in each region and to distinguish between object categories. The issue with RCNN is that there are too many overlapped proposals (over 2000 boxes from a single image), which makes inference very slow. There have been multiple improvements on the initial model (e.g., SPPNet, Fast and Faster RCNNs, etc, but they drew on the ideas of the original RCNN)
In 2015, YOLO (You Only Look Once) is introduced, bringing forward the idea of reframing the task as regression instead of classification and becoming one of the first one-shot detection models. One-shot detection refers to a setting where inference is done in a single step from the input to the output. Two-stage detection, in turn, performs various tasks in a pipeline, in a "coarse-to-fine" approach. YOLO models are fast and well-suited for deployment. The initial version had limitations with localization, which were addressed in the later iterations of the model.
SSD, or Single Shot MultiBox Detector, was also introduced in 2015. The innovation of SSD lies in the introduction of multi-reference and multi-resolution detection techniques. The introduction of these techniques has improved the accuracy of one-stage detectors, especially when detecting smaller objects. Unlike previous detectors that only run detection on the top layers, SSD detects objects of varying scales on different network layers.
Some of the most recent models incorporate attention mechanism instead of convolution operations (e.g., DETR), but this is outside of the scope of this topic.
Benchmark datasets and evaluation
In object detection, primarily two aspects matter: the inference speed (due to wide applicability in real world tasks) and the detection accuracy (which includes the classification accuracy and the localization accuracy). mAP of detection is one of the most popular metrics to evaluate the detection accuracy. mAP, or Mean Average Precision, is defined as the average precision of detection at varying recall levels, evaluated in a category-specific manner. The average mAP across all categories is commonly used as the measure of performance. Object localization accuracy corresponds to calculating the IoU between the predicted and actual box, determining it as "detected" if it is above a certain threshold, typically 0.5, or "missed" if below this threshold. Typically, the AP is averaged over multiple IoU thresholds in the [0.5, 0.95] range, which encourages more accurate object localization
The datasets are mostly competition-based, MS-COCO, ILSVRC, and Pascal VOC being some of the most popular ones.
Conclusion
As a result, you are now familiar with the following:
- Object detection is the task of identifying and locating objects in the image;
- Object detection can be split into two modes: one-shot and two-shot, the former of which is usually faster because the inference happens from a single input to the output;
- Main evaluation metric for the OD task is Mean Average Precision.