Machine learning model performance is often limited by the surrounding data pipeline, not just the model's inference speed.
The pipeline consists of three main stages: object detection with YOLO, object tracking with Strong Sort, and individual identification using ear tags.
Processing thousands of hours of video footage at scale makes a simple real-time processing approach completely impractical.
The initial single-threaded implementation processed video in real-time, resulting in a projected total runtime of over 1.3 years.
Combining Python profiling with CPU, GPU, and memory utilization metrics is crucial for identifying the true performance bottlenecks.
Quick wins were achieved by adding GPU acceleration, streaming video from cloud storage, and removing non-production debugging code.
Introducing multithreading allows data fetching and decoding to happen in parallel with model inference, reducing GPU idle time.
A producer thread fills a bounded frame buffer which is then consumed by the main inference thread, creating a smooth and balanced data flow.
Multiple worker processes, each running its own video processing thread, are used to feed a shared GPU and maximize its utilization.
Using unique frame IDs and timestamps ensures that results remain correct even when videos are processed out of order by parallel workers.
The choice between H.264 and H.265 video codecs creates a trade-off between storage costs and the computational cost of decoding.
The fully optimized pipeline can process an hour of video in approximately 40 seconds by running 32 parallel workers on a 4-GPU machine.
After massively improving the detection stage, the tracking stage emerges as the new dominant cost, demonstrating the iterative nature of optimization.
The main lessons include using multithreading for I/O, multiprocessing for parallel work, and always letting measurement guide optimization efforts.
Jobs that call for the skills explored in this talk.