A Deep Learning–Based Framework for Robust Real-Time Multi-Object Detection and Tracking

Abstract

The paper presents a person recognition and multi-object tracking system that is compatible with real-world video and allows individuals to view the results immediately. It operates on the Ultralytics YOLO to identify individuals in each frame and can operate a variety of BoxMOT trackers including DeepOCSORT, BoostTrack, StrongSORT, Bo Tsort and ByteTrack to track those individuals. It also attracts the tracks of the people on the video. It features a small web interface that allows uploading videos, monitoring the progress in the percentage, and downloading the completed video with the added information using Flask. In order to play the videos on any browser, the system automatically converts the output to an MP4 file that is H.264 and AAC encoded.The system is reliable as it has numerous features. It eliminates small confining boxes (smaller than approximately 500 pixels) that may introduce errors. It is able to append optional Re-Identification (ReID) models to retain the identical ID of a person when out of sight. It is also able to export the trained models to ONNX, TFLite, CoreML, and TorchScript to enable the system to run on phones and edge devices. We also tested the system using a number of real videos in busy street of cities, security footage in the indoor and moderately moving scenes in the outdoors. It was tested on a regular computer with an NVIDIA RTX 4070 graphics card and Intel Core i7 -13700K processor.It took approximately 70-90 milliseconds to process each frame, 10-14 frames per second. The video resolution (720p to 1080p), tracker selected (ByteTrack was the fastest and DeepOCSORT was the most accurate), and the use of ReID were the primary determinants of the speed. Our tests established the system as correct in identifying 524 different people and tracking 486 of them, with a track-retention rate of more than 92 percent. The use of memory remained constant even in long runs. The export pipeline also demonstrated that the framework is efficient on mobile devices, with TFLite models running at more than 25 frames per second on mid-range Android devices.