Evaluation of many trackers + FPS check + additional fixes

[Fleyderer]

I've added tracking/eval.py script for evaluation many detectors, reids, trackers at the same time with different FPS.

As a result of command:

python3 tracking/eval.py --yolo-models yolov8n.pt yolo11n.pt --reid-models osnet_x0_25_msmt17.pt --tracking-methods bytetrack deepocsort botsort --source ./tracking/val_utils/data/MOT17/train --verbose

You'll get results and plots of val.py for each combination under runs/eval. For example, plots are:

I've tried to add easy to understand progress bars and also fixed some problems while loading data etc.

[Fleyderer]

BTW, I also just returned from vacation and prepared a new PR 🙂

[mikel-brostrom]

Really nice!

{
    "yolov8n": {
        "osnet_x0_25_msmt17": {
            "bytetrack": {
                "original": {
                    "HOTA": 32.353,
                    "MOTA": 26.286,
                    "IDF1": 35.989
                },
                "5": {
                    "HOTA": 29.004,
                    "MOTA": 23.054,
                    "IDF1": 31.559
                },
                "10": {
                    "HOTA": 31.972,
                    "MOTA": 25.866,
                    "IDF1": 36.412
                },
                "15": {
                    "HOTA": 32.631,
                    "MOTA": 26.706,
                    "IDF1": 37.186
                },
                "20": {
                    "HOTA": 32.943,
                    "MOTA": 26.358,
                    "IDF1": 37.473
                }
            },
            "deepocsort": {
                "original": {
                    "HOTA": 29.931,
                    "MOTA": 23.195,
                    "IDF1": 33.956
                },
                "5": {
                    "HOTA": 25.282,
                    "MOTA": 17.378,
                    "IDF1": 25.458
                },
                "10": {
                    "HOTA": 28.084,
                    "MOTA": 21.197,
                    "IDF1": 30.739
                },
                "15": {
                    "HOTA": 31.02,
                    "MOTA": 23.993,
                    "IDF1": 35.447
                },
                "20": {
                    "HOTA": 28.885,
                    "MOTA": 22.551,
                    "IDF1": 32.295
                }
            },
            "botsort": {
                "original": {
                    "HOTA": 24.819,
                    "MOTA": 19.46,
                    "IDF1": 23.58
                },
                "5": {
                    "HOTA": 24.226,
                    "MOTA": 17.201,
                    "IDF1": 23.504
                },
                "10": {
                    "HOTA": 25.586,
                    "MOTA": 19.196,
                    "IDF1": 24.925
                },
                "15": {
                    "HOTA": 25.154,
                    "MOTA": 19.242,
                    "IDF1": 24.23
                },
                "20": {
                    "HOTA": 25.524,
                    "MOTA": 19.376,
                    "IDF1": 24.598
                }
            }
        }
    },
    "yolo11n": {
        "osnet_x0_25_msmt17": {
            "bytetrack": {
                "original": {
                    "HOTA": 30.292,
                    "MOTA": 22.719,
                    "IDF1": 32.475
                },
                "5": {
                    "HOTA": 26.048,
                    "MOTA": 19.9,
                    "IDF1": 27.156
                },
                "10": {
                    "HOTA": 30.705,
                    "MOTA": 22.225,
                    "IDF1": 33.701
                },
                "15": {
                    "HOTA": 31.448,
                    "MOTA": 22.997,
                    "IDF1": 34.726
                },
                "20": {
                    "HOTA": 30.927,
                    "MOTA": 22.844,
                    "IDF1": 33.815
                }
            },
            "deepocsort": {
                "original": {
                    "HOTA": 27.579,
                    "MOTA": 19.677,
                    "IDF1": 29.721
                },
                "5": {
                    "HOTA": 24.059,
                    "MOTA": 14.89,
                    "IDF1": 23.496
                },
                "10": {
                    "HOTA": 26.63,
                    "MOTA": 17.808,
                    "IDF1": 27.358
                },
                "15": {
                    "HOTA": 29.318,
                    "MOTA": 20.924,
                    "IDF1": 32.188
                },
                "20": {
                    "HOTA": 27.338,
                    "MOTA": 19.386,
                    "IDF1": 29.32
                }
            },
            "botsort": {
                "original": {
                    "HOTA": 22.59,
                    "MOTA": 16.261,
                    "IDF1": 19.823
                },
                "5": {
                    "HOTA": 21.834,
                    "MOTA": 14.395,
                    "IDF1": 20.21
                },
                "10": {
                    "HOTA": 22.991,
                    "MOTA": 15.964,
                    "IDF1": 20.827
                },
                "15": {
                    "HOTA": 23.031,
                    "MOTA": 16.203,
                    "IDF1": 20.563
                },
                "20": {
                    "HOTA": 23.415,
                    "MOTA": 16.315,
                    "IDF1": 20.939
                }
            }
        }
    }
}

Love the FPS feature. Maybe it should be an argument? Would help with experimentation

[Fleyderer]

Maybe it should be an argument?

I did this:

fps_parser = parser.add_mutually_exclusive_group(required=False)
fps_parser.add_argument('--fps-check', dest='fps_check', action='store_true',
                        help='Check dataset working on different FPS values.')
fps_parser.add_argument('--no-fps-check', dest='fps_check', action='store_false',
                        help='Do not check dataset working on different FPS values.')
parser.set_defaults(fps_check=True)

parser.add_argument('--fps-values', nargs='+',
                    default=FPS_DEFAULT_SET, type=str)

So, if you don't need to check FPS, you pass --no-fps-check, or if you want to use own values, you can do --fps-values 5 6 7 8 and we'll check 5 6 7 8 FPS instead of 5 10 15 20, which are default

UPD: Perhaps this should be described in more detail in the README.md, I didn't spend much time because I wasn't sure if you would merge it or not 🙂

[Fleyderer]

Hello! When are we going to merge these features? I have a discussion to start after this merge 🙂

[mikel-brostrom]

Sorry for my late response. I have been on a work trip this whole past week.
Ideally, this should be merged into val.py. Having a val.py and eval.py is confusing. But I can also see that this file will become massive.

[Fleyderer]

If you think it is better to merge into val.py, i see solution like:

1. Move current val.py into utils/val.py and move utils.py under utils folder too
2. Import functions from val.py into eval.py instead of running val.py script from eval
3. Rename eval.py to val.py and change docs. Or maybe dont rename, because eval sounds better for evaluating different detectors/trackers/reids on different datasets parts.

I'll start making these changes today, but tomorrow and until next week I won't be able to commit

[mikel-brostrom]

If you check the benchmark file, this functionality is already there, besides the FPS setting functionality. There may be a cleaner way of achieving this in python by combining:

python tracking/val.py generate_dets_embs --source ./assets/MOT17-mini/train --yolo-model yolov10n.pt --reid-model osnet_x0_25_msmt17.pt --imgsz 320
python tracking/val.py generate_mot_results  --yolo-model yolov10n.pt --reid-model osnet_x0_25_msmt17.pt --imgsz 320 --tracking-method $tracker

This is how the metrics table under README is generated for all the tracking methods. The FPS functionality needs to be added to generate_mot_results anyways. Not sure the current eval.py is the best way of achieving this. Seems very verbose to me.

[mikel-brostrom]

The current implementation makes it possible to parallelize all the tracking metrics generation after the detections and embeddings are generated.

[mikel-brostrom]

The detections and embeddings should be generated for all frames. But just the ones relevant to the FPS settings should be used. Then you can run everything parallelly. This execution should take the time it takes for the slowest tracker (which currently is strongsort at 7 minutes on MOT17, when not evaluating StrongSORT it should take a maximum of 3:30 minutes) to run.

[mikel-brostrom]

More thoughts. The prints will need to be controlled here, otherwise it will be just chaos when running the script.