add option for generation loss testing in benchmark_xl #3547
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an option
--generations=N
to do simple generation loss testing (same encode settings in each generation).This makes it easier to evaluate the impact of changes in encoder heuristics in multi-generation scenarios (which are of course often an indication of a poor workflow on the user end, but it happens in practice nonetheless), as well as to improve any inaccuracies in the encoder or decoder implementation (e.g. in color conversions, implementation of transforms, etc) that could lead to cumulative errors or some kind of drift.
For example (and also to demonstrate that this is in fact an issue):
Normal single-generation:
benchmark_xl v0.10.2 99663f9 [NEON]
12 total threads, 5 tasks, 5 threads, 0 inner threads
With
--generations 5
:benchmark_xl v0.10.2 99663f9 [NEON]
12 total threads, 5 tasks, 5 threads, 0 inner threads
Generation loss testing with 5 intermediate generations
With
--generations 50
:benchmark_xl v0.10.2 99663f9 [NEON]
12 total threads, 5 tasks, 5 threads, 0 inner threads
Generation loss testing with 50 intermediate generations