Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add option for generation loss testing in benchmark_xl #3547

Merged
merged 1 commit into from May 13, 2024

Conversation

jonsneyers
Copy link
Member

Adds an option --generations=N to do simple generation loss testing (same encode settings in each generation).

This makes it easier to evaluate the impact of changes in encoder heuristics in multi-generation scenarios (which are of course often an indication of a poor workflow on the user end, but it happens in practice nonetheless), as well as to improve any inaccuracies in the encoder or decoder implementation (e.g. in color conversions, implementation of transforms, etc) that could lead to cumulative errors or some kind of drift.

For example (and also to demonstrate that this is in fact an issue):

Normal single-generation:

benchmark_xl v0.10.2 99663f9 [NEON]
12 total threads, 5 tasks, 5 threads, 0 inner threads

005.png
Encoding                          kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
------------------------------------------------------------------------------------------------------------------------------------------------------------
jxl:d2                               1002   221010    1.7644664   2.749  50.339   2.79950030  70.88933231  33.78   1.23303327  2.175645741446   4.940      0
jxl:d1.77:gab0                       1002   219758    1.7544708   2.764  57.574   2.97714792  71.35176506  33.00   1.27466791  2.236367682335   5.223      0
jpeg:q78                             1002   361881    2.8891311  53.058 202.427   2.85090574  76.79926531  34.12   1.19507177  3.452718971749   8.237      0
jpeg:enc-jpegli:dec-jpegli:q85       1002   355152    2.8354091  32.260  34.470   2.11476969  78.18301538  36.04   0.91910006  2.606024669313   5.996      0
jxl:m:d2.5                           1002   227030    1.8125279   2.976  18.411   2.53045682  72.69585343  33.78   1.15891254  2.100561368278   4.587      0
Aggregate:                           1002   269330    2.1502326   8.271  51.783   2.63551298  73.92527446  34.13   1.14877734  2.470138491771   5.667      0

With --generations 5:

benchmark_xl v0.10.2 99663f9 [NEON]
12 total threads, 5 tasks, 5 threads, 0 inner threads
Generation loss testing with 5 intermediate generations

005.png
Encoding                          kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
------------------------------------------------------------------------------------------------------------------------------------------------------------
jxl:d2                               1002   214485    1.7123731   0.520   8.862   3.95412485  62.14050136  32.58   1.53259230  2.624369765655   6.771      0
jxl:d1.77:gab0                       1002   215034    1.7167561   0.534   9.886   3.43979791  64.56424948  32.08   1.50666937  2.586583804546   5.905      0
jpeg:q78                             1002   361679    2.8875184  13.716  37.776   3.41004999  76.25992557  34.04   1.23511902  3.566428843701   9.847      0
jpeg:enc-jpegli:dec-jpegli:q85       1002   310548    2.4793064   7.740  11.760   2.93604929  74.84523960  34.37   1.10255470  2.733570892787   7.279      0
jxl:m:d2.5                           1002   226926    1.8116976   0.586   3.341   3.64369397  70.12766239  33.67   1.38157288  2.502992335828   6.601      0
Aggregate:                           1002   259447    2.0713311   1.768  10.539   3.46042832  69.36547371  33.34   1.34148392  2.778657387286   7.168      0

With --generations 50:

benchmark_xl v0.10.2 99663f9 [NEON]
12 total threads, 5 tasks, 5 threads, 0 inner threads
Generation loss testing with 50 intermediate generations

005.png
Encoding                          kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
------------------------------------------------------------------------------------------------------------------------------------------------------------
jxl:d2                               1002   213410    1.7037906   0.063   1.063  16.50243217  23.38606784  31.23   4.10600005  6.995764446757  28.117      0
jxl:d1.77:gab0                       1002   215272    1.7186562   0.065   1.212  12.57214284  34.68192601  31.27   3.78103534  6.498299808125  21.607      0
jpeg:q78                             1002   361678    2.8875104   1.943   4.918  11.59559323  75.63766863  34.00   2.22209538  6.416323470545  33.482      0
jpeg:enc-jpegli:dec-jpegli:q85       1002   310410    2.4782046   1.017   1.427  11.72629471  73.29835767  34.31   2.36428681  5.859186544559  29.060      0
jxl:m:d2.5                           1002   226941    1.8118174   0.072   0.400   9.45330316  55.83525179  33.11   2.99474382  5.425928953456  17.128      0
Aggregate:                           1002   259224    2.0695507   0.225   1.294  12.16743193  47.85862907  32.76   3.00310450  6.215077093530  25.181      0

@jonsneyers jonsneyers added the testing Related to testing / avoiding regressions label Apr 30, 2024
@jonsneyers jonsneyers requested review from veluca93 and mo271 May 11, 2024 16:12
@jonsneyers jonsneyers added this pull request to the merge queue May 13, 2024
Merged via the queue into libjxl:main with commit d9a91bd May 13, 2024
101 of 116 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Related to testing / avoiding regressions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants