[bug] ColorCheckerDetection fails unpredictably after successful detection

[mikeedwards]

Describe the bug
When processing exr frames contained a color checker/Macbeth, sometimes the ColorCheckerDetection node will fail suddenly after reporting a successful detection. This happens on a small minority of frames (perhaps 8 out of 163 in a recent project). Deleting the frames that were reported as successful right before the crash resolves the problem, but no clear pattern has emerged for what created the failure.

To Reproduce
Steps to reproduce the behavior:

1. Create a pipeline with CameraInit and ColorCheckerDetection (SfmData -> Input)
2. Feed a large number of frames with a Macbeth in view into the init and compute the ColorCheckerDetection
3. See the log

Expected behavior
ColorCheckerDetection succeeds or it fails and reports an error message

Screenshots

Log

[2024-06-06 10:54:27.978155] [0x0000381c] [trace]   Embedded OCIO configuration file: 'C:\________\Meshroom-2023.3.0-win64\Meshroom-2023.3.0\aliceVision/share/aliceVision/config.ocio' found.
Program called with the following parameters:
 * debug = 0
 * input = "d:/projects/assets/scenes/________/meshroom/MeshroomCache/CameraInit/2bf4b2970ddb3de832a92107ad2dfc3df4bcf046/cameraInit.sfm"
 * maxCoresAvailable =  Unknown Type "unsigned int" (default)
 * maxCount =  Unknown Type "unsigned int"
 * maxMemoryAvailable = 18446744073709551615 (default)
 * outputData = "d:/projects/assets/scenes/________/meshroom/MeshroomCache/ColorCheckerDetection/b73f29d79ef13e6aba61b6dfcc7c578af3a1ded4/ccheckers.json"
 * verboseLevel = "trace"

Hardware : 
	Detected core count : 20
	OpenMP will use 20 cores
	Detected available memory : 39443 Mo

[10:54:27.980456][info] 1/2 - Process image at: 'D:/projects/assets/scenes/________/images/frame_0067.exr'.
[10:54:27.980456][debug] [IO] Read Image: D:/projects/assets/scenes/________/images/frame_0067.exr
[10:54:28.079709][trace] Read image D:/projects/assets/scenes/________/images/frame_0067.exr (encoded in Linear colorspace).
[10:54:28.794865][info] Checker not detected in image at: 'D:/projects/assets/scenes/________/images/frame_0067.exr'
[10:54:28.803053][info] 2/2 - Process image at: 'D:/projects/assets/scenes/________/images/frame_0068.exr'.
[10:54:28.803053][debug] [IO] Read Image: D:/projects/assets/scenes/________/images/frame_0068.exr
[10:54:28.905189][trace] Read image D:/projects/assets/scenes/________/images/frame_0068.exr (encoded in Linear colorspace).
[10:54:29.754674][info] Checker #1 successfully detected in 'frame_0068'

Desktop (please complete the following and other pertinent information):

• OS: win 11
• Python version: 3.11.9
• Meshroom version:
  • Binary version: 2023.3.0

Additional context
I have two exr files that fairly reliably demonstrate a success with no crash followed by a success with the crash, but they're huge (125 MB in a zip). For some reason, converting them to PNG does not reproduce the crash. If there's a preferred way to upload/distribute files for this, I'm happy to do that.

Oddly enough, if I just use one frame in the pipeline (the notorious frame_0068 above), I can only occasionally get it to fail (maybe half the time) by running Delete Data on the node and computing again and again. Other frames don't fail at all, though--only some "cursed" frames make this happen. Nothing is visually very different from problem frames and their neighbors, though.

When frame_0068 actually succeeds and doesn't crash, I get this additional line in the log:

...
[11:10:14.358898][info] 1/1 - Process image at: 'D:/projects/assets/scenes/________/images/frame_0068.exr'.
[11:10:14.358898][debug] [IO] Read Image: D:/projects/assets/scenes/________/images/frame_0068.exr
[11:10:14.424838][trace] Read image D:/projects/assets/scenes/________/images/frame_0068.exr (encoded in Linear colorspace).
[11:10:15.072248][info] Checker #1 successfully detected in 'frame_0068'
[11:10:15.183906][info] Found color checkers count: 1

which leads me to think there's something in the executable between the lines Checker #1 successfully detected in 'frame_0068' and Found color checkers count: 1 that is responsible. The inconsistency makes me think it's maybe a memory issue, but I'm stumped as to why only certain frames cause problems. Notorious "frame_0068" is a problem in both very small and very large sets of input images, so the total size of the input doesn't seem to be a factor.