Integer overflow in libaom internal function img_alloc_helper can lead to heap buffer overflow. This function can be reached via 3 callers:

• Calling aom_img_alloc() with a large value of the d_w, d_h, or align parameter may result in integer overflows in the calculations of buffer sizes and offsets and some fields of the returned aom_image_t struct may be invalid.
• Calling aom_img_wrap() with a large value of the d_w, d_h, or align parameter may result in integer overflows in the calculations of buffer sizes and offsets and some fields of the returned aom_image_t struct may be invalid.
• Calling aom_img_alloc_with_border() with a large value of the d_w, d_h, align, size_align, or border parameter may result in integer overflows in the calculations of buffer sizes and offsets and some fields of the returned aom_image_t struct may be invalid.

References

• https://nvd.nist.gov/vuln/detail/CVE-2024-5171
• https://issues.chromium.org/issues/332382766
• https://lists.fedoraproject.org/archives/list/[email protected]/message/6HYUEHZ35ZPY2EONVZCGO6LPT3AMLZCP
• https://lists.fedoraproject.org/archives/list/[email protected]/message/U5NRNCEYS246CYGOR32MF7OGKWOWER22