Test cases sometimes timeout

[rossburton]

I did this

We install the test suite so that we can run it on target as part of integration testing. Occasionally, one of the tests will timeout. For an example of this, see https://autobuilder.yocto.io/pub/non-release/20240409-14/testresults/qemux86-64-ptest/curl.log.

This shows test 340 fails after waiting two minutes for a response from the server. I initially blamed a heavily loaded host machine (as the tests are running inside a qemu-system) but the test execution started at 12:51, test 340 started at 12:52, and times out two minutes later at 12:54. This suggests that the other tests are all running quickly and the problem is not system load.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15268 is our tracker bug for this issue, it can be seen that there's no hugely obvious pattern (to me, at least) to what tests fail. We also pass !flaky !timing-dependent to runtests.pl to filter out tests marked as such.

My hunch is that there's a subtle race in the test harness, which is only visible when running the tests frequently.

I expected the following

No response

curl/libcurl version

curl 8.7.1

operating system

OpenEmbedded

[egorpugin]

I see all my curl-using programs experience major delays/stucks after 8.6.0->8.7.1 upgrade on linux (win & mac seems ok).
I'll see if I can debug them a bit.

Upd.:
All programs hang with this trace and after 2-5 mins continue normally.

(gdb) bt
#0  0x00007f1d3e65023e in __GI___select (nfds=16, readfds=0x7ffe250f0310, writefds=0x7ffe250f0390, exceptfds=0x7ffe250f0410, timeout=0x7ffe250f0300) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007f1d3f261f88 in Curl_poll.part.0 () from badger.curl.libcurl-8.7.1.so
#2  0x00007f1d3f24f86a in multi_wait.part () from badger.curl.libcurl-8.7.1.so
#3  0x00007f1d3f24fd5a in curl_multi_poll () from badger.curl.libcurl-8.7.1.so
#4  0x00007f1d3f215e5e in curl_easy_perform () from badger.curl.libcurl-8.7.1.so
#5  0x00007f1d3f9aae84 in download_file(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::filesystem::__cxx11::path const&, long) ()

Build is kinda custom, maybe I have issues with it.

[bagder]

Without being able to reproduce this locally, this feels next to impossible to fix.

[rossburton]

Absolutely agree. I guess this was more a a cry of has anyone else seen this before, or have any hunches.

I'll try building curl on my lowest machine and seeing if I can make it replicate there.