-
-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
syncthing/lib/api tests unreliable, failing ~50% of the time on Fedora Linux #9455
Comments
These tests started to be executed in parallel in 1.23.7, that would match your experience of these timeouts occuring more frequently. It becoming worse in 1.27.4 is a bit weird, as there was a change to make it less flaky: Anyhow, as I can't repro myself, could you please increase the default test timeout here and report back if that fixes the issue for you, and if so with what timeout - thanks. |
Mhm, when running this test-case enough times it eventually gets triggered. So I assume that it is not unrealistic to expect it to happen more often on different machines. With I increased the default timeout to 5s and added a logger in case it took longer than 1s (the actual default). The results;
Feels like it gets executed in 1s batches where for any given reason it can get pushed to a next one. As the average execute-time is like 1-3 ms. P.s. unless my timer is faulty;) |
Ah, that might indeed explain what's going on. Our build machines are pretty big - one of the failures I see was on a system with 2 x 12 CPUs (48 threads) available ... I will try to increase the timeout duration. |
As suggested, I raised the timeout here (to 60 seconds, just to be on the safe side - 5 seconds would probably be fine too), and I can no longer reproduce the failure for the lib/api tests. So it looks like the current 1 second timeout is definitely too tight when running these tests in parallel on many CPU cores. |
What happened?
I'm the package maintainer for syncthing on Fedora Linux (and for EPEL 9 and 8). We're running tests during package builds, and one of the tests (for the
github.com/syncthing/syncthing/lib/api
module) has been very unstable and unreliable. This started happening a few releases ago, but it is much worse with v1.27.4 now.The errors I see most often happen on Linux x86_64. I have seen then occur with both Go 1.22 (Fedora Rawhide and Fedora 40) and Go 1.21 (Fedora 39). With v1.27.4, about half of my attempted builds fail with the log output included below (or output very similar to it).
I am building from the official "source" distribution that includes your vendored dependencies (i.e. not against Go packages that are packaged by the distribution), so there should be no mismatch between the sources and dependencies you use and the ones used for my builds.
Syncthing version
v1.27.4
Platform & operating system
Linux x86_64
Browser version
N/A
Relevant log output
The text was updated successfully, but these errors were encountered: