Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix some flaky test failures on Windows #7453

Merged
merged 2 commits into from
Jun 28, 2024

Conversation

deivid-rodriguez
Copy link
Member

@deivid-rodriguez deivid-rodriguez commented Feb 2, 2024

What was the end-user or developer problem that led to this PR?

There's a longstanding flaky CI failure on Windows that reads like this:

4 processes for 172 specs, ~ 43 specs per process

Randomized with seed 26551


An error occurred in a `before(:suite)` hook.
Failure/Error: FileUtils.cp shipped_file, target_shipped_file, preserve: true

Errno::EACCES:
  Permission denied - read
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2280:in `copy_stream'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2280:in `block (2 levels) in copy_file'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2279:in `open'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2279:in `block in copy_file'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2278:in `open'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2278:in `copy_file'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:1078:in `copy_file'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:877:in `block in cp'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2469:in `block in fu_each_src_dest'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2485:in `fu_each_src_dest0'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:2467:in `fu_each_src_dest'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/3.3.0/fileutils.rb:876:in `cp'
# ./spec/support/builders.rb:432:in `block in _build'
# ./spec/support/builders.rb:426:in `each'
# ./spec/support/builders.rb:426:in `_build'
# ./spec/support/helpers.rb:280:in `with_built_bundler'
# ./spec/support/helpers.rb:260:in `block (2 levels) in system_gems'
# ./spec/support/helpers.rb:256:in `each'
# ./spec/support/helpers.rb:256:in `block in system_gems'
# ./spec/support/helpers.rb:289:in `block in with_gem_path_as'
# ./spec/support/helpers.rb:303:in `without_env_side_effects'
# ./spec/support/helpers.rb:284:in `with_gem_path_as'
# ./spec/support/helpers.rb:254:in `system_gems'
# ./spec/spec_helper.rb:91:in `block (2 levels) in <top (required)>'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:365:in `run'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2185:in `block in run_suite_hooks'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2183:in `each'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2183:in `run_suite_hooks'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2090:in `with_suite_hooks'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:116:in `block in run_specs'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:115:in `run_specs'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/bin/rspec:25:in `load'
# C:/hostedtoolcache/windows/Ruby/3.3.1/x64/bin/rspec:25:in `<main>'
*...*.......................................................................................................*.................*...............................................................................................................................................*...................................................................................................................................................................................................................................................................................................................................................................................................................................................*........................................**...............****.........*....................****.......***************.******.**.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*...........*............*.............*........**.*..**..*..***.**..*.*.*...............*......................................................................................................................................................................................................................................................................................................................................................*.......................................................................................................................................*............................**..............*........................................*...............*..................................................*.......*............*....................................*...........................................................*.......*.......*.......*.......................*................................*...*...................................................*.................*.......*......*....*...........*.......................*...............................................................................................................................................................................................................................................................................*............*......*...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

See for example https://github.com/rubygems/rubygems/actions/runs/9663003543/job/26654152938.

What is your fix for the problem, implemented in this PR?

My diagnosis is that when starting the parallelized test suite, multiple processes copy source files necessary to build bundler to its own folder in tmp/. On Windows, this can apparently result in permission errors, I guess because the file is in use by another process.

As a workaround, I changed building bundler in CI to happen just once at bin/rake spec:deps time, not on the fly when running specs.

I expect this to make this error not happen again.

Make sure the following tasks are checked

@deivid-rodriguez deivid-rodriguez force-pushed the deivid-rodriguez/more-windows-ci-fixes branch 6 times, most recently from 2f699e7 to a6e4de7 Compare February 6, 2024 17:27
@deivid-rodriguez deivid-rodriguez force-pushed the deivid-rodriguez/more-windows-ci-fixes branch 5 times, most recently from bd5b238 to 8a4771c Compare June 26, 2024 14:03
@deivid-rodriguez deivid-rodriguez changed the title Experimenting with more Windows CI fixes Fix some flaky test failures on Windows Jun 26, 2024
@deivid-rodriguez deivid-rodriguez marked this pull request as ready for review June 26, 2024 14:53
@deivid-rodriguez deivid-rodriguez force-pushed the deivid-rodriguez/more-windows-ci-fixes branch from 8a4771c to 2887285 Compare June 26, 2024 15:15
Instead, build it during setup when in CI.

This should avoid some Windows specific test failures when Bundler
copies the same files from multiple processes and runs into EACESS
errors.
@deivid-rodriguez deivid-rodriguez force-pushed the deivid-rodriguez/more-windows-ci-fixes branch from 2887285 to c194a1d Compare June 26, 2024 17:09
@deivid-rodriguez deivid-rodriguez merged commit f3d9cb0 into master Jun 28, 2024
82 checks passed
@deivid-rodriguez deivid-rodriguez deleted the deivid-rodriguez/more-windows-ci-fixes branch June 28, 2024 07:33
deivid-rodriguez added a commit that referenced this pull request Jul 5, 2024
…ci-fixes

Fix some flaky test failures on Windows

(cherry picked from commit f3d9cb0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant