-
-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: pytest scope to be function as tests might change server #1378
base: develop
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #1378 +/- ##
===========================================
- Coverage 84.27% 84.23% -0.04%
===========================================
Files 38 38
Lines 5315 5315
===========================================
- Hits 4479 4477 -2
- Misses 836 838 +2 ☔ View full report in Codecov by Sentry. |
So it seems there was a reason because the tests are now failing, but whatever the reason, it broke other tests. So we need to re-implement the way this works |
I think I did session because when run in parallel with "function", where it will keep entering and exiting the fixture which sets and unsets the test server config, a global variable. This caused the tests to fail. It's actually really hard to get around this since they all use the same config in parallel, where some tests want production and some want test server. In retro-spect, session doesn't make huge sense either, but it would essentially set the default to test server at the start which is most of them. Any production server tests however could change this global when they need to, but this then causes issues for the test server tests. Possible high-level solutions are really:
|
re 2,5,6: basically just use a pytest marker and run one with "not production" and one with "production", and that is a fairly transparent way of dealing with this. That said, it's still not ideal but from the top of my head I am not aware of a good solution. |
I am a bit confused about this. Previously (with unittest) this was never an issue, yet they still referred to the same global variable. |
but tests pass for me locally and as far as I can tell the original issue this PR tries to address also doesn't occur when run on CI/CD: https://github.com/openml/openml-python/actions/runs/12274682764/job/34248109797?pr=1384 |
I still don't fully understand by what miracle it does work though. Since all access is |
To make sure that each test runs on the test server by default, we need to change the scope to "function," as another test might overwrite this otherwise.
@eddiebergman Was there a reasons to set this to session?