Populate output xml file after every test on a suite is completed

[zebra993]

What's the problem this feature will solve?

I have been running tests suites on automated regressions via Jenkins using pytest for a while. Usually it runs without issues but from time to time due to timeouts an interrupt will kill the job and the output xml file will not be generated even thought some tests have been successfully run.

Is there a chance to extend/generate the output xml file after every test is completed? So if the suite face a timeout sigterm we still will be able to get the results for the tests that run without issues until that point.

I've run through the code (I'm not an expert on pytest) but I think it is related to this function.

Example: Lets say that we have a test suite with 10 tests, sometimes due timeout the job receive a sigterm that kills it at some point, for example, lets say on the 7th test , but all 6 previous tests have been run successfully. At this point the output xml file is not been created so the info for the previous 6 tests is lost.

Describe the solution you'd like

I would like to know if there is a flag o way to make the xml output generated every time a tests runs to avoid the issue of loosing information (specially if it is a big suite with far more tests and it takes a lot more of time to run).

I'm quite aware that this process (if possible or implemented) could add more time to the overall run time, but it will be appreciated and a reasonable trade off (at least in my case) instead of having to run a big suite of tests from scratch if a percentage or big part of it is already completed.

Additional context

Current versions related to pytest being used on my test environment:

pytest-randomly==3.4.0
pytest-repeat==0.8.0
pytest-timeout==1.3.4
pytest==6.2.5

I'll be glad to hear any suggestion from the pytest team or contribute with a PR for this one if needed.

[Zac-HD]

Given that this would probably be a substantial slowdown for many test suites, we're unlikely to put this in Pytest.

I'd suggest copying this logic to your own hook which will be run after each test function, probably pytest_runtest_makereport.

[RonnyPfannschmidt]

Altering the xml writer so it dumps and releases while updating a xml header by overwriting it would actually resolve the resource issues we currently face with captured output intensive test suites

[zebra993]

Hi, excellent!
In any way that you proceed I'll be waiting and if needing testing the new functionality or workaround.
This will be a very useful improvement for my team since we have quite big tests suites.

Glad to hear your suggestions and comments guys. Thanks in advance.

[RonnyPfannschmidt]

The topic is not something I'll personally touch

I haven't used junitxml output in almost a decade