JUnit XML files generated by meson are not parsed

[rohan-at-sentry]

The JUnit XML files generated by meson are not parsed: jluebbe/rauc#4 (comment)

In the web dashboard tab for tests, "No test results found for this branch" is shown.

Originally posted by @jluebbe in #304

[rohan-at-sentry]

@jluebbe

Do you mind posting a raw version of the JUnit xml that you tried to upload? Inspecting our logs, it seems like we're erroring out because a field we rely on is missing.

Specifically, the error message states "No duration found".

[jluebbe]

This is the output of running the RAUC testsuite: testlog.junit.xml.gz

Meson test runs should be written by default to build/meson-logs/testlog.junit.xml (see https://mesonbuild.com/Unit-tests.html#testlogjunitxml).

[rohan-at-sentry]

@jluebbe thanks for your patience on this. It looks like there wasn't a time attribute on the test case, which our parser was expecting. I'm unclear if this is unique to your setup or to the tool that is generating the report (which is meson).

At any case, we're trying to handle it on the parser side -> you can follow along codecov/test-results-parser#30

[jluebbe]

Thanks for implementing this so quickly! I've tried the build and test result upload again: https://github.com/jluebbe/rauc/actions/runs/10915969616/job/30296620556

So far, the results have now shown up on https://app.codecov.io/gh/jluebbe/rauc/tests/codecov. Do I need to wait until the fixed version is deployed?

[jluebbe]

I've tried this again (https://github.com/jluebbe/rauc/actions/runs/11345991363), but the results are not showing up. Is that something I can fix on my side?

Some details...

• the upload step: https://github.com/jluebbe/rauc/actions/runs/11345991363/job/31554073912#step:6:1
• the same junit.xml uploaded as a github artifact: https://github.com/jluebbe/rauc/actions/runs/11345991363/job/31554073912#step:7:21

[rohan-at-sentry]

@joseph-sentry can you please take a look here?

[joseph-sentry]

@jluebbe it looks like the generated JUnit XML doesn't have the time attribute on the testcase elements which is something our JUnit XML parser assumes is there.

I think the solution is that we can make the parser ignore this for now, but we don't have a way to process the duration of a testsuite, so there will be no reporting of test duration in the UI or PR comment.

I think this is a property of meson's JUnit reporting where a test maps to a testsuite in the JUnit report and the actual test functions in the test file map to testcase objects, which is different from how testing frameworks like pytest or jest report their test results.

EDIT: looking more into it, i wouldn't suggest using the tap protocol in your meson.build, i've tested it out and it seems like when you get a failing test it changes the name of the test in the junit.xml which will be reported bizarrely on the codecov UI and in the PR comment, I'd suggest using the exitcode protocol instead.

[jluebbe]

Even without the duration information, having the test results in codecov would be useful.

The name change issue seems like something we should fix on meson.