Errors on parsing valid metadata

[sagudev]

Example expectation:

[canvas_complex_rgba8unorm_store.https.html]
  expected: FAIL

gives

× found ' ' expected test section header, or indentation at the proper level
   ╭─[/servo/tests/wpt/webgpu/meta/webgpu/webgpu/web_platform/reftests/canvas_complex_rgba8unorm_store.https.html.ini:1:1]
 1 │ [canvas_complex_rgba8unorm_store.https.html]
 2 │   expected: FAIL
   · ─
   ╰────

[ErichDonGubler]

Hi there! Happy to see others using this tool. What version/commit are you reproducing this from?

[sagudev]

I am using https://github.com/sagudev/moz-webgpu-cts/tree/servo (based off main) on servo.

GitHub

GitHub - sagudev/moz-webgpu-cts at servo

Tools to analyze and adjust Web Platform Tests coverage in Firefox, currently catered exclusively towards the WebGPU team. - GitHub - sagudev/moz-webgpu-cts at servo

[sagudev]

Still fails on:

[canvas_composite_alpha_bgra8unorm_opaque_draw.https.html]
  expected: [PASS, FAIL, CRASH]

so we will probably need to handle such cases as subtest altogether.

Also failure on disabled (because it's valid to have any value):

[cts.https.html?q=webgpu:api,operation,adapter,requestDevice:features,known:*]
  disabled: reasons for being disabled

[ErichDonGubler]

Sigh, the lack of a good diagnostic is definitely #30. Sorry about that!

[ErichDonGubler]

@sagudev: RE: analyzer stuff: Does Servo actually use that tool? I had not expected it to be used by anybody but Mozilla. 😅

[ErichDonGubler]

@sagudev: RE: adding FAIL to the test outcome data model: This is an interesting weirdness in WPT. My understanding is that expected outcomes in tests are a strict superset of the ones that can be discovered in subtests (CC @jgraham for clarity). I think it's clear that we need to add it. PRs welcome! I don't think I'm going to be able to prioritize it for a while, but I should have bandwidth for review. The code you linked to (e90940f) LGTM (minus a nit or two I'd present in review), so I bet we can start with that.

[ErichDonGubler]

@sagudev: Can you please clarify what you mean by the following (original link)?

so we will probably need to handle such cases as subtest altogether.

[sagudev]

Does Servo actually use that tool? I had not expected it to be used by anybody but Mozilla

Currently no, but I am very pleased with experiments I did with it (it's so much faster than ./mach update-wpt we have), although I needed wptreports we don't normally do in CI. I am currently investigating some rare CRASHes, before I can return to my PR here.

[sagudev]

What I discovered is that all valid values for subtests are also valid for test that have no subtests. You can take a look at PR how I hacked around this.

[jgraham]

I think that's not quite true; you can't ever have NOTRUN as a top-level test status (for tests without subtests the test is always at least started); see https://searchfox.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/logtypes.py#192-222 for details.

logtypes.py - mozsearch