Invalid output always returned when running test script

[stxue1]

To get around #1, I'm running extract_tests.py without specifying an output folder:

python extract_tests.py -i ~/Documents/wdl/wdl/SPEC.md -d ~/Documents/wdl/wdl/tests/data/

I also changed this line to get around the data and test directory mismatching:

output_data_dir = output_dir

This lets me run run_tests_miniwdl.py, but for pretty much every test, I get an error similar to this:

declarations.wdl: ERROR
-----------------------
Return code:
Invalid output(s):
  dir: /home/heaucques/Documents/wdl/wdl-tests/scripts/tests/20240529_125428_declarations != None
  outputs: {'declarations.pi': 3.14} != None

I assume it's because of these lines, where the condition seems to be always true:

wdl-tests/scripts/run_tests_miniwdl.py

Lines 105 to 106 in 58ff362

	if key not in config["output"]:
	invalid.append((key, value, None))

This seems to be consistent with every test, as MiniWDL always has a consistent output. For example:

(venv3.12) heaucques@pop-os:~/Documents/wdl/wdl-tests/scripts/tests$ /home/heaucques/Documents/toil/venv3.12/bin/miniwdl run -p . -i {} test_sub.wdl
2024-05-29 13:01:44.709 wdl.w:test_sub workflow start :: name: "test_sub", source: "test_sub.wdl", line: 3, column: 1, dir: "/home/heaucques/Documents/wdl/wdl-tests/scripts/tests/20240529_130144_test_sub"
2024-05-29 13:01:44.711 wdl.w:test_sub miniwdl :: version: "v1.12.0", uname: "Linux pop-os 6.8.0-76060800daily20240311-generic #202403110203~1714077665~22.04~4c8e9a0 SMP PREEMPT_DYNAMIC Thu A x86_64"
2024-05-29 13:01:44.728 wdl.w:test_sub done
{
  "dir": "/home/heaucques/Documents/wdl/wdl-tests/scripts/tests/20240529_130144_test_sub",
  "outputs": {
    "test_sub.choco4": "I like chocolate when\nit's late",
    "test_sub.chocoearly": "I like chocoearly when\nit's early",
    "test_sub.chocoearlylate": "I like chocearly when\nit's late",
    "test_sub.chocolate": "I like chocolate when\nit's early",
    "test_sub.chocolove": "I love chocolate when\nit's late",
    "test_sub.no_newline": "I like chocolate when it's late"
  }
}

For example, in the hello.wdl test, when the subprocess output is parsed out into key value pairs, key = outputs and value = {'hello.matches': ['hello world', 'hello nurse']} but the key outputs is compared to {'hello.matches': ['hello world', 'hello nurse']} from the expected value supplied by test_config.json. So the condition key not in config["output"] will always be true, leading to an expected actual output of None. Maybe it should be the value from MiniWDL that is compared?