console.log() breaks zora reporter?

[mindplay-dk]

So I've had my test-suite going with something like ts-node src/test/test.ts | zr, but just ran into some problems.

I just realized, if I call console.log anywhere in my tests, that seems to break zr, which expects a stream of JSON objects - anything that doesn't look like the start of the JSON value makes it exit, without ever emitting the logged messages.

This may be one thing that the TAP protocol has going for it - out-of-band information, such as console.log message, just ends up as part of the message stream, without interrupting anything.

So is console.log with zora 5's new message stream format and zora-reporter just out of the question?

EDIT: solved - see notes at end of thread.

[lorenzofox3]

That's actually a bug or a missing feature.
The reporter should probably ignore message it cant understand/or print them as they come. We can change the parse function so it can be fault tolerant

[lorenzofox3]

you 'll have the same kind of issue with a tap parser: at some point you need to decide what to do with "unknown" messages

[mindplay-dk]

Making the parser error tolerant sounds complex and fragile.

Probably the "correct" thing to do is print diagnostic messages to stderr instead of stdout? "Don't cross the streams!" 😆

But there's no built-in way to do that in node, apparently - and I don't want to reinvent the console log pretty printer.

By far the simplest and safest thing would be to just run the reporter in-process. I know we've been over this, but it still seems like the benefits of running two parallel processes are largely hypothetical, while the problems are pretty real. 🤨

[mindplay-dk]

Just a quick take to explain exactly what I'm talking about:

master...mindplay-dk:no-undo

Just a quick POC! I haven't been able to run the tests (that for-loop in the bash doesn't work for me) - I don't know how to fix the typings, and I'm not sure if I'm doing package.json correctly...

[mindplay-dk]

(...er, sorry, wrong thread for that last comment!)

[lorenzofox3]

Making the parser error tolerant sounds complex and fragile.

That's on the contrary trivial for a simple parser such JSON.parse.

And that's pretty much what every parser do at some point (having a strict mode and tolerant mode): tap-parser, acorn, etc

const createJSONParser = ({strictMode = false}) => function*(lineStream) {
    for await (const line of lineStream){
         try {
             yield JSON.parse(line);
         } catch (e){
            if(stricMode){
               throw new Error(`could not parse line "${line}"`)
            } else {
              //ignore, etc
            }
         }
    }
}

then replace

map(JSON.parse)

by

createJSONParser(option) in the composition pipeline aforementioned at that's it

Probably the "correct" thing to do is print diagnostic messages to stderr instead of stdout? "Don't cross the streams!" 😆

But there's no built-in way to do that in node, apparently - and I don't want to reinvent the console log pretty printer.

That's very simple too process.stderr.write or console.error() 🤷‍♂️; but that does not seem to be a good idea to me... Moreover it is semantically incorrect and may confuse consumers: the testing program has no error, that's the tested program. The only valid case to me would be to print the error when the program bails out, but that's already the case with uncaught errors if I am not wrong.
If your intent is to filter failing assertions only, then grep is just fine:
ZORA_REPORTER=json node test/index.js | grep '"pass":false' (you can then pipe to fx for pretty print like in one of the example)

By far the simplest and safest thing would be to just run the reporter in-process. I know we've been over this, but it still seems like the benefits of running two parallel processes are largely hypothetical, while the problems are pretty real. 🤨

I don't think so. I mean no offence but you are over complicating pretty much everything; and technically you can already use the reporter in the same process. The problem comes to the fact you do not manage to run the program written in TS successfully, which is orthogonal to pretty much all the "problems" you have mentioned. The real issue is to make sure the package can be consumed by TS, yet another tool, if one wants to consume it programmatically.

[mindplay-dk]

To others finding this issue:

The latest release of zora-reporters (version 1.1.0) can be loaded as a CommonJS module, so now works with TypeScript and ts-node - follow the directions in the documentation to programmatically bootstrap the test-reporter and this now works beautifully, with console.log or any other console output. 👍

[lorenzofox3]

there is one PR left somehow related though:

The parser should be able to ignore string it can't parse: it is tracked in #108