Make grizzly work with mozregression

[farre]

Grizzly is great and so is mozregression, but they don't work well together. The reason is that on a successfully reproduced result grizzly exits with 0, but to be able to do:

mozregression -B debug --launch 2023-06-01 --command "python -m grizzly.replay --headless default {binary} /path/to/test/testcase.html"

then grizzly needs to exit with a none zero exit status.

A simple script works around this, but it's not very pretty:

#!/usr/bin/bash
if [ -z "$(python -m grizzly.replay --headless default $1 $2 2>&1 | grep "Result: Assertion failure")" ]
then
    exit 0
else
    exit 1
fi

and it eats all the log output :( It works for me, but adding e.g. a flag --return-error-on-result if returning errors directly is fighting with some other use case would also be a great solution.

[farre]

And I should note that:

• I tried to make a PR, but it felt like I'd make a mess of things. I'm happy to do it if I get some pointers
• This doesn't work well with ./mach mozregression because of virtual envs :(

[tysmith]

@farre Thanks for the suggestion, I'd be happy to look into adding support.

[mozfreddyb]

@farre My understanding is that the bugs we have filed automatically in bugzilla already have a regression range. Can you help me understand what else you need? If it's a question of granularity, we may be able to solve this in the provided regression range rather than having you run manual steps.

[farre]

@mozfreddyb so if I run grizzly replay on a testcase, it will bisect and find when a testcase starts failing? That's cool, how do I do that?

[mozfreddyb]

I think we do the bugzilla integration with our fuzzing tools via bugmon, which can do automated bisection or upon request

@pyoor, please help me understand - Do we run bugmon on all reported fuzz bugs that have a testcase? For those that we do, do we always provide a regression range or is that optional?

[farre]

I guess it's an old grizzly testcase, without a range that I've been trying to reproduce.

[tysmith]

Two things to note:

1. A test case created by Grizzly does not depend on Grizzly. It should work as expected if run from a local web server and the included prefs files is used, etc. Using Grizzly does simplify this a lot :)

2. Automatic bisection done by bugmon in automation only goes back one year because it depends on archived builds.

[pyoor]

I think we do the bugzilla integration with our fuzzing tools via bugmon, which can do automated bisection or upon request

@pyoor, please help me understand - Do we run bugmon on all reported fuzz bugs that have a testcase? For those that we do, do we always provide a regression range or is that optional?

We run bugmon on all bugs in Bugzilla with the bugmon keyword. As @tysmith mentioned however, bisection is limited to bugs less than a year old.