Unit-tests not registering as a fail

[spongeBri]

I would like to have a little assistance in why this particular scenario is not working.
We currently have 4 steps in our yaml

Deploy
Test (failed tests are rerun)
Publish (Publish the results of the initial run)
Publish Retry (Publish the results of the retry run, with the collected failed tests)

Currently even if the retry has failures, all steps are made to go green.
I am aiming for all to go green only if the retry does not have any failures when it publish's.

During the check the only step allowed to fail is the initial test publish, but the retry should not be allowed to fail as we want to catch if tests fail on the retry.
Not sure why its going green when its publishing failed results.

  check:
    if: always()

    needs:
    - upload-first-run-tests
    - upload-retry-run-tests

    runs-on: Ubuntu-latest

    steps:
    - name: Decide whether the needed jobs succeeded or failed
      uses: re-actors/alls-green@release/v1
      with:
        allowed-failures: upload-first-run-tests
        jobs: ${{ toJSON(needs) }}

Any help on solving this issue would be hugely appreciated

[webknjaz]

Wild guess — you have a continue-on-failure somewhere in that job. If that's the case, then GitHub sets the job status to success. You may want to get familiar with the differences of the outcome vs. conclusion — one is the actual result and the other is how GH treats it (and it is ultimately controlled by continue-on-failure value — on job or step level).
This also correlates with the green checkmarks in the left sidebar next to the listed jobs.
For this action to know that a job has failed, it shouldn't pretend to pass. The action cannot look into what's happening inside the job because GitHub does provide access to such a context.
FWIW if you have a public CI run / configs, I can take a look to know for sure.

[spongeBri]

Unfortunately we don't have continue-on-failure in the job but we are using - Github Publish Test Results

With this step, if test failures occur then this does not fail the step by default but it is configurable

[webknjaz]

Sounds like you've found your bug 🤷‍♂️