Not seeing reports when PR is merged even though reports are generated

[frankjkelly]

Describe the bug

When a PR is created or commits pushed I am seeing the reports created and linked e.g.

But when the PR is merged and the tests re-run and the report created I am not seeing them

NOTE: I am new to GHA and very possible I am doing something silly

To Reproduce

File

name: Entity Service and Client CI

on:
  pull_request: 
    types: [opened, synchronize, reopened, closed]
    branches: [master, release, feature/*]
.
.
jobs:
  build-gradle-project:
    runs-on: self-hosted
    steps:
    - name: Checkout project sources
      uses: actions/checkout@v3
      with:
        fetch-depth: '0'
        fetch-tags: 'true'
.
.
.
    - name: Unit Test
      run: ./gradlew test
    - name: Unit Test Report 
      if: success() || failure()
      uses: dorny/test-reporter@v1
      with:
        name: Unit Tests                  
        path: '**/build/reports/test/xml/*.xml' 
        reporter: java-junit             
.
.
.
    - name: Integration Test Report 
      if: success() || failure()
      uses: dorny/test-reporter@v1
      with:
        name: Integration Tests                  
        path: '**/build/test-results/integrationTest/*.xml'
        reporter: java-junit             
.
.
    - name: Contract Test Report 
      if: success() || failure()
      uses: dorny/test-reporter@v1
      with:
        name: Contract Tests                  
        path: '**/build/test-results/contract/*.xml'
        reporter: java-junit   

Expected behavior

I expect that the summary results show the links like they do on PR creation or new commits

Logs

Logs look like everything went OK

2024-01-26T13:32:09.8492303Z ##[group]Run dorny/test-reporter@v1
2024-01-26T13:32:09.8492703Z with:
2024-01-26T13:32:09.8492958Z   name: Unit Tests
2024-01-26T13:32:09.8493284Z   path: **/build/reports/test/xml/*.xml
2024-01-26T13:32:09.8493854Z   reporter: java-junit
2024-01-26T13:32:09.8494232Z   path-replace-backslashes: false
2024-01-26T13:32:09.8494586Z   list-suites: all
2024-01-26T13:32:09.8494866Z   list-tests: all
2024-01-26T13:32:09.8495144Z   max-annotations: 10
2024-01-26T13:32:09.8495466Z   fail-on-error: true
2024-01-26T13:32:09.8495761Z   fail-on-empty: true
2024-01-26T13:32:09.8496045Z   only-summary: false
2024-01-26T13:32:09.8496454Z   token: ***
2024-01-26T13:32:09.8496705Z env:
.
.
.
2024-01-26T13:32:09.8516755Z ##[endgroup]
2024-01-26T13:32:09.9658724Z Action was triggered by pull_request: using SHA from head of source branch
2024-01-26T13:32:09.9704422Z Check runs will be created with SHA=8f72e8f20b11e6ad550d29c10663689eaab3ee7b
.
.
.
2024-01-26T13:32:09.9945619Z Found 177 files tracked by GitHub
2024-01-26T13:32:09.9946234Z Using test report parser 'java-junit'
2024-01-26T13:32:10.0020655Z ##[group]Creating test report Unit Tests
2024-01-26T13:32:10.0022133Z Processing test results for check run Unit Tests
2024-01-26T13:32:10.2112541Z Creating check run Unit Tests
2024-01-26T13:32:10.6095676Z Creating report summary
2024-01-26T13:32:10.6096360Z Generating check run summary
2024-01-26T13:32:10.6175120Z Creating annotations
2024-01-26T13:32:10.6178602Z Updating check run conclusion (success) and output
2024-01-26T13:32:10.9441567Z Check run create response: 200
2024-01-26T13:32:10.9442881Z Check run URL: https://api.github.com/repos/cogitocorp/platform2-entity/check-runs/20901124339
2024-01-26T13:32:10.9444386Z Check run HTML: https://github.com/cogitocorp/platform2-entity/runs/20901124339
2024-01-26T13:32:10.9451740Z ##[endgroup]

I can click on the HTML page shown at the end and see the results

[timabell]

I think I might be seeing the same thing.

Here's a test report that was correctly generated and uploaded that doesn't show in the main view

It was working earlier and I was trying to work out what changed. Here's an older build where the report does show.

I wonder if it's because I turned on the CodeQL option in github between the two runs. It seems to add a new section, and the test report now seems to end up incorrectly under the codeql section instead of the build section.

And here's a newer PR run showing the test result in the right place https://github.com/timabell/schema-explorer/pull/65/checks?check_run_id=20998088207

[timabell]

Okay I think my hunch is off, I think it must be something in my workflow file. Having done some more tidying up it's now back again, and I have no idea why. Here's a newer build of main that has it in the right place https://github.com/timabell/schema-explorer/actions/runs/7705131292/job/20998613617

[timabell]

Okay I'm even more confused, it's just done it again, and this one is exactly as the issue describes - the PR build is fine, and then the build on main has the report missing.

---

PR that demonstrates the behaviour: timabell/schema-explorer#65

• PR build, all fine => https://github.com/timabell/schema-explorer/actions/runs/7705540824/job/20999641752
• merge to main, ended up in wrong place again
  • Where it should be: https://github.com/timabell/schema-explorer/actions/runs/7705547883
  • Where it actually is: https://github.com/timabell/schema-explorer/runs/20999661642 (found by looking at test report step raw output)

---

Tracking issue on my repo: timabell/schema-explorer#64

[jozefizso]

I would say it's a correct behavior. The Test Result is not a job. It reports a check.

I'm not sure why GitHub displays it as a Job for the PR builds.

[sszemeresky]

the action produces 2 links:
https://repo/check-runs/12345
https://repo/runs/12345
first one throws an error
{
"message": "Not Found",
"documentation_url": "https://docs.github.com/rest/checks/runs#get-a-check-run"
}
the second one redirects to some weird place where the report is displayed.

[EduPonz]

I think I might be seeing the same thing.

Here's a test report that was correctly generated and uploaded that doesn't show in the main view

It was working earlier and I was trying to work out what changed. Here's an older build where the report does show.

I wonder if it's because I turned on the CodeQL option in github between the two runs. It seems to add a new section, and the test report now seems to end up incorrectly under the codeql section instead of the build section.

And here's a newer PR run showing the test result in the right place https://github.com/timabell/schema-explorer/pull/65/checks?check_run_id=20998088207

This exact thing is happening to us here

[HofmeisterAn]

I think I run into the same issue.

I would say it's a correct behavior. The Test Result is not a job. It reports a check.

How is the check bound to the workflow? I use a workflow named Test Report that calls the action, but it appears that the report is associated with a different workflow named, CodeQL.

[EduPonz]

I've seen the report being associated with multiple different workflows actually (I think it's associated with whatever finished first or something like that). On the PRs is definitely a bother but not really a big problem as the check still appears on the list, although as child job of whatever other workflows. However, for cron jobs is a bit more unfortunate, as the check does not appear on the list and the way to find it is to go to the test report action ouptut and click on the HTML link there.

Can anyone with a bit more knowledge about the actions interface give some pointers to look for the culprit?

[timabell]

I've now seen this issue show up on a (private) project that's not even using this report uploader.

Edit - think it was https://github.com/phoenix-actions/test-reporting which turns out to be a fork of this repo

[shearer3000]

I'd like to report/follow this too. I see the test reports show under the CodeQL section rather than the workflow it was part of

[AyoubIssaad]

Same issue for me, all newly generated reports are shown under an old run and not the current one.

[alexremn]

Reporting the same. Check is bouncing across other workflows, though if you're making it as a required check for PR merge and it bounced, PR would never pass those checks as there would be waiting for results status.

[CRohm42]

+1 - Just added the action today and it is exactly what I was looking for, thank you! Same behavior, the job/Test Report is added to workflow based on commit number instead of the workflow run number.

Our team's preference is for the job/Test Report to be added to the current run number, regardless of commit.

[SHIINASAMA]

I had a similar problem. Multiple reports are created in different jobs. Then I got the following view...

[neotrow]

same issue here, really annoying and seemed to appear out of nowhere 🙃

did anyone find a workaround?

[AyoubIssaad]

same issue here, really annoying and seemed to appear out of nowhere 🙃

did anyone find a workaround?

I think we are seeing this behaviour due to a limitation on the github side and not on test-reporter, there is a workaround which is to use:
use-actions-summary: 'true'

This will publish the reports on the summary of your run and not as separate page. and it's accurate

[HofmeisterAn]

@AyoubIssaad do you have some more information? Isn't the default value already true:

test-reporter/action.yml

Lines 78 to 83 in 1a288b6

	use-actions-summary:
	description: |
	Allows you to generate reports for Actions Summary
	https://github.com/orgs/github/teams/engineering/discussions/871
	default: 'true'
	required: false

[timcassell]

This issue is basically a duplicate of #67. The use-actions-summary feature is only on main, there hasn't been a new release with it.