ci: whitelist paths, reorganize workflows & speed-up tests

[kmk3]

Main changes:

• Run workflows based on path whitelists rather than blacklists
• Reorganize jobs and workflows, mostly based on the paths used
• Run make in parallel where applicable
• docs: add missing CI badges to README.md
• Split tests into multiple jobs

Considering the most recent runs, this reduces the total amount of time
it takes to run the tests from about 9-10 minutes to about 3 minutes.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[kmk3]

@github-code-scanning on Aug 16:

This pull request sets up GitHub code scanning for this repository. Once the
scans have completed and the checks have passed, the analysis results for
this pull request branch will appear on this
overview.
Once you merge this pull request, the 'Security' tab will show more code
scanning analysis results (for example, for the default branch). Depending on
your configuration and choice of analysis tool, future pull requests will be
annotated with code scanning analysis results. For more information about
GitHub code scanning, check out the
documentation.

This was likely posted because this PR renames codeql-analysis.yml.

That was mostly done to avoid running the CodeQL Python check when it is not
needed, so that it only runs when a .py file is changed.

But it seems that the file has to be named exactly codeql-analysis.yml, or else
GitHub complains about it in the "Security" tab, even though the checks are
working normally:

From security/code-scanning/tools/CodeQL/status:

(Green checkmark) Check-C

Actions workflow · Last scan 9 minutes ago

(Green checkmark) Check-Python

Actions workflow · Last scan 10 minutes ago

(Warning) codeql-analysis.yml (Actions workflow missing)

Actions workflow · Last scan 8 hours ago

I think I'll undo the renaming.

Feel free to review this in any case.

[glitsj16]

Not that I understand much of the code in this PR, I do appreciate all the work being done to make CI-life more fun & less hasle!

[rusty-snake]

What about ci, m4 and contrib folders? And *.sh? Maybe tests folder?

[kmk3]

What about ci, m4 and contrib folders? And *.sh? Maybe tests folder?

Good catch, fixed the missing m4/ and added comments to clarify what each
workflow is supposed to do.

ci/ is currently only used for the profile checks, so it's only in
check-profiles.yml.

contrib/, test/ and some shell scripts are used in make dist, so they are in
build.yml (which in this PR checks both make dist and the normal build).

test/ is otherwise only used when running tests, so it's only in test.yml.

The shell scripts in the root directory do not seem to be used for anything
during the normal build, so they wouldn't affect the checks.

Is there any script in particular that you think would affect it?

[rusty-snake]

ci/ is currently only used for the profile checks, so it's only in
check-profiles.yml.

./ci/printenv.sh

[kmk3]

ci/ is currently only used for the profile checks, so it's only in
check-profiles.yml.

./ci/printenv.sh

That makes sense, though since this is only used for diagnostics in CI, how
about adding it only to build.yml to avoid re-running everything on every
change?

[kmk3]

On second thought, this file is unlikely to be changed often anyway, so I added
it to every workflow.

[topimiettinen]

@github-code-scanning on Aug 16:

This pull request sets up GitHub code scanning for this repository. Once the
scans have completed and the checks have passed, the analysis results for
this pull request branch will appear on this
overview.
Once you merge this pull request, the 'Security' tab will show more code
scanning analysis results (for example, for the default branch). Depending on
your configuration and choice of analysis tool, future pull requests will be
annotated with code scanning analysis results. For more information about
GitHub code scanning, check out the
documentation.

This was likely posted because this PR renames codeql-analysis.yml.

That was mostly done to avoid running the CodeQL Python check when it is not needed, so that it only runs when a .py file is changed.

But it seems that the file has to be named exactly codeql-analysis.yml, or else GitHub complains about it in the "Security" tab, even though the checks are working normally:

From security/code-scanning/tools/CodeQL/status:

(Green checkmark) Check-C
Actions workflow · Last scan 9 minutes ago

(Green checkmark) Check-Python
Actions workflow · Last scan 10 minutes ago

(Warning) codeql-analysis.yml (Actions workflow missing)
Actions workflow · Last scan 8 hours ago

I think I'll undo the renaming.

Feel free to review this in any case.

I think it's OK to rename, but then the old configuration must be deleted from Security tab. I did a similar rename recently.

[reinerh]

I personally don't like that most of these sections is now duplicated multiple times. When adjusting something you now need to remember to check and update multiple places.

[kmk3]

I personally don't like that most of these sections is now duplicated
multiple times. When adjusting something you now need to remember to check
and update multiple places.

I don't like it either, but unfortunately GitHub doesn't support YAML anchors:

• https://yaml.org/spec/1.0/index.html#syntax-anchor
• Support YAML Anchors actions/runner#1182

It looks like a core part of the language to me, but this problem was first
reported in 2019 and the above issue has been labeled as "enhancement" and
"papercut" rather than as a bug in their parser, so I wouldn't expect it to be
fixed any time soon.

Do you know of any other simple way to deduplicate them?

[reinerh]

autobuild no longer used

[kmk3]

autobuild no longer used

While the comment mentions the autobuild step specifically, it looks like that
permission still needed by the job to create/modify alerts in the Security tab:

From https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

security-events

Work with GitHub code scanning and Dependabot alerts. For example,
security-events: read permits an action to list the Dependabot alerts for
the repository, and security-events: write allows an action to update the
status of a code scanning alert. For more information, see "Repository
permissions for 'Code scanning alerts'" and "Repository permissions for
'Dependabot alerts'" in "Permissions required for GitHub Apps."

Also, when creating codeql-analysis.yml on a new project, it generates the
permissions without any comments:

jobs:
  analyze:
    name: Analyze
    # Runner size impacts CodeQL analysis time. To learn more, please see:
    #   - https://gh.io/recommended-hardware-resources-for-running-codeql
    #   - https://gh.io/supported-runners-and-hardware-resources
    #   - https://gh.io/using-larger-runners
    # Consider using larger runners for possible analysis time improvements.
    runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
    timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }}
    permissions:
      actions: read
      contents: read
      security-events: write

I have removed the comments to match the current generated file.

I could also remove the write permission if you're sure that it's not needed.

[reinerh]

autobuild no longer used

[kmk3]

@topimiettinen on Aug 16:

I think I'll undo the renaming.

Feel free to review this in any case.

I think it's OK to rename, but then the old configuration must be deleted
from Security tab. I did a similar rename recently.

I kept the real checks on separate files and left codeql-analysis.yml as a
dummy file with a no-op job, which seems to appease GitHub (all items have a
green checkmark in the status page now).

From the testing on my GitHub fork, there do not appear to be any duplicated
alerts or anything of the sort in the Security tab, so it seems like GitHub
correctly maps the alerts to the relevant tool rather than to a specific file.

I also don't see any option to delete anything in the following page:

• Security -> Code scanning -> Tool status -> CodeQL

Could you clarify how you did the renaming? Are you sure that deleting the
configuration would be needed?

[topimiettinen]

I also don't see any option to delete anything in the following page:

* Security -> Code scanning -> Tool status -> CodeQL

I think that was the place, but the option will only show up after the file is deleted.

Could you clarify how you did the renaming? Are you sure that deleting the configuration would be needed?

I just deleted the file (or rather, merged codeql.yml and codeql-analysis.yml):
topimiettinen/libaslrmalloc@ed48589

Some docs here.

[kmk3]

@topimiettinen on Aug 20:

I just deleted the file (or rather, merged codeql.yml and
codeql-analysis.yml):
topimiettinen/libaslrmalloc@ed48589

Some docs
here.

Thanks, I see what you mean now.

I removed codeql-analysis.yml in the PR again and I managed to delete the
configuration in my fork by doing the following:

• Security -> Code scanning -> Tool status -> (Setup Types) CodeQL ->
  (Configurations) language:python -> Delete configuration

It was really confusing because the CodeQL item had "language:python" in the
configurations (and nothing else) for some reason, even though it was mapped to
codeql-analysis.yml, which only had a dummy job (no cpp or python job).

Maybe it makes sense when using many scanning tools, but these steps seemed
really unintuitive; I would likely not have guessed how to do this through the
web UI alone.

[kmk3]

So it's been almost a week and from what I can tell all of the issues raised
were addressed.

Is there anything else?

If not I'll probably merge this later.

Note that the tests as they are already fail randomly and if a single test
fails, re-running it currently requires re-running all tests (which takes a
long time and which may need to be done multiple times).

While with this PR only the specific test job that fails needs to be
re-executed.

[kmk3]

Merged, thanks everyone for the feedback!

I managed to delete the configuration in my fork by doing the following:

• Security -> Code scanning -> Tool status -> (Setup Types) CodeQL ->
  (Configurations) language:python -> Delete configuration

Done in this repository as well.