Tracking issue: CI refactoring

[xopham]

Description

Initial refactoring step #622 resolved several topics already. Some remaining tasks are tracked in this issue.

Major changes:

• new build system
  • implement GHCR to store images as packages in pipeline
  • use buildkit for image build -> SBOMs, Provenance
  • sign packages using cosign
• modular CI (reusable workflows and actions)
• refactor CI

Improvements:

• integrated with GitHub Security
• modular CI
• faster builds
• signed builds
• nightly builds
• provenance and SBOMs added
• broader code scanning coverage

TBD:

• summary update
• rm trivy-released-image-scan into sca job
• join docs into main workflows
• update "required tests"
• matrix strategy for dockerhub-check

Open Question:

• OK to add registry to helm chart? seems more explicit and less reliant on a random default. -> yes
• OK to rename "nightly-scans" to "nightly"? yes
• can we drop tag engineering? -> yes
• should we make everything reusable with selector for which jobs to drop? -> choose reasonable level
• now only relying on Github security not pipeline fail. -> OK
• IMPORANT: mitigate risk of overwriting tags! -> Protect GHCR releases #915

Missing (create issues for remaining):

• release workflow + test
• somehow getRoot fails on nightly
• update public getRoot image
• rework all actions
• consider caching for build and integration tests
• speedup by reduction of integration tests
• remove update upload
• consider usage of https://github.com/marketplace/actions/docker-metadata-action
• use GH action specific parsing: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions
• add verify tag functionality for cosign validation to connaisseur
• sign scans
• add semgrep
• add checkov
• automated local testing with remote builds
• confirm dependabot update actions by commitSHA (checkout actions are purposely outdated by two minor versions)
• finetune checkov
• resolve OSSF Scorecard findings
• publish OSSF Scorecard findings
• merge upgrade int test into bash script ci: merge upgrade test into integration script #927
• why has cosign output changed?
• harmonize usage of yq (see ci: use ghcr, signatures, sboms, slsa, reusable workflows, ossf #622 (comment))
• consider dropping Uninstall when not necessary (see ci: use ghcr, signatures, sboms, slsa, reusable workflows, ossf #622 (comment))
• switch back to strict github security evaluation
• don't run integration tests on push to develop
• external MRs are not allowed to initiate push to packages? good? bad? Ci/fix git ref 2 tag #912