Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade actions and packages related to artifacts #3540

Merged
merged 8 commits into from
Dec 19, 2023
27 changes: 21 additions & 6 deletions .github/actions/load-img/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ name: openverse/load-img
description: Download and import Docker images from artifacts

inputs:
run_id:
description: The ID of the current workflow run
required: true

setup_images:
default: "upstream_db ingestion_server catalog api api_nginx frontend"
description: Space-separated list of image names
Expand All @@ -23,18 +27,29 @@ runs:
- name: Install `@actions/artifact`
shell: bash
run: |
pnpm install @actions/artifact@1 -w
pnpm install @actions/artifact -w
krysal marked this conversation as resolved.
Show resolved Hide resolved

- name: Download images
uses: actions/github-script@v6
with:
script: |
const artifact = require("@actions/artifact")
const artifactClient = artifact.create()
const images = "${{ inputs.setup_images }}".split(" ")
for (image of images) {
await artifactClient.downloadArtifact(image, "/tmp", { createArtifactFolder: false })
const { DefaultArtifactClient } = require('@actions/artifact')
const artifact = new DefaultArtifactClient()
const images = '${{ inputs.setup_images }}'.split(' ')
const [repositoryOwner, repositoryName] = '${{ github.repository }}'.split('/')
const findBy = {
repositoryOwner,
repositoryName,
workflowRunId: '${{ inputs.run_id }}',
token: '${{ github.token }}',
}
const workflowArtifacts = await artifact.listArtifacts({ findBy })
await Promise.all(
workflowArtifacts.artifacts
.filter((item) => images.includes(item.name))
.map((item) => artifact.downloadArtifact(item.id, { findBy, path: '/tmp' }))
)
console.log('Downloaded artifacts')

- name: Load images
shell: bash
Expand Down
25 changes: 18 additions & 7 deletions .github/workflows/ci_cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -231,11 +231,16 @@ jobs:
FRONTEND_PNPM_VERSION=${{ steps.prepare-build-args.outputs.frontend_pnpm_version }}

- name: Upload image `${{ matrix.image }}`
uses: actions/upload-artifact@v3
id: upload-img
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.image }}
path: /tmp/${{ matrix.image }}.tar

- name: Show artifact ID
run: |
echo '${{ matrix.image }} artifact ID is ${{ steps.upload-img.outputs.artifact-id }}'

###########
# Catalog #
###########
Expand Down Expand Up @@ -267,6 +272,7 @@ jobs:
- name: Load Docker images
uses: ./.github/actions/load-img
with:
run_id: ${{ github.run_id }}
setup_images: upstream_db catalog

- name: Run tests
Expand Down Expand Up @@ -298,6 +304,7 @@ jobs:
- name: Load Docker images
uses: ./.github/actions/load-img
with:
run_id: ${{ github.run_id }}
setup_images: upstream_db catalog

- name: Check DAG documentation
Expand Down Expand Up @@ -334,6 +341,7 @@ jobs:
- name: Load Docker images
uses: ./.github/actions/load-img
with:
run_id: ${{ github.run_id }}
setup_images: upstream_db ingestion_server

- name: Run ingestion-server tests
Expand All @@ -346,7 +354,7 @@ jobs:

- name: Upload ingestion test logs
if: success() || failure()
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: ing_logs
path: ingestion_server/test/ingestion_logs.txt
Expand Down Expand Up @@ -383,6 +391,7 @@ jobs:
- name: Load Docker images
uses: ./.github/actions/load-img
with:
run_id: ${{ github.run_id }}
setup_images: upstream_db ingestion_server api

- name: Start API, ingest and index test data
Expand All @@ -399,7 +408,7 @@ jobs:

- name: Upload API test logs
if: success() || failure()
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: api_logs
path: api_logs
Expand Down Expand Up @@ -447,6 +456,7 @@ jobs:
- name: Load Docker images
uses: ./.github/actions/load-img
with:
run_id: ${{ github.run_id }}
setup_images: upstream_db ingestion_server api

- name: Run check recipe
Expand All @@ -456,7 +466,7 @@ jobs:

- name: Upload schema
if: matrix.name == 'test_doc'
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: openapi.json
path: ./api/openapi.json
Expand Down Expand Up @@ -653,7 +663,7 @@ jobs:
- name: Run Playwright tests
run: just frontend/run ${{ matrix.script }} --workers=2

- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
if: failure()
id: test-results
with:
Expand Down Expand Up @@ -771,7 +781,7 @@ jobs:
# Docs will be located at `/tmp/docs`.

- name: Upload documentation
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: documentation
path: /tmp/docs/
Expand Down Expand Up @@ -809,7 +819,7 @@ jobs:

steps:
- name: Download documentation
uses: actions/download-artifact@v3
uses: actions/download-artifact@v4
with:
name: documentation
path: /tmp/docs
Expand Down Expand Up @@ -926,6 +936,7 @@ jobs:
- name: Load Docker image `${{ matrix.image }}`
uses: ./.github/actions/load-img
with:
run_id: ${{ github.run_id }}
setup_images: ${{ matrix.image }}

- name: Load and tag image `${{ matrix.image }}` (latest & sha)
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_automations_init.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
PR_NODE_ID: ${{ github.event.pull_request.node_id }}

- name: Upload event info as artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: event_info
path: /tmp/event.json
26 changes: 22 additions & 4 deletions .github/workflows/push_docker_image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,32 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Determine successful run ID
id: determine-run-id
uses: actions/github-script@v7
with:
github-token: ${{ secrets.ACCESS_TOKEN }}
script: |
const [owner, repo] = '${{ github.repository }}'.split('/')
const { data: runs } = await github.rest.actions.listWorkflowRunsForRepo({
owner,
repo,
branch: 'main',
event: 'push',
status: 'success',
head_sha: '${{ github.event.inputs.commit }}',
})
const runId = runs.workflow_runs.find((run) => run.name === 'CI + CD').id
core.setOutput('run_id', runId)

- name: Download image `${{ github.event.inputs.image }}`
uses: dawidd6/action-download-artifact@v2
uses: actions/download-artifact@v4
with:
workflow: ci_cd.yml
workflow_conclusion: success
commit: ${{ github.event.inputs.commit }}
name: ${{ github.event.inputs.image }}
path: /tmp
github_token: ${{ secrets.ACCESS_TOKEN }}
repository: ${{ github.repository }}
run_id: ${{ steps.determine-run-id.outputs.run_id }}

- name: Load and tag image `${{ github.event.inputs.image }}`
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/renovate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ jobs:
# To solve *that*, we'd have to extract to root (/), which isn't safe.
tar -czvf "$cache_archive" -C "$cache_dir" .

- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
if: github.event.inputs.repoCache != 'disabled'
with:
name: ${{ env.cache_key }}
Expand Down