proposal: Refactor CI pipleline into stages

[Xuanwo]

Hello, everyone!

CI is an important part of our community. Every contributor will meet them in every PR. Databend's CI didn't well organize in the past. In this proposal, I will discuss a new design of CI with the whole community.

Goal

• Fast: run needed CI only.
• Easy: easy to understand, easy to maintain.

How it works?

Github allows us to:

• Share artifacts between jobs in the same workflow.
• Reuse actions in different workflows/jobs.

So, instead of splitting jobs into different workflows, we can have a main.yml that organizes all related jobs together. In the case of databend, we will have two profiles:

• developing.yml: Used in all PRs and non-main branches
• production.yml: Used in main branch, build with --release and publish images.

Let's take a look at developing.yml for glance:

name: Developing

on: [push, pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/check

  build:
    runs-on: ${{ matrix.config.os }}
    needs: check
    strategy:
      matrix:
        config:
          - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu, cross: false }
          - { os: ubuntu-latest, target: aarch64-unknown-linux-gnu, cross: true }
          - { os: macos-11, target: x86_64-apple-darwin, cross: false }
          - { os: macos-11, target: aarch64-apple-darwin, cross: false }
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/build_debug

  test_unit:
    runs-on: ${{ matrix.config.os }}
    needs: build
    strategy:
      matrix:
        config:
          - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu }
          - { os: macos-11, target: x86_64-apple-darwin }
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/test_unit
      
  test_stateless_standalone:
    runs-on: ${{ matrix.config.os }}
    needs: build
    strategy:
      matrix:
        config:
          - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu }
          - { os: macos-11, target: x86_64-apple-darwin }
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/test_stateless_standalone

  test_stateless_cluster:
    runs-on: ${{ matrix.config.os }}
    needs: build
    strategy:
      matrix:
        config:
          - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu }
          - { os: macos-11, target: x86_64-apple-darwin }
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/test_stateless_cluster

  test_stateful_standalone:
    runs-on: ${{ matrix.config.os }}
    needs: build
    strategy:
      matrix:
        config:
          - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu }
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/test_stateful_standalone

In this way, we don't need to guess which workflow could be run. And we can have a global view of our jobs like:

Benefits

• With the needs support, we can only allow running build after check passed.
• We can share the artifacts that were built at build steps in tests so that we don't need to build them again.
• We can remove duplicated concurrency and paths-ignore in different workflows.

Demo

Here is a working demo: #3344

Welcome for any comments and reviews!

Tracking issues

#3453

[Xuanwo]

@tisonkun, @huachaohuang, this proposal may also apply to engula, PTAL.

[Xuanwo]

The running model is: workflow -> job -> step

Github's task scheduling is Job-based:

• Workflow is just a job container and no actual work happens at this level.
• Different jobs could be running at the same time in parallel and they will be run in a fresh new env.
• Steps in the same job will be run in sequences and they will share the same context.

[tisonkun]

If so, different job A and B with B needs A will run in the same env or different? I see the model is different, but then reuse the artifacts generated by A in B cannot be achieved?

[Xuanwo]

• Different
• Artifacts can be shared between the same workflows (across different jobs).

[tisonkun]

Artifacts can be shared between the same workflows (across different jobs).

Not directly (in the same env), but by cache or upload-artifacts?

[Xuanwo]

Not directly (in the same env), but by cache or upload-artifacts?

Yep, every job runs in a new VM.

[skyzh]

If there are multiple steps in one action (e.g. test, clippy, fmt), it seems that they won't be shown as separate steps on GitHub Action.

[Xuanwo]

Yes, github doesn't support nested actions so far, just like the following: