Trino CI CD

Pipeline performance

Job duration

Building and testing

Attempts at reducing build time:

• Only enable check plugins if air.check.skip-* is false - compilation (with checks) and test execution is done in separate steps, so test execution could skip checks
• merged Run git commit id Maven plugin only once - this plugin was being executed for every module
• merged Enable Maven cache in the CI workflow - TODO run the same query as in the description to see what impact it actually had (compare before and after 10th Dec)
• closed Speed up builds of freshly checked out branches - this PR tested the impact of adding more Maven flags, like -Dmaven.source.skip=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true, which is negligible; it also tested disabling test compilation, which turned out not to be possible with a clean local repository

Product tests

The heaviest jobs are product tests because they:

• set up complex environments: a Trino cluster, made of a coordinator and some workers, and other dependencies, like Hive
• generate test data
• execute a lot and/or complex queries

TODO to confirm the above, profile tests to be able to get average times for different steps.

Possible solutions:

• go through every test and see if they can be converted to integration tests - be careful not to reduce test coverage; historically, integration tests in Trino were not using containerized dependencies, so some tests were implemented as product tests, where dependencies were running in VMs

Queue

Trino's CI/CD pipelines are implemented as Github Workflows that use Github's public runners. Because there's a limit of concurrent jobs, runs are often queued for a long time before they execute.

The solution is to make the jobs shorter and (in PRs only) skip ones not relevant to the change.

• Make CI only build/test modules with any changes
• product tests cannot use Maven to identify dependencies, so they need to get a list of impacted connectors to skip irrelevant tests/groups/suites