In this project, I set up a continuous integration/continuous deployment (CI/CD) pipeline for processing data by implementing CI/CD methods with managed products on Google Cloud. Data scientists and engineers can adapt the methodologies from CI/CD practices to help ensure the high quality, maintainability, and adaptability of data processes and workflows. The methods that you can apply are as follows:
-
Version control of source code;
-
Automatic building, testing, and deployment of apps;
-
Environment isolation and separation from production;
-
Replicable procedures for environment setup.
NB: The folder ci-cd-for-data-processing-workflow contains source code for the guide on how to use Cloud Build and Cloud Composer to create a CI/CD pipeline for building, deployment and testing of a data processing workflow. The code is available in the github of the Google Cloud Platform organization for viewer. Refer to the solution guide for the steps to run the code in the folder ci-cd-for-data-processing-workflow.