Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing gcloud command or terraform code to create cloud build triggers #19

Open
hilliao opened this issue Apr 3, 2022 · 0 comments
Open

Comments

@hilliao
Copy link

hilliao commented Apr 3, 2022

Users like to have a way to create cloud build triggers in the Terraform folder or with gcloud commands. I found tensorflow transform is using Apache beam runner in the following cloud build file.

# Compile the pipeline.
- name: '$_CICD_IMAGE_URI'
  entrypoint: 'python'
  args: ['build/utils.py',
          '--mode', 'compile-pipeline',
          '--pipeline-name', '$_PIPELINE_NAME'
          ]
  dir: 'mlops-with-vertex-ai'
  env: 
  - 'PROJECT=$_PROJECT'  
  - 'REGION=$_REGION'
  - 'MODEL_DISPLAY_NAME=$_MODEL_DISPLAY_NAME'
  - 'DATASET_DISPLAY_NAME=$_DATASET_DISPLAY_NAME'  
  - 'GCS_LOCATION=$_GCS_LOCATION' 
  - 'TFX_IMAGE_URI=$_TFX_IMAGE_URI' 
  - 'BEAM_RUNNER=$_BEAM_RUNNER'
  - 'TRAINING_RUNNER=$_TRAINING_RUNNER'
  id: 'Compile Pipeline'
  waitFor: ['Local Test E2E Pipeline']

So far I gather an example from a notebook's section. But I can't find how to create the cloud build trigger.

  1. Run the training pipeline using Vertex Pipelines
    Set the pipeline configurations for the Vertex AI run
    os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME
    os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME
    os.environ["PIPELINE_NAME"] = PIPELINE_NAME
    os.environ["PROJECT"] = PROJECT
    os.environ["REGION"] = REGION
    os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}"
    os.environ["TRAIN_LIMIT"] = "85000"
    os.environ["TEST_LIMIT"] = "15000"
    os.environ["BEAM_RUNNER"] = "DataflowRunner"
    os.environ["TRAINING_RUNNER"] = "vertex"
    os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}"
    os.environ["ENABLE_CACHE"] = "1"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant