Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Code upload instead of docker push workflow #82

Open
marrrcin opened this issue Sep 23, 2022 · 4 comments
Open

[feature] Code upload instead of docker push workflow #82

marrrcin opened this issue Sep 23, 2022 · 4 comments

Comments

@marrrcin
Copy link
Contributor

This idea is borrowed from Azure ML (and this PR getindata/kedro-azureml#15 ) - where you define an Environment, which is a docker image which runs your image, but the code is not part of the image (only dependencies are present in the image).
The workflow for that will make Data Science iterations faster, as they will not have to build the docker image every time they want to run / debug something in Vertex AI. This issue itself will be partially addressed by #81 , but this would be a next iteration on that.

General workflow would work like this:

  1. Docker image with dependencies is uploaded to the container registry.
  2. User runs kedro vertexai run-once with some flag (or maybe we should have kedro vertexai run for docker and kedro vertexai run-once for this flow 💡)
  3. The code of Kedro project is copied to GCS (first packaged and compressed) and the job is started within the container. The container should have a modified entrypoint which will first download the code from GCS and then execute it.

Please discuss the design with @em-pe and @szczeles before implemeting.

@adrienpl
Copy link

adrienpl commented Jan 6, 2024

This feature is really a must have to have a functional development environment with kedro.

@szczeles
Copy link
Contributor

szczeles commented Jan 7, 2024

@adrienpl Agree! The current development cycle with docker images is so painful...

Actually, I've been working on some implementation of this feature a year ago. Once I find the local branch, I will push it so somebody can take it over (I'm not into kedro/vertex anymore).

@adrienpl
Copy link

adrienpl commented Jan 7, 2024

Thank you ! It will be really helpfull. What are you using right now ?

@szczeles
Copy link
Contributor

szczeles commented Jan 8, 2024

@adrienpl I'm no longer on a project in MLOps area, so I'm using just boring dev tools ATM ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants