Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Worker for Python Code? #151

Open
martinjol opened this issue Jan 2, 2020 · 0 comments
Open

Custom Worker for Python Code? #151

martinjol opened this issue Jan 2, 2020 · 0 comments

Comments

@martinjol
Copy link

Hi all,
thanks for the amazing work on the CRMint tool, I am using and recommending it a lot! From time to time, I come across situations where I wish I could use it as a pipeline management tool for broader use-cases than what is currently being possible with the integrated workers. For example, I need to pull weather forecast data daily from an API, do some data preparation with Pandas and then import it to BQ to process the data with BQML. The way I'm doing this now is with Cloud Scheduler, Pub/Sub and Cloud Functions, but this is obviously less than ideal for monitoring and error handling.
It would be amazing if there would be a "custom python" worker type in CRMint where I could simply paste my python code for tasks like these and have them as steps of these pipelines.

Thanks a lot!
Martin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant