I have multiple Python pipelines that I want to run on the Google Platform as on-demand jobs. To simplify the process of installing dependencies and downloading/preparing ML models, I created a Docker image containing all the necessary requirements. The entrypoint of the image is set to my main.py file, which takes unique arguments for each run. I want to be able to run the image using the GC Platform similar to a local run:
docker run -rm ${IMAGE} arg1 arg2 arg3
I have successfully uploaded the images, but I am unsure about how to proceed with setting up the next stage of my pipeline. Is it possible to store the image and run it when needed, or do I need to create an API endpoint and host the pipeline as an API?
>Solution :
You can’t run images like that dynamically. You have to deploy services (Cloud Run, Cloud Run Jobs) and then invoke them. Even if you use Kubernetes (GKE on Google Cloud) you have to deploy the pod before invoking it.
So, deploy your containers on the service that you which and invoke them with the arg that you need for your pipeline. Cloud Run Jobs with parameter override is the best place to achieve this IMO.
As orchestrator, you can use Cloud Workflow (my fav) or Composer (Managed Airflow).