Create Docker Image directory which deploys to Dockerhub about telemetry-airflow

[whd]

This structure would be a departure from our SOP. Why not have the Dockerfile and image generation CI configured in each container/projects's own repository? i.e. https://github.com/mozilla/mozilla-schema-generator should be built on CI and its Dockerfile and circleci configs should live within its own repo, uploading to Dockerhub at mozilla/mozilla-schema-generator:TAG. Likewise for any other projects that need to be used on airflow via the pod operator. This method of container management is how we currently maintain all Docker-based projects operated by Cloud and Data Operations.

Containers to be run by the pod operator on airflow likely don't have anything intrinsically to do with airflow itself. This change seems similar to me to putting the telemetry-batch-view jar uploading process and CI into telemetry-airflow. It is my understanding that the pod operator is generic and simply takes arbitrary containers to execute on pods. Unless we need custom images that change properties (e.g. the Dockerfile is an extension of mozilla-schema-generator's Dockerfile with some airflow-specific configuration) I don't think this is a structure we should adopt, and even then I would avoid this pattern if possible, instead making the standard image flexible enough to support usage both in a standard way and with airflow via docker entrypoints and additional executables/configuration.

There is the additional consideration of the reliability of docker images that needs to be addressed. Typically we mirror Dockerhub images via Jenkins to our internal container registry for production workflows. We would probably want a similar mechanism for containers used via pod operator, without necessarily mirroring via Jenkins, but this is tangential and something that can be considered separately from the main discussion.

[fbertsch]

Thanks for the feedback @whd. We'll close this and figure out an alternative solution.

[haroldwoo]

Do we still need dockerhub for this use case? We could configure circleci to push images to gcr directly so they are readily available for podoperators.