Comments (3)
This structure would be a departure from our SOP. Why not have the Dockerfile
and image generation CI configured in each container/projects's own repository? i.e. https://github.com/mozilla/mozilla-schema-generator should be built on CI and its Dockerfile
and circleci configs should live within its own repo, uploading to Dockerhub at mozilla/mozilla-schema-generator:TAG
. Likewise for any other projects that need to be used on airflow via the pod operator. This method of container management is how we currently maintain all Docker-based projects operated by Cloud and Data Operations.
Containers to be run by the pod operator on airflow likely don't have anything intrinsically to do with airflow itself. This change seems similar to me to putting the telemetry-batch-view
jar uploading process and CI into telemetry-airflow
. It is my understanding that the pod operator is generic and simply takes arbitrary containers to execute on pods. Unless we need custom images that change properties (e.g. the Dockerfile
is an extension of mozilla-schema-generator
's Dockerfile
with some airflow-specific configuration) I don't think this is a structure we should adopt, and even then I would avoid this pattern if possible, instead making the standard image flexible enough to support usage both in a standard way and with airflow via docker entrypoints and additional executables/configuration.
There is the additional consideration of the reliability of docker images that needs to be addressed. Typically we mirror Dockerhub images via Jenkins to our internal container registry for production workflows. We would probably want a similar mechanism for containers used via pod operator, without necessarily mirroring via Jenkins, but this is tangential and something that can be considered separately from the main discussion.
from telemetry-airflow.
Thanks for the feedback @whd. We'll close this and figure out an alternative solution.
from telemetry-airflow.
Do we still need dockerhub for this use case? We could configure circleci to push images to gcr directly so they are readily available for podoperators.
from telemetry-airflow.
Related Issues (20)
- Task 'event_events' in the dag 'copy_deduplicate' failing due to table/view not found exception (telemetry.event)) HOT 1
- Tests failing in CircleCI with `ModuleNotFoundError: No module named 'plugins.mozetl'`
- Task `crash_report_parquet` job is failing in socorro_import failing HOT 7
- fix requirements installed by dataproc_init.sh HOT 3
- Add back TAAR lite guids ranking job
- Migrate TAAR jobs from S3 to GCS
- DSRE-22 testing creating issue comment HOT 4
- Test issue for DSRE-22 Jira task HOT 1
- Airflow DAG tag validation is missing
- socorro import uses wrong schema HOT 1
- Add dag definition to schedule webcompat-kb job
- Add a dag definition to schedule broken-site-report-ml job
- Remove gke_command in favor of GKEPodOperator
- Remove GKENatPodOperator
- Create a "Wait for build" operator for dockerized components HOT 1
- Jobs run with GKEOperator need `get_logs=False`, otherwise job is likely to fail unless constantly logging to standard out HOT 2
- glam_org_mozilla_fenix dag failing on gke_command
- prio-processor dag fails to start gke cluster
- Specify exact container version for pipelines feeding public data HOT 1
- Add top-level option to bigquery_etl_query for replacing whole table
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from telemetry-airflow.