meltano / files-docker-compose

The declarative code-first data integration engine

Say goodbye to writing, maintaining, and scaling your own API integrations.
Unlock 600+ APIs and DBs and realize your wildest data and ML-powered product ideas.

---

Integrations

Meltano Hub is the single source of truth to find any Meltano plugins as well as Singer taps and targets. Users are also able to add more plugins to the Hub and have them immediately discoverable and usable within Meltano. The Hub is lovingly curated by Meltano and the wider Meltano community.

Installation

If you're ready to build your ideal data platform and start running data workflows across multiple tools, start by following the Installation guide to have Meltano up and running in your device.

Documentation

Check out the "Getting Started" guide or find the full documentation at https://docs.meltano.com.

Contributing

Meltano is a truly open-source project, built for and by its community. We happily welcome and encourage your contributions. Start by browsing through our issue tracker to add your ideas to the roadmap. If you're still unsure on what to contribute at the moment, you can always check out the list of open issues labeled as "Accepting Merge Requests".

For more information on how to contribute to Meltano, refer to our contribution guidelines.

Community

We host weekly online events where you can engage with us directly. Check out more information in our Community page.

If you have any questions, want sneak peeks of features or would just like to say hello and network, join our community of over +2,500 data professionals!

👋 Join us on Slack!

Responsible Disclosure Policy

Please refer to the responsible disclosure policy on our website.

License

This code is distributed under the MIT license, see the LICENSE file.

files-docker-compose's Issues

Update bundle/docker-compose.prod.yml - [merged]

Merges feat/make-docker-compose-airflow-consistent -> master

Migrated from GitLab: https://gitlab.com/meltano/files-docker-compose/-/merge_requests/4

---

Added airflow env vars to meltano ui container due to two reasons:

• DE will most likely connect to meltano-ui container and creating airflow melty user creates user in local sqlite file which is not shared with airflow container
• In case someone adds schedule in meltano-ui container, again the scripts will create it inside that sqlite db

This way, all containers use the same postgres db for airflow.

Document that Airflow needs psycopg2 in production - [merged]

Merges 1-docker-compose-prod-yml-with-airflow-requires-psycopg2-to-be-added-to-pip_url -> master

Migrated from GitLab: https://gitlab.com/meltano/files-docker-compose/-/merge_requests/3

---

Closes #1

Improve user experience with Docker and Compose file bundles

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3159

Originally created by @edgarrmondragon on 2022-01-13 21:16:57

---

• The current Compose file bundle binds the entire project directory as a volume. This can cause issues if the user has installed plugins in the host because virtual environments then get mounted as part of .meltano/ and may not be compatible or depend on user-level Python packages, etc.

• The Docker file bundle includes a Dockerfile with the following lines

  COPY ./meltano.yml . 
  RUN meltano install

  This means that if any plugins are defined inside of includes files, then they won't be installed during image build and require another meltano install after the container is created.

TODO: collate other DX issues with Docker and Compose

airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor

I'm getting the following error when trying to spin up docker with the the production config. Any help would be appreciated!

airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor

Config:

version: '3.8'

x-meltano-image: &meltano-image
  image: meltano-demo-project:dev # Change me to a name and tag that makes sense for your project
  build: .

x-meltano-env: &meltano-env
  MELTANO_DATABASE_URI: postgresql://postgres:postgres@meltano-system-db/meltano
  # Add any additional Meltano configuration environment variables here

# Uncomment if you are using the Airflow orchestrator, delete otherwise
x-airflow-env: &airflow-env
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgres://postgres:postgres@airflow-metadata-db/airflow
  AIRFLOW__CORE__EXECUTOR: LocalExecutor

services:
  meltano-ui:
    <<: *meltano-image
    command: ui
    environment:
      <<: *meltano-env
      # # Uncomment if you are using the Airflow orchestrator, delete otherwise
      # <<: *airflow-env
    volumes:
      - meltano_elt_logs_data:/project/.meltano/logs/elt
    expose:
      - 5000
    ports:
      - 5000:5000
    depends_on:
      - meltano-system-db
    networks:
      - meltano
    restart: unless-stopped

  meltano-system-db:
    image: postgres
    environment:
      POSTGRES_PASSWORD: password # CHANGE ME
      POSTGRES_DB: meltano
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - meltano_postgresql_data:/var/lib/postgresql/data
    expose:
      - 5432
    networks:
      - meltano
    restart: unless-stopped

  # Uncomment if you are using the Airflow orchestrator, delete otherwise
  airflow-scheduler:
    <<: *meltano-image
    command: invoke airflow scheduler
    environment:
      <<: *meltano-env
      <<: *airflow-env
    volumes:
      - meltano_elt_logs_data:/project/.meltano/logs/elt
    expose:
      - 8793
    depends_on:
      - meltano-system-db
      - airflow-metadata-db
    networks:
      - meltano
      - airflow
    restart: unless-stopped
  
  airflow-webserver:
    <<: *meltano-image
    command: invoke airflow webserver
    environment:
      <<: *meltano-env
      <<: *airflow-env
    expose:
      - 8080
    ports:
      - 8080:8080
    depends_on:
      - meltano-system-db
      - airflow-metadata-db
    networks:
      - meltano
      - airflow
    restart: unless-stopped
  
  airflow-metadata-db:
    image: postgres
    environment:
      POSTGRES_PASSWORD: password # CHANGE ME
      POSTGRES_DB: airflow
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - airflow_postgresql_data:/var/lib/postgresql/data
    expose:
      - 5432
    networks:
      - airflow
    restart: unless-stopped

networks:
  meltano:
  # Uncomment if you are using the Airflow orchestrator, delete otherwise
  airflow:

volumes:
  meltano_postgresql_data:
    driver: local
  meltano_elt_logs_data:
    driver: local
  # Uncomment if you are using the Airflow orchestrator, delete otherwise
  airflow_postgresql_data:
    driver: local

`docker-compose.prod.yml` with Airflow requires `psycopg2` to be added to `pip_url`

Migrated from GitLab: https://gitlab.com/meltano/files-docker-compose/-/issues/1

Originally created by @DouweM on 2021-02-10 17:50:18

---

https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/README.md#usage should refer to https://meltano.com/docs/production.html#airflow-orchestrator

Compose file update and readme addition - [merged]

Merges master -> master

Migrated from GitLab: https://gitlab.com/meltano/files-docker-compose/-/merge_requests/2

---

@DouweM First stab at a compose file for meltano and an accompanying readme. I am not using Airflow and I also don't have any experience with it, so if someone else could test the bit about it under "Adding services" to verify my instructions that would be great. I am also completely down with removing that section as well. Let me know if that helps and if there are any other service additions you would like me to add to the readme in the bundle, or anything else you think might be helpful.

Updating default docker compose file for meltano - [closed]

Merges master -> master

Migrated from GitLab: https://gitlab.com/meltano/files-docker-compose/-/merge_requests/1

---

Default to a more helpful docker compose file for meltano

Closes https://gitlab.com/meltano/meltano/-/issues/2106