astronomer / ap-airflow

Astronomer Core Docker Images

Astronomer Core Docker Images

Astronomer makes it easy to run, monitor, and scale Apache Airflow deployments in our cloud or yours. Source code is made available for the benefit of customers.

Terminology

Terms	Example	Description
edge build	main-dev	Built from the current main branch of astronomer/airflow
dev build	2.2.4-4-dev	Development build, released during ap-airflow changes, including pre-releases and version releases
nightly build	2.2.4-nightly-20220314	Nightly builds, regularly triggered by a CircleCI pipeline sometime during the midnight hour UTC
release build	2.2.4-4	Release builds, triggered by a release PR

Note: Edge builds are always development builds

Build matrix

Build	Nightly	Pre-release PR	Release PR
edge build	✅	✅	✅
nightly build	✅	✅
dev build	(only during pre-release)	✅	✅
release build			✅

Docker images

Docker images for deploying and running Astronomer Core are currently available on Quay.

We publish 2 variants for each AC Version (example: 2.3.4-7):

1. quay.io/astronomer/ap-airflow:2.3.4-7
2. quay.io/astronomer/ap-airflow:2.3.4-7-onbuild

The only difference between them is that the -onbuild images uses Docker ONBUILD commands to copy packages.txt, requirements.txt and the entire project directory (including dags, plugins folders etc) in the docker file.

We also publish a "floating" or movable tag that points at the latest release of the Airflow version:

1. quay.io/astronomer/ap-airflow:2.3.4
2. quay.io/astronomer/ap-airflow:2.3.4-onbuild

Version Life Cycle & Maintenance policy

The support and maintenance of the Docker images are described in the Version Life Cycle.

Contents of this repo

• The official Dockerfiles that build Astronomer Core Images
• Example docker-compose files for running various pieces and configurations of the platform.

Contribute

• Source Code: https://github.com/astronomer/ap-airflow
• Issue Tracker: https://github.com/astronomer/ap-airflow/issues

Step-by-step instructions for common activities

Release a new Astronomer Certified major, minor, or bugfix version (eg: X.Y.Z)

Click to expand Step-By-Step instructions

1. Remove the -dev part of the relevant version in IMAGE_MAP in .circleci/common.py.

   Example: The latest dev version is 2.2.1-1-dev, and we want to release 2.2.1-1.

   diff --git a/.circleci/common.py b/.circleci/common.py
   index xxxxxxx..yyyyyyy 100644
   --- a/.circleci/common.py
   +++ b/.circleci/common.py
   @@ -35,7 +35,7 @@ IMAGE_MAP = collections.OrderedDict([
        ("2.1.3-2", ["buster"]),
        ("2.1.4-2", ["buster"]),
        ("2.2.0-3-dev", ["bullseye", "buster"]),
   -    ("2.2.1-1-dev", ["bullseye", "buster"]),
   +    ("2.2.1-1", ["bullseye", "buster"]),
   ])
   
   # Airflow Versions for which we don't publish Python Wheels

2. Run the update-dockerfiles pre-commit hook (this should fail but it should change the relevant Dockerfile).

   Example:

   pre-commit run update-dockerfiles

3. Add the changed Dockerfile and commit (this should succeed).

   Example: The update-dockerfiles hook updated 2.2.1/bullseye/Dockerfile:

   git add 2.2.1/bullseye/Dockerfile; git commit

Release an existing Astronomer Certified version with an updated version of Airflow

Click to expand Step-By-Step instructions

1. Update the postfix version of the relevant version in IMAGE_MAP in .circleci/common.py.

   Example: The latest AC version is 2.2.0-1 and we want to release 2.2.0-2.

   diff --git a/.circleci/common.py b/.circleci/common.py
   index xxxxxxx..yyyyyyy 100644
   --- a/.circleci/common.py
   +++ b/.circleci/common.py
   @@ -35,7 +35,7 @@ IMAGE_MAP = collections.OrderedDict([
        ("2.1.3-2", ["buster"]),
        ("2.1.4-2", ["buster"]),
        ("2.2.0-3-dev", ["bullseye", "buster"]),
   -    ("2.2.1-1", ["bullseye", "buster"]),
   +    ("2.2.1-2", ["bullseye", "buster"]),
    ])
   
    # Airflow Versions for which we don't publish Python Wheels

2. Run the update-dockerfiles pre-commit hook (this should fail but it should change the relevant Dockerfile).

   Example:

   pre-commit run update-dockerfiles

3. Add the changed Dockerfile and commit (this should succeed).

   Example: The update-dockerfiles hook updated 2.2.0/bullseye/Dockerfile:

   git add 2.2.0/bullseye/Dockerfile; git commit

Add new Astronomer Certified development version

Click to expand Step-By-Step instructions

1. Add the Astronomer Certified version to IMAGE_MAP in .circleci/common.py.

   Example: The latest previous release was 2.2.1-1 and we're adding 2.3.0-1-dev.

   diff --git a/.circleci/common.py b/.circleci/common.py
   index xxxxxxx..yyyyyyy 100644
   --- a/.circleci/common.py
   +++ b/.circleci/common.py
   @@ -36,6 +36,7 @@ IMAGE_MAP = collections.OrderedDict([
        ("2.1.4-2", ["buster"]),
        ("2.2.0-3-dev", ["bullseye", "buster"]),
        ("2.2.1-1", ["bullseye", "buster"]),
   +    ("2.3.0-1-dev", ["bullseye"]),
    ])
   
    # Airflow Versions for which we don't publish Python Wheels

2. Edit the new CHANGELOG.md to show what has changed in this release.

   Example:

   nano 2.3.0/CHANGELOG.md

3. Add the new directory to the Git staging area.

   Example:

   git add 2.3.0

4. Run the update-dockerfiles pre-commit hook (this should fail but it should change the relevant Dockerfile).

   Example:

   pre-commit run update-dockerfiles

   The pre-commit hook should change some lines in the new Dockerfile.

   diff --git a/2.3.0/bullseye/Dockerfile b/2.3.0/bullseye/Dockerfile
   index xxxxxxx..yyyyyyy 100644
   --- a/2.3.0/bullseye/Dockerfile
   +++ b/2.3.0/bullseye/Dockerfile
   @@ -110,10 +110,10 @@ RUN apt-get update \
        && apt-get clean \
        && rm -rf /var/lib/apt/lists/*
   
   -ARG VERSION="2.2.1-1"
   +ARG VERSION="2.3.0-1.*"
    ARG SUBMODULES="async,azure,amazon,elasticsearch,google,password,cncf.kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
    ARG AIRFLOW_MODULE="astronomer_certified[${SUBMODULES}]==$VERSION"
   -ARG AIRFLOW_VERSION="2.2.1"
   +ARG AIRFLOW_VERSION="2.3.0"
   
    # Make pip look at our pip repo too, and force it to install these specific
    # versions when ever it installs a module.
   @@ -145,8 +145,8 @@ RUN apt-get update \
        && apt-get clean \
        && rm -rf /var/lib/apt/lists/*
   
   -ARG VERSION="2.2.1-1"
   -ARG AIRFLOW_VERSION="2.2.1"
   +ARG VERSION="2.3.0-1.*"
   +ARG AIRFLOW_VERSION="2.3.0"
    LABEL io.astronomer.docker.airflow.version="${AIRFLOW_VERSION}"
    LABEL io.astronomer.docker.ac.version="${VERSION}"
   

5. Stage the changes to the Dockerfile and commit (this should succeed).

   Example:

   git add 2.3.0/bullseye/Dockerfile && git commit

Add a new base build image (eg: new Debian stable release)

Click to expand Step-By-Step instructions

1. Add or adjust the Debian release name in IMAGE_MAP.

   Example: Previous Astronomer Certified versions only built with Debian Buster, but Debian Bullseye has just been released as the new Debian stable version and we'd like to add support for that.

   diff --git a/.circleci/common.py b/.circleci/common.py
   index xxxxxxx..yyyyyyy 100644
   --- a/.circleci/common.py
   +++ b/.circleci/common.py
   @@ -36,7 +36,7 @@ IMAGE_MAP = collections.OrderedDict([
        ("2.1.4-2", ["buster"]),
        ("2.2.0-3-dev", ["bullseye", "buster"]),
        ("2.2.1-1", ["bullseye", "buster"]),
   -    ("2.3.0-1-dev", ["buster"]),
   +    ("2.3.0-1-dev", ["bullseye", "buster"]),
    ])
   
    # Airflow Versions for which we don't publish Python Wheels

2. Add a new version directory for it.

   Example: There is currently a 2.3.0/buster directory that we need to copy to 2.3.0/bullseye and then modify that Dockerfile to use Debian Bullseye.

   cp -a 2.3.0/buster 2.3.0/bullseye

3. Adjust the relevant Dockerfile.

   Example: Update the 2.3.0/bullseye/Dockerfile to use the upstream Debian Bullseye image.

   diff --git a/2.3.0/bullseye/Dockerfile b/2.3.0/bullseye/Dockerfile
   index xxxxxxx..yyyyyyy 100644
   --- a/2.3.0/bullseye/Dockerfile
   +++ b/2.3.0/bullseye/Dockerfile
   @@ -14,7 +14,7 @@
    # limitations under the License.
    ARG APT_DEPS_IMAGE="airflow-apt-deps"
    ARG PYTHON_MAJOR_MINOR_VERSION="3.9"
   -ARG PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
   +ARG PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-bullseye"
   
    FROM ${PYTHON_BASE_IMAGE} as airflow-apt-deps
   

4. Stage the changes to the Dockerfile and commit (the pre-commit hooks should all succeed).

   Example:

   git add .circleci/common.py 2.3.0/bullseye && git commit

Changelog

All changes applied to available point releases will be documented in the CHANGELOG.md files within each version folder:

• 2.3.4 Changelog
• 2.4.3 Changelog

Testing

Local testing

This testing will run automatically in CI, but it will save some time to try it out locally first.

Airflow is launched into a local Kubernetes cluster using the project "kind" and the most recent version of the Astronomer airflow chart. Python's 'testinfra' module is used to perform system testing on the components while they are running in "kind".

Ensure prerequisites are met:

• docker
• python3
• virtualenv

Ensure docker installed, and user has permissions

docker run -it --rm hello-world

Ensure Python3 is installed and in PATH

python3 -c "print('Confirmed python3 installed.')"

Ensure virtualenv is installed

which virtualenv

Set up virtual environment

virtualenv --python=python3 venv
source venv/bin/activate
pip install -r .circleci/test-requirements.txt

Run system testing

Build the image you want to test

docker build -t airflow ./1.10.5/buster

Run system testing

.circleci/bin/test-airflow airflow

The first time you do the build, and the first time you do the system test it will take longer than subsequent runs. The system testing will install the tested versions of CI tools in /tmp/bin (helm, kubectl, kind). It will leave an airflow cluster running on your kind cluster in 'test-cluster'. When you run it again, it will delete the namespace of your most recent deployment and redeploy into a new namespace. If you make changes in the image, don't forget to re-build the image before testing it.

Use the newly installed tools

export PATH=/tmp/bin:$PATH

Ensure kubectl configured to use kind

kubectl cluster-info --context kind-test-cluster

Look at the pods

kubectl get pods --all-namespaces

Clean up

kind delete cluster --name test-cluster

Scheduled Tasks

The regularly scheduled tasks are:

Edge Builds

• Rebase the astro-main branch of astronomer/airflow onto main and push it to astronomer/airflow:astro-main (this then kicks off a GitHub Actions workflow that builds Airflow and Astronomer Certified Python packages/wheels) and pushes them to our PyPI package repository
• Build nightly Docker images for QA (using those nightly Airflow and Astronomer Certified wheels) and push them to the dev image repository

CircleCI Schedules

The CircleCI documentation on scheduled pipelines is very new and in slight disarray.

See also:

• The announcement for scheduled pipelines
• This relevant (for now) PR, which includes an example of branch filtering
• Documentation on parameter syntax
• Documentation on logic statements

The API for manipulating schedules is documented here, including examples.

Here is an example listing all schedules with HTTPie (and colorizing the response with jq):

$ http https://circleci.com/api/v2/project/gh/astronomer/ap-airflow/schedule \
       circle-token:<CIRCLECI_PERSONAL_ACCESS_TOKEN> \
       | jq

To create a new schedule (refer to the HTTPie docs about raw JSON):

$ http --verbose \
       https://circleci.com/api/v2/project/gh/astronomer/ap-airflow/schedule \
       circle-token:<CIRCLECI_PERSONAL_ACCESS_TOKEN> \
       name="every-morning-0200-UTC" \
       description="Every morning at 02:00 UTC" \
       attribution-actor="system" \
       parameters:='{ "branch": "master" }' \
       timetable:='{ "per-hour": 1, "hours-of-day": [2], "days-of-week": ["SUN", "MON", "TUE", "WED", "THU", "FRI", "SAT"]}'

Note that updating and deleting schedules uses a different URL path:

$ http PATCH \
       https://circleci.com/api/v2/schedule/<schedule_uuid> \
       circle-token:<CRCLECI_PERSONAL_ACCESS_TOKEN> \
       name=every-sunday-0100-utc \
       description="Every Sunday at 01:00 UTC"

You can create a CircleCI personal API token in your CircleCI user settings. Do note that a PAT will authenticate as you, and have full, read and write access on CircleCI, so keep your PAT secret and do not publish it anywhere!

License

Apache 2.0 with Commons Clause

ap-airflow's Issues

Support for Apple Silicon

Hello, do you plan support for images for Apple Silicon? Would be great to be able to run Astronomer on the new Macs! Right now the command astro dev start works but the containers (webserver, scheduler, database) refuse to work together.

log retention env var name misleading

ap-airflow/2.0.2/buster/include/clean-airflow-logs

Line 6 in ab60218

RETENTION=${ASTRONOMER__AIRFLOW__WORKER_LOG_RETENTION_DAYS:-15}

This environment variable implies that the retention is only relevant to the worker logs. however this script is also used to clean the scheduler logs in the helm chart. These should either be controlled by separate variables or this name should change to be ASTRONOMER__AIRFLOW__LOG_RETENTION_DAYS

error to run dev start

When run dev start with dockerfile image base quay.io/astronomer/ap-airflow:2.1.4-buster the image docker keeping restarting on ubuntu 20.04.3

Release date of 2.3.3-buster-onbuild

Could you say when 2.3.3-buster-onbuild image will be available?

Since we have got such notification, we would like to update astronomer ASAP, but we need only buster version.

Issue building 2.3.3 and 2.3.4 with dill library

Some customers are having issues with their builds when running astro deploy and astro dev start. The error is.

Here is the ticket to these issues on Zendesk

WARNING: Requested dill>=0.2.2 from https://files.pythonhosted.org/packages/3e/ad/31932a4e2804897e6fd2f946d53df51dd9b4aa55e152b5404395d00354d1/dill-0.3.1.tar.gz#sha256=d3ddddf2806a7bc9858b20c02dc174396795545e9d62f243b34481fd26eb3e2c (from apache-airflow>=2.2.0->apache-airflow-providers-opsgenie==4.0.0->-r requirements.txt (line 3)), but installing version 0.3.1.dev0
WARNING: Discarding https://files.pythonhosted.org/packages/3e/ad/31932a4e2804897e6fd2f946d53df51dd9b4aa55e152b5404395d00354d1/dill-0.3.1.tar.gz#sha256=d3ddddf2806a7bc9858b20c02dc174396795545e9d62f243b34481fd26eb3e2c (from https://pypi.org/simple/dill/) (requires-python:>=2.6, !=3.0.*). Requested dill>=0.2.2 from https://files.pythonhosted.org/packages/3e/ad/31932a4e2804897e6fd2f946d53df51dd9b4aa55e152b5404395d00354d1/dill-0.3.1.tar.gz#sha256=d3ddddf2806a7bc9858b20c02dc174396795545e9d62f243b34481fd26eb3e2c (from apache-airflow>=2.2.0->apache-airflow-providers-opsgenie==4.0.0->-r requirements.txt (line 3)) has inconsistent version: filename has '0.3.1', but metadata has '0.3.1.dev0'

Another customer has said that it is stalling with this message

[#12](https://astronomer.zendesk.com/agent/tickets/12) 97.08 Collecting dill>=0.2.2
[#12](https://astronomer.zendesk.com/agent/tickets/12) 97.09   Downloading dill-0.3.5.1-py2.py3-none-any.whl (95 kB)
[#12](https://astronomer.zendesk.com/agent/tickets/12) 173.8 INFO: pip is looking at multiple versions of distro to determine which version is compatible with other requirements. This could take a while.
[#12](https://astronomer.zendesk.com/agent/tickets/12) 173.8   Downloading dill-0.3.5-py2.py3-none-any.whl (95 kB)
[#12](https://astronomer.zendesk.com/agent/tickets/12) 249.8   Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB)
[#12](https://astronomer.zendesk.com/agent/tickets/12) 268.8 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
[#12](https://astronomer.zendesk.com/agent/tickets/12) 325.8   Downloading dill-0.3.3-py2.py3-none-any.whl (81 kB)
[#12](https://astronomer.zendesk.com/agent/tickets/12) 400.2   Downloading dill-0.3.2.zip (177 kB)
[#12](https://astronomer.zendesk.com/agent/tickets/12) 474.7   Downloading dill-0.3.1.1.tar.gz (151 kB)
[#12](https://astronomer.zendesk.com/agent/tickets/12) 550.5   Downloading dill-0.3.1.tar.gz (151 kB)

Here are the respective requirements.txt files and Dockerfiles

First Customer

FROM quay.io/astronomer/ap-airflow:2.3.4-onbuild

# first error
--extra-index-url https://pip.astronomer.io/simple

apache-airflow-providers-opsgenie==4.0.0
apache-airflow-providers-snowflake==2.7.0
memory-profiler>=0.55.0
numpy>=1.17.4
pycurl>=7.43.0.2
jinja2==2.11.3
sqlalchemy==1.4.9
snowflake-sqlalchemy==1.2.4
packaging==21.0
pyyaml==5.4.1
colorlog==4.0.2
pyparsing==2.4.7
google-auth==1.34.0
types-python-dateutil==2.8.3
types-pytz==2021.3.3
types-PyYAML==6.0.1
types-croniter==1.0.3
types-requests==2.26.2
types-pycurl==7.44.6

Second Customer

FROM quay.io/astronomer/ap-airflow:2.3.4-onbuild

# stalling
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-microsoft-mssql==3.2.0
black==22.6.0
mandrill==1.0.60
python-gnupg==0.5.0
slackclient==2.9.4
user-agents==2.2.0
spectacles==2.2.0
dbt-core==1.2.1
dbt-snowflake==1.2.0
snowflake-connector-python[pandas]==2.7.12

One customer tested with Docker build and verified that worked, but could not get it working with astro deploy or astro dev start.

Unable to build latest Docker images

I am no longer able to build the latest Airflow image on Docker for Desktop version 4.6.1 (76265), running Mac OS Monterey v12.3.1.

I previously had been able to successfully build the 2.2.5 release, but that now fails. I also tried building the 2.3.0 release, and that also failed. Both due to pip being unable to find a version that satisfied requirements for astronomer-certified[.....]==2.2.5-3-* (and 2.3.0 similarly). Full stack trace:

> DOCKER_BUILDKIT=0 docker build --build-arg PYTHON_MAJOR_MINOR_VERSION=3.8 -t ap-airflow:py38 https://github.com/astronomer/ap-airflow.git#master:2.3.0/bullseye/
Sending build context to Docker daemon  30.21kB
Step 1/67 : ARG APT_DEPS_IMAGE="airflow-apt-deps"
Step 2/67 : ARG PYTHON_MAJOR_MINOR_VERSION="3.9"
Step 3/67 : ARG PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-bullseye"
Step 4/67 : FROM ${PYTHON_BASE_IMAGE} as airflow-apt-deps
 ---> 654e18c3f59c
Step 5/67 : LABEL maintainer="Astronomer <[email protected]>"
 ---> Using cache
 ---> 1c96172c8065
Step 6/67 : ARG ASTRONOMER_USER="astro"
 ---> Using cache
 ---> 204eabfc3b98
Step 7/67 : ARG ASTRONOMER_UID="50000"
 ---> Using cache
 ---> 83b112225176
Step 8/67 : LABEL io.astronomer.docker=true
 ---> Using cache
 ---> e46957e97ba2
Step 9/67 : LABEL io.astronomer.docker.distro="debian"
 ---> Using cache
 ---> 12ae0a3d4812
Step 10/67 : LABEL io.astronomer.docker.module="airflow"
 ---> Using cache
 ---> dc8791d042f7
Step 11/67 : LABEL io.astronomer.docker.component="airflow"
 ---> Using cache
 ---> 50829d950c62
Step 12/67 : LABEL io.astronomer.docker.uid="${ASTRONOMER_UID}"
 ---> Using cache
 ---> b9d8acf44b83
Step 13/67 : ARG ORG="astronomer"
 ---> Using cache
 ---> a999463fbeb3
Step 14/67 : ENV AIRFLOW_HOME="/usr/local/airflow"
 ---> Using cache
 ---> bb61da75bc45
Step 15/67 : ENV PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}${AIRFLOW_HOME}
 ---> Using cache
 ---> b971365c26b6
Step 16/67 : ENV ASTRONOMER_USER=${ASTRONOMER_USER}
 ---> Using cache
 ---> af6e48608745
Step 17/67 : ENV ASTRONOMER_UID=${ASTRONOMER_UID}
 ---> Using cache
 ---> 71b33d71cab5
Step 18/67 : ARG PYTHON_BASE_IMAGE
 ---> Using cache
 ---> 3e5b784bb97b
Step 19/67 : ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
 ---> Using cache
 ---> 6d795160f3c2
Step 20/67 : ARG PYTHON_MAJOR_MINOR_VERSION
 ---> Using cache
 ---> 639c51da0550
Step 21/67 : ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
 ---> Using cache
 ---> d15a535a7564
Step 22/67 : ARG PIP_VERSION="21.2.4"
 ---> Using cache
 ---> 408e6dc75a18
Step 23/67 : ENV PYTHON_PIP_VERSION=${PIP_VERSION}
 ---> Using cache
 ---> c36cc72b62d8
Step 24/67 : ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8     LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
 ---> Using cache
 ---> 40057402fe74
Step 25/67 : ARG DEPENDENCIES_EPOCH_NUMBER="2"
 ---> Using cache
 ---> 676cbb5c9126
Step 26/67 : ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
 ---> Using cache
 ---> 5f6cc1bed465
Step 27/67 : RUN apt-get update     && apt-get install -y --no-install-recommends            apt-utils            curl            libmariadb3            freetds-bin            gosu            libffi7            libkrb5-3            libpq5            libsasl2-2            libsasl2-modules            libssl1.1            locales             netcat            rsync            sasl2-bin            sudo            tini     && apt-get autoremove -yqq --purge     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 9cf0c3ac1a3b
Step 28/67 : RUN pip install --upgrade pip=="${PYTHON_PIP_VERSION}"
 ---> Using cache
 ---> 9461618a6cb2
Step 29/67 : RUN useradd --uid $ASTRONOMER_UID --create-home ${ASTRONOMER_USER}     && groupadd astrogroup --gid 101     && usermod --append --groups astrogroup ${ASTRONOMER_USER}
 ---> Using cache
 ---> a34cf6859437
Step 30/67 : FROM ${APT_DEPS_IMAGE} as devel
 ---> a34cf6859437
Step 31/67 : SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
 ---> Using cache
 ---> eefe33e545d3
Step 32/67 : ENV PIP_NO_CACHE_DIR="true"
 ---> Using cache
 ---> b26b8b1eec8d
Step 33/67 : RUN apt-get update     && curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -     && apt-get install -y --no-install-recommends         build-essential         default-libmysqlclient-dev         libffi-dev         libkrb5-dev         libpq-dev         libsasl2-dev         libssl-dev     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 44d9a1d678b5
Step 34/67 : ARG VERSION="2.3.0-1-*"
 ---> Running in fe07adf2edb0
Removing intermediate container fe07adf2edb0
 ---> ce1b787845ca
Step 35/67 : ARG SUBMODULES="async,azure,amazon,elasticsearch,google,password,cncf.kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
 ---> Running in 466474de123b
Removing intermediate container 466474de123b
 ---> 3d5e855f1f0b
Step 36/67 : ARG AIRFLOW_MODULE="astronomer_certified[${SUBMODULES}]==$VERSION"
 ---> Running in f53f2b4b1a74
Removing intermediate container f53f2b4b1a74
 ---> 48d77d872f25
Step 37/67 : ARG AIRFLOW_VERSION="2.3.0"
 ---> Running in e295064c7e0f
Removing intermediate container e295064c7e0f
 ---> c91edae1de05
Step 38/67 : ARG ASTRONOMER_FAB_SECURITY_MANAGER_VERSION="1.9.1"
 ---> Running in 21bea3cde6ff
Removing intermediate container 21bea3cde6ff
 ---> 5459a53eb263
Step 39/67 : COPY include/pip.conf /etc/pip.conf
 ---> 70ee97bac288
Step 40/67 : COPY include/pip-constraints.txt /usr/local/share/astronomer-pip-constraints.txt
 ---> a0467429a7df
Step 41/67 : COPY build-time-pip-constraints.txt /tmp/build-time-pip-constraints.txt
 ---> 0a82e8991f77
Step 42/67 : RUN pip install "${AIRFLOW_MODULE}" celery flower         --constraint /tmp/build-time-pip-constraints.txt     && pip install "https://github.com/astronomer/astronomer-airflow-scripts/releases/download/v0.0.5/astronomer_airflow_scripts-0.0.5-py3-none-any.whl"     && pip install "astronomer-fab-security-manager==${ASTRONOMER_FAB_SECURITY_MANAGER_VERSION}"
 ---> Running in 7ac63e8d3338
+ pip install 'astronomer_certified[async,azure,amazon,elasticsearch,google,password,cncf.kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==2.3.0-1-*' celery flower --constraint /tmp/build-time-pip-constraints.txt
Looking in indexes: https://pypi.org/simple, https://pip.astronomer.io/simple/
ERROR: Could not find a version that satisfies the requirement astronomer_certified[amazon,async,azure,cncf.kubernetes,elasticsearch,google,mysql,password,postgres,redis,slack,ssh,statsd,virtualenv]==2.3.0-1-* (from versions: 1.10.7.post7, 1.10.7.post8, 1.10.7.post9, 1.10.7.post10, 1.10.7.post11, 1.10.7.post12, 1.10.7.post13, 1.10.7.post14, 1.10.7.post15, 1.10.7.post16, 1.10.7.post17, 1.10.7.post18, 1.10.7.post19, 1.10.10.post1, 1.10.10.post2, 1.10.10.post3, 1.10.10.post4, 1.10.10.post5, 1.10.10.post6, 1.10.10.post7, 1.10.10.post8, 1.10.10.post9, 1.10.10.post10, 1.10.12.post1, 1.10.12.post2, 1.10.12.post3, 1.10.12.post4, 1.10.12.post5, 1.10.12.post6, 1.10.13.post1, 1.10.14.post1, 1.10.14.post2, 1.10.14.post3, 1.10.14.post4, 1.10.14.post5, 1.10.15.post1, 1.10.15.post2, 1.10.15.post3, 1.10.15.post4, 1.10.15.post5.dev3, 1.10.15.post5, 1.10.15.post6.dev1, 1.10.15.post6, 1.10.15.post7.dev1, 1.10.15.post7, 2.0.0.post1, 2.0.0.post2, 2.0.0.post3, 2.0.0.post4, 2.0.0.post5, 2.0.0.post6, 2.0.0.post7, 2.0.0.post8, 2.0.0.post9, 2.0.0.post10, 2.0.1.post1, 2.0.2.post1, 2.0.2.post2, 2.0.2.post3, 2.0.2.post4, 2.0.2.post5, 2.0.2.post6, 2.1.0.post1, 2.1.0.post2, 2.1.0.post3, 2.1.0.post4, 2.1.0.post5, 2.1.0.post6, 2.1.0.post7.dev2, 2.1.0.post7.dev5, 2.1.0.post7, 2.1.1.post1, 2.1.1.post2, 2.1.1.post3, 2.1.1.post4, 2.1.1.post5, 2.1.1.post6.dev2, 2.1.1.post6.dev5, 2.1.1.post6, 2.1.3.post1, 2.1.3.post2, 2.1.3.post3, 2.1.3.post4.dev2, 2.1.3.post4.dev5, 2.1.3.post4, 2.1.4.post1, 2.1.4.post2, 2.1.4.post3, 2.1.4.post4.dev2, 2.1.4.post4.dev7, 2.1.4.post4, 2.1.4.post5.dev4, 2.1.4.post5, 2.1.4.post6.dev1, 2.1.4.post6, 2.1.4.post7.dev1, 2.1.4.post7.dev2, 2.1.4.post7, 2.2.0.post1, 2.2.0.post2, 2.2.0.post3, 2.2.0.post4, 2.2.0.post5.dev3, 2.2.0.post5, 2.2.1.post1, 2.2.1.post2, 2.2.1.post3.dev2, 2.2.1.post3, 2.2.2.post1, 2.2.2.post2.dev2, 2.2.2.post2, 2.2.3.post1.dev918, 2.2.3.post1.dev919, 2.2.3.post1.dev924, 2.2.3.post1.dev929, 2.2.3.post1.dev942, 2.2.3.post1, 2.2.3.post2.dev1, 2.2.3.post2, 2.2.4.post1.dev1180, 2.2.4.post1.dev1190, 2.2.4.post1.dev1192, 2.2.4.post1, 2.2.4.post2.dev1, 2.2.4.post2, 2.2.4.post3.dev1, 2.2.4.post3, 2.2.4.post4.dev1, 2.2.4.post4.dev2, 2.2.4.post4, 2.2.5.post1.dev1264, 2.2.5.post1.dev1265, 2.2.5.post1, 2.2.5.post2, 2.2.5.post3.dev2, 2.3.0.dev20220323, 2.3.0.dev20220325, 2.3.0.dev20220326, 2.3.0.dev20220329, 2.3.0.dev20220330, 2.3.0.dev20220331, 2.3.0.dev20220401, 2.3.0.dev20220402, 2.3.0.dev20220405, 2.3.0.dev20220406, 2.3.0.dev20220407, 2.3.0.dev20220408, 2.3.0.dev20220409, 2.3.0.dev20220411, 2.3.0.dev20220412, 2.3.0.dev20220413, 2.3.0.dev20220414, 2.3.0.dev20220416, 2.3.0.dev20220419, 2.3.0.dev20220420, 2.3.0.dev20220421, 2.3.0.dev20220422, 2.3.0.post1.dev308, 2.3.0.post1.dev334, 2.3.0.post1.dev356)
ERROR: No matching distribution found for astronomer_certified[amazon,async,azure,cncf.kubernetes,elasticsearch,google,mysql,password,postgres,redis,slack,ssh,statsd,virtualenv]==2.3.0-1-*
WARNING: You are using pip version 21.2.4; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
The command '/bin/bash -o pipefail -e -u -x -c pip install "${AIRFLOW_MODULE}" celery flower         --constraint /tmp/build-time-pip-constraints.txt     && pip install "https://github.com/astronomer/astronomer-airflow-scripts/releases/download/v0.0.5/astronomer_airflow_scripts-0.0.5-py3-none-any.whl"     && pip install "astronomer-fab-security-manager==${ASTRONOMER_FAB_SECURITY_MANAGER_VERSION}"' returned a non-zero code: 1

should I use BULLSEYE Dockerfile for runtime6.0.4? 2.4.3/bullseye/Dockerfile

I currently use astro dev init, and then astro dev start to get airflow 2.4.3 up and running locally.

Have been looking at the parameters that can be set in Dockerfile (which basically only has a FROM line in it after astro dev init) - and found 2.4.3/bullseye/Dockerfile which is fully loaded. Is the intent behind that so we can use those parameters?

Seems like I should use the Dockerfile & requirements.txt files in this github repo locally to keep things constrained to that 2.4.3 version - is that the idea?

Docker pip install freezes when building images based on 2.2.0-1

I have tried to deploy both locally on my mac and on circleci, but in both cases it never gets past the pip install command pip install --no-cache-dir -q -r requirements.txt. Not really sure how to debug this since it uses the quiet flag.

What is the use of '[email protected]' in the docker images?

Seems a little suspicious

certificate of `https://pip.astronomer.io/simple/` is expired

Hi, I'm using astronomerinc/ap-airflow:2.1.3-3-buster image.
It seems certificate of https://pip.astronomer.io/simple/ is expired.

Add info steps after docker build

Description

One thing that has been useful elsewhere and seems useful here is to add some info steps to CI after docker build. For instance we've done docker inspect afterwards to show metadata about the build. The build URL shows up in the metadata of a docker image, which makes a nice chain to follow to get info.

Example

https://quay.io/repository/astronomer/ap-airflow/manifest/sha256:23ccc960a0db114aadfa27d81f7bb7f947c576c45269340f0b7a8bf365ad9567 shows < io.astronomer.ci.build_url=https://circleci.com/gh/astronomer/ap-airflow/44315>, and if we go there, it would be great to see things like docker inspect and pip freeze and stuff, in order to know which components were included in that built docker image.

References

https://quay.io/repository/astronomer/ap-base/manifest/sha256:ba3b7812f0091cead3bc674d24ad9cce95a2d706f1be9c33cdac57f18139a221 shows < io.astronomer.build.url=https://circleci.com/gh/astronomer/ap-base/512> which shows docker inspect output https://app.circleci.com/pipelines/github/astronomer/ap-base/113/workflows/29aea806-ebbc-4203-95c7-f10e2eca93e1/jobs/512/parallel-runs/0/steps/0-103

Reasoning

Today I was trying to find a list of packages that were contained in a docker image, and this seems like a good method for having a historical record of that information.

Connections to MSSQL fail with encryption (solution in the comment)

The issue seems to be with the version of pymssql included with either Airflow or these images. After a few days of poking around, we found a solution (thanks to this question).

FROM astronomerinc/ap-airflow:1.10.10-buster-onbuild
USER root
RUN apt-get update
RUN apt-get install -y \
      build-essential \
      freetds-bin \
      freetds-dev

RUN pip3 install Cython
RUN pip3 install --no-binary pymssql pymssql

I didn't put in a PR because I'm not sure if you'd want that backported to existing builds, or just wait until Airflow 2 is out (there's no issue with pyodbc).

No module named 'typing_extensions'

https://issues.apache.org/jira/browse/AIRFLOW-6854

1.10.10.* crashes when running with python 3.8:
File "/home/asadaoui/.virtualenvs/airflow/lib/python3.8/site-packages/airflow/serialization/json_schema.py", line 26, in
from typing_extensions import Protocol
ModuleNotFoundError: No module named 'typing_extensions'

Docker image building failure when used as base image

Hi, I was trying to use this image as the base image for our project and bumped into this issue. Here's the build log:

Step 1/18 : FROM quay.io/astronomer/ap-airflow:1.10.12-buster-onbuild
1.10.12-buster-onbuild: Pulling from astronomer/ap-airflow
852e50cd189d: Pulling fs layer
334ed303e4ad: Pulling fs layer
92c4d2410afa: Pulling fs layer
2ad460f59b0e: Pulling fs layer
029a44c5c02e: Pulling fs layer
44900377c3a4: Pulling fs layer
94e6315e644c: Pulling fs layer
0a3d218e3763: Pulling fs layer
d29d7c782ecb: Pulling fs layer
51e74141e244: Pulling fs layer
4547faff47df: Pulling fs layer
87430c4bbefd: Pulling fs layer
f8eb859db472: Pulling fs layer
a56073fffedd: Pulling fs layer
423471916159: Pulling fs layer
afd942d2534b: Pulling fs layer
30495f695d7a: Pulling fs layer
850a45aaed63: Pulling fs layer
eb3d3e6dbcba: Pulling fs layer
2ad460f59b0e: Waiting
029a44c5c02e: Waiting
44900377c3a4: Waiting
94e6315e644c: Waiting
0a3d218e3763: Waiting
d29d7c782ecb: Waiting
51e74141e244: Waiting
4547faff47df: Waiting
87430c4bbefd: Waiting
f8eb859db472: Waiting
a56073fffedd: Waiting
423471916159: Waiting
afd942d2534b: Waiting
30495f695d7a: Waiting
850a45aaed63: Waiting
eb3d3e6dbcba: Waiting
334ed303e4ad: Verifying Checksum
334ed303e4ad: Download complete
92c4d2410afa: Verifying Checksum
92c4d2410afa: Download complete
2ad460f59b0e: Verifying Checksum
2ad460f59b0e: Download complete
852e50cd189d: Verifying Checksum
852e50cd189d: Download complete
029a44c5c02e: Verifying Checksum
029a44c5c02e: Download complete
94e6315e644c: Verifying Checksum
94e6315e644c: Download complete
0a3d218e3763: Verifying Checksum
0a3d218e3763: Download complete
d29d7c782ecb: Verifying Checksum
d29d7c782ecb: Download complete
44900377c3a4: Verifying Checksum
44900377c3a4: Download complete
4547faff47df: Verifying Checksum
4547faff47df: Download complete
87430c4bbefd: Verifying Checksum
87430c4bbefd: Download complete
f8eb859db472: Verifying Checksum
f8eb859db472: Download complete
a56073fffedd: Verifying Checksum
a56073fffedd: Download complete
423471916159: Verifying Checksum
423471916159: Download complete
afd942d2534b: Verifying Checksum
afd942d2534b: Download complete
30495f695d7a: Verifying Checksum
30495f695d7a: Download complete
850a45aaed63: Verifying Checksum
850a45aaed63: Download complete
eb3d3e6dbcba: Verifying Checksum
eb3d3e6dbcba: Download complete
51e74141e244: Verifying Checksum
51e74141e244: Download complete
852e50cd189d: Pull complete
334ed303e4ad: Pull complete
92c4d2410afa: Pull complete
2ad460f59b0e: Pull complete
029a44c5c02e: Pull complete
44900377c3a4: Pull complete
94e6315e644c: Pull complete
0a3d218e3763: Pull complete
d29d7c782ecb: Pull complete
51e74141e244: Pull complete
4547faff47df: Pull complete
87430c4bbefd: Pull complete
f8eb859db472: Pull complete
a56073fffedd: Pull complete
423471916159: Pull complete
afd942d2534b: Pull complete
30495f695d7a: Pull complete
850a45aaed63: Pull complete
eb3d3e6dbcba: Pull complete
Digest: sha256:25c05d46fd4e6f00678bffd77f59fa046bdfc22bef873b0f42f55b7924317b1d
Status: Downloaded newer image for quay.io/astronomer/ap-airflow:1.10.12-buster-onbuild
# Executing 7 build triggers
---> Running in f5924208b34e
Removing intermediate container f5924208b34e
---> Running in 12e04f12762d
�[91m+ [[ -s packages.txt ]]
�[0mRemoving intermediate container 12e04f12762d
---> Running in 732e91387793
�[91m+ grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt
+ pip install --no-cache-dir -q -r requirements.txt
�[0m�[91mWARNING: You are using pip version 19.3.1; however, version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
�[0mRemoving intermediate container 732e91387793
---> Running in c2cb9bd1d200
Removing intermediate container c2cb9bd1d200
---> 67f7e6cc9e06
Step 2/18 : RUN apt-get update && apt-get -y install build-essential sudo git && git clone https://github.com/ncopa/su-exec.git /tmp/su-exec && cd /tmp/su-exec && make && chmod 770 su-exec && mv ./su-exec /usr/local/sbin/
---> Running in 737a1ad5d22f
�[91m+ apt-get update
�[0mReading package lists...
�[91mE: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
�[0mThe command '/bin/bash -o pipefail -e -u -x -c apt-get update && apt-get -y install build-essential sudo git && git clone https://github.com/ncopa/su-exec.git /tmp/su-exec && cd /tmp/su-exec && make && chmod 770 su-exec && mv ./su-exec /usr/local/sbin/' returned a non-zero code: 100
build hook failed! (100)

Microsoft ODBC Driver for SQL Server not supported in Astronomer Certified 2.2.0-1

While testing out an upgrade from Airflow 2.1.4 to 2.2.2 in my dev environment I noticed a breaking change while the image was building.

The reason the image failed to build is because I am installing msodbcsql17 in my Dockerfile using the described method as shown here: https://www.astronomer.io/docs/enterprise/v0.26/astronomer-certified/install-packages. The build fails because it is unable to locate the package msodbcsql17. If I take out installing msodbcsql17 the image builds successfully. This was not an issue with Astronomer Certified 2.1.4.

I found out Astronomer Certified 2.2.0-1 upgrades from Debian 10 ("Buster") to Debian 11 ("Bullseye") (PR for this is #298), however the Microsoft ODBC Driver for SQL Server is not supported on Debian Bullseye (see https://github.com/MicrosoftDocs/sql-docs/issues/6804), so we can't use Airflow’s OdbcHook with Microsoft SQL Server anymore in Astronomer Certified 2.2.0+ until Microsoft releases a msodbcsql version that supports Bullseye (possibly in January 2022 according the twice a year release frequency).

Also, is it possible you guys might have prematurely upgraded to Debian Bullseye as Airflow seems like it currently does not support Debian Bullseye due to this same MSSQL issue? (see discussion on this Airflow PR: apache/airflow#18279)

Question: Is it possible to keep Astronomer Certified 2.2.0+ images for Buster until there is a msodbcsql version that supports Bullseye?

In the meantime I am switching all of my DAGs from using OdbcHook to MsSqlHook so that I can upgrade Airflow past 2.1.4.

Document or remove DEV_ALLOWLIST from CircleCI config

It's not super clear what DEV_ALLOWLIST is intended for or whether or not it's still relevant, and it complicates a bunch of logic in our Jinja template (see below). We should either document it's use case(s), or we should remove it entirely.

1. ap-airflow/.circleci/config.yml.j2

   Line 50 in 5214a32

   {%- if "dev" in ac_version and airflow_version not in dev_allowlist %}

2. ap-airflow/.circleci/config.yml.j2

   Line 115 in 5214a32

   {%- if ("dev" not in ac_version or airflow_version in dev_allowlist) and not edge_build %}

3. ap-airflow/.circleci/config.yml.j2

   Line 162 in 5214a32

   {%- if ("dev" not in ac_version or airflow_version in dev_allowlist) and not edge_build %}

4. ap-airflow/.circleci/config.yml.j2

   Line 189 in 5214a32

   {%- if ("dev" not in ac_version or airflow_version in dev_allowlist) and not edge_build %}

5. ap-airflow/.circleci/config.yml.j2

   Line 232 in 5214a32

   {%- if airflow_version not in dev_allowlist %}

Change Snyk and Trivy scan steps to advisory for dev builds

Our workflows currently require that the Trivy and Snyk scans all pass before pushing any builds at all, even dev builds.

But, at the moment, that's compounding our issues, because we not only have a bad constraints file that needs updating, we also now need to resolve a CVE in gzip/liblzma5 to push any images at all, even images for dev versions:

I think we should still run the Snyk and Trivy checks, but we shouldn't gate dev image pushes on them. Turn them into advisory checks.

We should still (eg: leave this alone) require those checks to pass when we push out images for actual release versions.

Edit: fix should -> shouldn't

Release 2.2.5

Better spacing or more controls over CSS for web UI

I am not sure if this is the correct place for this request, so let me know if i need to move it somewhere better.
I recently started using tags in our Astronomer instance and while they are very useful, the way the UI is currently styled makes it feel very busy and cramped. I tried to find a way to just override some of the styling, which seems possible in base airflow but not in astronomer. Can we get some better styling for the DAG list, or atleast a better way to apply styles ourselves? Here's an example of my concern.

without styling

Changing the tags font-size to .75em and the DAG name font-size to 1.25em. I feel like this greatly improves the ability to quickly skim the names while keeping the tags readable.

If there is a way to do this and i just missed it, please let me know.

clean_airflow_logs.sh does not handle SIGTERM properly

The script clean-airflow-logs.sh performs log trimming inside an infinite while loop. The loop uses sleep to wait until the next scheduled cleanup and exits when SIGINT or SIGTERM is recieved. However, Bash does not execute traps until the current command has finished. [0] That means, when SIGTERM arrives, the signal handler will not be called until sleep has finished.

When this scripts runs as a container in a Kubernetes pod this behavior causes potentially large delays when terminating the Pod. Kubernetes sends SIGTERM to all containers of a Pod when terminating it. Since the container running the clean-airflow-logs.sh script is currently sleeping, it may take a long time for it to react to SIGTERM. As a result, the container exceeds the configured termination grace period and gets killed by the Kubernetes Pod controller.

I'll happily provide a patch to address the issue. Are you willing to accept a contribution for this?

[0] see Bash Manual section 3.7.6

Edge builds: Include more data from CircleCI build environment in Docker images

We currently use the latest-main.build.json file to collect data from the package build environment when we add labels to the Docker images. However, we do not include any/much information from the CircleCI build environment in the built Docker image.

It would be smart to include more information from the CircleCI environment, possibly including a select few environment variables, in the Docker image labels. At the very least we should include a link to the CircleCI build workflow and/or build job so we and QA have an easier time "following the breadcrumbs" back if/when they are diagnosing issues possibly caused by the build environment.

Update pip version installed in images

As discussed here, older versions of pip cause sqlalchemy to update beyond what we want when apache-airflow-providers-snowflake is installed. There is a workaround to pair apache-airflow-providers-snowflake with a restraint of sqlalchemy at the same time, but it would be simpler to just have pip figure it out.

Enable providing PIP_INDEX_URL as a Docker --build-arg

Currently, it's not possible to add Python packages from internal pip indexes to the requirements.txt file and have them installed when building images FROM astronomerinc/ap-airflow:*-onbuild.

Adding the following line right before the ONBUILD RUN pip install -r requirements.txt command:

ONBUILD ARG PIP_INDEX_URL

would enable users of the Dockerfile to build an image specifying a custom PIP_INDEX_URL (e.g. pointing to an internal company pip index) by adding a --build-arg PIP_INDEX_URL https://username:[email protected]/ to the docker build command, and include internal packages in the requirements.txt file.

A use case for this is for example adding an on_failure_callback to DAGs, that uses internal Python libraries for custom logging and incident handling.

Consolidate or simplify the Slack notifications for releases

The current Slack notifications/release approvals are sent in multiple separate messages for each potential version release. This eats up a lot of vertical space and makes it difficult to pick out individual versions.

It would be nice to either consolidate the multiple version approvals into a single Slack message, or to just include a link to the CircleCI workflow to manually approve releases.

I have used Slack's Block Kit builder to prototype a new message format. The content is the same, the only difference is dark/light theme.

It might be cool to render the Astronomer logo instead of an extra large ✋ emoji to differentiate and separate release approval notifications.

The JSON for this prototype is inlined below.

Click to expand JSON content (it's a lot)

{
  "blocks": [
    {
      "type": "section",
      "fields": [
        {
          "type": "plain_text",
          "text": "New build request"
        },
        {
          "type": "mrkdwn",
          "text": "*New build request (Markdown)!*"
        }
      ]
    },
    {
      "type": "divider"
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*CircleCI Workflow*:\n    <https://app.circleci.com/pipelines/github/astronomer/ap-airflow/5678/workflows/11112222-3333-4444-5555-666677778888|CircleCI#4321>\nPull Request:\n    <https://github.com/astronomer/ap-airflow/pull/1234|astronomer/ap-airflow#1234>"
      },
      "accessory": {
        "type": "image",
        "image_url": "https://i.imgur.com/3jf0wLj.png",
        "alt_text": "hand thumbnail"
      }
    },
    {
      "type": "context",
      "elements": [
        {
          "type": "image",
          "image_url": "https://api.slack.com/img/blocks/bkb_template_images/notificationsWarningIcon.png",
          "alt_text": "notifications warning icon"
        },
        {
          "type": "mrkdwn",
          "text": "Edge builds not included"
        }
      ]
    },
    {
      "type": "divider"
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Approve a release:*"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`1.10.10`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`1.10.12`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`1.10.14`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`1.10.15`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.0.0`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.0.2`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.1.0`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.1.1`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.1.3`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.1.4`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.2.0`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.2.1`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.2.2`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "`2.2.3`"
      },
      "accessory": {
        "type": "button",
        "text": {
          "type": "plain_text",
          "emoji": true,
          "text": "Approve :white_check_mark:"
        },
        "value": "click_me_123"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*<fakelink.ToMoreTimes.com|Show more releases>*"
      }
    }
  ]
}

Broken Dag issue when importing from outside of dag

How do I install pip installable modules? do i need to pass the requirement.txt in the dags folder?

Add Slack notifications for failed nightly builds

We currently have no way of notifying people about failed builds (I'm not sure who all gets emailed about CircleCI nightly build failures, but I'm not included). Additionally, notifying a Slack channel would allow more people to more easily subscribe to these updates.