how to use variables ? about airflow

[mistercrunch]

I need to add an entry in the documentation to explain this, but variables are general purpose and used to control or alter the behavior of specific pipelines that define logic based on the content of variables.

For instance there could be a convention that an "aggregation framework" would look for variables starting with the prefix aggfwk_ and perhaps read a JSON config from the variable. The JSON config would define the metadata for the aggregation framework needs to operate. Adding new variables would create new pipelines or sets of pipes.

So it's just a place to store metadata that pipelines can read from. Arguably most of this metadata should be code and live in source control, but sometimes we need some settings to be more reactive and that's when we use variables.

Our only internal use case now is a backfilling framework that allows us to run huge backfills efficiently.

Another thought is Airflow doesn't expose a way to communicate across tasks, or even for an upstream task influence the behavior of a downstream tasks. Maybe variables could be used for that purpose, but that's somewhat messy.

[m1racoli]

How can variables be accessed?
Is there an API to access variables defined in the UI?
I assume they are stored in the database?

We want to use variables for providing development and production flags within the creation of the DAG or within the task template.

[mistercrunch]

Variable is a sqlalchemy model and is located at airflow.models.Variable

I just added a simple utility classmethod to easily Variable.get(key, deser_json=False)
9c36068

This makes it a very concise call to get a variable.

from airflow.models import Variable
foo = Variable.get("foo")

Though this is not in the current pypi release, it will make it to 1.1.0 . In the meantime you can look at the method for the more verbose way to get the variable.

[m1racoli]

Thank's a lot! That was, what I was looking for.
I just discovered the 1.1.0 release on pypi as well.

[Yongyiw]

Just asking if it is possible to set Variables in code(like in some hook class). Cause I want to save the http connection cookies as variables.

[Yongyiw]

Never mind, I found my answer here: #220