puckel / docker-airflow Goto Github PK
View Code? Open in Web Editor NEWDocker Apache Airflow
License: Apache License 2.0
Docker Apache Airflow
License: Apache License 2.0
What I'm trying to accomplish
connect to a Teradata database using the jdbc connection type.
Error message I'm receiving
java.lang.RuntimeException: Class com.teradata.jdbc.TeraDriver not found
Setup
volumes:
- /usr/local/airflow_project/airflow/:/usr/local/airflow/
mdir /usr/local/airflow/drivers/teradata/
copy terajdbc4.jar
copy tdgssconfig.jar
driver path
/usr/local/airflow_project/airflow/drivers/teradata/terajdbc4.jar:/usr/local/airflow_project/airflow/drivers/teradata/tdgssconfig.jar
Steps I've taken that moved the issue forward
&& apt-get install -yqq default-jre \
to the dockerfile. I was getting a java error previouslySteps I've tried since then
Python code
"""
General hook for jdbc db access.
If a connection id is specified, host, port, schema, username and password will be taken from the predefined connection.
Raises an airflow error if the given connection id doesn't exist.
Otherwise host, port, schema, username and password can be specified on the fly.
:param jdbc_url: jdbc connection url
:type jdbc_url: string
:param jdbc_driver_name: jdbc driver name
:type jdbc_driver_name: string
:param jdbc_driver_loc: path to jdbc driver
:type jdbc_driver_loc: string
:param conn_id: reference to a predefined database
:type conn_id: string
:param sql: the sql code to be executed
:type sql: string or string pointing to a template file. File must have
a '.sql' extensions.
"""
import jaydebeapi
conn_name_attr = 'jdbc_conn_id'
default_conn_name = 'jdbc_default'
supports_autocommit = True
def get_conn(self):
#conn = self.get_connection(getattr(self, self.conn_name_attr))
#host = conn.host
#login = conn.login
#psw = conn.password
#jdbc_driver_loc = conn.extra_dejson.get('extra__jdbc__drv_path')
#jdbc_driver_name = conn.extra_dejson.get('extra__jdbc__drv_clsname')
conn = jaydebeapi.connect(jdbc_driver_name,
[str(host), str(login), str(psw)],
jdbc_driver_loc,)
return conn
conn = jaydebeapi.connect('com.teradata.jdbc.TeraDriver',
['server','user','password'],
['/Users/user/drivers/tdgssconfig.jar',
'/Users/user/drivers/terajdbc4.jar'])
Hi,
when trying to build puckel/docker-airflow i got this error: Cython.Compiler.Errors.InternalError: Internal compiler error: 'algos_common_helper.pxi' not found
Downloading/unpacking pandas<1.0.0,>=0.15.2 (from airflow[celery,hdfs,hive,postgres]==1.7.1.3)
Running setup.py (path:/tmp/pip-build-fReTFb/pandas/setup.py) egg_info for package pandas
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
pandas/index.pyx: cannot find cimported module 'datetime'
pandas/index.pyx: cannot find cimported module 'util'
./pandas/hashtable.pxd: cannot find cimported module 'khash'
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip-build-fReTFb/pandas/setup.py", line 680, in <module>
**setuptools_kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "<string>", line 15, in replacement_run
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 306, in find_sources
mm.run()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 533, in run
self.add_defaults()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 562, in add_defaults
sdist.add_defaults(self)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 19, in finalize_options
self.distribution.ext_modules)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 809, in cythonize
aliases=aliases)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 714, in create_extension_list
kwds = deps.distutils_info(file, aliases, base).values
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 590, in distutils_info
return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 600, in transitive_merge
node, extract, merge, seen, {}, self.cimported_files)[0]
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 605, in transitive_merge_helper
deps = extract(node)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 581, in distutils_info0
externs = self.cimports_and_externs(filename)[1]
File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper
res = cache[args] = f(self, *args)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 491, in cimports_and_externs
for include in self.included_files(filename):
File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper
res = cache[args] = f(self, *args)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 474, in included_files
include_path = self.context.find_include_file(include, None)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 274, in find_include_file
error(pos, "'%s' not found" % filename)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Errors.py", line 177, in error
raise InternalError(message)
Cython.Compiler.Errors.InternalError: Internal compiler error: 'algos_common_helper.pxi' not found
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/pandas.egg-info
writing requirements to pip-egg-info/pandas.egg-info/requires.txt
writing pip-egg-info/pandas.egg-info/PKG-INFO
writing top-level names to pip-egg-info/pandas.egg-info/top_level.txt
writing dependency_links to pip-egg-info/pandas.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/pandas.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
package init file 'pandas/io/tests/sas/init.py' not found (or not a regular file)
pandas/index.pyx: cannot find cimported module 'datetime'
pandas/index.pyx: cannot find cimported module 'util'
./pandas/hashtable.pxd: cannot find cimported module 'khash'
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip-build-fReTFb/pandas/setup.py", line 680, in
**setuptools_kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "", line 15, in replacement_run
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 306, in find_sources
mm.run()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 533, in run
self.add_defaults()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 562, in add_defaults
sdist.add_defaults(self)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 19, in finalize_options
self.distribution.ext_modules)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 809, in cythonize
aliases=aliases)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 714, in create_extension_list
kwds = deps.distutils_info(file, aliases, base).values
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 590, in distutils_info
return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 600, in transitive_merge
node, extract, merge, seen, {}, self.cimported_files)[0]
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 605, in transitive_merge_helper
deps = extract(node)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 581, in distutils_info0
externs = self.cimports_and_externs(filename)[1]
File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper
res = cache[args] = f(self, *args)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 491, in cimports_and_externs
for include in self.included_files(filename):
File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 44, in wrapper
res = cache[args] = f(self, *args)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 474, in included_files
include_path = self.context.find_include_file(include, None)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 274, in find_include_file
error(pos, "'%s' not found" % filename)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Errors.py", line 177, in error
raise InternalError(message)
Cython.Compiler.Errors.InternalError: Internal compiler error: 'algos_common_helper.pxi' not found
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-build-fReTFb/pandas
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c set -ex && buildDeps=' python-dev python-pip libkrb5-dev libsasl2-dev libssl-dev libffi-dev build-essential libblas-dev liblapack-dev ' && echo "deb http://http.debian.net/debian jessie-backports main" >/etc/apt/sources.list.d/backports.list && apt-get update -yqq && apt-get install -yqq --no-install-recommends
any ideas ?
Hi, great progress on packaging airflow as containers, thanks!
After pulling and building the latest version of the docker images I'm still not able to access the data-profiling features. For ad-hoc queries and charts I'm getting this error:
(2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
Any advise?
This is really cool, thanks for putting it together. What in your view would be the best way to update DAGs in the context of this deployment? Would you suggest just forking this repo and updating the whole docker-compose deployment every time one needs to update a DAG, or possibly placing the DAGs in a container volume mounted in the Airflow containers?
If i clone this repo then run docker-compose up
it seems everything starts OK until an example DAG is run. Then the whole thing fails.
scheduler_1 | [2016-03-08 10:38:03,426] {jobs.py:632} ERROR - (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT task_instance.task_id AS task_instance_task_id, task_instance.dag_id AS task_instance_dag_id, task_instance.execution_date AS task_instance_execution_date, task_instance.start_date AS task_instance_start_date, task_instance.end_date AS task_instance_end_date, task_instance.duration AS task_instance_duration, task_instance.state AS task_instance_state, task_instance.try_number AS task_instance_try_number, task_instance.hostname AS task_instance_hostname, task_instance.unixname AS task_instance_unixname, task_instance.job_id AS task_instance_job_id, task_instance.pool AS task_instance_pool, task_instance.queue AS task_instance_queue, task_instance.priority_weight AS task_instance_priority_weight, task_instance.operator AS task_instance_operator, task_instance.queued_dttm AS task_instance_queued_dttm \nFROM task_instance \nWHERE task_instance.dag_id = %s AND task_instance.task_id = %s AND task_instance.execution_date = %s \n LIMIT %s'] [parameters: ('example_bash_operator', 'also_run_this', datetime.datetime(2016, 3, 1, 0, 0), 1)]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 629, in _execute
scheduler_1 | self.process_dag(dag, executor)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 475, in process_dag
scheduler_1 | ti.refresh_from_db()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 667, in refresh_from_db
scheduler_1 | TI.execution_date == self.execution_date,
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first
scheduler_1 | ret = list(self[0:1])
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
scheduler_1 | return list(res)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
scheduler_1 | context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
scheduler_1 | context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
scheduler_1 | cursor.execute(statement, parameters)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute
scheduler_1 | self.errorhandler(self, exc, value)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
scheduler_1 | raise errorvalue
scheduler_1 | OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT task_instance.task_id AS task_instance_task_id, task_instance.dag_id AS task_instance_dag_id, task_instance.execution_date AS task_instance_execution_date, task_instance.start_date AS task_instance_start_date, task_instance.end_date AS task_instance_end_date, task_instance.duration AS task_instance_duration, task_instance.state AS task_instance_state, task_instance.try_number AS task_instance_try_number, task_instance.hostname AS task_instance_hostname, task_instance.unixname AS task_instance_unixname, task_instance.job_id AS task_instance_job_id, task_instance.pool AS task_instance_pool, task_instance.queue AS task_instance_queue, task_instance.priority_weight AS task_instance_priority_weight, task_instance.operator AS task_instance_operator, task_instance.queued_dttm AS task_instance_queued_dttm \nFROM task_instance \nWHERE task_instance.dag_id = %s AND task_instance.task_id = %s AND task_instance.execution_date = %s \n LIMIT %s'] [parameters: ('example_bash_operator', 'also_run_this', datetime.datetime(2016, 3, 1, 0, 0), 1)]
scheduler_1 | [2016-03-08 10:38:03,496] {jobs.py:653} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 624, in _execute
scheduler_1 | dag = dagbag.get_dag(dag.dag_id)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 155, in get_dag
scheduler_1 | orm_dag = DagModel.get_current(dag_id)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1907, in get_current
scheduler_1 | obj = session.query(cls).filter(cls.dag_id == dag_id).first()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first
scheduler_1 | ret = list(self[0:1])
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
scheduler_1 | return list(res)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,508] {jobs.py:602} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 600, in _execute
scheduler_1 | self.prioritize_queued(executor=executor, dagbag=dagbag)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/utils.py", line 132, in wrapper
scheduler_1 | result = func(*args, **kwargs)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 500, in prioritize_queued
scheduler_1 | pools = {p.pool: p for p in session.query(models.Pool).all()}
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
scheduler_1 | return list(self)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,517] {jobs.py:653} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 621, in _execute
scheduler_1 | paused_dag_ids = dagbag.paused_dags()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 339, in paused_dags
scheduler_1 | DagModel.is_paused == True)]
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,523] {jobs.py:602} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 600, in _execute
scheduler_1 | self.prioritize_queued(executor=executor, dagbag=dagbag)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/utils.py", line 132, in wrapper
scheduler_1 | result = func(*args, **kwargs)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 500, in prioritize_queued
scheduler_1 | pools = {p.pool: p for p in session.query(models.Pool).all()}
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
scheduler_1 | return list(self)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,528] {jobs.py:653} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 621, in _execute
scheduler_1 | paused_dag_ids = dagbag.paused_dags()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 339, in paused_dags
scheduler_1 | DagModel.is_paused == True)]
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,531] {jobs.py:602} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 600, in _execute
scheduler_1 | self.prioritize_queued(executor=executor, dagbag=dagbag)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/utils.py", line 132, in wrapper
scheduler_1 | result = func(*args, **kwargs)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 500, in prioritize_queued
scheduler_1 | pools = {p.pool: p for p in session.query(models.Pool).all()}
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
scheduler_1 | return list(self)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,536] {jobs.py:653} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 621, in _execute
scheduler_1 | paused_dag_ids = dagbag.paused_dags()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 339, in paused_dags
scheduler_1 | DagModel.is_paused == True)]
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,540] {jobs.py:602} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 600, in _execute
scheduler_1 | self.prioritize_queued(executor=executor, dagbag=dagbag)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/utils.py", line 132, in wrapper
scheduler_1 | result = func(*args, **kwargs)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 500, in prioritize_queued
scheduler_1 | pools = {p.pool: p for p in session.query(models.Pool).all()}
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
scheduler_1 | return list(self)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,551] {jobs.py:653} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 621, in _execute
scheduler_1 | paused_dag_ids = dagbag.paused_dags()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 339, in paused_dags
scheduler_1 | DagModel.is_paused == True)]
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,557] {jobs.py:602} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 600, in _execute
scheduler_1 | self.prioritize_queued(executor=executor, dagbag=dagbag)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/utils.py", line 132, in wrapper
scheduler_1 | result = func(*args, **kwargs)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 500, in prioritize_queued
scheduler_1 | pools = {p.pool: p for p in session.query(models.Pool).all()}
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
scheduler_1 | return list(self)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,563] {jobs.py:653} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 621, in _execute
scheduler_1 | paused_dag_ids = dagbag.paused_dags()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 339, in paused_dags
scheduler_1 | DagModel.is_paused == True)]
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.is_paused = true'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,565] {jobs.py:602} ERROR - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | Traceback (most recent call last):
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 600, in _execute
scheduler_1 | self.prioritize_queued(executor=executor, dagbag=dagbag)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/utils.py", line 132, in wrapper
scheduler_1 | result = func(*args, **kwargs)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 500, in prioritize_queued
scheduler_1 | pools = {p.pool: p for p in session.query(models.Pool).all()}
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
scheduler_1 | return list(self)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
scheduler_1 | return self._execute_and_instances(context)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
scheduler_1 | result = conn.execute(querycontext.statement, self._params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
scheduler_1 | return meth(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
scheduler_1 | return connection._execute_clauseelement(self, multiparams, params)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
scheduler_1 | compiled_sql, distilled_params
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
scheduler_1 | None, None)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
scheduler_1 | exc_info
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
scheduler_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1071, in _execute_context
scheduler_1 | conn = self._revalidate_connection()
scheduler_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 391, in _revalidate_connection
scheduler_1 | "Can't reconnect until invalid "
scheduler_1 | StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description \nFROM slot_pool'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,567] {models.py:124} INFO - Filling up the DagBag from /usr/local/airflow/dags
scheduler_1 | [2016-03-08 10:38:03,570] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_http_operator.py
scheduler_1 | [2016-03-08 10:38:03,644] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,646] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_python_operator.py
scheduler_1 | [2016-03-08 10:38:03,662] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,667] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_branch_operator.py
scheduler_1 | [2016-03-08 10:38:03,689] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,693] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_xcom.py
scheduler_1 | [2016-03-08 10:38:03,700] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,703] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_short_circuit_operator.py
scheduler_1 | [2016-03-08 10:38:03,711] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,713] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_bash_operator.py
scheduler_1 | [2016-03-08 10:38:03,715] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,720] {models.py:197} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/tutorial.py
scheduler_1 | [2016-03-08 10:38:03,725] {models.py:324} WARNING - (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE dag.dag_id = %s \n LIMIT %s'] [parameters: [immutabledict({})]]
scheduler_1 | [2016-03-08 10:38:03,726] {jobs.py:611} ERROR - Failed at reloading the dagbag
webserver_1 | [2016-03-08 10:38:03 +0000] [41] [ERROR] Exception in worker process:
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | [2016-03-08 10:38:03 +0000] [44] [ERROR] Exception in worker process:
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | [2016-03-08 10:38:03 +0000] [41] [INFO] Worker exiting (pid: 41)
webserver_1 | [2016-03-08 10:38:03 +0000] [43] [ERROR] Exception in worker process:
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | [2016-03-08 10:38:03 +0000] [44] [INFO] Worker exiting (pid: 44)
webserver_1 | [2016-03-08 10:38:03 +0000] [43] [INFO] Worker exiting (pid: 43)
webserver_1 | [2016-03-08 10:38:03 +0000] [42] [ERROR] Exception in worker process:
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
webserver_1 | worker.init_process()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
webserver_1 | self.load_wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
webserver_1 | self.wsgi = self.app.wsgi()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1 | self.callable = self.load()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
webserver_1 | return self.load_wsgiapp()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
webserver_1 | return util.import_app(self.app_uri)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 368, in import_app
webserver_1 | app = eval(obj, mod.__dict__)
webserver_1 | File "<string>", line 1, in <module>
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 119, in cached_app
webserver_1 | app = create_app(config)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 39, in create_app
webserver_1 | from airflow.www import views
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1687, in <module>
webserver_1 | class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 1771, in ChartModelView
webserver_1 | .group_by(models.Connection.conn_id)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
webserver_1 | return self._execute_and_instances(context)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
webserver_1 | close_with_result=True)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
webserver_1 | **kw)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
webserver_1 | execution_options=execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
webserver_1 | engine, execution_options)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
webserver_1 | conn = bind.contextual_connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
webserver_1 | self._wrap_pool_connect(self.pool.connect, None),
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
webserver_1 | e, dialect, self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
webserver_1 | exc_info
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
webserver_1 | return fn()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
webserver_1 | return _ConnectionFairy._checkout(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
webserver_1 | fairy = _ConnectionRecord.checkout(pool)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
webserver_1 | rec = pool._do_get()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
webserver_1 | self._dec_overflow()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
webserver_1 | compat.reraise(exc_type, exc_value, exc_tb)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
webserver_1 | return self._create_connection()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
webserver_1 | return _ConnectionRecord(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
webserver_1 | self.connection = self.__connect()
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
webserver_1 | connection = self.__pool._invoke_creator(self)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
webserver_1 | return dialect.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
webserver_1 | return self.dbapi.connect(*cargs, **cparams)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
webserver_1 | return Connection(*args, **kwargs)
webserver_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
webserver_1 | super(Connection, self).__init__(*args, **kwargs2)
webserver_1 | OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'mysql' (111)")
webserver_1 | [2016-03-08 10:38:03 +0000] [42] [INFO] Worker exiting (pid: 42)
I tried to re-build docker-airflow, and I get the following error:
Installing collected packages: cryptography, idna, pyasn1, setuptools, enum34, ipaddress, cffi, appdirs, packaging, pycparser, pyparsing
Running setup.py install for cryptography
Installed /tmp/pip-build-WXBrEA/cryptography/cffi-1.9.1-py2.7-linux-x86_64.egg
Searching for pycparser
Reading https://pypi.python.org/simple/pycparser/
Best match: pycparser 2.17
Downloading https://pypi.python.org/packages/be/64/1bb257ffb17d01f4a38d7ce686809a736837ad4371bcc5c42ba7a715c3ac/pycparser-2.17.tar.gz#md5=ca98dcb50bc1276f230118f6af5a40c7
Processing pycparser-2.17.tar.gz
Writing /tmp/easy_install-P3qata/pycparser-2.17/setup.cfg
Running pycparser-2.17/setup.py -q bdist_egg --dist-dir /tmp/easy_install-P3qata/pycparser-2.17/egg-dist-tmp-nrAJxO
warning: no previously-included files matching 'yacctab.*' found under directory 'tests'
warning: no previously-included files matching 'lextab.*' found under directory 'tests'
warning: no previously-included files matching 'yacctab.*' found under directory 'examples'
warning: no previously-included files matching 'lextab.*' found under directory 'examples'
zip_safe flag not set; analyzing archive contents...
pycparser.ply.yacc: module references __file__
pycparser.ply.yacc: module MAY be using inspect.getsourcefile
pycparser.ply.yacc: module MAY be using inspect.stack
pycparser.ply.lex: module references __file__
pycparser.ply.lex: module MAY be using inspect.getsourcefile
pycparser.ply.ygen: module references __file__
Installed /tmp/pip-build-WXBrEA/cryptography/pycparser-2.17-py2.7.egg
no previously-included directories found matching 'docs/_build'
warning: no previously-included files matching '*' found under directory 'vectors'
generating cffi module 'build/temp.linux-x86_64-2.7/_padding.c'
generating cffi module 'build/temp.linux-x86_64-2.7/_constant_time.c'
generating cffi module 'build/temp.linux-x86_64-2.7/_openssl.c'
building '_openssl' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_openssl.so
building '_constant_time' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_constant_time.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_constant_time.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_constant_time.o -o build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_constant_time.so
building '_padding' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_padding.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_padding.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_padding.o -o build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_padding.so
Found existing installation: setuptools 5.5.1
Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, owned by OS
Running setup.py install for cffi
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 12, in <module>
import setuptools.version
File "/usr/local/lib/python2.7/dist-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 70, in <module>
import packaging.version
ImportError: No module named packaging.version
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-WXBrEA/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-7DvuKF-record/install-record.txt --single-version-externally-managed --compile:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 12, in <module>
import setuptools.version
File "/usr/local/lib/python2.7/dist-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 70, in <module>
import packaging.version
ImportError: No module named packaging.version
Looks like the version of pip is pip 1.5.6
whereas upgrading to pip 9+ seems to fix the issue. (cf. pypa/setuptools#937).
Hi @puckel ,
i wanted to ask how can i get the last Airflow commits to my container ?
Is there like a frequent update option from the Airflow's master branch ?
Hi @puckel ,
are you planning a new docker container for the new Airflow 1.8.0 that will be out in a couple of days?
I have been using you airflow docker container for months now. great job and thanks
Hello, I had no problem setting up ldap authentication before,
and after couple modification to the code, I am no longer able to bind ldap connection with webserver showing AuthenticationError: Cannot bind to ldap server
.
I am able to telnet from the container, so I know this isnt the network problem..
If I can get any help or way to debug this, it would be really really helpful since I've been stuck with this for couple days now.
Thanks in advance for any help.
[2017-03-28 14:53:07,932] {models.py:266} ERROR - Failed to import: /usr/local/lib/python3.5/site-packages/airflow/example_dags/example_http_operator.py
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/airflow/models.py", line 263, in process_file
m = imp.load_source(mod_name, filepath)
File "/usr/local/lib/python3.5/imp.py", line 172, in load_source
module = _load(spec)
File "", line 693, in _load
File "", line 673, in _load_unlocked
File "", line 673, in exec_module
File "", line 222, in _call_with_frames_removed
File "/usr/local/lib/python3.5/site-packages/airflow/example_dags/example_http_operator.py", line 20, in
from airflow.operators.sensors import HttpSensor
File "/usr/local/lib/python3.5/site-packages/airflow/operators/sensors.py", line 33, in
from airflow.hooks.hdfs_hook import HDFSHook
File "/usr/local/lib/python3.5/site-packages/airflow/hooks/hdfs_hook.py", line 20, in
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
File "/usr/local/lib/python3.5/site-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
^
SyntaxError: invalid syntax
Hi @puckel ,
if i may ask, what are the pro/cons of using redis with airflow in comparison to RabbitMQ?
I'm borrowing a lot from your docker compose to deploy in an openshift environment and the Flower pod crashes consistently with the following error...
Traceback (most recent call last):
File "/usr/local/bin/flower", line 11, in <module>
load_entry_point('flower==0.9.1', 'console_scripts', 'flower')()
File "/usr/local/lib/python2.7/dist-packages/flower/__main__.py", line 11, in main
flower.execute_from_commandline()
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 311, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/local/lib/python2.7/dist-packages/flower/command.py", line 59, in handle_argv
return self.run_from_argv(prog_name, argv)
File "/usr/local/lib/python2.7/dist-packages/flower/command.py", line 36, in run_from_argv
self.apply_env_options()
File "/usr/local/lib/python2.7/dist-packages/flower/command.py", line 71, in apply_env_options
value = option.type(value)
ValueError: invalid literal for int() with base 10: 'tcp://172.30.195.95:5555'
I think this is due to the fact that the pod is trying to access celery via IP rather than service name. The flower pod works fine and actually connects to http://localhost:5555
when I import the service from using the kube-airflow
project. I'm having trouble finding where the IP address is passed into the Flower pod so I can change it from IP to service name. Can you point me in the right direction as to which file I should modify to fix the issues with the Flower pod?
Hi,
Thanks for the docker image. When I run a pipeline , more often I find that the scheduler is quite unstable., it starts of with the error : ERROR - [Errno 104] Connection reset by peer
The error log for your ref;
ERROR - [Errno 104] Connection reset by peer
2016-06-17T11:42:51.799483436Z Traceback (most recent call last):
2016-06-17T11:42:51.799494203Z File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
2016-06-17T11:42:51.799501615Z executor.heartbeat()
2016-06-17T11:42:51.799507897Z File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
2016-06-17T11:42:51.799514286Z self.sync()
2016-06-17T11:42:51.799520586Z File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
2016-06-17T11:42:51.799527236Z state = async.state
2016-06-17T11:42:51.799533355Z File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
2016-06-17T11:42:51.799540325Z return self._get_task_meta()['status']
2016-06-17T11:42:51.799546778Z File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
2016-06-17T11:42:51.799553135Z return self._maybe_set_cache(self.backend.get_task_meta(self.id))
2016-06-17T11:42:51.799559187Z File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
2016-06-17T11:42:51.799565969Z binding.declare()
2016-06-17T11:42:51.799571830Z File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
2016-06-17T11:42:51.799578902Z self.exchange.declare(nowait)
2016-06-17T11:42:51.799585186Z File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
2016-06-17T11:42:51.799591931Z nowait=nowait, passive=passive,
2016-06-17T11:42:51.799598206Z File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
2016-06-17T11:42:51.799604873Z self._send_method((40, 10), args)
2016-06-17T11:42:51.799611275Z File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
2016-06-17T11:42:51.799631815Z self.channel_id, method_sig, args, content,
2016-06-17T11:42:51.799639848Z File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
2016-06-17T11:42:51.799646601Z write_frame(1, channel, payload)
2016-06-17T11:42:51.799652763Z File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
2016-06-17T11:42:51.799659065Z frame_type, channel, size, payload, 0xce,
2016-06-17T11:42:51.799665133Z File "/usr/lib/python2.7/socket.py", line 224, in meth
2016-06-17T11:42:51.799671254Z return getattr(self._sock,name)(*args)
2016-06-17T11:42:51.799676859Z error: [Errno 104] Connection reset by peer
2016-06-17T11:42:51.799967150Z [2016-06-17 11:42:51,799] {jobs.py:759} ERROR - Tachycardia!
2016-06-17T11:45:01.136584726Z Fri Jun 17 11:45:01 UTC 2016 - waiting for RabbitMQ... 1/10
2016-06-17T11:47:13.872341907Z Fri Jun 17 11:47:13 UTC 2016 - waiting for RabbitMQ... 2/10
2016-06-17T11:49:26.352538395Z Fri Jun 17 11:49:26 UTC 2016 - waiting for RabbitMQ... 3/10
Kindly direct me with the fix.
Regards,
Vijay Raajaa G S
Hi @puckel ,
I just noticed that the fernet keys are different across containers, which makes the encryption not reversible. This is probably because each container will generate a new key when launching the image and running entrypoint.sh.
I was thinking a solution to move the key generation to Dockerfile from entrypoint.sh, but this could leave security flaws as the key will possibly be built into the image and pushed remote afterwards.
Please let me know what you have in mind about this issue, or correct me if I misunderstood anything.
Thanks,
Tianlong
As I already have redis and postgres containers in my server, so I rewrote the docker-compose as below:
version: '2'
services:
webserver:
image: puckel/docker-airflow:latest
restart: always
network_mode: bridge
external_links:
- bi-postgres:postgres
- redis
environment:
- LOAD_EX=n
- FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- EXECUTOR=Celery
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
volumes:
- /root/notebooks/dags:/usr/local/airflow/dags
ports:
- "5002:8080"
command: webserver
flower:
image: puckel/docker-airflow:latest
restart: always
network_mode: bridge
external_links:
- redis
environment:
- EXECUTOR=Celery
ports:
- "5555:5555"
command: flower
scheduler:
image: puckel/docker-airflow:latest
restart: always
depends_on:
- webserver
volumes:
- /root/notebooks/dags:/usr/local/airflow/dags
environment:
- LOAD_EX=n
- FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- EXECUTOR=Celery
command: scheduler -n 5
worker:
image: puckel/docker-airflow:latest
restart: always
depends_on:
- scheduler
volumes:
- /root/notebooks/dags:/usr/local/airflow/dags
environment:
- FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- EXECUTOR=Celery
command: worker
And I got issue on running initdb (DB: postgresql+psycopg2://airflow:***@postgres/airflow), but the redis get connected well. Please help.
I'm seeing this in the webinterface when trying to access the logs of a task, but only when running LocalExecutor - could this be a misconfiguration?
*** Log file isn't local.
*** Fetching here: http://33d456e47018:8793/log/g_pipelines/rawlogs_pipeline/2016-10-24T05:30:00
*** Failed to fetch log file from worker.
*** Reading remote logs...
*** Unsupported remote log location.
Curious if there is any way control installing custom pip packages? I was hoping that you could perhaps use something like the ONBUILD
triggers on the Python image
To be more specific why I'm asking is that I tried using the DockerOperator
in a DAG and I'm running into the following error: No module named docker
My guess is that we need to install the docker
package.
Since the airflow version 1.7.1.3 the scheduler stuck probably due to a deadlock.
It seems like that setting the number of runs to 5 & auto restart didn't solve the problem.
possible solutions:
docker-compose restart scheduler
from time to time.reference:
https://issues.apache.org/jira/browse/AIRFLOW-401
Is there a way to create docker image with pre-configured airflow "Connections"?
Hi, i got the following error when trying to load a new DAG(logs from the scheduler ):
[2016-11-16 10:44:39,687] {jobs.py:680} INFO - Starting the scheduler [2016-11-16 10:44:39,687] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags [2016-11-16 10:44:39,794] {base_hook.py:53} INFO - Using connection to: 10.1.100.71 [2016-11-16 10:44:39,794] {models.py:250} ERROR - Failed to import: /usr/local/airflow/dags/osf_dag_v3.py Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 247, in process_file m = imp.load_source(mod_name, filepath) File "/usr/local/airflow/dags/osf_dag_v3.py", line 29, in <module> ssh_hook = SSHHook(conn_id='ssh_conn') File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/ssh_hook.py", line 55, in\__init\__ self.key_file = conn.extra_dejson.get('key_file', None) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 590, in extra_dejson if self.extra: File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 293, in\__get\__ return self.descriptor.\__get\__(instance, owner) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 534, in get_extra return FERNET.decrypt(bytes(self._extra, 'utf-8')).decode() File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 103, in decrypt raise InvalidToken InvalidToken
Hi @puckel,
I just noticed that it is suggested to uncomment LOAD_EX=n
in two services both inside docker-compose-CeleryExecutor.yml
and docker-compose-LocalExecutor.yml
. Is it how it is supposed to be? I'd intuitively expect this ENV variable to be needed only once because I don't think that more than one service can be responsible for adding extra DAGs. How about removing an extra occurrence of LOAD_EX=n
if not needed?
Thanks a lot for crafting the docker recipe for airflow - sooo cool to be able to launch airflow in under a minute without any preps!
I'm having a small issue with opening logs for the post_op task in the example_http_operator DAG. This seems to work fine on my local machine running with SequentialExecutor, but after a fresh setup with the docker container it seems to crash.
I tried a bunch of things including setting the locales in the Dockerfile but that doesn't seem to help. I'm a bit lost whether this is my own incapabilities (new to Docker) or if this is reproduceable by someone else.
The following stacktrace is displayed when opening a log from the failed post_op task:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python2.7/dist-packages/flask_admin/base.py", line 68, in inner
return self._run_view(f, _args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/flask_admin/base.py", line 359, in _run_view
return fn(self, _args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/flask_login.py", line 755, in decorated_view
return func(_args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/airflow/www/utils.py", line 103, in wrapper
return f(_args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/airflow/www/views.py", line 786, in log
log = log.decode('utf-8') if PY2 else log
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 5872: ordinal not in range(128)
I tried adding this to the docker-compose.yml
image: puckel/docker-airflow
restart: always
links:
- mysqldb:mysqldb
- rabbitmq:rabbitmq
command: scheduler
but the db doesn't get setup in time and the scheduler has a hard time.
scheduler_1 | sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'airflow.job' doesn't exist") [SQL: u'INSERT INTO job (dag_id, state, job_type, start_date, end_date, latest_heartbeat, executor_class, hostname, unixname) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (None, 'running', 'SchedulerJob', datetime.datetime(2015, 9, 10, 4, 50, 21, 405881), None, datetime.datetime(2015, 9, 10, 4, 50, 21, 405910), 'CeleryExecutor', '25a5024ff2aa', 'root')]
Using the docker-compose up
command, most of it works except I get the following error:
mariadb_1 | 150722 14:09:17 [Warning] Aborted connection 4 to db: 'airflow' user: 'airflow' host: '172.17.0.20' (Unknown error)
Going to mess around with it, but wanted to post the issue.
Is there a reason we you are not using the official python image as your base image
I am currently looking into porting docker-airflow onto Kubernetes and I looked through the work of kube-airflow. One thing that puzzled me is that the exit of the scheduler makes Kubernetes think it crashed and gets into a CrashLoopBackOff
.
An alternative would be to run airflow_scheduler_autorestart.sh instead. Is there a reason why this script is not used in the docker container for the scheduler?
I'm running into an issue where the PythonOperator just straight up fails when running airflow backfills. Even an extremely simple use case (https://gist.github.com/fbdiehlad/9e356d9a303339eb3c3f1be3475efce2) fails immediately when I try to run a backfill against it. Oddly enough, when I run the task using airflow test, it works fine. I'm thinking this is some kind of permissions issue from running the setup with docker? Has anyone else run into this before?
Hi Puckel,
i´ve got a Problem with Airflow SSH Hook. When i start my dag, i got an error, that Airflow "Failed to create remote temp file". I want to open a ssh-connection via Airflow. I set all settings in Airflow Connections and my DAG. Maybe you can help me ?!
When i run this DAG (sshjonas.py) outside of a docker-container on a local Airflow-Installation then everythink works...
BR,
Jonas
For some fixes: https://issues.apache.org/jira/browse/AIRFLOW-812
(Apparently, 1.8.1 is not out yet :()
Running on VirtualBox (4GB, 4 Cores) on MacOS:
Version: '5.7.12' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2016-06-01T16:50:31.139441Z 4 [Note] Aborted connection 4 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:31.567596Z 3 [Note] Aborted connection 3 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:32.419459Z 6 [Note] Aborted connection 6 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:32.421728Z 7 [Note] Aborted connection 7 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:33.186267Z 10 [Note] Aborted connection 10 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:33.224555Z 9 [Note] Aborted connection 9 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:33.974580Z 13 [Note] Aborted connection 13 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:34.039175Z 12 [Note] Aborted connection 12 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
2016-06-01T16:50:36.152814Z 16 [Note] Aborted connection 16 to db: 'airflow' user: 'airflow' host: '172.17.0.4' (Got an error reading communication packets)
scheduler_1 | ./entrypoint.sh: line 15: [: too many arguments
scheduler_1 | ./entrypoint.sh: line 15: [: too many arguments
scheduler_1 | ./entrypoint.sh: line 15: [: too many arguments
scheduler_1 | ./entrypoint.sh: line 15: [: too many arguments
scheduler_1 | ./entrypoint.sh: line 29: [: too many arguments
scheduler_1 | ./entrypoint.sh: line 29: [: too many arguments
scheduler_1 | ./entrypoint.sh: line 29: [: too many arguments
The scheduler seems to be restarted every few seconds, the rest of the services less often
Hi.. I've been working on a Kubernetes deployment.
Is that something you'd be interested in adding in here ?
Hi,
how come that Airlow can generate 140 GB Logs under /stor/OSF_Tracking/docker/overlay2/d90dc3c63c3849bef95d82a47e67c138b54db7316860252a75ca308351cd00fe/diff/usr/local/airflow/logs ?
My other questions are is there a way to suppres that? or make like log rotation?
How to put those logs as a volume in docker ?
Any suggestion is very much appreciated
Hi everybody,
I just get a problem that the rabbitmq service is always restarting, with status: Restarting(1)
The Problem happened as there was not enough space disk, and the postgres service was shut down.
As i have the whole Airflow docker container restarted, i got this problem
Docker version 1.12.3 on a Ubuntu Linux version 14.04.5 LTS
Hello,
I have a airflow check that performs BashOperator with below command.
ssh root@xxxxx python2.7 xxx.py
Task failed so I went in to the worker container and checked that ssh
is missing in container.
I can't find online how to install ssh on docker container. If there is anything you can help, very much appreciated.
Not sure whats going on here but after doing the following steps I can't seem to have the passwords in connections encrypted.
git clone https://github.com/puckel/docker-airflow.git
docker-compose -f docker-compose-CeleryExecutor.yml up -d
# check http://localhost:8080/admin/connection/
# nothing is encrypted
My understanding is that these should all be encrypted as
I've tried changing the fernet key but that doesn't seem to work either.
whats really strange is that if I don't use the celery compose and instead just run it the connections are encrypted. Any ideas whats going on?
Relevant Resource?
https://airflow.incubator.apache.org/faq.html#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto
Python 2.7.9 has a bug on boto library,
What would be the way to install different version of Python on the containers?
Just to warn you that updating your scrip to fetch the HEAD (given a COMMIT ID in the build args) gives the following error:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in <module>
__import__('pkg_resources').run_script('airflow==1.9.0.dev0+apache.incubating', 'airflow')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 738, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1499, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow", line 28, in <module>
args.func(args)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/bin/cli.py", line 734, in webserver
app = cached_app(conf)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/www/app.py", line 161, in cached_app
app = create_app(config)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/www/app.py", line 60, in create_app
from airflow.www import views
File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/www/views.py", line 27, in <module>
import bleach
File "/usr/local/lib/python2.7/dist-packages/bleach-2.0.0-py2.7.egg/bleach/__init__.py", line 5, in <module>
from bleach.linkifier import (
File "/usr/local/lib/python2.7/dist-packages/bleach-2.0.0-py2.7.egg/bleach/linkifier.py", line 4, in <module>
import html5lib
File "build/bdist.linux-x86_64/egg/html5lib/__init__.py", line 16, in <module>
File "build/bdist.linux-x86_64/egg/html5lib/html5parser.py", line 2, in <module>
ImportError: cannot import name viewkeys
Submitting of new workflows is not supported yet by Airflow.
Make a workaround to support new DAG.
Whenever I try to access the logs endpoint in the admin console, webserver has the following exception:
2015-09-29 18:12:58,901 - airflow.www.app - ERROR - Exception on /admin/airflow/log [GET]
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/lib/python2.7/dist-packages/flask_admin/base.py", line 68, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/flask_admin/base.py", line 359, in _run_view
return fn(self, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/airflow/www/app.py", line 795, in log
loc = loc.format(**locals())
KeyError: 'AIRFLOW_HOME'
2015-09-29 18:12:58,902 - tornado.access - ERROR - 500 GET /admin/airflow/log?task_id=run_this_last&dag_id=example_bash_operator&execution_date=2015-09-28T00:00:00 (192.168.99.1) 5.89ms
The reason for this error is that Airflow is trying to format a string with {AIRFLOW_HOME}
in it and failing because the key AIRFLOW_HOME
does not appear in locals. The root of this problem seems to be that the {AIRFLOW_HOME}
key is in the docker-airflow config/airflow.cfg
under base_log_folder
. It's easy enough to fix this problem (by cloning the repo and modifying the config), but I didn't know if there was a step which I was missing that would correctly format the config in the process of bringing the containers up. Really the only step that I'm performing is docker-compose up -d
.
Hello,
Our current airflow static folder is residing under root path. We are looking for how to change the static root path [/static]. Is any specific configuration file using [/static]. If so, how to change it. Highly appreciated if anybody can help to sort out this issue?
*** Log file isn't local.
*** Fetching here: http://:8793/log/tutl1/print_date1/2017-04-03T09:27:24
*** Failed to fetch log file from worker.
*** Reading remote logs...
*** Unsupported remote log location.
I'm using airflow-1.8.0
While i'm trying to run it as airflow user the logs are coming, but when i modify the dockerfile to run as root , the logs are not coming . Can you help me out in finding the issue.
I'm using Celery executor
I have tried to run docker-compose -f docker-compose-CeleryExecutor.yml up -d
But recieved errors
ERROR: for webserver No such image: sha256:3631d65918560df72f335194d391370a09ea944ab9f7a9f225611829bd883bdf
ERROR: for flower No such image: sha256:3631d65918560df72f335194d391370a09ea944ab9f7a9f225611829bd883bdf
Hi there,
We are running Airflow inside Docker since about 1.5 month without any bigger issues. However since a few days the worker and Scheduler keeps restarting without executing any of the DAGs.
The log of the worker shows an Unrecoverable error: TypeError
:
[2016-11-09 14:41:04,432: CRITICAL/MainProcess] Unrecoverable error: TypeError('unorderable types: NoneType() <= int()',)
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line 115, in start
self.on_start()
File "/usr/local/lib/python3.4/dist-packages/celery/apps/worker.py", line 143, in on_start
self.emit_banner()
File "/usr/local/lib/python3.4/dist-packages/celery/apps/worker.py", line 159, in emit_banner
string(self.colored.reset(self.extra_info() or '')),
File "/usr/local/lib/python3.4/dist-packages/celery/apps/worker.py", line 188, in extra_info
if self.loglevel <= logging.INFO:
TypeError: unorderable types: NoneType() <= int()
I have no idea what caused this issue. Do you have an idea or a clue what could go wrong?
flower_1 | [2016-03-24 00:43:30,633] {__init__.py:36} INFO - Using executor CeleryExecutor
flower_1 | Traceback (most recent call last):
flower_1 | File "/usr/local/bin/flower", line 5, in <module>
flower_1 | from pkg_resources import load_entry_point
flower_1 | File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2876, in <module>
flower_1 | working_set = WorkingSet._build_master()
flower_1 | File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 451, in _build_master
flower_1 | return cls._build_from_requirements(__requires__)
flower_1 | File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 464, in _build_from_requirements
flower_1 | dists = ws.resolve(reqs, Environment())
flower_1 | File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 639, in resolve
flower_1 | raise DistributionNotFound(req)
flower_1 | pkg_resources.DistributionNotFound: pytz==2015.7
dockerairflow_flower_1 exited with code 0
Hey there,
Loving the docker setup you put together here, it's been very helpful in getting me closer to a production airflow configuration, and learn more about the components!
I am having some difficulty running the docker-compose-LocalExecutor.yml setup though, using docker v1.12.1 and running on osx. I attached the docker-compose logs output here. I tried this with the CeleryExecutor as well, and had some very similar results. I tried removing the local volumes just to ensure I was in a clean environment, but things just don't seem to work :/
If you have any suggestions on how I might get things running (including the scheduler and web UI components for flower/airflow running) please let me know!
hi,
i am using the last version of this docker-airflow, and i noticed that flower is always restarting and i don't have acces to the webserver
HTTP/1.1 200 OK
[2016-11-15 16:45:46,526] {init.py:36} INFO - Using executor CeleryExecutor
[2016-11-15 16:45:46,584] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-11-15 16:45:46,607] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 15, in
args.func(args)
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 576, in flower
os.execvp("flower", ['flower', '-b', broka, port, api])
File "/usr/lib/python2.7/os.py", line 346, in execvp
_execvpe(file, args)
File "/usr/lib/python2.7/os.py", line 382, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory
Hi there!
I've installed your docker-airflow images and they run examples fine (except http requests, but it's for another ticket).
Now I would like to run my own DAG, could you please explain how to do it with your images?
a) can/should I use running containers to launch DAGs?
b) can/should I rather install airflow distribution, configure & use that as a client?
I think it is valuable information to others, could you please write the instructions in your README?
Thanks!
Krzysiek
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.