kjam / data-pipelines-course Goto Github PK

Hi Kjam,

I have issue when try to fetch the finance data.

it seems like some authority issue, not sure if you have encountered it and got it resolved.

Best regards,

YX

Hi, I am new to this subject and I'm following the course on safari, I installed airflow on my local machine and a remote server, on both cases I can't see my files after $airflow list_dags command, I exported the path for AIRFLOW_HOME , there is a dag folder in the airflow home directory, what is point that I'm missing? Also web UI doesn't show the files too, web server is printing "Filling up the DagBag from /Users/myuser_name/airflow/dags" but doesn't let me see the dags.

note: I didn't use the requirements.txt in the virtualenv, I just followed what you've done in the videos.

Update: $airflow list_dags works now , It was my fault that I used a same dag_id with the examples, unfortunately interface still does not update.

Update2: Now I can see the dag, it took a while to show that but on/off button is invisible for my dag.

I can not find py3_requirements.txt in the repo.

Hi Katharine,

I want to thank you for the data-pipeline course on Safari O'Reilly as your tutorials have been helpful.

I have a question about Luigi log history function. I followed what you did in the luigi.cfg file:

[worker]
keep_alive=True
task_limit=10

[scheduler]
retry_count=4
record_task_history=True

[task_history]
db_connection=sqlite:///tasks.db

However, when I go to localhost:8082/history, it only saves the most recent history. Is there a way to save all the histories from the previous attempts? For example, let's say that I run the pipeline 10 times. Is there a way to save all the history from the 10 pipeline executions?

Any help would be greatly appreciated.

Best,
Daniel

Hi,

After creating python2 virtual environment, and then trying to pip install requirements.txt , an error showed:

Collecting jupyterhub==0.6.1 (from -r requirements.txt (line 64))
  Using cached jupyterhub-0.6.1.tar.gz
    Complete output from command python setup.py egg_info:
    ERROR: Jupyter Hub requires Python version 3.3 or above.   

Clearly JupyterHub has a requirement that it is only compatible with Python >= 3.3

Thanks !