kjam / data-pipelines-course Goto Github PK
View Code? Open in Web Editor NEWCourse materials for my data pipeline video course with O'Reilly
Course materials for my data pipeline video course with O'Reilly
Hi Katharine,
I want to thank you for the data-pipeline course on Safari O'Reilly as your tutorials have been helpful.
I have a question about Luigi log history function. I followed what you did in the luigi.cfg file:
[worker]
keep_alive=True
task_limit=10
[scheduler]
retry_count=4
record_task_history=True
[task_history]
db_connection=sqlite:///tasks.db
However, when I go to localhost:8082/history, it only saves the most recent history. Is there a way to save all the histories from the previous attempts? For example, let's say that I run the pipeline 10 times. Is there a way to save all the history from the 10 pipeline executions?
Any help would be greatly appreciated.
Best,
Daniel
Hi Kjam,
I have issue when try to fetch the finance data.
it seems like some authority issue, not sure if you have encountered it and got it resolved.
Best regards,
YX
Hi,
After creating python2 virtual environment, and then trying to pip install requirements.txt , an error showed:
Collecting jupyterhub==0.6.1 (from -r requirements.txt (line 64))
Using cached jupyterhub-0.6.1.tar.gz
Complete output from command python setup.py egg_info:
ERROR: Jupyter Hub requires Python version 3.3 or above.
Clearly JupyterHub has a requirement that it is only compatible with Python >= 3.3
Thanks !
I can not find py3_requirements.txt in the repo.
Hi, I am new to this subject and I'm following the course on safari, I installed airflow on my local machine and a remote server, on both cases I can't see my files after $airflow list_dags command, I exported the path for AIRFLOW_HOME , there is a dag folder in the airflow home directory, what is point that I'm missing? Also web UI doesn't show the files too, web server is printing "Filling up the DagBag from /Users/myuser_name/airflow/dags" but doesn't let me see the dags.
note: I didn't use the requirements.txt in the virtualenv, I just followed what you've done in the videos.
Update: $airflow list_dags works now , It was my fault that I used a same dag_id with the examples, unfortunately interface still does not update.
Update2: Now I can see the dag, it took a while to show that but on/off button is invisible for my dag.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.