To make automation easy, it is recommended that you use vex to manage your python environment.
From your virtual environment:
pip install -r requirements.txt
pip install ./ # OR pip install -e ./
# Done
Logs are collected:
[your home]/Downloads/pudl_scrapers/scraped/
Data from the scrapers is stored:
[your home]/Downloads/pudl_scrapers/scraped/[source_name]/[today #]
The general pattern is scrapy crawl [source_name] for one of the supported
sources. Typically and additional "year" argument is available, in the form
scrapy crawl [source_name] -a year=[year].
See below for exact commands and available arguments.
scrapy crawl census
No other options.
For full instructions:
epacems --help
To collect all the data:
scrapy crawl eia860
To collect a specific year (eg, 2007):
scrapy crawl eia860 -a year=2007
To collect all the data:
scrapy crawl eia860m
To collect a specific month & year (eg, August 2020):
scrapy crawl eia860 -a month=August -a year=2020
To collect all the data:
scrapy crawl eia861
To collect a specific year (eg, 2007):
scrapy crawl eia861 -a year=2007
To collect all the data:
scrapy crawl eia923
To collect a specific year (eg, 2007):
scrapy crawl eia923 -a year=2007
To collect all the data:
scrapy crawl ferc1
To collect a specific year (eg, 2007):
scrapy crawl ferc1 -a year=2007
To collect the data:
scrapy crawl ferc714
There are no subsets, that's it.
To collect all the data:
scrapy crawl epaipm
No additional arguments are supported.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent