hail-cali / db_modeling

Package for DB processor and Asyncio Web Scraper based on coroutine

Coroutines Web Scrapper & DB Processor

What is Coroutines ?

• asynchronous programming
• stable, also useful to exception handling

Based on asyncio Stream

Feature

• coroutines web scraper: run_web_scrapper.py in test
• coroutines selenium scrapper : run_selenium.py in test
• db processor: DBConnector in db_connector
• query builder: dev for sql query builder & http query builder

---

How to use

Web Scraper (crawling)

• run code inside test dir
• when use tasks.csv

python run_web_scrapper.py --tasks path/to/tasks.csv --save_file crawling  \
                            --result_path result --result_type text

• when edit url list inside code, skip tasks option

python run_web_scrapper.py  --save_file crawling  \
                            --result_path result --result_type text

• sample run sh

python run_web_scrapper.py --tasks ../tasks.csv --save_file crawling  \
                            --result_path result --result_type text

Selenium Scrapper

python run_selenium.py --save_file selenium  \
                            --result_path result --result_type text

DB processor

• shell 'dev'

python run.py

Modules

• stream with request module: Reader, Writer, Stream, Session in stream.map

---

How to Custom

• inherit stream.map.BaseSession, make CustomSession

• edit code inside async def aenter

• edit params base_session of func asyncio_scraper in run_web_scrapper.py

• Same as selenium scrapper

---

Task lists

• selenium scrapper
• db processor code
• web scrapper code
• Dev crawler using API

---