chusiang / crawler-book-info

A crawler for quick parser the book information

Crawler Book Info

A sample crawler for quick parser some books information.

Initialization

1. Install the virtualenv.

   [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ sudo pip3 install virtualenv
   

2. create virtualenv.

   [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ virtualenv -p python3 .venv
   

3. Enter the virtualenv.

   [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ . .venv/bin/activate
   

4. Install packages with pip.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ pip3 install -r requirements.txt
   

Usage

tenlong.com.tw

1. Run crawler with ISBN-13.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ python3 tenlong.py 9781491915325
   

2. (option) Run crawler via make.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ make telong 9781491915325
   

books.com.tw

1. Run crawler with url.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ python3 books.py https://www.books.com.tw/products/0010810939
   

2. Run crawler with product number.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ python3 books.py 0010810939
   

Not support the ISBN-13 args yet on books.com.tw.

View Result

1. Open html via Firefox on GNU/Linux.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ firefox index.html
   

   

2. We can see the https://www.tenlong.com.tw/products/9781491915325, it is clean, now.

Run local Nginx for Evernote Web Clipper

The Evernote Web Clipper is not support local files, so we can clip it with Nginx.

1. Run Nginx container.

   $ docker run --name nginx -v "$(pwd)":/usr/share/nginx/html/ -p 80:80 -d nginx
   

2. Open html via Firefox on GNU/Linux.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ firefox http://localhost
   

3. (option) Run Nginx container via make.

   $ make run_containers
   

4. (option) Open web via make.

   $ make review_serve
   

5. Finally, we can clip the information to Evernote with Evernote Web Clipper.

License

Copyright (c) chusiang from 2017-2022 under the MIT license.

crawler-book-info's Issues

modify for new tenlong website

天瓏網路書局店於 2017/02/28 進行大幅度的改版，故 crawler_tenlong.py 這支簡易的爬蟲程式也得跟著改版哩！

url diff:

• old: https://www.tenlong.com.tw/items/7121298120
• new: https://www.tenlong.com.tw/products/9787121298127

Support parser the books dot com

Lose parser author and outline of tenlong

目前爬出來的書籍資料，少了「作者簡介」和「目錄大綱」的部份。

Copy parsed context to clipboard via pyperclip module

Example:

In [1]: import pyperclip

In [2]: pyperclip.copy('Hello World!')

In [3]: pyperclip.paste()
Out[3]: 'Hello World!'

Reference:

1. pyperclip | PyPI
2. Automate the Boring Stuff with Python Programming / 8. Python's Built-In Functions | Udemy