crawl-me 是一个基于plugin的网页图片下载工具。crawl-me通过简单的命令行就可以用你想要的方式下载各个网站下的图片。目前暂时只支持gamersky(游明星空), pixiv(P站),更多plugin尽请期待,欢迎为它添加新的plugin。
-
####Ubuntu下安装
由于代码依赖了pyquery,安装前请确保libxslt-devel libxml2-devel已被安装
sudo apt-get install libxml2-dev sudo apt-get install libxslt1-dev
然后请确保安装了setuptools, Ubuntu下你可以:
sudo apt-get install python-setuptools
然后从github clone source到本地
$ git clone https://github.com/nyankosama/crawl-me.git $ cd crawl-me/ $ sudo python setup.py install
-
####Windows下安装
首先你需要安装python2.7和pip,python2.7可以通过windows installer安装。安装pip首先下载get-pip.py, 然后执行下面命令。
python get-pip.py
然后,你需要安装pyquery的所依赖的lxml,选择对应的lxml installer下载并安装
最后从github clone 到本地
$ git clone https://github.com/nyankosama/crawl-me.git $ cd crawl-me/ $ sudo python setup.py install
-
下载gamersky下的http://www.gamersky.com/ent/201404/352055.shtml 的第1页到第10页的所有图片到当前目录的gamersky-crawl文件夹下
crawl-me gamersky http://www.gamersky.com/ent/201404/352055.shtml ./gamersky-crawl 1 10
-
下载pixiv中id为3878890的用户的所有作品到pixiv-crawl文件下
crawl-me pixiv 3878890 ./pixiv-crawl <your pixiv id> <your password>
-
general help
$ crawl-me -h usage: crawl-me [-h] plugin positional arguments: plugin plugin the crawler uses optional arguments: -h, --help show this help message and exit available plugins: ----gamersky ----pixiv
-
gamersky
$ crawl-me gamersky -h usage: crawl-me [-h] plugin authorId savePath pixivId password positional arguments: plugin plugin the crawler uses authorId the author id you want to crawl savePath the path where the imgs ars saved pixivId your pixiv login id password your pixiv login password optional arguments: -h, --help show this help message and exit
-
pixiv
$ crawl-me pixiv -h usage: crawl-me [-h] plugin authorId savePath pixivId password positional arguments: plugin plugin the crawler uses authorId the author id you want to crawl savePath the path where the imgs ars saved pixivId your pixiv login id password your pixiv login password optional arguments: -h, --help show this help message and exit
添加通过pip安装的支持