eduardschaeli / wget-image-scraper

Scrapes images from a list of urls with wget

Image scraper

This shellscript scrapes all images, recursively, from the urls listed in the sites.txt file.

It will be used in an art project by Stefan Baltensperger.

Usage:

• Add urls to the sites.txt file like this http://www.yoursite.ch The HTTP:// part ist important!
• Make the scraper.sh shell script executable: chmod +x scraper.sh
• Run it like this: ./scraper.sh

wget-image-scraper's Issues

NO IMAGES

I cant see images from that folder

Only downloading thumbnails

Good day sir

The script only seems to download thumbnails instead of full resolution wallpapers:

http://www.goodfon.com/catalog/aviation/

Are you able to assit or advise? I am not too familiar with the wget switches and what they do.

Thank-you

Kind Regards

Bradley

Doesn't work for HTTPS without tweaking

You might want to add a note.

Also for the level setting.

Syntax error: end of file unexpected (expecting "done")

im getting below error while running the program

Syntax error: end of file unexpected (expecting "done")

[feature] parallel

Possible to mod this so it runs each download in parallel?

for example this works...

cat sites.txt | xargs -n 1 -P 24 wget -q -P ./downloads

Can't timestamp and not clobber old files at the same time.

hi, I get the following error

Scraping leveldesigninspirationmachine.tumblr.com
-----------------------------------
> creating folder..
downloads/leveldesigninspirationmachine.tumblr.com
> scraping leveldesigninspirationmachine.tumblr.com
Can't timestamp and not clobber old files at the same time.
Usage: wget [OPTION]... [URL]...


> Finished scraping leveldesigninspirationmachine.tumblr.com

scraping js-injected images

Is there any way to scrap js injected images?