amol9 / imagebot Goto Github PK

Hello!
I am behind the firewall, so failed to run "python ./setup.py install". I had to install every missing python package one-by-one. Is there are way to specify proxy information?

D:>imagebot crawl http://www.bbc.com -m -s 640x480 -l info -is "E:\Photos\BBC"
2017-06-04 19:32:30 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: imagebot)
2017-06-04 19:32:30 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'imagebot.spiders', 'LOG_LEVEL': 20, 'SPIDER_MODULES': ['imagebot.spiders'], 'HTTPCACHE_ENABLED': True, 'BOT_NAME': 'imagebot', 'USER_AGENT': 'imagebot', 'HTTPCACHE_DIR': 'C:\Users\CP\.imagebot\httpcache', 'HTTPCACHE_POLICY': 'scrapy.extensions.httpcache.RFC2616Policy'}
2017-06-04 19:32:30 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2017-06-04 19:32:30 [root] ERROR: No module named gi
Unhandled error in Deferred:
2017-06-04 19:32:30 [twisted] CRITICAL: Unhandled error in Deferred:

2017-06-04 19:32:30 [twisted] CRITICAL:
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\crawler.py", line 95, in crawl
six.reraise(*exc_info)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\crawler.py", line 77, in crawl
self.engine = self._create_engine()
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\crawler.py", line 102, in create_engine
return ExecutionEngine(self, lambda : self.stop())
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\core\engine.py", line 69, in init
self.downloader = downloader_cls(crawler)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\core\downloader_init.py", line 88, in init
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\utils\misc.py", line 44, in load_object
mod = import_module(module)
File "c:\python27\lib\importlib_init.py", line 37, in import_module
import(name)
File "c:\python27\lib\site-packages\scrapy-1.4.0-py2.7.egg\scrapy\downloadermiddlewares\retry.py", line 20, in
from twisted.web.client import ResponseFailed
File "c:\python27\lib\site-packages\twisted\web\client.py", line 42, in
from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
File "c:\python27\lib\site-packages\twisted\internet\endpoints.py", line 37, in
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\python27\lib\site-packages\twisted\internet\stdio.py", line 30, in
from twisted.internet import _win32stdio
File "c:\python27\lib\site-packages\twisted\internet_win32stdio.py", line 9, in
import win32api
ImportError: DLL failed to load:% 1 is not a valid Win32 application.

Hello, You creating a great software.

It's works great but when I trying to crawl some website like 17173.com which domain starts with number, it wouldn't works.

I found the problem is in the submodule "web", the regex that in class AbsUrl would not match such domains. Could you change it for me? Thanks.

It looks like the library requires the Python PIL to work properly, but it is not listed in the project dependencies.
Could you tell me which one is the right version to use?

I'm having the following issues with pillow ^7.1.2 (possibly unrelated..?):

Traceback (most recent call last):
  File "d:\projects\.virtualenvs\imagescraper-uzbe7zgj-py3.8\lib\site-packages\scrapy\pipelines\files.py", line 481, in media_downloaded
    checksum = self.file_downloaded(response, request, info)
  File "d:\projects\.virtualenvs\imagescraper-uzbe7zgj-py3.8\lib\site-packages\scrapy\pipelines\images.py", line 106, in file_downloaded
    return self.image_downloaded(response, request, info)
  File "d:\projects\.virtualenvs\imagescraper-uzbe7zgj-py3.8\lib\site-packages\scrapy\pipelines\images.py", line 110, in image_downloaded
    for path, image, buf in self.get_images(response, request, info):
  File "d:\projects\.virtualenvs\imagescraper-uzbe7zgj-py3.8\lib\site-packages\scrapy\pipelines\images.py", line 123, in get_images
    orig_image = Image.open(BytesIO(response.body))
  File "d:\projects\.virtualenvs\imagescraper-uzbe7zgj-py3.8\lib\site-packages\PIL\Image.py", line 2895, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x0000017497F73A90>

Windows, Py 3.8, installed from master at 8738efc

Every time the script is launced it raises a SyntaxError. I checked on multiple devices with different OS and different Python versions (3.8, 3.9, 3.10).

Here is with virtualenv on 3.8

Traceback (most recent call last):
  File "C:\Users\xstre\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\xstre\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\xstre\python3.8\Scripts\imagebot.exe\__main__.py", line 4, in <module>
  File "C:\Users\xstre\python3.8\lib\site-packages\imagebot\main.py", line 11, in <module>
    from imagebot.clear import clear_cache, clear_db, clear_duplicate_images
  File "C:\Users\xstre\python3.8\lib\site-packages\imagebot\clear.py", line 28
    print ose.message
    ^^^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?