m-rey / immoscraper Goto Github PK

Traceback (most recent call last):
  File "./immoscraper.py", line 51, in <module>
    page_count = get_page_count(mainpage_result)
  File "./immoscraper.py", line 18, in get_page_count
    .find("select", {"class": "select font-standard"})
AttributeError: 'NoneType' object has no attribute 'find_all'

The script gets the page count from the dropdown menu. When there is no such menu ( because all the results fit in one page), the script fails.
Implementing #1 to manage page traversal should fix this.

ImmobilienScout24 has a lot of data in machine readable form. We can use that data to simplify the scraper. For example, the script with the XPath /html/body/script[1] contains a lot of information that could be used.

See https://github.com/balzer82/immoscraper/blob/master/immoscraper.ipynb

Once there's enough collected data, it would be great to analyze it. This blog post has some info about how data could be analyzed and used: https://giansegato.com/essays/how-i-beat-the-berlin-rental-market-with-a-python-script/

This project looks like a good fit to learn how to use scrapy.