chrismuir / zillow

Zillow Scraper for Python using Selenium

Zillow Scraping with Python

ATTENTION:

As of 2019, this code no longer works for most users. Zillow is now able to detect the use of all/most automated webdrivers, and will display an unlimited number of CAPTCHA's when the site is launched in a webdriver. I have no interest in putting more work into this project, but am leaving it up to serve as an example of how to webscrape using Selenium with Python.

WARNING: Use this code at your own risk, scraping is against Zillow's TOC

Basic tool for scraping current home listings from Zillow, written in Python using Selenium. The code takes as input search terms that would normally be entered on the Zillow home page. It creates 11 variables on each home listing from the data, saves them to a dataframe, and then writes the df to a CSV file that gets saved to your working directory. Using zip codes as search terms seems to yield the best results, the scraper works at a rate of about 75 zip codes per hour (compared to the Zillow API limit of 1000 homes per 24h).

There are two files, zillow_runfile.py and zillow_functions.py. Clone this repo to your working directory, open the runfile and step through the code line-by-line. The zillow functions are sourced at the top of the runfile.

This tool uses a for loop to iterate over a list of input search terms, scrape the listings of each, and append the results to a dataframe. Function zipcodes_list() allows the user to compile a large list of zip codes to use as search terms, using the package zipcode. For example, st = zipcodes_list(['10', '11', '770']) will yield every US zip code that begins with '10', '11', or '770' as a single list. Object st could then be passed to the scraper.

Some things to keep in mind

• You will need to edit the input parameter of function init_driver within zillow_runfile.py to point to the local path of your web driver program (required by Selenium).
• The max return for each search term (i.e. each zip code) is 520 home listings.
• Zillow will periodically throw up a CAPTCHA page. The script is designed to pause scraping indefinitely until the user has manually completed the CAPTCHA requirements (at which point it should resume scraping).
• There tends to be a small amount of NA's on every search, however foreclosure properties seem to be more likely to return NA's. So the more foreclosures there are in a search, the more NA's there will be.

Software Requirements/Info

• This code was written using Python 3.5.
• Selenium (this can be PIP installed, written using v3.0.2).
• The Selenium package requires a webdriver program. This code was written using Chromedriver v2.25.

Example of the output dataframe

df.head(n = 6)

                 address     city state    zip    price  sqft bedrooms  \
0      3011 Bissonnet St  Houston    TX  77005   575000  1820        3   
1          4229 Drake St  Houston    TX  77005   615000  2611        3   
2        2237 Wroxton Rd  HOUSTON    TX  77005  2095000  5492        4   
3      4318 Childress St  Houston    TX  77005   540000  2438        4   
4       2708 Werlein Ave  Houston    TX  77005  1449000  3905        4   
5  5402 Buffalo Speedway  Houston    TX  77005  1995000  4658        3   

  bathrooms days_on_zillow           sale_type  \
0         2             NA      House For Sale   
1         3             NA   For Sale by Owner   
2         5             NA      House For Sale   
3         4              2  Townhouse For Sale   
4         5              1      House For Sale   
5         4              5      House For Sale   

                                                 url  
0  http://www.zillow.com/homes/for_sale//homedeta...  
1  http://www.zillow.com/homes/for_sale//homedeta...  
2  http://www.zillow.com/homes/for_sale//homedeta...  
3  http://www.zillow.com/homes/for_sale//homedeta...  
4  http://www.zillow.com/homes/for_sale//homedeta...  
5  http://www.zillow.com/homes/for_sale//homedeta...  

zillow's Issues

ImportError: No module named 'zipcode'

Trying to run this code however it seems its missing a module called 'zipcode'.

DOM Not Working

Hello,

It seems the class for DOM has changed to (if input into console)
document.getElementsByClassName('zsg-list_inline zsg-photo-card-badge');

I tried updating this myself, but don't have enough python knowledge to do that. Can you fix it?

Specify encoding in pandas.DataFrame.to_csv()

This was suggested in issue #10 by addieg. Add encoding = "UTF-8" when writing to csv via pandas. So this line:

pd.DataFrame(output_data, columns = columns).to_csv(file_name, index = False)

would become this:

pd.DataFrame(output_data, columns = columns).to_csv(file_name, index = False, encoding = "UTF-8")

Per the pandas docs, the default encoding for both read_csv and to_csv is ascii for Python 2, so making this edit would mean that anyone using Python 2 will need to explicitly specify encoding = "UTF-8" when reading any of the csv files back into Python.

ReCaptcha with chromedriver on Mac can't be beaten

I've spent the last several minutes solving reCaptcha problems. It won't go away. Somehow Zillow or Google know Chrome is being controlled by automated test software (chromedriver).

Finally it gave up and told me "Try again later
Your computer or network may be sending automated queries. To protect our users, we can't process your request right now. For more details visit our help page"

ImportError Zillow

import Zillow.zillow_functions as zl

ImportError: No module named 'Zillow'

add another final output variable: Saves

Would be convenient to scrape for how many saves a home has also!

is_displayed() always False for zsg-pagination-next element.

Great repo, Chris!

Unfortunately I can only get one page of output (26 homes) for each zip code.

Problem appears to be with the "driver.find_element_by_class_name('zsg-pagination-next').is_displayed()" query in get_html(). It always returns False for me.

I've tried the versions of selenium and chromedriver you suggested and several other newer versions using Python 3.5.2 on both Windows 7 and Ubuntu 16.04. Same problem in both cases.

I'm assuming you are not seeing the same issue. Any suggestions?

-Rick

Capcha is immediate and impossible to solve

The crawler runs for me, but capcha comes up immediately, and it's very hard to solve. It almost seems like they're using a capcha that is designed to just waste time and not be solveable. Anyone else seeing this problem? Occasionally I can pass the captcha and get some data, but this is very hard to achieve.

Could not find a version

When trying to install zillow_functions I keep getting the message: Could not find a version that satisfies the requirement zillow_functions (from versions: )No matching distribution found for zillow_functions

! pip install zillow_functions

ZIPCODE_CONNECTION_STRING not set

Installed zipcode package, and getting following error when attempting to run:

Exception: ZIPCODE_CONNECTION_STRING not set, http://docs.sqlalchemy.org/en/latest/core/engines.html#sqlalchemy.create_engine

Python knowledge too limited to tell me what's going wrong here, and looking at the link didn't provide me any insights. Help?

CAPTCHA does not go away

Hi there!

I'm using your software for a personal project and when Zillow throws up a CAPTCHA it takes a really long time and dozens of iterations to get rid of it. I basically complete the CAPTCHA and the page reloads a new one. It goes on for about 10/15 minutes no matter how many times I do it. Do know why this is happening? Is there a workaround to this issue?

Thanks,
Giulio

Unknown error after the program runs for a while

Hi there,

I am using your script, and after it runs for a while (searching around 150 zips), it throws an unknown error:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (336, 77). Other element would receive the click:

...

(Session info: chrome=64.0.3282.186)

It did this three times already -- any idea what might be going on?
Many thanks,
Andreea

keyerror on pandas import

I assume the error is mine rather than the code but when I step through I am getting a keyerror on import of pandas. I am a novice programmer but spent several hours on stackoverflow and google looking for an answer with no luck. Any help is appreciated.

No longer functioning

Zillow, reformatted the site. No longer functioning.

Periodically write data to file while scraping

This was suggested in issue #10 by addieg. Currently, all of the scraped data is written to file as the last step in the process. It makes sense to write scraped data to file at some regular interval while the script is running....should probably dump to file after each individual search term.