codyburker / postmates_webscraper

Data Files

• scraped_data_raw\: This folder is where the script to scrape restaurants from addresses outputs to.
• 11-19-2021 730pm ET\: Files with these naming patterns contain scrapes at points in time copied over from the scraped_data_raw folder.
  • The csv files with the longer names in them have the timestamp of when the data was written, and the rows (corresponding to OAK_Berk_geocodio_filtered.csv) that are in that file.
  • df_merged.csv is these files appended to each other, and joined with OAK_Berk_geocodio_filtered.csv. These files are generated by running process_raw_data.ipynb.
• alemada_ca_gb_geocodio.csv
• OAK_Berk_geocodio.csv: The addresses to check, with lattitude, longitude, and Geoid.
• OAK_Berk_geocodio_filtered.csv: A shortened version of OAK_Berk_geocodio.csv with some rows removed.
• ACS\: This folder contains data from the 2019 Census.

Scripts

• posty2_redo.py: The script that uses selenium to take an address, and return delivery time, name, and delivery fee for a restaurant.
• distance2.py: Given the link to a restaurant on postmates and the lattitude and longitude of an address, find the distance between those two (will be used to normalize delivery time).
• distance.py: Obsolete.
• distance_scrape.py
• posty2_redo.py and join with OAK_Berk_geocodio_filtered.csv into a single file called df_merged.csv.
• posty2.py: Obsolete.

Notebooks

• process_raw_data.ipynb: Used to combine multiple CSVs from
• Filter_Geocodio.ipynb: Used to filter OAK_Berk_geocodio.csv into valid addresses in OAK_Berk_geocodio_filtered.csv.
• cody_analysis.ipynb: Attempt at linking joined_wDistance.csv to holc_grade.csv

Aux Files

• chromedriver: Used for selenium Mac
• chromedriver.exe: Used for selenium Windows