A Basic python script that, taking a list of base URLs will scrape /sitemap.xml and return a CSV of all pages in each site
Copy base_urls.csv.sample
into base_urls.csv
and update it with your base urls
Run scraper.py
to create a sites.csv
file containing the list of urls.