navbryce / warship-scraper Goto Github PK

Log scraping errors to file to make debugging easier

Nagato

As the title states, the coding conventions (the style of the code) are a little rough...because this was my first python project. Need to be fixed at some point.

Dates in the format of {{year}}-{{other year}} will throw an error. Usually associated with "in-commission" property of ship.

The scraper can't handle values in the following format: "{{value}}{{unit}}". Note: no space between value and unit.

Some ships have information that is missing. The scraper should attempt to find this information through its class or a sister-ship.

Scrape ship's nationality and draw edge if two ships share a nationality.

After scraping a ship, check for errors and verify the ship was scraped successfully before saving it to the database.

• Weird caption on thumbnail image

• Wikipedia images have two sizes. Currently scrapes the smaller size. Should scrape both.

Things that can/should be Parallelize:

• Script in ABoatScraping.ship_compare.ship_network_algo.run_algo_on_network.py

Add unit converting for standard values AKA choose standard units. Already has been implemented for armor and gun size.