Feature request: scraping of recipes about animal-crossing-scraper

[sungyeonu]

Please check the updated data files

[AustinTSchaffer]

@sungyeonu

I have also built some code around scraping recipes and raw crafting materials from the Wiki, if you're interested in merging any of that code into your project.

https://github.com/AustinTSchaffer/animal-crossing-crafting-recipe-database

[sungyeonu]

Cool!

I checked out your project and it is definitely helpful. The wiki is often inconsistent in the way they post the data of different things and it was interesting to see how you implemented it. I think I definitely should store the JSONs the way you do. Mine is just array of objects in the order of what the wiki has while yours has a name reference first.

[AustinTSchaffer]

Hey thanks for the nice comment! It took me about a week just to figure out how I wanted to format the data for an application that I've pretty much abandoned at this point... I used the recipe names in that way so that I could use each recipe's name as a primary/foreign key and an index.

Not sure if you saw this part too, but if you scroll down to the bottom you'll see that I also have a separate table for "raw_materials", which normalizes the data somewhat.

[sungyeonu]

Hey friend,

Scraping DIY recipe definitely sounds like a great feature to add on top of the project!

For now I think it'll be better to scrape the DIY_recipes page. Even if the list is incomplete, it will be expanded, just like fish and bugs pages were. I see that seasonal recipes list are complete but the wiki page will be keep adding new events and it will be harder to maintain that way.

One thing: when data is incomplete, we can't directly scrape and assign data into Python objects. Each field should be using error handling (try & except).

I will get this started later today

[sungyeonu]

I believe Tools are scraped well now. Pages containing the DIY recipes are slightly different so not everything are scraped perfectly. For example, in equipments.json, the first element (Acorn pochette)'s sizeLink is not scraped. wallMountedsList and wallpaperRugsFlooringsList could not be scraped with my current function.

[Ma-ve]

Sweet progress! Cool stuff man. The Wiki is a bit inconsistent, but great work so far!

[sungyeonu]

@AustinTSchaffer
I've been updating my scrape functions so the JSONS have the key! The structure looks much improved.