sungyeonu / animal-crossing-scraper

Web scraper for Animal Crossing - New Horizons data using bs4

animal-crossing-scraper

Python webscraper to extract various data of the game Animal Crossing - New Horizons.

Data source: https://animalcrossing.fandom.com/wiki/

Feel free to use the JSON files! Some of the pages on the wiki are incomplete, so if some data is missing please check back later.

Output

All JSON files are stored in /data

-- Completed --
/museums

• bugs.json
• fish.json
• fossils.json
• artwork.json

/characters

• villagers.json

/flowers

• availability.json
• genetics_others.json
• genetics_rose.json
• hybridization_advanced.json
• hybridization_simple.json

/music

-- Incomplete --
/crafting

• tools.json
• equipments.json
• housewares.json
• miscellaneous.json
• wallMounteds.json
• wallpaperRugsFloorings.json
• others.json

/clothing

• tops.json
• bottoms.json
• dresses.json
• hats.json
• accessories.json
• socks.json
• shoes.json
• bags.json
• umbrellas.json

JSON fields

Price: -1 indicates the data is not available yet.

Set Up

1. In the animal-crossing-scraper root directory, clone the project using

git clone https://github.com/sungyeonu/animal-crossing-scraper.git

2. Set up a virtual environment

python -m venv venv

3. Activate the virtual environment

• Windows: venv\Scripts\activate
• Linux/Mac: source venv/bin/activate

4. Install required packages

pip install -r requirements.txt

Usage

To run:

python scrapy.py

To test:

python test_scrapy.py

animal-crossing-scraper's Issues

Create json data info page

Critter object key error

I saw that the the critter objects has dashes as the monthKey.
"seasons-northern-hemisphere"
Can you change it to underscores? Because some programming languages has problems when array/object key has dash in it.

obtainedFrom should be an array

Many of the items can be obtained from multiple sources (see https://animalcrossing.fandom.com/wiki/DIY_recipes/Miscellaneous) so the obtainedFrom property should be an array of strings rather than a single string

Feature request: Scrape records

https://animalcrossing.fandom.com/wiki/Music_(New_Horizons)

Feature request: Scraping of villagers

Can you add a JSON for the villager data?

Feature request: Scrape the villager's main page for additional villager info

Suggested aditional information:

• The main image
• Caption under the main image
• Gender (Separated from the personality)
• Initial clothes
• Skill
• Goal
• Coffee
• Favorite song
• Style
• The text from the Appearance section
• The text from the House section
• The text from the Trivia section

Missing data on villlagers

• home_request: ['Admiral', 'Audie', 'Bea', 'Billy', 'Biskit', 'Boots', 'Boris', 'Boyd', 'Buzz', 'Candi', 'Cashmere', 'Chester', 'Claudia', 'Cleo', 'Cyd', 'Dobie', 'Dom', 'Ellie', 'Gonzo', 'Hopkins', 'Hornsby', 'Huck', 'Ike', 'Jacob', 'Judy', 'Julia', 'June', 'Ketchup', 'Leopold', 'Lily', 'Megan', 'Mott', 'Murphy', 'Nibbles', 'Norma', 'Olive', 'Paolo', 'Peewee', 'Piper', 'Plucky', 'Raddle', 'Raymond', 'Reneigh', 'Rex', 'Rory', 'Sandy', 'Shep', 'Sherb', 'Snooty', 'Spike', 'Sprocket', 'Stella', 'Stu', 'Sylvana', 'Tad', 'Tammi', 'Tank', 'Tasha', 'Tybalt', 'Ursala', 'Vivian', 'Wade', 'Weber', 'Zucker']

• goal: ['Biskit', 'Dobie', 'Lily']

• skill: ['Biskit', 'Dobie', 'Lily']

• coffee: ['Biskit', 'Dobie', 'Lily']

• initial_clothes: ['Dobie', 'Lily']

• initial_phrase: ['Dobie']

• favorite_song: ['Biskit', 'Dobie', 'Lily', 'Louie', 'Paolo', 'Spike']

• style: ['Admiral', 'Bea', 'Billy', 'Biskit', 'Boots', 'Boris', 'Boyd', 'Buzz', 'Carrie', 'Cashmere', 'Claude', 'Cleo', 'Dobie', 'Ellie', 'Gonzo', 'Hornsby', 'Huck', 'Jacob', 'Julia', 'June', 'Ketchup', 'Leopold', 'Lily', 'Maddie', 'Maggie', 'Murphy', 'Norma', 'Paolo', 'Piper', 'Plucky', 'Raddle', 'Sandy', 'Sherb', 'Spike', 'Sprocket', 'Stella', 'Stu', 'Sylvana', 'Tad', 'Tasha', 'Tybalt', 'Ursala', 'Wade', 'Weber']

• appearance: ['Biskit', 'Dobie', 'Lily']

• birthday: ['Dobie']

• species: ['Dobie']

Villagers such as Dom and Judy with field values = null are not included.

Feature request: Scrape furnitures

Can you scrape the non-DIY furnitures, too?

Feature request: Scrape deep-sea creatures

https://animalcrossing.fandom.com/wiki/Deep-sea_creatures_(New_Horizons)

Feature request: Scrape flowers

Could you scrape flowers, colors, breeding data, etc.?

Bug: Missing villager images

For Spork and Jacob the image url is missing.
There is only /wiki/Crackle and /wiki/Jacob in the json as the url.

Add astrological signs to villagers.json

Species error for some villagers

I found an error in the villager.json.
The first letter is missing for the following villager species:

• Ankha
• Charlise
• Dom

Group variations and materials and their imageLinks

I think these two data fields should be paired, something like this:

"variations": [
"Black",
"Red",
],
"variationImageLinks": [
"https://vignette.wikia.nocookie.net/animalcrossing/images/e/e2/NH-Tops-Silk_shirt_4.png/revision/latest/scale-to-width-down/50?cb=20200414145617",
"https://vignette.wikia.nocookie.net/animalcrossing/images/e/e7/NH-Tops-Silk_shirt_1.png/revision/latest/scale-to-width-down/50?cb=20200414145454",
],

to

"variations": [{
"color": "black",
"imageLink": "https://vignette.wikia.nocookie.net/animalcrossing/images/e/e2/NH-Tops-Silk_shirt_4.png/revision/latest/scale-to-width-down/50?cb=20200414145617",
},
.
.
.

Create a separate completed_fossils.json file

Then nest the parts in it

Create testing

Create unit tests for DIY page, fish, bugs, and fossils. Manually grab test data from wikis (maybe first two objects for each category)

Replace spaces with underscores in keys

Add full name in "name" tag instead

Add full-size-image, catch-phrase, museum-phrase to fish and bugs?

Example: https://animalcrossing.fandom.com/wiki/Bitterling

• catch-phrase: "I caught a bitterling! It's mad at me, but only a little."

• museum-phrase: "Bitterlings hide their eggs inside large bivalves—like clams—where the young can stay safe until grown. The bitterling isn't being sneaky. No, their young help keep the bivalve healthy by eating invading parasites! It's a wonderful bit of evolutionary deal making, don't you think? Each one keeping the other safe... Though eating parasites does not sound like a happy childhood... Is that why the fish is so bitter?"

• full-size-image: https://vignette.wikia.nocookie.net/animalcrossing/images/e/e9/Nh-bitterling.jpg/revision/latest?cb=20200331062501

acnhapi.com uses image and icon to seperate the two images. Currently this repo only scrapes the icon. Also, the full-size-image on animal crossing.fandom has a background color while acnhapi.com offers the full-size-image like https://acnhapi.com/v1/images/fish/1 without background colors.

Scrape artworks

Wiki link: https://animalcrossing.fandom.com/wiki/Artwork_(New_Horizons)

each object should contain: painting name, forgery link, genuine link, description.

Wiki renamed the category "Critter" to "Museum". Change the category in scrapy.py as well

Scrape clothing

wiki link: https://animalcrossing.fandom.com/wiki/Clothing_(New_Horizons)
Clothing is broken down in 9 categories: scrape all of them

Bug Report: DIY items with broken imageLink

I tried several images in the DIY jsons, but for every try I got the "broken image" image as the result
For example this is the "Ironwood DIY Workbench"'s image: https://vignette.wikia.nocookie.net/animalcrossing/images/2/2e/NH-Ironwood_DIY_workbench.png/revision/latest?cb=20200329061925

Update request: Use the original image link without the downscale tag

In some pages there's a downscale query in the image url, for example on DIY pages materials section:
https://vignette.wikia.nocookie.net/animalcrossing/images/d/d4/NH-acorn_icon.png/revision/latest/scale-to-width-down/18?cb=20200407141759
There the part /scale-to-width-down/18 downscales the acorn icon to a small 18x18 icon. If you skip that part of the URL you'll got the original image.

As I saw this tag is used on the DIY page at the materials part and on the villagers page at the villager images, but maybe they're used at other places at well.

remove scale down in image_url in villagers

Feature request: scraping of recipes

Heya!

I could really use recipes in a proper format -- however, animalcrossing.fandom.com doesn't make it easy.

They have an incomplete list at https://animalcrossing.fandom.com/wiki/DIY_recipes (and sub categories). But, event wise, for the Cherry-blossom season event, this one is complete:

https://animalcrossing.fandom.com/wiki/Cherry-blossom_petal

Then again, the Bunny Day recipes are available at https://animalcrossing.fandom.com/wiki/DIY_recipes/Housewares, but only if you look for a [a-zA-Z0-9] egg.

I've found no other website that has a properly formatted list of differing event recipes.

In any case, is there any use for this, in this repo? Maybe just scrape the DIY_recipes page?

Modification request: Separate the critter "Active Hours"

There are several critters with two active time ranges and it shared in the same string.
For example: '4 AM - 8 AM & 4 PM - 7 PM'
Can you separate the two time ranges as separate strings?

Feature request: Add hemisphere based critter data

Can you add both hemispheres to the fish and to the bug data?