mtgjson / mtgjson3

MTGJSON repository for Magic Cards

Intro

This is the code used to generate the JSON for mtgjson.com

I didn't plan on open-sourcing this, so there are rough edges. See 'Other' section below.

Requirements

• Linux or MacOS (never tested under Windows)
• Node.js v0.10.0 or later. It will NOT work with 0.9.9 or earlier.

Setup

npm install
# See 'Other' below

Usage

cd build
node buildSet.js <expansion code>

This creates the file:

json/<expansion code>.json

You will need to run this twice for any 'new' sets. First pass doesn't include foreign langauges.

Troubleshooting

If something doesn't work after doing a git pull update, make sure you also update NPM modules with

npm update

Including a new set

If you're building mtgjson data on your own, you should follow the following steps to be able to grab the new set data:

• make sure the new set is already available on gatherer
• add a file describing the set in shared/set_configs or an appropriate subdirectory
• node build/buildSet NEWSETCODE
• check your json files to see if everything is in order
• node util/updatePrintingsFromSet NEWSETCODE -- This will update the printings on the previous sets, using the data from the new set.
• node util/updateRulingsFromSet NEWSETCODE -- Same as above, but for rulings.
• node util/updateLegalitiesFromSet NEWSETCODE -- Same as above, but for legalities.

Other

All gatherer page requests are cached to the cache directory and any future requests will use the cached version instead.

The 'verify' directory is a little private local webpage I use to verify that my JSON is correct by comparing it side by side with card images.

The 'web' directory is used to generate the mtgjson.com website.

Note that both verify and web have some symbolic links to files that only exist on my local dev workstation. Sorry.

mtgjson3's Issues

Duel Decks: Elspeth vs. Kiora is currently on gatherer…

but it's under the name Duel Decks: Kiora vs. Elspeth.

empty colors

colorless cards have their color value represented in two different ways: as an empty array (the majority), or as a non-existent key (eg ghostfire)

Add a foil information per card

There should be foil information on a per card level. Early sets had no foils, later sets had foils, some sets like Planeshift had just a few cards that could be foil and some sets are foil only.

Right now there is no way to identify using solely the MTGJSON data which sets/cards are available in foil format so this information would be useful.

foreign multiverse id

it might be useful to include the multiverse id in the foreign names field. this would allow me to fetch images in different languages

Add 'deck' or 'subset' field

This can be used to identify which cards are part of which decks (such as 'elf' deck in the 'elves vs goblins' set). The existing 'source' field is used a bit like this for some promos sets like CPK, but it's not quite the same thing, but it probably would work just fine for this.

Magic: Origins now on Gatherer

Thought I'd leave you a note 😸

[Gatherer] Fate Reforged watermark errors

Gatherer is wrong about something. 😏

FRF Abzan cards think they have the Temur watermark.
FRF Temur cards think they have no watermark.

---

I don't suppose you've ever considered sending your ever-growing list of corrections directly to rules manager Matt Tabak?

Add additional Duels sets

Should add the additional Duels of the Planeswalkers sets (2012/1013/etc) and other 'Duels' digital sets, such as 'Magic Duels: Origins'. Will need to make sure I get the 'booster' property set correctly as it differs significantly from physical MtG.

Some language's name are not included in AllCards

Is this a Gatherer or script problem?

For example, Mana Crypt has no 'foreignLanguages' array even thought there is a pt-BR version of it and Scavenging Ooze has the pt-BR name in the 'portuguese' index, though there is no 'portuguese (Brazil)' index.

Thanks for the awesome work.

unglued basics

unglued basic lands have the rarity 'common'; could this be changed to 'basic land', like every other set?

http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=9680 - UGL plains
http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=73963 - UNH plains

Move tokens to a new 'tokens' array

As per Rule 108.2 tokens are not considered cards. Thus I should move them out of the 'cards' array and add a new 'tokens' array along side it. The easiest way to do this would probably be AFTER all processing on the set and cards is done, simply move any card where layout==="token" out of the cards array and into the new tokens array.

[unh] super secret tech rarity

http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=74272

the unhinged card 'super secret tech' has the rarity 'special'. this probably should be rare instead

DKM (Deckmasters) set does not contain multiverseid

They're not present in DKM.json or Allsets.json

Cache Issues

This is a great service. I was actually in the middle of scraping the data myself when I came across this. There is a fairly large problem with the caching implementation though, and that is that it never updates the cache. There should be some rule for how long a cache file stays valid, otherwise you'll never pick up on ruling updates. I believe this is what is causing your Theros file to be missing a LOT of rulings. Some are as far back as 9-15-2013 like the Whip of Erebos. I would be happy to help out with this, but I really don't want to dig into the java side of the project!

There's no such set as Elves vs. Dragons.

As possibly the biggest fail of all time — Duel Decks Anthology: Elves vs. Goblins is currently in gatherer as Duel Decks Anthology: Elves vs. Dragons.

CON.json invalid file handle

Looks like the json starts many characters into the file and is missing a good chunk of information including the opening json braces.

Add deck card restriction info

Some of the duels decks have different restrictions as to how many cards of each rarity can be put into deck, namely: common-4, uncommon-3, rare-2, mythic/planeswalker-1.

Once I do Issue #53 I should also look into adding this 'deck card restriction' info too. Will need to figure out what it all is though. Also, maybe it doesn't belong, because the restriction may be more about the 'format' rather than the 'set'. Will need to see what my research shows when the time comes.

Add 'color identity'

A user requested the following: "Commander/EDH, and color identity is what determines what cards can go in your decks for Commander. It refers to any colored mana symbol on the card, be it in the cost or text box. So, Yasova is URG, Talisman of Dominance is UB, Karn, Silver Golem is 'Colorless', the Pacts and Evermind are their respective colors as indicated by their Color Indicator, etc. Most cards are pretty straightforward, and just the color in the mana cost, but everything from Quagmire Druid to Memnarh to Lands (Sandsteppe Citadel) to Rattleclaw Mystic have color identities that differ than their casting cost.

http://mtgcommander.net/rules.php has an explanation of color identity (the official source of commander info/rules); it's also addressed in the comprehensive rules at 903.4 (http://magiccards.info/rule/903-commander.html)."

Add tokens

Add Tokens. One place to get them is magiccards.info
Token List: http://magiccards.info/extras.html

Sadly magiccards.info doesn't have all of them and they are very slow to add new ones.

More token info:
https://github.com/Psithief/Cockatrice-tokens/blob/master/tokens.
http://mtgsalvation.gamepedia.com/Token/Full_List

Should do Issue #46 before this one, as the new tokens will go into the new tokens array.

Note: Afterwards, in addition to updating test_urls.js I also need to look at the tokens folder in DevLab/images/originals/mtg/CCGHQ/full/

Creatures have type "Summon" in some sets

This isn't so important, but... a few cards in pCEL have 'types': ["Summon"]... shouldn't this be ["Creature"]? By the way, the same is true for creatures in Unglued (and one in Unhinged, "Old Fogey" :-). The 'type' attribute has the same issue.

I realize these are special sets, so maybe the usual "normalization" doesn't apply here?

Commander 14 is up

Whenever you get a chance, can you add the Commander '14 product (C14)? :)

Update for M15

Just a request to update for the newly spoiled M15 :)
Good job, you guys, keep it up! (tell me if there is something I can do)

Add TCGPlayer set names for each set

This would allow people who use MTGJSON to more easily integrate into TCGPlayer prices.

m14

it's on gatherer - http://gatherer.wizards.com/Pages/Search/Default.aspx?output=spoiler&method=text&action=advanced&set=+%5b%22Magic+2014%22%5d

Reserved list

It would be cool if the JSON for cards on the reserved list contained a field indicating this. Something like
{"reserved": true} ?

See: http://archive.wizards.com/Magic/tcg/article.aspx?x=magic/products/reprintpolicy

Could we add a scraper to craw this page?

Problem with mhash

I'm trying to get the mtgjson application to work and for some reason it's throwing an error.

root@farragut:/var/www/isleofgames/mtgjson/mtgjson# node build/buildSet.js allsets

Doing sets: 10E,2ED,3ED,4ED,5DN,5ED,6ED,7ED,8ED,9ED,ALA,ALL,APC,ARB,ARC,ARN,ATQ,AVR,BNG,BOK,BRB,BTD,C13,C14,CHK,CHR,CM1,CMD,CNS,CON,CSP,DD2,DDC,DDD,DDE,DDF,DDG,DDH,DDI,DDJ,DDK,DDL,DDM,DDN,DGM,DIS,DKA,DRB,DRK,DST,EVE,EVG,EXO,FEM,FUT,GPT,GTC,H09,HML,HOP,ICE,INV,ISD,JOU,JUD,KTK,LEA,LEB,LEG,LGN,LRW,M10,M11,M12,M13,M14,M15,MBS,MD1,ME2,ME3,ME4,MED,MIR,MMA,MMQ,MOR,MRD,NMS,NPH,ODY,ONS,PC2,PCY,PD2,PD3,PLC,PLS,PO2,POR,PTK,RAV,ROE,RTR,S00,S99,SCG,SHM,SOK,SOM,STH,THS,TMP,TOR,TSB,TSP,UDS,UGL,ULG,UNH,USG,V09,V10,V11,V12,V13,V14,VAN,VIS,VMA,WTH,WWK,ZEN

Ripping Set: Tenth Edition
Getting card list...
Error ripping: Tenth Edition
TypeError: undefined is not a function
at Object.generateCacheFilePath (/var/www/isleofgames/mtgjson/mtgjson/shared/shared.js:357:16)
at getURLAsDoc (/var/www/isleofgames/mtgjson/mtgjson/build/rip.js:466:25)
at Function.getListHTML (/var/www/isleofgames/mtgjson/mtgjson/build/rip.js:46:4)
at next (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:48:22)
at tiptoe (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:111:2)
at Object.ripSet (/var/www/isleofgames/mtgjson/mtgjson/build/rip.js:24:2)
at Function.build (/var/www/isleofgames/mtgjson/mtgjson/build/buildSet.js:36:8)
at next (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:48:22)
at tiptoe (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:111:2)
at CBRunner.fun (/var/www/isleofgames/mtgjson/mtgjson/build/buildSet.js:33:2)

Line 357 in shared.js is:
var urlHash = hash("whirlpool", targetUrl);

I've confirmed that targetUrl is a valid string, so there seems to be something about the hash function that is throwing the error. I even tried to bring down your mhash project code and rebuild it but it had no effect. What am I doing wrong?

zip path

AllSets.json.zip unzips into mnt/compendium/DevLab/mtgjson/web/json/AllSets.json. would be nice if it gave just the json file, without the nested directories

Several promo sets have invalid number fields

Go through each 'promo' set (where isMCISet is set to true) and 'verify' whether or not the cards have card numbers and 'remove' the 'number' field from any promo sets that shouldn't have a number field. Verify against card images

WPN and Gateway should be split

WPN and Gateway were two separate programs, despite magiccards.info combining them into one. Yes, WPN replaced Gateway, but they are not the same program and right now as they are combined there is no way to determine which set each card is from as the years overlap.

So I should manually figure out which ones were part of which and separate them into pWPN and pGTW

Split Vanguard (VAN) into two sets

Vanguard (VAN) consists of 32 physical cards and then a bunch of avatar cards that were online only. These should be split into two separate sets, making sure the online only one has an 'onlineOnly' set to true

Gatherer Updated with MM2

Just thought I'd leave you a note :)

Add MTGO only sets

There are several missing MTGO only sets that could be added:

• PRM set. (promo cards)
• Mirrodin Pure vs. New Phyrexia

Theme Decks:

• Boltslinger
• Exiler
• Deathdancer Xira
• Enchantress Rubinia

More sets?

I have an e-mail from someone that lists out a lot of the PRM cards, but I would need to find a more automated way of updating this list, probably looking into the MTGO game files directly to see if I can parse this info out or not.

Mishra's Toy Factory

Can you update pHHO to include Mishra's Toy Factory when you get a chance? :)

Thanks for your hard work!

Request: Json that defines current formats

Hiya,

It'd be great if there was a way to cross-section the sets with the current formats, i.e. to find out what's currently "standard" or "modern".

http://magic.wizards.com/en/content/standard-formats-magic-gathering
http://magic.wizards.com/en/gameinfo/gameplay/formats/modern

holiday cards

http://wiki.mtgsalvation.com/article/Promotional_cards#Happy_Holidays_cards

these cards are not in gatherer, but could they be added to mtgjson?

Add full foreign language support

Add full foreign language scrapes.

Implementation info:
Need to use ?printed=true on Gatherer to get this data.
If the cards have unique multiverseids probably have totally seperate json. Files like AllCards-x-ch.json or what not for each language.
When doing this, keep the 'foreignNames' field, and make sure it includes 'English' in non-english printings

Fated Reforged

Just letting you know Gatherer is updating as i type this message and that it's time for an update. Thanks for your work dude :)

Update: It's live on Wizards' Gatherer now 👍 !

docs: linkify sections

eg http://mtgjson.com/#cards

Ertai the Corrupted 29292 is a promo card. Change set?

Would you consider changing the set for Ertai, the Corrupted?
This is actually a promo card, and should be in one of the promo sets.
The actual card from the set is here with id 25614.

Adding In "Number" to AllSets.json

I see how and why you don't include the number field in the AllSets, but maybe it could be possible to include it.

Seeing as how the arrays printingCodes and printings are in the same order, is it possible to add a number array that follows the same format, holding the same object type as originally designated in the docs? In this case, the values in number are Strings, as they would be in the single set files.

Fictitious Example:

Variable	Value
printingCodes	[ "ICE", "CHR" ]
printings	[ "Ice Age", "Chronicles" ]
number	["3", "185"]

If the array idea above could work, then it could also potentially be applied to the other missing fields: artist, border, flavor, multiverseid, originalText, originalType, rarity, releaseDate, reserved, timeshifted, variations, and watermark

I'd really like to hear your opinion on this ⛵

Subdirectories in cache path

Writing a cache file will attempt to save to non-existant one-letter subdirectory in cache.

originalCardText has invalid activation costs in earlier cards

Many of the early 4th edtion and earlier cards have messed up activation costs in the 'originalCardText' field (such as '1oRooR' on gatherer for 1RR) and also missing colons for almost all of them. These are gatherer errors that I'll need to correct by hand and add to SET_CORRECTIONS (though some may be able to be fixed in an automated fashion if I can determine a pattern to the errors)

Being that these errors are in the originalCardText and not in the oracle 'text' the priority on fixing this issue is much lower

Artist Corrections

Submitted corrections from a user:

1. Certain printings of "Armored Griffin", "Just Fate", "Piracy", and "Town Sentry" have Brad Williams, though he goes by Bradley Williams on most cards and most printings.
2. Certain printings of "Hamletback Goliath" have Brian Snoddy, without the accent found on all other instances.
3. Certain printings of "Plains" have Christoper Rush, without an 'h'.
4. "Keeper of Tresserhorn" has D.A. Gregory, instead of D. Alexander Gregory found on all other instances.
5. "Curse of Thirst" promo printing has Dave Rendall, though it's actually Dave Kendall.
6. "Rakdos Cackler" promo printing has Daven Kendall, though it's still just Dave Kendall. =]
7. Certain printings of "Brainstorm" have DiTerrlizzi, with an extra 'r'.
8. "Urborg Mindsucker" has both Tony Diterlizzi and Tony DiTerlizzi.
9. "Counterspell" has Dom!, with an exclamation point, which matches the printed card but not the name on his other three cards.
10. Certain printings of basic lands have Edward P. Beard Jr., without the comma found on his other cards.
11. Certain printings have G. Darrow, instead of Geofrey Darrow.
12. Certain printings of "Worn Powerstone" have Henry G. Higginbotham, instead of Henry G. Higgenbotham.
13. Certain printings of "Mycoid Shepherd" have Jarreau Wimberley, instead of Jarreau Wimberly.
14. "Fireball" and "Incinerate" have Jock, while "Cinder Cloud", "Dwarven Miner", and "Kukemssa Pirates" have JOCK in all caps.
15. Certain printings of "Man-o'-War" have John J. Muth, instead of Jon J. Muth.
16. "Mortal Wound", and "Ovinomancer" have Kevin Walker, though most cards have Kev Walker.
17. "Boggart Ram-Gang" promo printing has Lucio Patrillo, instead of Lucio Parrillo.
18. "Dark Hatchling", "Mana Leech", and "Spined Fluke" have Mark Nelson, instead of Mark A. Nelson.
19. "Mana Crypt" has Matt Steward, instead of Matt Stewart.
20. Certain printings have Matthew Wilson, instead of Matthew D. Wilson.
21. Certain printings of "Grave Titan" have Nils Hamm (Lucas Graciano), though it's painted by Lucas Graciano and misprinted as Nils Hamm.
22. Certain printings have Parente, instead of Paolo Parente.
23. Certain printings have Randy Asplund, instead of Randy Asplund-Faith.
24. "Swamp", and "Untamed Wilds" have Romas, instead of Romas Kukalis.
25. Certain printings have Scott M. Fisher, instead of Scott M. Fischer.
26. "Gitaxian Probe" promo printing has Trevon Claxton, instead of Trevor Claxton.

vma has no 'booster' field

Add additional planes/schemes/vanguards

yawgatog.com has a few planes/schemes/vanguards that are not present on mtgjson
planes -- Celestine Reef, Horizon Boughts, Mirrored Depths, Tember City
schemes -- Drench the Soil in Their Blood, Imprison This Insolent Wretch, Perhaps You've Met My Cohort, Plots That Span Centuries, Your Inescapable Doom
vanguards -- Jace Beleren Avatar, The Avenger Avatar, The Harvester Avatar, The Hunter Avatar, The Philosopher Avatar, The Protector Avatar, The Warrior Avatar

Additional info:
Missing oversize cards. the planes and schemes are all promos. the planes are listed as the bottom 4 cards on -- http://magiccards.info/query?q=celestine reef&v=card&s=cname -- and the schemes
are listed as the bottom 5 cards on -- http://magiccards.info/query?q=plots that span centuries&v=card&s=cname -- the missing vanguards however i could not find anywhere on gatherer or magiccards.info. they appear to also be promos, but only available online.

Lastly, I should also look into additional data yawgatog may have that would be useful

imageName with numbers appended to card name.

In a good number of cards, the json imageName field is a string that ends with either a 2, 3, or 4. In these cases, without fail, the image is not found at mtgimage.com/card/[card name].jpg but is instead at mtgimage.com/card/[card name with number omitted].jpg.

I recognize that this might, instead be an issue with mtgimage, but as I cannot be sure, I opted for posting the issue here, as it appears to have more traffic than mtgimage.

Urza's Power Plant isn't getting the correct subtypes.

Urza's Power Plant will parse like this:

u'subtypes': [u'Urza\u2019s', u'Power']

when it should look like:

u'subtypes': [u'Urza\u2019s', u'Power-Plant']

pack size

would be nice to have the pack size of each set. see: http://wiki.mtgsalvation.com/article/Booster_pack

Who/What/When/Where/Why

http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=74358

mtgjson has this card's layout property as normal, instead of split

add github link to docs

mtgjson.com does not link to this github repo; imo it should