tsubakibotpad / pad-data-pipeline Goto Github PK

No. 6293 One For All, All Might has jp name 「ワン・フォー・オール」オールマイト and the automatic transliteration of the jp name is just fo which is the part between the dots, フォー. Unclear if this is a data issue with this one card and needs to be changed in the crud or a code issue where we have to change handling of special characters for a more general case.

Create a table called monster_series since now it'll be one-to-many relation

The ingame text changed recently, and for once it's actually clear:

We need to figure out what to do about monsters that aren't automatically found as drops

Currently farmable status is joining monsters to drops. So we could either:

1. Create fake rows in dropdata table
2. Add some new column override_farmable in monsters (probably this)
3. Make a new table with manual farmable data (not sure if there's any merit to this option vs the other two)

Probably extra data is sticking around in the pipeline and needs to be dropped from the awakenings table (or awoken skills? I don't remember which is which gg)

Example file is: https://d1kpnpud0qoyxf.cloudfront.net/ilmina/download_card_data.json

The last time it update was Jan 21, it's missing New Years in NA, there's some line missing from some updating script probably

Add field to monsters whether name override is guessed translation

(ticket requested by aradia)

Please do #39 at the same time

I think we add a column internal_monster_id which is XXXX_na or XXXX_jp whenever there's a difference for that one card, and then we add an extra entry for that one card; otherwise this column is the normal id

then we use internal_monster_id as the PK/FK and use this in all of our joins.

Export new tables monsters_na, active_skills_na, leader_skills_na, evolutions_na, exchanges_na (maybe don't bother with drops_na)

In MonsterGraph instead of node[model] we have node[models] = { 'na': , 'jp': } and these will normally point to the same thing but may point to different. Then graph.get_monster takes an optional param of server (defaults to jp and returns the na model if we want it to.

That should be like everything needed to fill TsubakiBotPad/pad-cogs#81

Make a new table called monster_name_overrides with fields name_en and is_translation

Also foreign key monster_id

is_translation is expected to be true always but this anticipates the possibility of it sometimes maybe not being true just in case; and the separate table moves manual data out of monsters. Also this opens up support for a name_ko idk.

Anyway once we do this we can automate fixing stuff like Maggie automatically, see TsubakiBotPad/pad-cogs#1010

We can put in better descriptions with stats etc and use this to auto populate "advanced help" for awoken skills

Potentially we can also use this to auto generate the padglobal command statawoken

see #23

In particular I want to have a "last modified" field in monsters otherinfo, of course this will have to be ??? at first but we can start populating

That is an ongoing issue I also reported to tactical_retreat previously. The maximum level for skills get changed from time to time. Unfortunately, that is not in sync across the JP, KR and NA version. NA usual gets these changes much later (sometimes months later). Here is a current example (20.12.2020):

Monster 2512 (Snow Star Byakko, Haku) has active skill 634 (Western Seven-Star Formation). In the database "turn_min" is 7 (Skill Level 8). Which is probably true for JP version (and maybe KR) but in the NA version the minimum turn is still 8 (Skill Level 7).

To fix this manually, find the offending IDs (they'll be listed in the human_fix_errors) and run this SQL to get all of the bad awakening IDs.

SELECT a1.awakening_id FROM awakenings AS a1 LEFT JOIN awakenings AS a2 ON a1.monster_id = a2.monster_id AND a1.order_idx < a2.order_idx  WHERE a1.monster_id IN (<PUT MONSTER IDS HERE>) AND a2.order_idx IS NULL;

from aradia:

DELETE FROM evolutions WHERE evolution_id IN (30766, 30764, 30754, 30555, 30757, 30763, 30557, 30753, 30761, 30759, 30556, 30756, 30769, 30758, 30760, 30755, 30768, 30765, 30767, 30762);

Example for Aljae:

10,000,000 additional damage when matching 4 or more colors; reduce damage taken by 50% when matching 5 orbs in an L shape; 20x ATK when matching 4 or more colors; 1.5x HP for Wood Att.

->

10,000,000 additional true damage when matching 4 or more colors; reduce damage taken by 50% when matching 5 orbs in an L shape; 20x ATK when matching 4 or more colors; 1.5x HP for Wood Att.

will need jp translation as well

Sometimes, dungeons can have more than one name prefix indicating their type. I think this has only been due to a typo by GHNA.

At one point, dungeon 4407 had this NA name, indicating (twice) that it was a quest dungeon:
#Q##Q#November Quest Dungeon-Int

This was eventually fixed, and the name is now:
#Q#November Quest Dungeon-Int
There are currently no dungeons with more than one type marker.

If the second marker is not stripped, you store the name as #Q#November Quest Dungeon-Int. Not a HUGE deal.

There is one that currently has two identical color markers, but the code doesn't break from it. It will probably be fixed at some point.
d=4429 f=7: $47ae64$$47ae64$Kaguya-Hime!-No RCV/Awoken Skills

The code which handles this:

This is currently out in NA with the name Constance Maggie but this isn't just a one-time issue, we really really need an automated log of these so that we can find out when they need to be replaced.

[Fixed 5 second movetime] 2x all stats for Attacker and Devil type; 2x ATK when matching 2+ Dark combos; 6.5x ATK when matching 1+ of Poison and Mortal Poison at once

should be Poison OR mortal poison

card id is 6801

https://github.com/TsubakiBotPad/pad-data-pipeline/blob/4a50f89665327814cdb69081d9255b8117ad7c8f/etl/pad/raw/exchange.py#L36-L39

[5] is level of the resulting monster.

[6] is bitflags for the resulting monster:

• &1: Max-skilled.
• &2: Max-awoken.

https://github.com/TsubakiBotPad/pad-data-pipeline/blob/4a50f89665327814cdb69081d9255b8117ad7c8f/etl/pad/raw/exchange.py#L68-L71
Bitflags.

• &1: No dupes.
• &2: "Restricted"/"One-Time Only!"
• &4: Multiexchange. Brings up the menu where you can exchange up to 30 times at once.

Are we resetting CORS when we re-upload files? do we need to change something in the pipeline code? in bash? in aws?

https://d1kpnpud0qoyxf.cloudfront.net/media/animated_portraits/06156.mp4 / https://d1kpnpud0qoyxf.cloudfront.net/media/animated_portraits/06156.gif

as well as in the still: https://d1kpnpud0qoyxf.cloudfront.net/media/portraits/06156.png?cachebuster=2

/images/awakenings is missing at least 1 new awakening

https://github.com/TsubakiBotPad/pad-data-pipeline/blob/4a50f89665327814cdb69081d9255b8117ad7c8f/etl/pad/raw_processor/crossed_data.py#L75-L78

6254. Pixel Volcano Dragon's Gem is 5902. Pixel Volcano Dragon's evo mat, rather than the result of an exchange (despite having the Gem LS: "A special Evo Material you can obtain through [Monster Exchange]").

In order to not need to process the raw enemy_skill data in the bot itself, the parameters for the skills must be stored in the database or the processing to emoji text needs to be done in the pipeline.
Example of the parameters:

Skill Type 105: DebuffRCV
Parameters: 1 and 2
p1: turns it is available
p2: buff/debuff amount
The parameters are super scuffed and are stored as an array of 16 parameters. Unused parameters are null. No I do not know why Gungho did it this way.

To store the processed text it should be in theory as easy as making a new converter (similar to EnESConverter) that extends the BaseConverter. Instead of outputting english/jp/kr you would output emojis/condensed text.

Regardless the skill type should probably also be stored in the enemy_skills table just in case something is missing/some weird edge case needs to be handled.

Finally (not related to Enemy Skills), MonsterBehaviors should maaaybe be stored as a JSON/XML...maybe...right now I am simply deserializing the behavior blob by copying the MonsterBehavior class from the pipeline. It might be impossible to fix this without fully restructuring how MonsterBehaviors are made.

This code is in monster_model:

if self.name_en == self.name_ja:
    self.roma_subname = self.make_roma_subname(self.name_ja)
else:
    # Remove annoying stuff from NA names, like Jörmungandr
    self.name_en = tsutils.rmdiacritics(self.name_en)

Since monsters.pem_egg and monsters.rem_egg don't work and have never worked and are being deleted (#30 / #39) and `egg_machines) doesn't store the data in a queryable way, create a table that can actually be joined

Enemy Shikokuen was released before Card Shikokuen in NA and KR servers, so GH didn't care enough to actually release their leader skill data.

rlr7d509/28/2020
There is a missing update in the current sqlite database file. Yian Kut-Ku (6463) is missing as encounter for the Skill Leveling Dungeon Monster Hunter (4151); *5 Monsters subdungeon (4151001).
Downloaded the sqlite file one hour ago.

rlr7d509/30/2020
I am not sure if you plan to look into this. But it sounds similar to a problem I had when I was using tactical_retreats SQL database. The issue was that they would reuse the skill leveling dungeons and add new monsters to it. In his case tactical_retreats would not touch a dungeon once was in his database. He had to fix it to make an update if they added new monsters.

rlr7d5Today at 10:01 PM
There is an "encounters" table with two foreign keys. One to dungeon and one to sub_dungeon
well and monster_id of course

"increase combo by 1 when matching 3 or more colors; 1000000 additional damage when matching 3 or more colors; 24x ATK and reduce damage taken by 60% when matching 5 Heal orbs in a cross formation"