hearthsim / python-hsreplay

Python library for creating and parsing HSReplay XML files

python-hsreplay

A python module for HSReplay support.

https://hearthsim.info/hsreplay/

Installation

The library is available on PyPI. pip install hsreplay will install it.

Dependencies:

• hearthstone
• hslog
• lxml (optional) for faster XML parsing and writing. Will use xml.etree if not available.
• aniso8601 or dateutil for timestamp parsing

Usage

The main document class is hsreplay.document.HSReplayDocument. That class contains all the necessary functionality to import and export HSReplay files.

Reading/Writing HSReplay XML files

The classmethod from_xml_file(fp) takes a file-like object and will return a document. If you already have an ElementTree object, you can call the from_xml(xml) classmethod instead.

To export to an HSReplay XML document, the HSReplayDocument.toxml(pretty=False) method can be used to obtain a UTF8-encoded string containing the document.

Reading directly from a log file

The library integrates directly with the python-hearthstone library to produce HSReplayDocument objects directly from a log file or a parser instance.

Use the helper classmethods from_log_file(fp, processor="GameState", date=None, build=None) and from_parser(parser, build=None), respectively.

Exporting back to a PacketTree

It is possible to export HSReplayDocument objects back into a PacketTree with the to_packet_tree() method. This therefore allows lossless conversion from a PacketTree, into HSReplayDocument, then back into a PacketTree.

This is especially interesting because of the native functionality in python-hearthstone which is able to export to a Game tree and allows exploring the game state. By converting HSReplayDocument objects to a PacketTree, it's very easy to follow the replay at a gameplay level, explore the state of the various entities and even hook into the exporter in order to programmatically analyze it.

python-hsreplay's Issues

Player names missing when converting reconnected games

In the second game (after the reconnect).

powerlog: http://files.hsdecktracker.net/powerlog.txt
hsreplay: http://hsreplay.net/joust/replay/d43a810d-3b40-419f-ad8b-0aa57a47cc31

Fix missing entity in TargetOption

@beheh commented on Sun Aug 27 2017

The following options block from this power.log contains a player name not matching the Battletag. The generated HSReplay file is therefore missing the entity attribute on the Target block.

D 23:49:45.8759909 GameState.DebugPrintOptions() -   option 1 type=POWER mainEntity=[name=Gadgetzan Socialite id=43 zone=HAND zonePos=1 cardId=CFM_659 player=2] error=NONE errorParam=
D 23:49:45.8759909 GameState.DebugPrintOptions() -     target 0 entity=[name=Anduin Wrynn id=64 zone=PLAY zonePos=0 cardId=HERO_09 player=1] error=NONE errorParam=
D 23:49:45.8759909 GameState.DebugPrintOptions() -     target 1 entity=[name=Magni Bronzebeard id=66 zone=PLAY zonePos=0 cardId=HERO_01a player=2] error=NONE errorParam=
D 23:49:45.8759909 GameState.DebugPrintOptions() -     target 2 entity=[name=Emerald Reaver id=47 zone=PLAY zonePos=1 cardId=UNG_803 player=2] error=NONE errorParam=
D 23:49:45.8759909 GameState.DebugPrintOptions() -     target 3 entity=[name=Bloodsail Corsair id=33 zone=PLAY zonePos=1 cardId=NEW1_025 player=1] error=NONE errorParam=
D 23:49:45.8759909 GameState.DebugPrintOptions() -     target 4 entity=GameEntity error=REQ_HERO_OR_MINION_TARGET errorParam=
D 23:49:45.8759909 GameState.DebugPrintOptions() -     target 5 entity=dan conley error=REQ_HERO_OR_MINION_TARGET errorParam=
D 23:49:45.8764913 GameState.DebugPrintOptions() -     target 6 entity=Carrot error=REQ_HERO_OR_MINION_TARGET errorParam=
D 23:49:45.8764913 GameState.DebugPrintOptions() -     target 7 entity=[name=Lesser Heal id=65 zone=PLAY zonePos=0 cardId=CS1h_001 player=1] error=REQ_HERO_OR_MINION_TARGET errorParam=
D 23:49:45.8764913 GameState.DebugPrintOptions() -     target 8 entity=[name=Armor Up! id=67 zone=PLAY zonePos=0 cardId=CS2_102_H1 player=2] error=REQ_HERO_OR_MINION_TARGET errorPar

HSReplay document with missing entity attribute on <Target/>:

<Option EntityName="Gadgetzan Socialite" entity="43" index="1" type="3">
	<Target EntityName="Anduin Wrynn" entity="64" index="0"/>
	<Target EntityName="Magni Bronzebeard" entity="66" index="1"/>
	<Target EntityName="Emerald Reaver" entity="47" index="2"/>
	<Target EntityName="Bloodsail Corsair" entity="33" index="3"/>
	<Target entity="1" error="33" index="4"/>
	<Target error="33" index="5"/>
	<Target entity="3" error="33" index="6"/>
	<Target EntityName="Lesser Heal" entity="65" error="33" index="7"/>
	<Target EntityName="Armor Up!" entity="67" error="33" index="8"/>
</Option>

The resulting XML cannot be reimported using HSReplayDocument.from_xml_file, since it erros with the following stacktrace:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
helpers.py:126: in __init__
    self.exporter = get_exporter_from_replay(path, is_power_log)
helpers.py:111: in get_exporter_from_replay
    packet_tree = replay.to_packet_tree()[0]
../../../redshift-env/lib/python3.4/site-packages/hsreplay/document.py:71: in to_packet_tree
    ret.append(game.export())
../../../redshift-env/lib/python3.4/site-packages/hsreplay/elements.py:90: in export
    tree.packets.append(node.export())
../../../redshift-env/lib/python3.4/site-packages/hsreplay/elements.py:343: in export
    packet.options.append(node.export(i))
../../../redshift-env/lib/python3.4/site-packages/hsreplay/elements.py:360: in export
    packet.options.append(node.export(i))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <OptionTargetNode>, id = 5

    def export(self, id):
            optype = "target"
            type = None
            packet = self.packet_class(
>                   self.ts, int(self.entity), id, type, optype,
                    self.error, self.errorParam
            )
E           TypeError: int() argument must be a string or a number, not 'NoneType'

../../../redshift-env/lib/python3.4/site-packages/hsreplay/elements.py:392: TypeError

Player.initial_deck is sometimes missing cards

Downstream issue: HearthSim/HSReplay.net#43

This log file is missing cards in the initial_deck of player "Red1010". The replay itself contains a lot more cards (a seen in the replay).

I'm testing with the following script:

import sys
from datetime import datetime
from hsreplay.dumper import parse_log

file = open(sys.argv[1], "r")
parser = parse_log(file, processor="GameState", date=datetime.now())
file.close()
game_tree = parser.games[0]

for player in game_tree.game.players:
    print("Found player %s" % player.name)
    for card in player.initial_deck:
        print("...with card %s" % card)

...and am only getting the following cards for the second player:

Found player Red1010
...with card Card(id=60, card_id='EX1_391')
...with card Card(id=61, card_id=None)
...with card Card(id=62, card_id=None)
...with card Card(id=63, card_id=None)
...with card Card(id=64, card_id=None)
...with card Card(id=65, card_id=None)
...with card Card(id=66, card_id=None)
...with card Card(id=67, card_id='OG_283')

Not sure if this should be here on in python-hearthstone. Just copy over if necessary.

Require enum IDs for tags

The spec currently allows for tags to be set by name but this is problematic when it comes to forward-compatibility (we have seen tags be renamed in the past, such as DEATH_RATTLE -> DEATHRATTLE).

This is blocked by jleclanche/fireplace#167

Figure out why META_ is sometimes prepended to DAMAGE/TARGET

See commit f9ec23d

Difference between c# and python implementation of Options?

In c#, OptionsHandler forces options to be at the root of the element:

var match = Regexes.OptionsEntityRegex.Match(data);
if(match.Success)
{
    var id = match.Groups[1].Value;
    state.Options = new Options {Id = int.Parse(id), OptionList = new List<Option>(), TimeStamp = timestamp};
    state.UpdateCurrentNode(typeof(Game));
    if(state.Node.Type == typeof(Game))
        ((Game)state.Node.Object).Data.Add(state.Options);
    else
        throw new Exception("Invalid node " + state.Node.Type + " -- " + data);
    return;
}

The effect of this is that if you are within an Action, the Options node will cause the parser to set the current node as the Game, and all subsequent calls will be added to the Game, instead of the previous Action node.

I don't know enough Python to point out where in the code that happens, but looking at the attached files you can clearly see at line 793 that the output XML diverges

python.out.xml.txt
csharp.out.xml.txt

The initial log is also attached, and from reading it (line 878 is where the option is declared), there seems to be no reason not to attach the Options to the root Action instead of the Game node.

Power_2.log.txt

So my question is: what is the correct output?

Thanks!

META_DATA line not parsed correctly

When i try to parse a file generated by the game, HSReplay throws an error in DataHandler.cs on line 114

state.Node.Type == Game

attached log file

Replacing SendChoices by ChosenEntities

The SendChoiceselement we get from GameState.SendChoices is very symmetrical to ChosenEntities (GameState.DebugPrintEntitiesChosen), which we only get from other players. The problem is that we don't get the exactly the same data in each case, so we have to keep them separate for now.

Ideally we should use ChosenEntities for both. But then we lack the choice type.. this would require some changes on Blizzard's side.

Importing output_log: object has not attribute 'game'

Hey,

I've tried to generate an hsreplay from the output_log.txt file, but got a 'PowerLogParser' object has no attribute 'game', and indeed it looks like the generated log files miss the game declaration at the beginning (I've attached an example of output_log.txt I get using HDT).

Is there something to configure that I've missed?

Thanks,
Seb

AssertionError: assert self.players[entity] == id

python3.4 /root/projects/HSReplay/python/main.py /root/projects/md5/9285997e3519b5cab46576799461acdd.log > /root/projects/xml/9285997e3519b5cab46576799461acdd.xml
Traceback (most recent call last):
  File "/root/projects/HSReplay/python/main.py", line 671, in <module>
    main()
  File "/root/projects/HSReplay/python/main.py", line 665, in main
    parser.read(f)
  File "/root/projects/HSReplay/python/main.py", line 331, in read
    self.add_data(*sre.groups())
  File "/root/projects/HSReplay/python/main.py", line 342, in add_data
    self.handle_data(ts, data)
  File "/root/projects/HSReplay/python/main.py", line 490, in handle_data
    self.game.update_current_player(entity, str(value))
  File "/root/projects/HSReplay/python/main.py", line 112, in update_current_player
    self.register_player_id(entity, self.first_player)
  File "/root/projects/HSReplay/python/main.py", line 102, in register_player_id
    assert self.players[entity] == id
AssertionError

Creates a blank 9285997e3519b5cab46576799461acdd.xml.

Input log: https://files.spectralcoding.com/files/projects/hscs/hsreplay/issue20151017/9285997e3519b5cab46576799461acdd.log
Log file from Oct 13th, 2015, probably HS 3.1.0.10357

Missing "state.UpdateCurrentNode(typeof(Game), typeof(Action));" when handling ACTION_END tag

When converting the following output_log to XML using the C# implementation, you get an incorrect XML: the last action is nested, while it should really be at the root.

forgottentorch.txt
actual_forgottentorch.xml.txt

It looks like DataHandler.cs isn't updating the currentnode before setting the new node value:

if(data == "ACTION_END")
{
    state.Node = state.Node.Parent ?? state.Node;
    return;
}

adding the following line seems to fix the issue (at least on my Java implementation which is a direct port of the c# one):

if(data == "ACTION_END")
{
    state.UpdateCurrentNode(typeof(Game), typeof(Action));
    state.Node = state.Node.Parent ?? state.Node;
    return;
}

as it results in the following XML:
forgottentorch.xml.txt

As I have no way to run / test C#, I'll leave it to you to see if the fix is needed :)

[Feature Request] Make in-play secrets viewable

When watching replays, please it possible to mouse over the "?" logo for secrets on the character portrait to see what secrets are in play.

Or better yet, come up with a way to display which secrets are currently in effect so mousing over isn't necessary.

Otherwise it's a small nightmare trying to remember which secrets are currently in play when reviewing games with secret-heavy players.

Unhandled Methods in Power.log (Possibly related to Spectator mode)

I get these warnings when parsing a large Power.log file. The application also crashes before creating an XML file.

Warning: Unhandled method 'PowerSpellController [taskListId=43].InitPowerSpell'
Warning: Unhandled method 'GameState.OnEntityChoices'
Warning: Unhandled method 'GameState.OnEntityChoices'
Warning: Unhandled choices: 'id=1 Player=GameEntity TaskList= ChoiceType=MULLIGAN CountMin=0 CountMax=3'
Traceback (most recent call last):
  File "/home/redacted/projects/hscs/HSReplay/python/main.py", line 671, in <module>
    main()
  File "/home/redacted/projects/hscs/HSReplay/python/main.py", line 665, in main
    parser.read(f)
  File "/home/redacted/projects/hscs/HSReplay/python/main.py", line 331, in read
    self.add_data(*sre.groups())
  File "/home/redacted/projects/hscs/HSReplay/python/main.py", line 348, in add_data
    self.handle_choices(ts, data)
  File "/home/redacted/projects/hscs/HSReplay/python/main.py", line 446, in handle_choices
    self.current_choice_node.source = entity
AttributeError: 'PowerLogParser' object has no attribute 'current_choice_node'

Full command and output: https://gist.github.com/SpectralCoding/235340f9e0fc2b60a628
Full power.log file (38MB): https://files.spectralcoding.com/files/projects/hscs/hsreplay/issue20151011/fe9a9b1fb14dafa308bf0ab94befcd18.log

TypeError: argument of type 'NoneType' is not iterable

python3.4 /root/projects/HSReplay/python/main.py /root/projects/md5/f0a75baa2d7c28bee146cb806fe8b28d.log > /root/projects/xml/f0a75baa2d7c28bee146cb806fe8b28d.xml
Warning: Unimplemented options: 'target 0 entity='
Warning: Unimplemented options: 'target 0 entity='
Warning: Unhandled method 'PowerSpellController [taskListId=22].InitPowerSpell'
Warning: Unhandled method 'PowerSpellController [taskListId=38].InitPowerSpell'
Traceback (most recent call last):
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 1081, in _escape_attrib
    if "&" in text:
TypeError: argument of type 'NoneType' is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/projects/HSReplay/python/main.py", line 671, in <module>
    main()
  File "/root/projects/HSReplay/python/main.py", line 667, in main
    print(parser.toxml())
  File "/root/projects/HSReplay/python/main.py", line 657, in toxml
    return pretty_xml(root)
  File "/root/projects/HSReplay/python/main.py", line 50, in pretty_xml
    ret = ElementTree.tostring(xml)
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 1126, in tostring
    short_empty_elements=short_empty_elements)
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 778, in write
    short_empty_elements=short_empty_elements)
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 943, in _serialize_xml
    short_empty_elements=short_empty_elements)
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 935, in _serialize_xml
    v = _escape_attrib(v)
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 1093, in _escape_attrib
    _raise_serialization_error(text)
  File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 1059, in _raise_serialization_error
    "cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize None (type NoneType)

Produces a blank f0a75baa2d7c28bee146cb806fe8b28d.xml.

Input Log: https://files.spectralcoding.com/files/projects/hscs/hsreplay/issue20151017/f0a75baa2d7c28bee146cb806fe8b28d.log
Log file from Oct 12th, 2015, probably HS 3.1.0.10357

Handles changes in chosen entities

See DebugPrintEntitiesChosen

Parse extra game attributes before serializing

We should parse various attributes when serializing a GameNode/PlayerNode to xml to allow us to set an initial deck and various extra attributes (rank, legendRank etc)

Generated Game entity has attribute "timestamp" instead of "ts"

Looking at the c# code, it looks like the Game entity generate an attribute named timestamp in the XML, like

public class Game
{
    [XmlAttribute("timestamp")]
    public string TimeStamp { get; set; }

    [XmlAttribute("type")]
    public int Type { get; set; }

    [XmlElement("Action", typeof(Action))]
    [XmlElement("Choices", typeof(Choices))]
    [XmlElement("FullEntity", typeof(FullEntity))]
    [XmlElement("GameEntity", typeof(GameEntity))]
    [XmlElement("ShowEntity", typeof(ShowEntity))]
    [XmlElement("HideEntity", typeof(HideEntity))]
    [XmlElement("Options", typeof(Options))]
    [XmlElement("Player", typeof(PlayerEntity))]
    [XmlElement("SendChoices", typeof(SendChoices))]
    [XmlElement("SendOption", typeof(SendOption))]
    [XmlElement("TagChange", typeof(TagChange))]
    [XmlElement("MetaData", typeof(MetaData))]
    [XmlElement("ChosenEntities", typeof(ChosenEntities))]
    public List<GameData> Data { get; set; }
}

According to the spec, the attribute should be name ts

Edit: GameData entities are fine, only Game entity is affected

Look into carrying timezones from the default datetime

Current node not updated when processing HideEntity regex (c#)?

Hey,

In DataHandler.cs, there is:

match = Regexes.ActionHideEntityRegex.Match(data);
if(match.Success)
{
    var rawEntity = match.Groups[1].Value;
    var tagName = match.Groups[2].Value;
    var value = match.Groups[3].Value;
    var entity = Helper.ParseEntity(rawEntity, state);
    var zone = Helper.ParseTag(tagName, value);
    var hideEntity = new HideEntity {Entity = entity, Zone = zone.Value, TimeStamp = timestamp};
    if(state.Node.Type == typeof(Game))
        ((Game)state.Node.Object).Data.Add(hideEntity);
    else if(state.Node.Type == typeof(Action))
        ((Action)state.Node.Object).Data.Add(hideEntity);
    else
        throw new Exception("Invalid node: " + state.Node.Type);
    return;
}

Contrary to all the other processes, there is no call to state.UpdateCurrentNode(typeof(Game), typeof(Action)); before checking the type of the current node. Is this intentional?

HDT integration

• Add command line argument to use PowerTaskList
• Add frozen script download to releases (cx_freeze)

Also: Do we want to keep the csharp implementation on this repo (will most likely not be maintained)?

c# testdata - Power_2.log.txt vs Power_2.log.xml

In Power_2.log.txt, starting from line 558, you can see (in order):

• ACTION_START
• Choices
• TAG_CHANGE
• Choices
• TAG_CHANGE
• ACTION_START
• ACTION_END
• etc.

The XML present in the test data however doesn't nest these elements, and produces:

<Action entity="1" index="-1" type="5">
    <TagChange entity="3" tag="305" value="1"/>
    <Choices entity="1" max="3" min="0" playerID="1" source="1" type="MULLIGAN">
        <Choice entity="8" index="0"/>
        <Choice entity="16" index="1"/>
        <Choice entity="33" index="2"/>
    </Choices>
</Action>
<TagChange entity="2" tag="305" value="1"/>
<Choices entity="2" max="5" min="0" playerID="2" source="1" type="MULLIGAN">
    <Choice entity="54" index="0"/>
    <Choice entity="49" index="1"/>
    <Choice entity="55" index="2"/>
    <Choice entity="43" index="3"/>
    <Choice entity="68" index="4"/>
</Choices>

while I would expect something like

<Action entity="1" index="-1" target="0" ts="01:41:27.9112918" type="5">
    <TagChange entity="2" tag="305" value="1"/>
    <Choices entity="1" max="3" min="0" playerID="1" source="1" taskList="0" ts="01:41:27.9312929" type="1">
        <Choice entity="8" index="0"/>
        <Choice entity="16" index="1"/>
        <Choice entity="33" index="2"/>
    </Choices>
    <TagChange entity="3" tag="305" value="1"/>
    <Choices entity="2" max="5" min="0" playerID="2" source="1" taskList="0" ts="01:41:28.0212980" type="1">
        <Choice entity="54" index="0"/>
        <Choice entity="49" index="1"/>
        <Choice entity="55" index="2"/>
        <Choice entity="43" index="3"/>
        <Choice entity="68" index="4"/>
    </Choices>
    <TagChange entity="2" tag="305" value="2"/>
    <Action entity="2" index="-1" target="0" ts="01:41:47.1423917" type="5">
        <TagChange entity="25" tag="49" value="3"/>
        <TagChange entity="25" tag="263" value="1"/>
        <TagChange entity="8" tag="49" value="2"/>
        <TagChange entity="8" tag="263" value="0"/>
        <TagChange entity="28" tag="49" value="3"/>
        <TagChange entity="28" tag="263" value="2"/>
        <TagChange entity="16" tag="49" value="2"/>
        <TagChange entity="16" tag="263" value="0"/>
        <TagChange entity="2" tag="305" value="3"/>
    </Action>
        ...

Is this the intended behaviour?

Thanks!

Seb

[Feature Request] Attack direction arrows

Twould make it even more intuitive when watching a game if arrows appeared between a minion or character making an attack and the intended target of their attack.

Optional but useful: A arrow that displays redirected attacks caused by secrets, certain minions on the board, etc. (Ie; minion A attacks minion B, but then the paladin secret causes the minion to redirect to the summoned 2/1.)

Missing return statements in DataHandler.cs?

Hey,

Looking through the code in DataHandler.cs, and two of the match.Success cases don't have a return statement. Is this intentional?

No idea if this could cause a bug or if this is intentional, just thought I would raise the point in case it might help.

Cheers,
Seb

AttributeError: 'str' object has no attribute 'isoformat'

Traceback (most recent call last):
  File "python/convert.py", line 13, in <module>
    main()
  File "python/convert.py", line 9, in main
    print(log_to_xml(f))
  File "/home/benedict/Dokumente/HSReplay/python/hsreplay/__init__.py", line 314, in log_to_xml
    root.append(game_element.xml())
  File "/home/benedict/Dokumente/HSReplay/python/hsreplay/__init__.py", line 50, in xml
    element.append(node.xml())
  File "/home/benedict/Dokumente/HSReplay/python/hsreplay/__init__.py", line 50, in xml
    element.append(node.xml())
  File "/home/benedict/Dokumente/HSReplay/python/hsreplay/__init__.py", line 59, in xml
    element.attrib["ts"] = self.ts.isoformat()
AttributeError: 'str' object has no attribute 'isoformat'

Power.log.

Unhandled method 'GameState.DebugPrintChoice'

https://etc.spectralcoding.com/output_log.txt

What does the "m" symbol mean

In opponent's hand there is a "m" symbol when they replace cards. "M" = Replace? What does it stand for.

the Block element is not declared

the title says it all, in the dtd file, no Block element is defined

Fix FULL_ENTITY regexes for create/update

See GameState.cs -> DebugPrintPower()

Player IDs treated as entity IDs

In 10357, in DebugPrintEntitiesChosen and SendChoices, the Player argument is bugged: A Player ID is being interpreted as an entity ID and the corresponding entity is output, meaning GameEntity is Player 1 and such.

For the time being we retain PlayerID but ideally Blizzard will phase PlayerID out entirely and we can use entity IDs for this. Or they fix this inconsistency and we keep using player IDs like we do now...