Water Elemental + Betrayal

Emperor Cobra + Betrayal

http://hearthstone.gamepedia.com/Betrayal

So maybe the 'freezing attack' and 'poisonous attack' should be implemented via event triggers

Q learning + deep neural network

https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0

When a card is drawn
remove all its enchantments

restore the aura to default value in card database
(maybe it got silenced before going to graveyard)

In this sense,
maybe we can just store card_id when cards are in deck

or, in other words,
we only need to store the whole Cards::CardData when cards are in HAND / PLAY zone

严重性 代码 说明 项目 文件 行 禁止显示状态
错误 CS0103 当前上下文中不存在名称“GameEngineCppWrapper” GameEngineUI D:\Code\hearthstone-ai-master\hearthstone-ai-master\vs_projects\GameEngineUI\Form1.cs 32 活动的

Runtime analysis of a execution hot path is crucial. It sometimes make a JIT out-performed a static program like C++.

Profile-guided optimization maybe a rescue for this.
https://textslashplain.com/2016/01/10/getting-started-with-profile-guided-optimization/

g++ optimization flags:
-fprofile-generate
-fprofile-use
-fprofile-dir

also,
As with Stealth, Taunt minions that are Immune have their Taunt ability temporarily suppressed, and can thus be bypassed.

Currently the minions are stored in a c++ list container,

when insert, the other iterators are still valid (as opposite to the std::vector)

In current code, we have implemented our own consistency-checking mechanism,

and when a minion is inserted, the consistency-checking framework will invalidate all other iterators

this is not necessary as long as the std::list is used.

Note: when a minion is removed, the other iterators are still valid in std::list,

BUT!! the iterators pointing to the removed minion should be invalidated.

However, since we have no tracking info for such iterators, all iterators are marked invalidated.

This is a desired behavior since the game engine should not introduce such behavior.

Cards manipulator: set zone, set zone position, etc.
Characters manipulator: all above, attack/defend
Minion manipulator: all above, enchant, aura,

They should have hierarchy like this.

A /wd4127 compiler option is added to suppress this warning

Use hash table to identify which tree nodes can be shared

TODO: do we really want to share tree nodes?

• The play history is important in control decks (or even mid-range decks)
• AMAF or RAVE already relief the slow-start issue
• Bright side is: this can shrink the memory print of the game tree

Death rattle is triggered by play order
Maybe we can use event trigger to check death
Then no need for play order anymore

https://msdn.microsoft.com/en-us/library/ms164759.aspx

Don't use vector, use hash table instead

Client cards should consider to use categorized event triggers instead

Check all battlecry
--> they should all apply Targetable() filter

Check all spell target
--> they should all apply SpellTargetable() filter

What situations lead to an invalid state?

1. not enough resource
   a. cost health, but with no enough health
   b. not cost health, but with no crystal
   c. [NOTE] cost might be reduced/added due to some effects
2. client card cannot be played
3. client card needs target, but failed
4. no space for minion
5. secret already exists
6. GetDefender() callback returns invalid target
7. attacker is not attackable
8. defender VANISHED before attack
9. hero power is not usable

specify event type in template parameter
check enchantment existence in framework

As title

In current implementation,
before applying each action (i.e., play-card, attack, hero-power, or end-turn)
the game state is saved on stack
This game state is restored if the action is actually an invalid action and make the application failed.

We can make the game state to be copy-on-write,
and support the fast response for those methods which are probably invoked when applying an invalid action

Since there are many discussions on the efficiency of copy-on-write data structures,
it's better to delay after we've done some profiling.

Two lines affected in common.h:

1. https://github.com/peter1591/hearthstone-ai/blob/master/pilot_run/include/game-engine/cards/common.h#L118 which redeclares https://github.com/peter1591/hearthstone-ai/blob/master/pilot_run/include/game-engine/cards/common.h#L112
   and

2. https://github.com/peter1591/hearthstone-ai/blob/master/pilot_run/include/game-engine/cards/common.h#L161 which redeclares https://github.com/peter1591/hearthstone-ai/blob/master/pilot_run/include/game-engine/cards/common.h#L155

aura enchantment should be applied in-order

You play an Amani Berserker and Enrage it, giving it 5 Attack. You then play Humility on it, giving it 1 Attack. You then heal and Enrage it a second time - the new Enrage is at the end of order of play, going after the Humility effect and it now has 4 Attack.

use std::variant
each enchantment entry is either an 'normal enchantment', or an 'aura enchantment'

state::State state;

Implement state.Manipulate() to replace: state::Manipulators::StateManipulator(state)

Refines on state::State:

• PrepareValidActions() --> returns ValidActionHelper()
  • Do some process
  • Selection stage can save the board after this is called
• ApplyAction(ValidActionHelper const* = nullptr)

Requirement

• No overhead. Keep the simulation quick.

Notes

• Even in selection stage, the board after 'PrepareValidActions()' is not saved in memory
  • Only save the BoardView
• Since there's hidden information, a determination phase runs before each episode. So even the board is saved, we will not run that exactly board at following episodes.
• But, in fact, the hidden information should get nothing to do with the prepare action.
• So, maybe the valid action helper should not be implemented within the state::State. It should be related to BoardView.

First I added log.config to C:\Users[username]\AppData\Local\Blizzard\Hearthstone and added

[Achievements]
LogLevel=1
FilePrinting=true
ConsolePrinting=true
ScreenPrinting=false

[Power]
LogLevel=1
FilePrinting=true
ConsolePrinting=true
ScreenPrinting=false

If this is needed please add to documentation.

The I follow your directions: Opened C# project under path\hearthstone-ai-master\vs_projects\GameEngineUI

Then I run it. Everything compiles and runs. I see a window with 1 button. I press the button a file picker appears. What do I do?

I see from the code it wants to know where the cpp dll is. I click on it, but then it just pops up with a number (427549). What does this mean? How can I get what the best move is?

https://jguegant.github.io/blogs/tech/sfinae-introduction.html

already replaced by return value
return false: remove it from container

Similar to tag framework

Pros:

1. To get attributes, we only need to operate on entities
2. state::Board acts as a index of the card references. E.g., to quickly enumerate over all minions.

Cons:

1. Many fields on entity

Current decision:

• Write all properties on entity (i.e., cards::RawCard)

Need to store (un)enchanted states in entity?

Or, when we need to update/re-calculate enchanted states, we...

1. Load the raw card information from database
2. If the minion is silenced, add a 'SILENCED' enchantment
   --> which remove divine-shield / charge / spell damage / etc.
3. Apply all enchantments

In some states, only a subset of actions are valid

1. Minions cannot attack (just summoned, or attacked)
2. No hand card can be played (not enough resource, no required target)
3. etc.

In current implementation

1. All actions are numbered from 1
2. Invalid actions are pre-filtered out as much as possible
3. The left (hopefully) valid actions are re-numbered from 1

Since later, a policy network might be used to

• pick up the most promising action
• The re-numbering process might not be a good idea
  • E.g., State 1 has a promising action 'PLAY 3RD CARD'
  • State 2, which is similar to State 1, also has a promising action 'PLAY 3RD CARD'
  • But, since the re-number, the PLAY CARD action might be with a different number
  • Might make the underlying policy network (e.g., deep neural network) a hard time to learn

Some thoughts

1. Do not re-number valid actions. Just filter them out if later the action is picked up.
2. state::State support find valid actions more deeply.

Gladiator's Longbow

Gorehowl

CardManipulator's Damage() calls
BoardManipulator's CalculateFinalDamageAmount()
which calculate spell damage for spell/secret cards

So, client cards should not add spell damage by itself?

Hello @peter1591, I wanted to ask you about the project structure. So I've been trying to resolve https://github.com/zappybiby/hearthstone-ai/issues/1 but I am not seeing any obvious issues with compiling or anything like that.

Now I'm wondering if any of the projects you have in the main repo (HearthstoneAI, MCTS, and vs_projects) are linked together in some way. I've never dealt with a repo with multiple projects (and I am new to coding as well) so the structure here confuses me. Maybe we should rename the solutions? Sorry for being a newbie! I hope to add more to this project soon after I get this resolved.

manipulator postfix can be removed,

and the underlying POD structures can be added with the 'Data' postfix

• client cards use manipulators, not directly using state::State or FlowControl::FlowContext

  • if an enchantment is bound with an event, the event should be triggered correctly after a minion became a copy of it.

• enchantments should have a method:

  • AfterAdded()
  • event can only be registered there
    • bring a event manager pointer as a context field
  • called after a minion is copied / transformed-as

Restart mechanism for invalid actions

Problem

Some of the choices might lead to an invalid action.

Current Design

Remember a tree in both selection and simulation stage.
This tree is rooted from the last main action,
and will be traversed again from this root once an invalid
game state is detected.

Issues in current design

• When an invalid state is detected, we restarted from the last main action
  • The selection/simulation policy is re-calculated again, and then applied
    • Issue: [FIXED] we should re-apply the first few choices, except for the last sub-action?
• Issue: Cannot switch to simulation stage during sub-actions
  • Discussion: is this really beneficial?
• The tree structure for the selection stage and the simulation stage are totally different
  • Issue: The restart algorithm are totally different. Make some unification?
  • Can we unify the restart steps, and write in TreeBuilder?
  • Define some interface for the selection/simulation stages
    • GetBoardForMainAction() <-- maybe this should in TreeBuilder
    • GetPendingSubActions()

Analysis

Why an invalid state?

• No playable hand card
  • Cannot be easily detected beforehand, since card might be played by costing health.
• No available attacker
  • Most case can be pre-detected by game simulation engine with ValidActionGetter.
  • Special flags: cannot-attack-to-hero
• No available defender
  • Most case can be pre-detected by game simulation engine with ValidActionGetter.
  • hero is immune
• No available target
  • A card requires a target, but no target is available

Deal with invalid state

When an invalid state is reached, we cannot finish the current MCTS episode, since that particular move is actually invalid.

Probability of an invalid state

• No playable hand card
  • No pre-checking for playable hand card.
  • So, all hand cards are considered as playable
  • If a player has no crystal left, then all hand cards are not playable
  • Conclusion: high chance, nearly 100% if no crystal left (except cost health instead of crystals).

Several approaches can be done in this situation.

1. Discard current MCTS episode, and restart again.

• The action can be marked as invalid in selection stage
• But, in simulation stage, there's no tree to remark this.

2. Restart from the last main action
3. Restart from the last sub action

• It's possible that this sub action has no any valid action. Need to restart from the previous sub action.

Selection stage

A tree is established in selection stage, so we can mark a child as invalid easily.

Simulation stage

As discuss in the issue #45, the simulation engine should be able to generate valid actions. At least, with a high probability to generate a valid action.

Discussions

Need tree for simulation?

If we have a tree for simulation, we can remember which action is invalid, and restart quickly.

Since there's a high chance to have an invalid action when picking up a playable card (happens when no crystal left), we should make it fast.

But, for performance, we should lower down the rate of an invalid state as much as possible.

What happens if there's no tree for simulation, and an invalid state is reached during simulation? We can have a linear (not a tree) data structure to record the black-list choices along the path.

Record black-list for choices

A linear data structure to record the black-list choices along the chosen path.

Random node

Assumption: If a state is valid before random. Then, ALL random outcome should yields a valid state.

That is,

• If ANY random outcome yields an invalid state. Then, the state before the random is invalid.

Interface for stage handler

`
// Make a choice, and modify the progress accordingly
//@return the choice
int Select(Progress & progress);

// Report if a choice leading to an invalid state
void ReportInvalid(Progress & progress)
`

Data structure to record black list choices

• A linear structure to record all nodes traversed
• Re-apply the sub-actions from a saved board
• Each node consists
  • ActionType --> random / manual
  • Choices --> consistency check only
  • variant<selection::Progress, simulation::Progress>
• The 'Progress' class should be copyable
  • It's guaranteed that, only the last progress will be used for restore

Task list

Selection and simulation stage handler

• Refactor out progress class
• Follow new interface
• DONE

Implement data structure to record black list

• DONE

Unify logics in TreeBuilder

• DONE

Analysis

• Should we switch to simulation within sub-actions?
  • Create another issue

Code refine

• Simulation stage handler
  • ChooseAction() and ApplyAction() are too similar

Lower down simulation invalid rate

• Currently, the action applied in a simulation stage is with success rate about 21%
• This mainly due to that we cannot check if we can play a card or not before applying the action
• Should modify game engine to support this kind of queries.

unify logic in tree builder

• unify for both selection and simulation
  • extract 'Progress' from selection class?
  • the whole selection/simulation stage handler can be seen as the Progress class
  • but, the simulation stage handler needs a ChoiceBlacklist on the stack

the event listened by the enchantment should be registered after a new minion became a copy of the minion.

add a new test:

1. minion: gain +1/+1 after turn end
2. faceless manipulator: become the minion
3. turn end
4. check both gained +1/+1

Do we need to switch to simulation mode within a main action?

For example,
A main action is to decide from (PLAY-CARD, HERO-POWER, END-TURN)

Assume we were in selection mode at this main action node, the UCB policy is used to determined from these choices.
Assume we choose the PLAY-CARD action
Assume this is the FIRST TIME we make this choice, so a new node is added to the game tree.

Now, do we want to switch to simulation mode?

In current design, we only switch to simulation mode after this MAIN ACTION + SUB ACTIONS are done.
That is, we switch to simulation after

1. added a node for PLAY-CARD
2. added a node for CHOOSE-HAND-CARD
3. added a node for CHOOSE-TARGET (if any)
4. more nodes for callback (if any)
   Now, after this main action is done, we switched to simulation mode.

https://www.youtube.com/watch?v=YlaP_kF823k

Client card needs:

1. FlowControl::Manipulators
2. state::EventManager

but should not touch:

1. state::Cards (zone changer, etc.)
   since manipulators might need to trigger events when zone changed

Freezed twice --> thaw at once

Taunted twice --> broken at once

Divine shield twice --> broken at once

stealth twice --> shown at once

Also, the minion stat can be reduced to below zero since some stats (e.g., taunt) can be removed during game flow (e.g., attack)

In StaticEventTriggerrer.h line 17, class 'Invoker' doesn't resolve. Did you mean to use class 'Invoker2'?

Refer to Gladiator's Longbow
Apply an enchantment (make it immune), and remove it after attack

Dead entries should call aura update at least one more time,
to remove the aura enchantments.

Maybe a aura manager?

Some cost modifiers are attached to the entire game (i.e., board), or attached to a particular player

Some of them are permanent effects, some of them are one-turn effects, and some of them are aura effects.

How to deal with them?

Pros:

1. All targetable objects are now of type 'Card'
2. Unify logic for attacker / defender

Cons:

1. Weapon mechanism should be re-design
2. One more card type? Say, kCardTypeHero?

Notes:

1. Hero can be replaced by a card
2. When hero is placed/replaced, weapon status should be updated

D3 JS
https://skillsmatter.com/skillscasts/7460-visualising-game-trees-with-d3-js
https://bl.ocks.org/mbostock/4062045