Equation of state-action value function in seminar_vi week 02 about practical_rl

[dniku]

For posterity: the equations you posted were invalid because they did not specify where you got the action $a$ from.

1. $V_i(s) = \sum_s' P(s' |s,a) \dot [r_{i+1} + \gamma V_i(s')$: here the $a$ in $P(s' |s,a)$ is undefined, and it has to come from somewhere.
2. $Q_i(s, a) = \sum_{s', a'} P(s' | s,a) \cdot [ r(s,a,s') + \gamma Q_{i}(s', a')]$: similar problem here, but $a'$ is unspecified.

These are actually substantial issues: once you fix them by saying $a$ is a parameter or something you sum over, you come to the mixed usage of $V$ and $Q$ we advertise in seminar_vi.ipynb.

[AI-Ahmed]

hello @dniku , This was an old question when It was the first time me learning RL. Don't worry about it, as you can see I have already closed it!