A Budgeted Markov Decision Process (BMDP) is an extension of a Markov Decision Process to critical applications requiring safety constraints. It relies on a notion of risk implemented in the shape of a cost signal constrained to lie below an — adjustable — threshold. So far, BMDPs could only be solved in the case of finite state spaces with known dynamics. This work extends the state-of-the-art to continuous spaces environments and unknown dynamics. We show that the solution to a BMDP is a fixed point of a novel Budgeted Bellman Optimality operator. This observation allows us to introduce natural extensions of Deep Reinforcement Learning algorithms to address large-scale BMDPs. We validate our approach on two simulated applications: spoken dialogue and autonomous driving.
Carrara N., Leurent, E., Laroche R., Urvoy T., Maillard, O-A., and Pietquin O., 2019.
Budgeted Reinforcement Learning in Continuous State Space. In Advances in Neural Information Processing Systems.
@incollection{Carrara2019,
title = {Budgeted Reinforcement Learning in Continuous State Space},
author = {Carrara, Nicolas and Leurent, Edouard and Laroche, Romain
and Urvoy, Tanguy and Maillard, Odalric-Ambrym and Pietquin, Olivier},
booktitle = {Advances in Neural Information Processing Systems 32},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and
F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {9299--9309},
year = {2019},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/
9128-budgeted-reinforcement-learning-in-continuous-state-space.pdf}
}
We show samples of driving styles emerging from constraining the time spent on the opposite lane with different constraint budget values. The budget enables to control in real-time the tradeoff between efficiency and safety.
Budget | Behaviours |
---|---|
In the following table, we display two dialogues done with the same BFTQ policy. The policy is given two budgets to respect in expectation, and . For budget 0, one can see that the system never uses the ask_num_pad
action. Instead, it uses ask_oral
, an action subject to recognition errors. The system keeps asking for the same slot 2, because it has the lowest speech recognition score. It eventually summarizes the form to the user, but then reaches the maximum dialogue length and thus faces a dialogue failure. For budget 0.5, the system first asks in a safe way, with ask_oral
. It may want to ask_num_pad
if one of the speech recognition score is low. Then, the system proceeds to a confirmation of the slot values. If it is incorrect, the system continues the dialogue using unsafe the ask_num_pad
action to be certain of the slot values.
We compare two approaches for contructing a batch of samples. The animations display the trajectories collected in each intermediate subbatch. The first row corresponds to a classical risk neutral epsilon-greedy exploration policy while the second row showcases a risk-sensitive exploration strategy introduced in the paper. Each animation corresponds to a different seed.
We display the evolution in the budgeted policy behavior with respect to the budget. The policies have been learnt with a risk-sensitive exploration. When the budget is low, the agent takes the safest path on the left. When the budget increases, it gradually switches to the other lane, earning higher rewards but also costs. This gradual process could not be achieved with a deterministic policy as it would chose either one path or the other. Each animation corresponds to a different seed.
First install the following conventional libraries for python3: pycairo, numpy, scipy and pytorch. Install the highway-env environment. Then, use one of these methods:
Install requirements:
- The agents: eleurent@rl-agents
Run the training:
cd <path-to-rl-agents>/scripts/
python experiments.py evaluate scripts/configs/TwoWayEnv/env.json \
scripts/configs/TwoWayEnv/agents/BFTQAgent/baseline.json \
--train --episodes=1000
- Clone ncarrara@budgeted-rl
- Change python path to the path of this repository
export PYTHONPATH="budgeted-rl/ncarrara"
- Navigate to budgeted-rl folder
cd budgeted-rl/ncarrara/budgeted-rl
- Run main script using any config file. Choose the range of seeds you want to test on:
python main/egreedy/main-egreedy.py config/slot-filling.json 0 6
python main/egreedy/main-egreedy.py config/corridors.json 0 4
python main/egreedy/main-egreedy.py config/highway-easy.json 0 10
Parameter | Description | Value |
---|---|---|
- | Size of the environment | 7 x 6 |
- | Standard deviation of the Gaussian noise applied to actions | (0.25, 0.25) |
H | Episode duration | 9 |
Parameter | Description | Value |
---|---|---|
Number of vehicles | 2 - 6 | |
Standard deviation of vehicles initial positions | 100 m | |
Standard deviation of vehicles initial velocities | 3 m/s | |
H | Episode duration | 15 s |
This work has been supported by CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020, the French Ministry of Higher Education and Research, INRIA, and the French Agence Nationale de la Recherche (ANR). We thank Guillaume Gautier, Fabrice Clerot, and Xuedong Shang for the helpful discussions and valuable insights.