juliapomdp / pomcp.jl

Julia Implementation of the POMCP algorithm for solving POMDPs

POMCP (Deprecated)

This package is now unmaintained and DEPRECATED in favor of BasicPOMCP.jl and POMCPOW.jl

The Partially Observable Monte Carlo Planning (POMCP) online solver for POMDPs.jl.

Described in

Silver, D., & Veness, J. (2010). Monte-Carlo Planning in Large POMDPs. In Advances in neural information processing systems (pp. 2164–2172). Retrieved from http://discovery.ucl.ac.uk/1347369/

Installation

Using POMDPs.jl (should automatically take care of dependencies)

Pkg.add("POMDPs")
import POMDPs
POMDPs.add("POMCP")

OR (some optional dependencies may be missing)

Pkg.clone("https://github.com/juliapomdp/POMCP.jl.git")

Documentation

Documentation can be found at http://juliapomdp.github.io/POMCP.jl/latest/ (generated by Documenter.jl)

A minimal example showing the functions required to use the package is here: https://github.com/JuliaPOMDP/POMCP.jl/blob/master/notebooks/Minimal_Example.ipynb

pomcp.jl's Issues

Particle filter tester

It might be nice to have some code to test particle filters without running all of pomcp

Prior Knowledge

Need to provide a mechanism to incorporate additional prior knowledge like in MCTS once we figure out the best way to do that.

Overhaul

Recent experience shows that this package can be significantly improved. There are two really important things that should be done:

1. This package can be greatly simplified by by making it only an implementation of the vanilla POMCP algorithm. DPW and the capability to use arbitrary belief updaters should be in separate packages. They are sufficiently different.
2. The speed could probably be improved by an order of magnitude for problems with fast state updates by getting rid of the small dictionaries at every node and investigating type stability

No need to update beliefs on tree in some cases

If a custom belief updater is used, there is usually no need to update the beliefs in the tree (this is only the case if the belief is used in the rollout).

I should add an option to turn this off.

Add standard belief node interface

there should be a small interface for getting info from belief nodes that can also be used by POMCPOW, etc.

running tree visualization breaks profileview

If you run tree visualization, and then attempt to run pofileview, the javascript breaks and the labels for profileview don't show.

A quick fix is to refresh the kernel and refresh the browser window, and rerun only the profileview code.

Switch to using Particle Collection from ParticleFilters.jl

once ParticleFilters.jl becomes more accessible, use its particle collection types instead of the ones already in POMCP

Make it so that default_action can be just an action

right now it has to be a function or an object that a function is called on - it should be possible to make it just an action

Infrastructure for dealing with particle depletion

There should be an infrastructure for dealing with particle depletion

trees visualized in different cells get rendered in the first cell's output

Add requirements

The requirements for this solver should be specified with a @POMDP_require block, see instructions

Fix visualization

Fix the visualization using stuff from MCTS

Improved Documentation

• Particle filtering and reinvigoration
• Belief Updating

Precompilation

For an online solver like POMCP that is stopped based on a timeout, I need to make sure that compilation time is not counted against the timeout for the first step. Does anyone know if there is a standard way to make sure that certain functions are precompiled in julia?

Error with -Inf rewards

As @juliohm found, If rewards of -Inf are used, solver.jl line 137 will end up with NaN and there will eventually be hard-to-diagnose errors.

Behavior for this needs to be defined and tested.

Warnings about particle depletion

There should be better warnings about particle depletion so that a user knows what is happening.

sample() should be moved to POMDPToolbox