Header logo is

Recurrent Policy Gradients

2010

Article

ei


Reinforcement learning for partially observable Markov decision problems (POMDPs) is a challenge as it requires policies with an internal state. Traditional approaches suffer significantly from this shortcoming and usually make strong assumptions on the problem domain such as perfect system models, state-estimators and a Markovian hidden system. Recurrent neural networks (RNNs) offer a natural framework for dealing with policy learning using hidden state and require only few limiting assumptions. As they can be trained well using gradient descent, they are suited for policy gradient approaches. In this paper, we present a policy gradient method, the Recurrent Policy Gradient which constitutes a model-free reinforcement learning method. It is aimed at training limited-memory stochastic policies on problems which require long-term memories of past observations. The approach involves approximating a policy gradient for a recurrent neural network by backpropagating return-weighted characteristic eligibilities through time. Using a ‘‘Long Short-Term Memory’’ RNN architecture, we are able to outperform previous RL methods on three important benchmark tasks. Furthermore, we show that using history-dependent baselines helps reducing estimation variance significantly, thus enabling our approach to tackle more challenging, highly stochastic environments.

Author(s): Wierstra, D. and Förster, A. and Peters, J. and Schmidhuber, J.
Journal: Logic Journal of the IGPL
Volume: 18
Number (issue): 5
Pages: 620-634
Year: 2010
Month: October
Day: 0

Department(s): Empirical Inference
Bibtex Type: Article (article)

Digital: 0
DOI: 10.1093/jigpal/jzp049
Language: en
Organization: Max-Planck-Gesellschaft
School: Biologische Kybernetik

Links: PDF
Web

BibTex

@article{5879,
  title = {Recurrent Policy Gradients},
  author = {Wierstra, D. and F{\"o}rster, A. and Peters, J. and Schmidhuber, J.},
  journal = {Logic Journal of the IGPL},
  volume = {18},
  number = {5},
  pages = {620-634},
  organization = {Max-Planck-Gesellschaft},
  school = {Biologische Kybernetik},
  month = oct,
  year = {2010},
  doi = {10.1093/jigpal/jzp049},
  month_numeric = {10}
}