Boosting reinforcement learning with sparse and rare rewards using Fleming-Viot particle systems - Archive ouverte HAL Access content directly
Conference Papers Year : 2022

Boosting reinforcement learning with sparse and rare rewards using Fleming-Viot particle systems

(1, 2) , (3) , (1, 4, 5) , (6, 1)
1
2
3
4
5
6

Abstract

We consider reinforcement learning control problems under the average reward criterion in which non-zero rewards are both sparse and rare, that is, they occur in very few states and have a very small steady-state probability. Using Renewal Theory and Fleming-Viot particle systems, we propose a novel approach that exploits prior knowledge on the sparse structure of the environment to boost exploration of the non-zero rewards. We also demonstrate how to combine the methodology with a policy gradient algorithm to construct the FVRL method that is able to efficiently solve structured control problems under these scenarios. We provide theoretical guarantees of the convergence of both the steady-state probability estimator and the policy gradient learner. Finally, we illustrate the method on an M/M/1/K queue control problem where the objective is to determine the optimum blocking threshold K. Our results show that FVRL learns the optimum blocking threshold much more efficiently than vanilla Monte-Carlo reinforcement learning.
Fichier principal
Vignette du fichier
2022 - EWRL - Mastropietro, Majewski, Ayesta, Jonckheere - Boosting Reinforcement Learning with Sparse and Rare Rewards with Fleming-Viot Particle Systems.pdf (1 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03772025 , version 1 (07-09-2022)

Identifiers

  • HAL Id : hal-03772025 , version 1

Cite

Daniel Mastropietro, Szymon Majewski, Urtzi Ayesta, Matthieu Jonckheere. Boosting reinforcement learning with sparse and rare rewards using Fleming-Viot particle systems. 15th European Workshop on Reinforcement Learning (EWRL 2022), Sep 2022, Milano, Italy. ⟨hal-03772025⟩
31 View
0 Download

Share

Gmail Facebook Twitter LinkedIn More