Deep reinforcement learning for weakly coupled MDP's with continuous actions

Francisco Robledo (1, 2, 3) , Urtzi Ayesta (4, 5, 3, 6) , Konstantin Avrachenkov (7)

1 LMAP - Laboratoire de Mathématiques et de leurs Applications [Pau]
2 UPPA - Université de Pau et des Pays de l'Adour
3 UPV / EHU - Universidad del País Vasco [Espainia] / Euskal Herriko Unibertsitatea [España] = University of the Basque Country [Spain] = Université du pays basque [Espagne]
4 IRIT-RMESS - Réseaux, Mobiles, Embarqués, Sans fil, Satellites
5 Ikerbasque - Basque Foundation for Science
6 CNRS - Centre National de la Recherche Scientifique
7 NEO - Network Engineering and Operations

This paper introduces the Lagrange Policy for Continuous Actions (LPCA), a reinforcement learning algorithm specifically designed for weakly coupled MDP problems with continuous action spaces. LPCA addresses the chal- lenge of resource constraints dependent on continuous actions by introducing a Lagrange relaxation of the weakly coupled MDP problem within a neural network framework for Q-value computation. This approach effectively decouples the MDP, enabling efficient policy learning in resource-constrained environments. We present two variations of LPCA: LPCA-DE, which utilizes differential evolu- tion for global optimization, and LPCA-Greedy, a method that incrementally and greadily selects actions based on Q-value gradients. Comparative analysis against other state-of-the-art techniques across various settings highlight LPCA’s robust- ness and efficiency in managing resource allocation while maximizing rewards.

Mots clés: Computing methodologies → Sequential decision making Machine learning; Reinforcement Learning; Lagrangian relaxation; Markov Decision Problem; Weakly Coupled MDP; Continuous Actions; Lagrange Policy; Neural Networks; Differential Evolution; Resource Allocation; Policy Optimization

Domaines: Système multi-agents [cs.MA]; Intelligence artificielle [cs.AI]; Apprentissage [cs.LG]; Système multi-agents [cs.MA]

Francisco Robledo, Urtzi Ayesta, Konstantin Avrachenkov. Deep reinforcement learning for weakly coupled MDP's with continuous actions. ACM SIGMETRICS / ASMTA 2024, Jun 2024, Venise, Italy. ⟨hal-04594762v2⟩