Reinforcement Learning and Stochastic Games

This course addresses the following question: How can an agent learn to act towards achieving a certain goal in a dynamic, uncertain, and partially observable environment, possibly in the presence of other acting agents? This question and its specific variants (single agent, knowledge of the environment, stochasticity, learning from prior knowledge, common objectives, and coordination among agents) are of central importance in a broad range of modern applications, including computer vision, robotics, autonomous driving, natural language processing, games (board games, card games, and video games), and wireless communications.

The course focuses on the theory, methods, and algorithms of reinforcement learning dealing with the above question. Considering that the interaction between agents and their environment is continuous, the former observe the state of the environment through relevant measurements, and indirectly obtain information about the actions of other agents in the overall system. Some information is shared among all agents, while other information remains private. Based on the available observations, each agent selects and executes an action. The state of the environment evolves over time, generally in an uncertain (probabilistic) manner, depending on its previous evolution and the actions of the agents. Each agent makes decisions sequentially based on learned rules; consequently, learning itself is sequential. This interaction cycle is repeated continuously, allowing agents to learn how to act so as to optimize not merely their immediate reward, but their overall long-term return.

The course 's content:

• Multi-armed bandit problems, with or without context, which simplify the decision-making process.
• Markov decision processes (MDPs) with known, possibly stochastic, system models; the perspective of dynamic programming and the Bellman equations will be studied, extending the formulation to partially observable MDPs (POMDPs).
• Algorithms for learning optimal policies (Q-learning, Monte Carlo/REINFORCE) when the environment model is unknown.
• Deep reinforcement learning algorithms that learn approximately optimal decisions using artificial neural networks.
• Extensions of deep reinforcement learning algorithms to world models and multi-agent environments involving cooperation or competition.

Programming environments for deep reinforcement learning algorithms in Python/PyTorch using the Gymnasium and Stable-Baselines libraries.