Papers

You can choose to focus your project on an RL research paper. Your goal is to thoroughly understand the paper, replicate its experiments, and optionally extend/improve it with your own new idea. 

(You are allowed to come up with your own paper of interest, but do discuss it with the teachers. Make sure the paper comes with 1) a public codebase (from the official authors) and 2) a smaller toy experiment that is for sure computationally feasible to reproduce.)

RL for (sequential) generative modelling

Explanation: 

• Generative modelling is a major topic in AI. We can also approach generation as a sequential process, to be trained with RL-techniques: 

Possible papers: 

• Generative Flow Networks as Entropy-Regularized RL (Paper, Code)

  • This paper studies the (tight) relation between flow-based sequential generation and (entropy-regularized) reinforcement learning. 

  • You could try to understand these methods and replicate the Hypergrid experiments (Sec 4.1) and possibly the Small Molecule Generation (Sec 4.2)

Safe reinforcement learning

Explanation: 

• Safety is crucial in real-world deployment. Safe RL tries to maximize the expected return while respecting seperate cost constraints. 

Possible papers: 

• CRAX: Fast Safe Reinforcement Learning Benchmarking (Paper, Code)

  • This recent paper provides a fast benchmarking suite for a range of Safe RL algorithms. It does come in Jax (which has a learning curve). 

  • The benchmark currently measures safety only through cumulative cost. Maybe you could implement and compare different ways to specify cost (for example probabilistic, such as the chance the agent is in an unsafe region). 

RL for (sequential) social dilemmas

Explanation: 

• Sequential social dilemmas (based on game theory) are a central aspect of societal interaction, negotiation and the evolution of cooperation. We can use RL to learn good strategies in such settings.
  (Note: Social dilemmas are a type of multi-agent RL. However, whereas standard multi-agent RL is typically fully cooperative or fully competitive, social dilemmas are 'mixed motive': there are both incentives to cooperate and to defect.) 

Possible papers: 

• SocialJax (Paper, Code) 

• This recent paper provides a fast benchmarking suite for RL in social dilemmas. It does come in Jax (which has a learning curve). 

• You could try to compare i) a plain RL algorithm (IPPO), ii) a multi-agent RL algorithm (MAPPO), and iii) a social dilemma specific RL algorithm, such as SVO or IPPO-RE. You could then reproduce (some of) Table 1 and Figure 2.