Papers

You can choose to focus your project on an RL research paper. Your goal is to thoroughly understand the paper, replicate its experiments, and optionally extend/improve it with your own new idea. 

(You can come up with your own paper of interest, but do discuss this with the teachers. Make sure the paper comes with 1) a public codebase and 2) a smaller toy experiment that is for sure computationally feasible to reproduce.)

Model-free RL

Explanation: 

• Model-free RL is arguably the workhorse of reinforcement learning. Most research focuses on stabalizing/speeding up the learning process.

Possible papers: 

• PQN (Paper, Code, (PureJax))

  • The pure Jax version of PQN, which fully executes on the GPU, runs blazingly fast. 

  • This could therefore be an interesting base algorithm for one of the Applications .

Generative modelling

Explanation: 

• Generative modelling is a major topic in AI. We can also approach generation as a sequential process, to be trained with RL-techniques: 

Possible papers: 

• Generative Flow Networks (Paper, Code)

  • This paper studies the (tight) relation between flow-based sequential generation and (entropy-regularized) reinforcement learning. 

  • You could try to understand these methods and replicate the Hypergrid experiments (Sec 4.1) and possibly the Small Molecule Generation (Sec 4.2)

Evolution

Explanation: 

• Most reinforcement (& supervised) learning is based on gradient-based optimization. However, evolutionary (gradient-free) optimization could be an interesting alternative. 

Possible papers: 

• EGGROLL (Paper, Code) 

• This paper introduces a Low-Rank Adaptation (LoRA) method (known from LLMs), for evolutionary strategies, making them way more efficient. 

• You could try to understand these methods and reproduce the RL experiments (Sec. 6.3 and Fig 4a).