Papers

You can choose to focus your project on an RL research paper. Your goal is to thoroughly understand the paper, replicate its experiments, and optionally extend/improve it with your own new idea. 

(You are allowed to come up with your own paper of interest, but do discuss it with the teachers. Make sure the paper comes with 1) a public codebase (from the official authors) and 2) a smaller toy experiment that is for sure computationally feasible to reproduce.)

RL for (sequential) generative modelling

Explanation: 

• Generative modelling is a major topic in AI. We can also approach generation as a sequential process, to be trained with RL-techniques: 

Possible papers: 

• Generative Flow Networks as Entropy-Regularized RL (Paper, Code)

  • This paper studies the (tight) relation between flow-based sequential generation and (entropy-regularized) reinforcement learning. 

  • You could try to understand these methods and replicate the Hypergrid experiments (Sec 4.1) and possibly the Small Molecule Generation (Sec 4.2)

Safe reinforcement learning

Explanation: 

• Safety is crucial in real-world deployment. Safe RL tries to maximize the expected return while respecting seperate cost constraints. 

Possible papers: 

• CRAX: Fast Safe Reinforcement Learning Benchmarking (Paper, Code)

  • This recent paper provides a fast benchmarking suite for a range of Safe RL algorithms. It does come in Jax (which has a learning curve). 

  • The benchmark currently measures safety only through cumulative cost. Maybe you could implement and compare different ways to specify cost (for example probabilistic, such as the chance the agent is in an unsafe region). 

Evolutationary Strategies combined with integer neural networks

Explanation: 

• A recent line of work (Paper) explored the usage of evolutionary strategies (ES) on RL tasks. Thesis. 

• Moreover, this work proposes to use pure integer neural networks for training, as ES do not require gradients through the model.

Possible papers: 

• The work in the paper was extended to combine the ES with integer neural networks on RL tasks by a Bachelor student (Thesis & Code to follow).

• The thesis reveals that this strategy is fruitful and has impressively low wallclock-times. As a potential topic the existing algorithm could be refined, by introducing other techniques to make the integer neural network approach more robust and/or extend the results to new tasks.

Model-based RL

Explanation: 

• Model-based reinforcement combines planning and learning to improve training efficiency. 

Possible papers: 

• PO-MPC (Paper, Code) is an recent internally developed approach to model-based reinforcement learning. 

• The first author, Alvaro Serra-Gomez (from LIACS), is willing to discuss and supervise a possible extension project with you.