Snapshot-Driven Deep Reinforcement Learning
Abstract
Deep reinforcement learning (DRL) has suggested many effective solutions to complex problems. Despite the impressive achievements of DRL, we have limited insights into why DRL is effective. DRL is also known as a black-box model with high complexity which makes DRL becomes difficult to be interpreted in a human-understandable way to discover the rationale behind DRL's predictions. Many papers have proposed different techniques intended to improve interpretability and measured the effect of different interpretability methods on user trust, the ability to simulate models, and the ability to detect mistakes. Understanding DRL through interpretation methods can unlock knowledge to improve the quality of prediction, understand internal errors to be able to fix, answer why a model does not work and improve the overall learning process.Most existing interpretation approaches only assume a converged model, which can't interpret the learning process of DRL and important concepts shaped. Therefore, the previous approaches are slow to produce a robust interpretation instantly. The sparsity of interpretation is another problem in the previous approaches because most of the known interpretation requires enormous human effort. Also, interpretation frameworks have not been used for supporting the learning process to have better performance. To address these challenges in DRL, a snapshot-driven approach, extracting a small number of examples for interpretation and using these snapshots for learning improvement, is proposed. Utilizing the snapshots, we can interpret the change in the learning process's behavior. The snapshots can also provide instant feedback to new samples for a faster understanding of the decisions of DRL. With the understanding of DRL, the snapshots can provide improvement to the learning methods in different settings. In order to fully accomplish the snapshot-driven learning, I propose methods 1) to extract snapshots from a DRL agent and understand the behavior, 2) to improve the learning process of the DRL agent and evaluate the effectiveness of the snapshots on it, 3) to reduce the number of snapshots for easier interpretation, and 4) to apply snapshots usage in continual DRL settings for latent stabilization to prevent catastrophic forgetting problem.