Learning Agent State Online with Recurrent Generate-and-Test

Abstract

The concept of state is fundamental to a reinforcement learning agent. The state is the input to the agent's action-selection policy, value functions, and environmental model. A reinforcement learning agent interacts with the environment by performing actions and receiving observations, resulting in the agent's data stream of experience. In many cases, the observations only provide partial information about the state, and the agent needs to learn the state directly based on the data stream of experience. We refer to the state directly learned from the data stream of experience as the agent state. Existing methods based on gradient descent, including Real-Time Recurrent Learning (RTRL) and Backpropagation Through Time (BPTT), can learn the agent state. However, these methods are computationally expensive, making them unsuitable for online learning of the agent state. In this thesis, we propose computationally efficient methods based on the generate-and-test approach to learn the agent state. We study the effectiveness of our generate-and-test methods for learning the agent state on two partially observable multi-step prediction problems---the trace conditioning problem and the trace patterning problem. The trace conditioning problem focuses on the agent's ability to remember a cue presented in the past to predict a signal in the future. The trace patterning problem is an extension of the trace conditioning problem in which a non-linear combination of observation signals triggers the arrival of a temporally distant signal. In the trace patterning problem, the agent must learn non-linear configurations of observation signals to predict the signal of interest accurately. Our experiments show that the proposed generate-and-test methods can learn the agent state online and make accurate predictions in both problems mentioned above.