Agent-State Construction with Auxiliary Inputs

Abstract

In most, if not every, realistic sequential decision-making tasks, the decision-making agent is not able to model the full complexity of the world. In reinforcement learning, the environment is often much larger and more complex than the agent, a setting also known as partial observability. In such settings, the agent must leverage more than just the current sensory inputs; it must construct an agent state that summarizes the agent’s previous interactions with the world. Currently, the most common approach to tackle such a problem is to learn the agent-state function with a recurrent network. This is done with the agent’s sensory stream as input, which is often augmented with transformations of the agent’s observation. These augmentations are done in multiple ways, from simple approaches like concatenating observations to more complex ones such as uncertainty estimates or predictive representations. Nevertheless, although ubiquitous in the field, these additional inputs, which we term auxiliary inputs, are rarely emphasized, and it is not clear what their role or impact is. In this work we formalize agent-state construction with auxiliary inputs and present several examples of auxiliary inputs that incorporate information from the past, present, and/or future of the agent-environment interaction. We show that auxiliary inputs allow an agent to discriminate between observations that would otherwise be aliased, leading to more expressive features that smoothly interpolate between different states. We empirically evaluate this agent-state construction with different function approximators, using different instantiations of these auxiliary inputs across a variety of tasks. This approach is complementary to state-of-the-art methods such as recurrent neural networks, and acts as a heuristic that facilitates longer temporal credit assignment, reducing the number of time steps needed when performing truncated backpropagation through time and leading to better performance.