Realization of Efficient Exploration by Self-Generating Evaluation Considering Curiosity and Fear Indices Based on Prediction Error

In this paper, we propose a reinforcement-learning method in which an agent generates intrinsic rewards by evaluating its environment based on sensor data. The evaluation is perormed from three perspectives, one of which concerns the predictability of sensor inputs. This perspective focuses on the magnitude of the prediction error: a large error suggests environment uncertainty and leads to a low evaluation. However, prediction errors can arise from two distinct causes that previous method fail to distinguish. The first is the intrinsic unpredictability of a situation, where high prediction error persists even after sufficient learning. Such states are likely to be dangerous and should be assigned low evaluations to discourage the agent from entering them. The second is insufficient learning of the predictor itself, where high prediction error simply reflects the need for further exploration. These states should be assigned high evaluations to encourage learning. To address these cases this study introduced a method that evaluates environmental states by jointly considering the degree of danger and the necessity of further predictor learning with the learning progress for each sensor input. Two complimentary evaluation indices were proposed curiosity, which promotes exploration of insufficiently learned states, and fear, which discourages entry into unpredictable or hazardous states. The effectiveness of the proposed method was demonstrated through simulation experiments in a two-dimensional discrete environment.