Study of Social Presence While Interacting in Metaverse with an Augmented Avatar during Autonomous Driving

In this paper, we studied the effects of using Microsoft HoloLens 2 in a Metaverse-based collaborative mixed reality environment on the driver’s social presence while using an autonomous driving system. In (semi-) autonomous vehicles the driver is the system’s monitor, and the driving process becomes a secondary task. Our approach is motivated by the advent of Microsoft Mesh XR technology that enables immersion in multi-person, shared mixed reality environments. We conducted a user study comparing the effects on social presence in two scenarios: baseline and mixed reality collaboration. During the baseline condition, participants communicated and interacted with another person using Skype/Meet which was installed on a mobile tablet. In the second scenario the participants used the Microsoft Mesh application installed on HoloLens 2 to collaborate in a mixed reality environment where each user is represented by an augmented 3D avatar. During the experiment, the participant had to perform a social interaction tell-a-lie task and a remote collaborative tic-tac-toe game, while also monitoring the vehicle’s behavior. The social presence was measured using the Harms and Biocca questionnaire, one of the most widely used tools for evaluating the user’s experience. We found that there are significant statistical differences for Co-presence, Perceived Emotional Interdependence, and Perceived Behavioral Interdependence, and participants were able to easily interact with the avatar in the mixed reality scenario. The proposed study procedure could be taken further to assess the driver’s performance during handover procedures, especially when the autonomous driving system encounters a critical situation.