Take-Over Requests after Waking in Autonomous Vehicles
Source abstract
Abstract
Autonomous vehicles (AVs) enable drivers to devote their primary attention to non-driving-related tasks (NDRTs). Consequently, AVs must provide intelligibility services appropriate to drivers’ in-situ states and in-car activities to ensure driver safety, and accounting for the type of NDRT being performed can result in higher intelligibility. We discovered that sleeping is drivers’ most preferred NDRT, and this could also result in a critical scenario when a take-over request (TOR) occurs. In this study, we designed TOR situations where drivers are woken from sleep in a high-fidelity AV simulator with motion systems, aiming to examine how drivers react to a TOR provided with our experimental conditions. We investigated how driving performance, perceived task workload, AV acceptance, and physiological responses in a TOR vary according to two factors: (1) feedforward timings and (2) presentation modalities. The results showed that when awakened by a TOR alert delivered >10 s prior to an event, drivers were more focused on the driving context and were unlikely to be influenced by TOR modality, whereas TOR alerts delivered <5 s prior needed a visual accompaniment to quickly inform drivers of on-road situations. This study furthers understanding of how a driver’s cognitive and physical demands interact with TOR situations at the moment of waking from sleep and designs effective interventions for intelligibility services to best comply with safety and driver experience in AVs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human-Automation Interaction and Safety · Sleep and Work-Related Fatigue · Older Adults Driving Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Gwangju Institute of Science and Technology, School of Integrated Technology, Gwangju 61005, Korea
Funding
National Research Foundation of Korea
References (100+)
Show source-provided references
1. Naus, G., Vugts, R., Ploeg, J., Van de Molengraft, M., and Steinbuch, M. (2009, January 21–25). Towards on-the-road implementation of cooperative adaptive cruise control. Proceedings of the 16th World Congress and Exhibition on Intelligent Transport Systems 2009, Stockholm, Sweden.
2. Helgath, J., Braun, P., Pritschet, A., Schubert, M., Böhm, P., and Isemann, D. (2018, January 15–20). Investigating the effect of different autonomy levels on user acceptance and user experience in self-driving cars with a VR driving simulator. Proceedings of the International Conference of Design, User Experience, and Usability, Las Vegas, NV, USA.10.1007/978-3-319-91806-8_19 ↗
3. Investigating the importance of trust on adopting an autonomous vehicle10.1080/10447318.2015.1070549 ↗
4. Rödel, C., Stadler, S., Meschtscherjakov, A., and Tscheligi, M. (2014, January 17–19). Towards autonomous cars: The effect of autonomy levels on acceptance and user experience. Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Seattle, WA, USA.10.1145/2667317.2667330 ↗
5. Forster, Y., Hergeth, S., Naujoks, F., Krems, J.F., and Keinath, A. (2019, January 24–28). Empirical validation of a checklist for heuristic evaluation of automated vehicle HMIs. Proceedings of the International Conference on Applied Human Factors and Ergonomics, Washington, DC, USA.10.1007/978-3-030-20503-4_1 ↗
6. Transitioning to manual driving requires additional time after automation deactivation10.1016/j.trf.2018.03.019 ↗
7. Effect of different alcohol levels on take-over performance in conditionally automated driving10.1016/j.aap.2018.03.001 ↗
8. Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance10.1007/s12008-014-0227-2 ↗
9. Kim, W., Cho, K., Kim, M., Yeo, D., Lee, J., and Kim, S. (2019, January 13–16). A Study on Driver’s Attitude according to Combination of Modality and Message Type in Providing Driving Information in Autonomous Driving. Proceedings of the HCI Korea, Jeju, Korea.
10. Ruijten, P.A., Terken, J., and Chandramouli, S.N. (2018). Enhancing trust in autonomous vehicles through intelligent user interfaces that mimic human behavior. Multimodal Technol. Interact., 2.10.3390/mti2040062 ↗
11. Lim, B.Y., Dey, A.K., and Avrahami, D. (2009, January 4–9). Why and why not explanations improve the intelligibility of context-aware intelligent systems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.10.1145/1518701.1519023 ↗
12. Intelligibility and accountability: Human considerations in context-aware systems10.1207/S15327051HCI16234_05 ↗
13. Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., and Kankanhalli, M. (2018, January 21–26). Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3173574.3174156 ↗
14. Fröhlich, P., Baldauf, M., Meneweger, T., Erickson, I., Tscheligi, M., Gable, T., Ruyter, B., and Paternò, F. (2019, January 4–9). Everyday automation experience: Non-expert users encountering ubiquitous automated systems. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290607.3299013 ↗
15. Fröhlich, P., Baldauf, M., Palanque, P., Roto, V., Meneweger, T., Tscheligi, M., Becerra, Z., and Paternò, F. (2020, January 25–30). Automation experience across domains: Designing for intelligibility, interventions, interplay and integrity. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3334480.3375178 ↗
16. Asleep at the automated wheel—Sleepiness and fatigue during highly automated driving10.1016/j.aap.2018.03.013 ↗
17. Sleep in highly automated driving: Takeover performance after waking up10.1016/j.aap.2020.105617 ↗
18. Yeo, D., Lee, J., Kim, W., Kim, M., Cho, K., Ataya, A., and Kim, S. (2019, January 13–16). A Hand-Over Notification System of Vehicle Driving Control according to Driver’s Condition based on Cognitive Load in Autonomous Driving Situation. Proceedings of the HCI Korea, Jeju, Korea.
19. Situation awareness and workload in driving while using adaptive cruise control and a cell phone10.1016/j.ergon.2005.04.002 ↗
20. Behavioural effects of mobile telephone use during simulated driving10.1080/00140139508925285 ↗
21. Speech-based e-mail and driver behavior: Effects of an in-vehicle message system interface10.1518/hfes.46.4.625.56814 ↗
22. Mok, B., Johns, M., Lee, K.J., Miller, D., Sirkin, D., Ive, P., and Ju, W. (2015, January 15–18). Emergency, automation off: Unstructured transition timing for distracted drivers of automated vehicles. Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, Gran Canaria, Spain.10.1109/ITSC.2015.396 ↗
23. Borojeni, S.S., Weber, L., Heuten, W., and Boll, S. (2018, January 3–6). From reading to driving: Priming mobile users for take-over situations in highly automated driving. Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services, Barcelona, Spain.10.1145/3229434.3229464 ↗
24. Borojeni, S.S., Chuang, L., Heuten, W., and Boll, S. (2016, January 24–26). Assisting drivers with ambient take-over requests in highly automated driving. Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Ann Arbor, MI, USA.10.1145/3003715.3005409 ↗
25. How traffic situations and non-driving related tasks affect the take-over quality in highly automated driving10.1177/1541931214581434 ↗
26. Surrogate in-vehicle information systems and driver behaviour: Effects of visual and cognitive load in simulated rural driving10.1016/j.trf.2005.04.002 ↗
27. Effects of driver and secondary task characteristics on lane change test performance10.1002/hfm.20342 ↗
28. Effects of non-driving related task modalities on takeover performance in highly automated driving10.1177/0018720818768199 ↗
29. Kim, H.S., Yoon, S.H., Kim, M.J., and Ji, Y.G. (2015, January 1–3). Deriving future user experiences in autonomous vehicle. Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Nottingham, UK.10.1145/2809730.2809734 ↗
30. Mok, B., Johns, M., Miller, D., and Ju, W. (2017, January 6–11). Tunneled in: Drivers with active secondary tasks need more time to transition from automation. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA.10.1145/3025453.3025713 ↗
31. Is take-over time all that matters? The impact of visual-cognitive load on driver take-over quality after conditionally automated driving10.1016/j.aap.2016.04.002 ↗
32. Köhn, T., Gottlieb, M., Schermann, M., and Krcmar, H. (2019, January 17–20). Improving take-over quality in automated driving by interrupting non-driving tasks. Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA.10.1145/3301275.3302323 ↗
33. Wintersberger, P., Riener, A., Schartmüller, C., Frison, A.K., and Weigl, K. (2018, January 23–25). Let me finish before I take over: Towards attention aware device integration in highly automated vehicles. Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Toronto, ON, Canada.10.1145/3239060.3239085 ↗
34. Drivers’ visual-distracted take-over performance model and its application on adaptive adjustment of time budget10.1016/j.aap.2021.106099 ↗
35. Purucker, C., Naujoks, F., Wiedemann, K., Neukum, A., and Marberger, C. (2018, January 15–17). Effects of Secondary Tasks on Conditional Automation State Transitions While Driving on Freeways: Judgements and Observations of Driver Workload. Proceedings of the 6th International Conference on Driver Distraction and Inattention, Gothenburg, Sweden.
36. Kim, S., Chun, J., and Dey, A.K. (2015, January 18–23). Sensors know when to interrupt you in the car: Detecting driver interruptibility through monitoring of peripheral interactions. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.10.1145/2702123.2702409 ↗
37. Frison, A.K., Wintersberger, P., Liu, T., and Riener, A. (2019, January 17–20). Why do you like to drive automated? a context-dependent analysis of highly automated driving to elaborate requirements for intelligent user interfaces. Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA.10.1145/3301275.3302331 ↗
38. Ataya, A., Kim, W., Elsharkawy, A., and Kim, S. (2021). How to Interact with a Fully Autonomous Vehicle: Naturalistic Ways for Drivers to Intervene in the Vehicle System while Performing Non-Driving Related Tasks. Sensors, 21.10.3390/s21062206 ↗
39. Sleep and take-over in automated driving10.1016/j.ijtst.2019.09.003 ↗
40. Sleep inertia in automated driving: Post-sleep take-over and driving performance10.1016/j.aap.2020.105918 ↗
41. Take-over requests for automated driving10.1016/j.promfg.2015.07.788 ↗
42. Kaduk, S.I., Roberts, A.P., and Stanton, N. (2015). I just woke up behind the wheel: Sleep inertia as a new risk in semi-automated cars. Contemporary Ergonomics and Human Factors, Proceedings of the International Conference on Ergonomics & Human Factors, Daventry, UK, 13–16 April 2015, Taylor & Francis.
43. Vigilance monitoring for operator safety: A simulation study on highway driving10.1016/j.jsr.2005.11.003 ↗
44. Evaluation of lane departure warnings for drowsy drivers10.1177/154193120605002211 ↗
45. The utility of automated measures of ocular metrics for detecting driver drowsiness during extended wakefulness10.1016/j.aap.2015.11.033 ↗
46. The impact of drowsiness on in-vehicle human-machine interaction with head-up and head-down displays10.1007/s11042-018-5966-9 ↗
47. A comparison of driver sleepiness in the simulator and on the real road
48. Predictors of objective sleep tendency in the general population10.1093/sleep/26.6.678 ↗
49. Effect of a relatively long afternoon nap on autonomous nervous activity, sleep architecture, and subjective sleep quality10.1002/tee.22702 ↗
50. Kim, W., and Kim, S. (2019, January 20–23). A New Approach to Studying Sleep in Autonomous Vehicles: Simulating the Waking Situation. Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19), New Orleans, LA, USA.10.1145/3332167.3357098 ↗
51. Interrupting Drivers for Interactions: Predicting Opportune Moments for In-vehicle Proactive Auditory-verbal Tasks10.1145/3287053 ↗
52. Driver inattention monitoring system for intelligent vehicles: A review10.1109/TITS.2010.2092770 ↗
53. What determines the take-over time? An integrated model approach of driver take-over after automated driving10.1016/j.aap.2015.02.023 ↗
54. Ali, S.F., Aslam, A.S., Awan, M.J., Yasin, A., and Damaševičius, R. (2021). Pose Estimation of Driver’s Head Panning Based on Interpolation and Motion Vectors under a Boosting Framework. Appl. Sci., 11.10.3390/app112411600 ↗
55. Psychophysiological responses to takeover requests in conditionally automated driving10.1016/j.aap.2020.105804 ↗
56. Predicting driver takeover performance in conditionally automated driving10.1016/j.aap.2020.105748 ↗
57. Solovey, E.T., Zec, M., Garcia Perez, E.A., Reimer, B., and Mehler, B. (May, January 26). Classifying driver workload using physiological and driving performance data: Two field studies. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, ON, Canada.
58. Perello-March, J.R., Burns, C.G., Woodman, R., Elliott, M.T., and Birrell, S.A. (2021). Driver State Monitoring: Manipulating Reliability Expectations in Simulated Automated Driving Scenarios. IEEE Trans. Intell. Transp. Syst., 1–11.
59. Sheng, S., Pakdamanian, E., Han, K., Kim, B., Tiwari, P., Kim, I., and Feng, L. (2019, January 27–30). A case study of trust on autonomous driving. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.10.1109/ITSC.2019.8917251 ↗
60. Physiological and behavioural changes associated to the management of secondary tasks while driving10.1016/j.apergo.2009.01.007 ↗
61. Non-driving-related tasks, workload, and takeover performance in highly automated driving contexts10.1016/j.trf.2018.11.015 ↗
62. Indicators of Fatigue in Truck Drivers
63. Workload-adaptive cruise control–A new generation of advanced driver assistance systems10.1016/j.trf.2013.06.001 ↗
64. Autonomic responses of young passengers contingent to the speed and driving mode of a vehicle10.1016/S0169-8141(01)00059-2 ↗
65. Sandhaus, H., and Hornecker, E. (2018, January 14). A WOZ Study of Feedforward Information on an Ambient Display in Autonomous Cars. Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings, Berlin, Germany.10.1145/3266037.3266111 ↗
66. Directional tactile alerts for take-over requests in highly-automated driving10.1016/j.trf.2019.07.025 ↗
67. Takeover requests in simulated partially autonomous vehicles considering human factors10.1109/THMS.2017.2674998 ↗
68. Kutchek, K., and Jeon, M. (2019, January 4–9). Takeover and Handover Requests using Non-Speech Auditory Displays in Semi-Automated Vehicles. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290607.3313078 ↗
69. Take-over requests in highly automated driving: A crowdsourcing survey on auditory, vibrotactile, and visual displays10.1016/j.trf.2018.04.001 ↗
70. Assessment of cognitive workload of in-vehicle systems using a visual peripheral and tactile detection task setting10.3233/WOR-2012-0786-4919 ↗
71. A comparison of tactile, visual, and auditory warnings for rear-end collision prevention in simulated driving10.1518/001872008X250674 ↗
72. (2019, August 25). Driver Asleep at the Wheel of His Tesla on Busy Freeway in Los Angeles. Available online: https://www.youtube.com/watch?v=ZhObsMnipS8.
73. (2019, September 10). Tesla Driver Caught on Camera Apparently Asleep at the Wheel | NBC Nightly News. Available online: https://www.youtube.com/watch?v=NHUZxeSUFUk.
74. Design and evaluation for situation awareness enhancement10.1177/154193128803200221 ↗
75. Van Den Beukel, A.P., and Van Der Voort, M.C. (2013, January 6–9). The influence of time-criticality on situation awareness when retrieving human control after automated driving. Proceedings of the 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), The Hague, The Netherlands.10.1109/ITSC.2013.6728523 ↗
76. Takeover transition in autonomous vehicles: A YouTube study10.1080/10447318.2019.1634317 ↗
77. Task interruption and control recovery strategies after take-over requests emphasize need for measures of situation awareness10.1177/0018720819866976 ↗
78. Wu, Y., Kihara, K., Takeda, Y., Sato, T., Akamatsu, M., and Kitazaki, S. (2019, January 27–30). Assessing the Mental States of Fallback-Ready Drivers in Automated Driving by Electrooculography. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.10.1109/ITSC.2019.8917029 ↗
79. SAE On-Road Automated Vehicle Standards Committee (2014). Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. SAE Stand. J., 3016, 1–16.
80. Holländer, K., and Pfleging, B. (2018, January 25–28). Preparing drivers for planned control transitions in automated cars. Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia, Cairo, Egypt.10.1145/3282894.3282928 ↗
81. Naujoks, F., Forster, Y., Wiedemann, K., and Neukum, A. (2017). A human-machine interface for cooperative highly automated driving. Advances in Human Aspects of Transportation, Springer.10.1007/978-3-319-41682-3_49 ↗
82. Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Jina, S., Iqbal, S.T., Bennett, P.N., and Inkpen, K.M. (2019, January 4–9). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290605.3300233 ↗
83. Urgent and non-urgent takeovers during conditional automated driving on public roads: The impact of different training programmes10.1016/j.trf.2021.06.001 ↗
84. Methodology for the design of automotive HUD graphical interfaces10.15446/dyna.v85n207.71437 ↗
85. Human memory: A proposed system and its control processes10.1016/S0079-7421(08)60422-3 ↗
86. Forster, Y., Naujoks, F., and Neukum, A. (2017, January 11–14). Increasing anthropomorphism and trust in automated driving functions by adding speech output. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.10.1109/IVS.2017.7995746 ↗
87. A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems10.1177/0018720816634228 ↗
88. Shi, Y., Ruiz, N., Taib, R., Choi, E., and Chen, F. (May, January 28). Galvanic skin response (GSR) as an index of cognitive load. Proceedings of the CHI ’07 Conference on Human Factors in Computing Systems, San Jose, CA, USA.
89. Kim, S., Hong, J.H., Li, K.A., Forlizzi, J., and Dey, A.K. (2012, January 18–22). Route guidance modality for elder driver navigation. Proceedings of the International Conference on Pervasive Computing, Newcastle, UK.10.1007/978-3-642-31205-2_12 ↗
90. How to measure cerebral correlates of emotions in marketing relevant tasks10.1007/s12559-014-9304-x ↗
91. Wickens, C.D. (1991). Processing resources and attention. Multiple-Task Performance, Taylor & Francis.
92. Parasuraman, R., and Davies, D.R. (1984). Varieties of Attention, Academic Press.
93. Comparison of driving errors between on-the-road and simulated driving assessment: A validation study10.1080/15389580902894989 ↗
94. Aboalayon, K.A.I., Faezipour, M., Almuhammadi, W.S., and Moslehpour, S. (2016). Sleep stage classification using EEG signal analysis: A comprehensive survey and new investigation. Entropy, 18.10.3390/e18090272 ↗
95. A novel multi-class EEG-based sleep stage classification system10.1109/TNSRE.2017.2776149 ↗
96. Using electroencephalography to measure cognitive load10.1007/s10648-010-9130-y ↗
97. Deep in thought while driving: An EEG study on drivers’ cognitive distraction10.1016/j.trf.2014.08.001 ↗
98. Real-time sensing of trust in human-machine interactions10.1016/j.ifacol.2016.12.188 ↗
99. Akash, K., Hu, W.L., Reid, T., and Jain, N. (2017, January 24–26). Dynamic modeling of trust in human-machine interactions. Proceedings of the 2017 American Control Conference (ACC) IEEE, Seattle, WA, USA.10.23919/ACC.2017.7963172 ↗
100. Gupta, K., Hajika, R., Pai, Y.S., Duenser, A., Lochner, M., and Billinghurst, M. (2019, January 12–15). In AI we trust: Investigating the relationship between biosignals, trust and cognitive load in VR. Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology, Parramatta, Australia.10.1145/3359996.3364276 ↗