Real‐Time Non‐Driving Behavior Recognition Using Deep Learning‐Assisted Triboelectric Sensors in Conditionally Automated Driving

Abstract Real‐time recognition of non‐driving behaviors is of great importance in conditionally automated driving, as it determines the takeover time budget, which in turn has a huge impact on the performance of the takeover. Here, a novel real‐time non‐driving behavior recognition system (RNBRS) integrating self‐powered, low‐cost, easy‐to‐manufacture triboelectric sensors, and a deep learning model is proposed. The structure, working mechanism, and electrical characteristics of triboelectric sensors are investigated and analyzed. Through the ingenious structural design of single‐electrode triboelectric sensors and driving simulation experiments under conditional automated driving, non‐driving behaviors are captured in the form of electrical signals. A well‐trained long short‐term memory network model is adopted to recognize the five most typical non‐driving behaviors, including phone, console touchpad, driving, monitoring driving, and no operation, and test accuracy of 93.5% is achieved. Demonstration of a set of controlled experiments shows that RNBRS enables vehicles with conditional automation to dynamically adjust takeover time budget based on driver behavior, therefore significantly improving both safety and stability of takeover. This study opens new frontiers for the development of self‐powered electronics and inspires new thoughts on human‐machine interaction and the safety of autonomous vehicles.