Bio-Inspired Gaze and Neural Command Fusion for Assistive Smartphone Interaction
Source abstract
Abstract
This paper presents an assistive smartphone interaction system that combines mobile gaze tracking with EEG-based BCI commands. The Android application estimates the user’s gaze with the front camera, MediaPipe facial landmarks, a TinyTrackerS TFLite model, temporal smoothing, and a calibrated mapping from model output to screen coordinates. The gaze point is used to locate the intended screen area, while the BCI layer uses Emotiv Cortex commands for click, scroll, and back actions. A FastAPI and MongoDB backend manages profiles, calibration data, validation reports, runtime data, and WebSocket control events. Android Accessibility is used to execute the selected actions, with raw tap fallback when needed. The system was tested with 36 student volunteers during a short Patient Assist task. In the evaluation, 33 out of 36 gaze mappings were promoted to the active profile. The average static mean error was 390.02 px, and the average static p95 error was 789.09 px. BCI command success was 89.58% for click, 72.22% for scroll, and 77.78% for back. The Android layer acknowledged 228 out of 234 accepted control events. The average usability score was 4.21 out of 5.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Gaze Tracking and Assistive Technology · Interactive and Immersive Displays · Assistive Technology in Communication and Mobility
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Industrial Engineering and Robotics, National University of Science and Technology POLITEHNICA Bucharest, 060042 Bucharest, Romania
Faculty of Engineering in Foreign Languages, National University of Science and Technology POLITEHNICA Bucharest, 060042 Bucharest, Romania
Department of Entrepreneurship and Management, Faculty of Entrepreneurship Business Engineering and Management, National University of Science and Technology POLITEHNICA Bucharest, 060042 Bucharest, Romania
Funding
National University of Science and Technology POLITEHNICA Bucharest
References (27)
Show source-provided references
1. Bonanno, M., Saracino, B., Ciancarelli, I., Panza, G., Manuli, A., Morone, G., and Calabrò, R.S. (2025). Assistive technologies for individuals with a disability from a neurological condition: A narrative review on the multimodal integration. Healthcare, 13.10.3390/healthcare13131580 ↗
2. Saha, S., Karlsson, P., Anderson, C., Kavehei, O., and McEwan, A. (2026). Individualized brain-computer interface for people with disabilities: A review. Front. Hum. Neurosci., 20.10.3389/fnhum.2026.1738876 ↗
3. A real-time eye movement-based computer interface for people with disabilities10.1016/j.smhl.2024.100521 ↗
4. Brain–computer interfaces in 2023–202410.1002/brx2.70024 ↗
5. The history, current state and future possibilities of the non-invasive brain computer interfaces10.1016/j.medntd.2025.100353 ↗
6. Brain-Computer Interfaces in Assistive Technologies: A Bibliometric Review
7. Belwafi, K., and Ghaffari, F. (2024). Thought-controlled computer applications: A brain–computer interface system for severe disability support. Sensors, 24.10.3390/s24206759 ↗
8. Huang, X., Xu, Y., Hua, J., Yi, W., Yin, H., Hu, R., and Wang, S. (2021). A review on signal processing approaches to reduce calibration time in EEG-based brain–computer interface. Front. Neurosci., 15.10.3389/fnins.2021.733546 ↗
9. Recent applications of EEG-based brain-computer-interface in the medical field
10. Drăgoi, M.-V., Nisipeanu, I., Frimu, A., Tălîngă, A.M., Hadăr, A., Dobrescu, T.G., Suciu, C.P., and Manea, A.R. (2024). Real-time home automation system using BCI technology. Biomimetics, 9.10.20944/preprints202407.0353.v1 ↗
11. English and romanian brain-to-text brain-computer interface word prediction system
12. Fischer-Janzen, A., Wendt, T.M., and Van Laerhoven, K. (2024). A scoping review of gaze and eye tracking-based control methods for assistive robotic arms. Front. Robot. AI, 11.10.3389/frobt.2024.1326670 ↗
13. Real-time human-computer interaction using eye gazes
14. Smartphone-based eye tracking system using edge intelligence and model optimisation10.1016/j.iot.2024.101481 ↗
15. Bonazzi, P., Rüegg, T., Bian, S., Li, Y., and Magno, M. (2023). Tinytracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation. Proceedings of the 2023 IEEE SENSORS, Vienna, Austria, 29 October–1 November 2023, IEEE.10.1109/SENSORS56945.2023.10325167 ↗
16. Towards an Eye-Brain-Computer Interface: Combining Gaze with the Stimulus-Preceding Negativity for Target Selections in XR
17. Coutray, K., Barbel, W., Groth, Z., and LaViola, J.J. (2025). NeuroGaze: A hybrid EEG and eye-tracking brain-computer interface for hands-free interaction in virtual reality. Front. Hum. Neurosci., 19.10.3389/fnhum.2025.1695446 ↗
18. Batista, P.G., Vieira, A.C., and Gaspar, P.D. (2025). Robotic Prostheses and Neuromuscular Interfaces: A Review of Design and Technological Trends. Machines, 13.10.3390/machines13090804 ↗
19. Continuous neural control of a bionic limb restores biomimetic gait after amputation10.1038/s41591-024-02994-9 ↗
20. Tissue-integrated bionic knee restores versatile legged movement after amputation10.1126/science.adv3223 ↗
21. Camera-based interactive wall display using hand gesture recognition
22. Real-time hand gesture recognition: A comprehensive review of techniques, applications, and challenges
23. Control the Robot Arm through Vision-Based Human Hand Tracking10.5937/fme2401037P ↗
24. (2026, March 27). Google AI Edge. Available online: https://ai.google.dev/edge/mediapipe/solutions/guide.
25. (2026, April 24). Emotiv Cortex API. Available online: https://emotiv.gitbook.io/cortex-api.
26. (2026, March 25). Android Developers. Available online: https://developer.android.com/.
27. Directedeness, correlations, and daily cycles in springbok motion: From data via stochastic models to movement prediction10.1103/PhysRevResearch.5.043129 ↗