Categorizing Touch-Input Locations from Touchscreen Device Interfaces via On-Board Mechano-Acoustic Transducers
Source abstract
Abstract
Many mobile electronics devices, including smartphones and tablets, require the user to interact physically with the device via tapping the touchscreen. Conveniently, these compact devices are also equipped with high-precision transducers such as accelerometers and microphones, integrated mechanically and designed on-board to support a range of user functionalities. However, unintended access to these transducer signals (bypassing normal on-board data access controls) may allow sensitive user interaction information to be detected and thereby exploited. In this study, we show that acoustic features extracted from the on-board microphone signals, supported with accelerometer and gyroscope signals, may be used together with machine learning techniques to successfully determine the user’s touch input location on a touchscreen: our ensemble model, namely the random forest model, predicts touch input location with up to 86% accuracy in a realistic scenario. Accordingly, we present the approach and techniques used, the performance of the model developed, and also discuss limitations and possible mitigation methods to thwart possible exploitation of such unintended signal channels.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Music and Audio Processing · User Authentication and Security Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Information Systems Technology and Design (ISTD), Singapore University of Technology & Design, 8 Somapah Rd, Singapore 487372, Singapore
Science, Mathematics and Technology (SMT), Singapore University of Technology & Design, 8 Somapah Rd, Singapore 487372, Singapore
References (28)
Show source-provided references
1. Nist special publication 800-57 part 1, revision 4
2. Panda, S., Liu, Y., Hancke, G.P., and Qureshi, U.M. (2020). Behavioral Acoustic Emanations: Attack and Verification of PIN Entry Using Keypress Sounds. Sensors, 20.10.3390/s20113015 ↗
3. Bucicoiu, M., Davi, L., Deaconescu, R., and Sadeghi, A.R. (2015, January 14–17). XiOS: Extended application sandboxing on iOS. Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security, Singapore.10.1145/2714576.2714629 ↗
4. Owusu, E., Han, J., Das, S., Perrig, A., and Zhang, J. (2012, January 28–29). Accessory: Password inference using accelerometers on smartphones. Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications, San Diego, CA, USA.10.1145/2162081.2162095 ↗
5. Zhu, T., Ma, Q., Zhang, S., and Liu, Y. (2014, January 3–7). Context-free attacks using keyboard acoustic emanations. Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security, Scottsdale, AZ, USA.10.1145/2660267.2660296 ↗
6. Compagno, A., Conti, M., Lain, D., and Tsudik, G. (2017, January 2–6). Do not Skype & Type! Acoustic Eavesdropping in Voice-Over-IP. Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, Abu Dhabi, United Arab Emirates.10.1145/3052973.3053005 ↗
7. Keyboard acoustic emanations revisited10.1145/1609956.1609959 ↗
8. Asonov, D., and Agrawal, R. (2004, January 12–12). Keyboard acoustic emanations. Proceedings of the IEEE Symposium on Security and Privacy, Berkeley, CA, USA.
9. On the best sensor for keystrokes inference attack on android10.1016/j.protcy.2013.12.285 ↗
10. Narain, S., Sanatinia, A., and Noubir, G. (2014, January 23–25). Single-stroke language-agnostic keylogging using stereo-microphones and domain specific machine learning. Proceedings of the 2014 ACM Conference on Security and Privacy in Wireless & Mobile Networks, Oxford, UK.10.1145/2627393.2627417 ↗
11. Lu, L., Yu, J., Chen, Y., Zhu, Y., Xu, X., Xue, G., and Li, M. (May, January 29). Keylistener: Inferring keystrokes on qwerty keyboard of touch screen through acoustic signals. Proceedings of the IEEE INFOCOM 2019-IEEE Conference on Computer Communications, Paris, France.10.1109/INFOCOM.2019.8737591 ↗
12. Zhou, M., Wang, Q., Yang, J., Li, Q., Jiang, P., Chen, Y., and Wang, Z. (2019). Stealing your Android patterns via acoustic signals. IEEE Trans. Mob. Comput.
13. Shumailov, I., Simon, L., Yan, J., and Anderson, R. (2019). Hearing your touch: A new acoustic side channel on smartphones. arXiv.
14. SonarSnoop: Active acoustic side-channel attacks10.1007/s10207-019-00449-8 ↗
15. Cano, P., Batle, E., Kalker, T., and Haitsma, J. (2002, January 9–11). A review of algorithms for audio fingerprinting. Proceedings of the 2002 IEEE Workshop on Multimedia Signal Processing, St. Thomas, VI, USA.
16. Wang, A. (2003, January 26–30). An Industrial Strength Audio Search Algorithm. Proceedings of the ISMIR 2003: Proceedings of the fourth International Conference on Music Information Retrieval, Baltimore, MD, USA.
17. Capturing the acoustic fingerprint of stratospheric ash injection10.1029/2008EO400001 ↗
18. Teo, K.R., Balamurali, B.T., Chen, J.m., and Zhou, J.Y. (February, January 30). Retrieving Input from Touch Interfaces via Acoustic Emanations. Proceedings of the 2021 IEEE Conference on Dependable and Secure Computing, Aizuwakamatsu, Japan.10.1109/DSC49826.2021.9346271 ↗
19. Culurciello, E. (2017, October 13). e-Lab VideoSensors. Available online: https://github.com/e-lab/VideoSensors.
20. Giannakopoulos, T. (2015). pyaudioanalysis: An open-source python library for audio signal analysis. PLoS ONE, 10.10.1371/journal.pone.0144610 ↗
21. Teo, K.R., Balamurali, B.T., Ng, T.S., and Chen, J.m. (2018, January 6–9). Exploring DíZi Performance Parameters With Machine Learning. Proceedings of the 2018 ACMC Conference of the Australasian Computer Music Association, Perth, Australia.
22. Circularity in judgments of relative pitch10.1121/1.1919362 ↗
23. Scikit-learn: Machine learning in Python
24. Semiconductors, N. (2017, February 02). Xtrinsic MAG3110 Three-Axis, Digital Magnetometer. Available online: https://www.nxp.com/docs/en/data-sheet/MAG3110.pdf.
25. Visualizing data using t-SNE
26. Miluzzo, E., Varshavsky, A., Balakrishnan, S., and Choudhury, R.R. (2012, January 26–29). Tapprints: Your finger taps have fingerprints. Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services, Lake District, UK.10.1145/2307636.2307666 ↗
27. Gupta, H., Sural, S., Atluri, V., and Vaidya, J. (2016, January 18–20). Deciphering text from touchscreen key taps. Proceedings of the IFIP Annual Conference on Data and Applications Security and Privacy (DBSec), Trento, Italy.10.1007/978-3-319-41483-6_1 ↗
28. Side-channel inference attacks on mobile keypads using smartwatches10.1109/TMC.2018.2794984 ↗