journal article·2025-04-30Open access

Machine Learning-Based Smartphone Grip Posture Image Recognition and Classification

Dohoon Kwon, Xin Cui, Yejin Lee, Younggeun Choi, Aditya Subramani Murugan, Eunsik Kim, Heecheon You

Applied Sciences· Volume 15· Issue 9

Abstract

Uncomfortable smartphone grip postures resulting from inappropriate user interface design can degrade smartphone usability. This study aims to develop a classification model for smartphone grip postures by detecting the positions of the hand and fingers on smartphones using machine learning techniques. Seventy participants (35 males and 35 females with an average of 38.5 ± 12.2 years) with varying hand sizes participated in the smartphone grip posture experiment. The participants performed four tasks (making calls, listening to music, sending text messages, and web browsing) using nine smartphone mock-ups of different sizes, while cameras positioned above and below their hands recorded their usage. A total of 3278 grip posture images were extracted from the recorded videos and were preprocessed using a skin color and hand contour detection model. The grip postures were categorized into seven types, and three models (MobileNetV2, Inception V3, and ResNet-50), along with an ensemble model, were used for classification. The ensemble-based classification model achieved an accuracy of 95.9%, demonstrating higher accuracy than the individual models: MobileNetV2 (90.6%), ResNet-50 (94.2%), and Inception V3 (85.9%). The classification model developed in this study can efficiently analyze grip postures, thereby improving usability in the development of smartphones and other electronic devices.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Ergonomics and Musculoskeletal Disorders · Hand Gesture Recognition Systems

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Industrial and Management Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea

School of Industrial Engineering, University of Ulsan, Ulsan 44610, Republic of Korea

Mechanical, Automotive, and Materials Engineering, University of Windsor, Windsor, ON N9B 3P4, Canada

Funding

research programs (2017M3C1B6070526; 2018R1A2A2A 05023299) through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (MEST) · Ministry of Trade, Industry, and Energy (MOTIE) under Industrial Technology Innovation Program, and the Biomedical Research Institute Fund, Chonbuk National University Hospital

References (68)

Show source-provided references
1. Choi, Y., Yang, X., Park, J., Lee, W., and You, H. (2020). Effects of Smartphone Size and Hand Size on Grip Posture in One-Handed Hard Key Operations. Appl. Sci., 10.10.3390/app10238374 ↗
2. The Performance of Hand Postures in Front- and Back-of-Device Interaction for mobile computing10.1016/j.ijhcs.2008.03.004 ↗
3. Effects of Grip Type and Wrist Posture on Forearm EMG Activity, Endurance Time and Movement Accuracy10.1016/j.ergon.2012.11.012 ↗
4. Smartphone Form Factors: Effects of Width and Bottom Bezel on Touch Performance, Workload, and Physical Demand10.1016/j.apergo.2017.10.002 ↗
5. Mobile Input Device Type, Texting Style and Screen Size Influence Upper Extremity and Trapezius Muscle Activity, and Cervical Posture While Texting10.1016/j.apergo.2015.03.003 ↗
6. Two-Handed Grip on a Mobile Phone Affords Greater Thumb Motor Performance, Decreased Variability, and a More Extended Thumb Posture Than a One-Handed Grip10.1016/j.apergo.2015.06.025 ↗
7. Lumsden, J. (2008). Understanding One-Handed Use of Mobile Devices. Handbook of Research on User Interface Design and Evaluation for Mobile Technology, IGI Global.10.4018/978-1-59904-871-0 ↗
8. Self-Organizing Maps for Hand and Full Body Tracking10.1016/j.neucom.2013.10.041 ↗
9. Achenbach, P., Laux, S., Purdack, D., Müller, P.N., and Göbel, S. (2023). Give Me a Sign: Using Data Gloves for Static Hand-Shape Recognition. Sensors, 23.10.20944/preprints202311.1385.v1 ↗
10. Hand Gesture Recognition for Human Computer Interaction10.1016/j.procs.2017.09.092 ↗
11. Singh, A., Singh, A., Rani, R., Dev, A., and Sharma, A. (2022, January 5–6). Hand Gesture Detection Using Convexity Hull and Convolutional Neural Network. Proceedings of the 2022 International Conference on Machine Learning, Computer Systems and Security (MLCSS), Bhubaneswar, India.10.1109/MLCSS57186.2022.00027 ↗
12. Chung, H.X., Hameed, N., Clos, J., and Hasan, M.M. (2022, January 2–4). A Framework of Ensemble CNN Models for Real-Time Sign Language Translation. Proceedings of the 14th International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Phnom Penh, Cambodia.
13. Niranjan, S.K., Rana, A., and Khurana, H. (2019, January 12–14). A Complete Study of Methodology of Hand Gesture Recognition System for Smart Homes. Proceedings of the 2019 International Conference on Contemporary Computing and Informatics (IC3I), Amity Global Institute, Singapore.
14. Deep CNN Models-Based Ensemble Approach to Driver Drowsiness Detection10.1007/s00521-020-05209-7 ↗
15. Phung, V.H., and Rhee, E.J. (2019). A High-Accuracy Model Average Ensemble of Convolutional Neural Networks for Classification of Cloud Image Patches on Small Datasets. Appl. Sci., 9.10.3390/app9214500 ↗
16. Human-Computer Interaction with Hand Gesture Recognition Using ResNet and MobileNet10.1155/2022/8777355 ↗
17. Tang, C., Ou, Y., Jiang, G., Xie, Q., and Xu, Y. (2012, January 11–14). Hand Tracking and Pose Recognition via Depth and Color Information. Proceedings of the 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), Guangzhou, China.10.1109/ROBIO.2012.6491117 ↗
18. Face Detection in Color Images
19. Hand Gestures Recognition with Improved Skin Color Segmentation in Human-Computer Interaction Applications
20. Shape-Based Hand Recognition10.1109/TIP.2006.873439 ↗
21. Contour Model-Based Hand-Gesture Recognition Using the Kinect Sensor10.1109/TCSVT.2014.2302538 ↗
22. Minnen, D., and Zafrulla, Z. (2011, January 6–13). Towards Robust Cross-User Hand Tracking and Shape Recognition. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130392 ↗
23. Hand Detection by Two-Level Segmentation with Double-Tracking and Gesture Recognition Using Deep-Features10.1007/s11220-022-00379-1 ↗
24. Approach to hand posture recognition based on hand shape features for human–robot interaction10.1007/s40747-021-00333-w ↗
25. Hand Gesture Recognition for Characters Understanding Using Convex Hull Landmarks and Geometric Features10.1109/ACCESS.2023.3300712 ↗
26. Dewi, C., Chen, A.P.S., and Christanto, H.J. (2023). Deep Learning for Highly Accurate Hand Recognition Based on Yolov7 Model. Big Data Cogn. Comput., 7.10.3390/bdcc7010053 ↗
27. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L. (2018, January 18–29). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
28. Recognition of Pear Leaf Disease Under Complex Background Based on DBPNet and Modified MobilenetV210.1049/ipr2.12855 ↗
29. Sun, Q., and Luo, X. (2022, January 2–4). A New Image Recognition Combining Transfer Learning Algorithm and MobileNet V2 Model for Palm Vein Recognition. Proceedings of the 2022 4th International Conference on Frontiers Technology of Information and Computer (ICFTIC), Qingdao, China.10.1109/ICFTIC57696.2022.10075212 ↗
30. Dabwan, B.A., Jadhav, M.E., Al Yami, M., Hassan, E.A., Almula, S.M., and Ali, Y.A. (2024, January 26–27). Classifying Hand Gestures for People with Disabilities Utilizing the MobileNetV2 Model. Proceedings of the 2024 1st International Conference on Innovative Sustainable Technologies for Energy, Mechatronics, and Smart Systems (ISTEMS), Dehradun, India.10.1109/ISTEMS60181.2024.10560333 ↗
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
32. Object Detection Using ResNet50
33. Yildirim, M.E., Salman, Y.B., and Genc, E. (November, January 29). Person Identification by Using ResNet on Hand Images. Proceedings of the 2024 24th International Conference on Control, Automation and Systems (ICCAS), Jeju, Republic of Korea.10.23919/ICCAS63016.2024.10773175 ↗
34. Practice of Gesture Recognition Based on Resnet5010.1088/1742-6596/1574/1/012154 ↗
35. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
36. An Efficient and Robust Hand Gesture Recognition System of Sign Language Employing Finetuned Inception-V3 and EfficientNet-B0 Network10.32604/csse.2023.037258 ↗
37. mIV3Net: Modified inception V3 network for hand gesture recognition10.1007/s11042-023-15865-1 ↗
38. Anusha, S.B., Samyama Gunjal, G.H., and Manjushree, N.S. (2021). Static Hand Gesture Prediction Using Inception V3. Cognitive Science and Technology, Proceedings of the International Conference on Cognitive and Intelligent Computing, Hyderabad, India, 11–12 December 2021, Springer.
39. A Survey on Ensemble Learning10.1007/s11704-019-8208-z ↗
40. Zhou, Z.H. (2012). Ensemble Methods: Foundations and Algorithms, Chapman and Hall/CRC.10.1201/b12207 ↗
41. A Novel Hand Gesture Detection and Recognition System Based on Ensemble-Based Convolutional Neural Network10.1007/s11042-022-11909-0 ↗
42. Ewe, E.L.R., Lee, C.P., Kwek, L.C., and Lim, K.M. (2022). Hand Gesture Recognition via Lightweight VGG16 and Ensemble Classifier. Appl. Sci., 12.10.3390/app12157643 ↗
43. Soft Voting-based Ensemble Model for Bengali Sign Gesture Recognition10.33166/AETiC.2022.02.003 ↗
44. Advanced Threat Detection Using Soft and Hard Voting Techniques in Ensemble Learning
45. (2013, May 02). Size Korea Report on the Fifth Survey of Korean Anthropometry. Available online: http://sizekorea.kats.go.kr/.
46. The Assessment and Analysis of Handedness: The Edinburgh Inventory10.1016/0028-3932(71)90067-4 ↗
47. Achilleas, M., Eleni, D., Paris-Alexandros, K., and Minas, D. (2017, January 28–30). Real-Time Detection of Suspicious Objects in Public Areas Using Computer Vision. Proceedings of the 21st Pan-Hellenic Conference on Informatics, Larissa, Greece.10.1145/3139367.3139429 ↗
48. Smys, S., Iliyasu, A.M., Bestak, R., and Shi, F. (2020). Hand Gesture Recognition Using OpenCv and Python. New Trends in Computational Vision and Bio-Inspired Computing, Springer International Publishing.10.1007/978-3-030-41862-5 ↗
49. OpenCV Team (2024, August 20). Open Source Computer Vision Library. Available online: http://opencv.org/.
50. Research on Hand Recognition Method Based on Markov Random Field10.1016/j.proeng.2017.01.170 ↗
51. Chung, H.Y., Chung, Y.L., and Tsai, W.F. (2019, January 13–15). An Efficient Hand Gesture Recognition System Based on Deep CNN. Proceedings of the 2019 IEEE International Conference on Industrial Technology (ICIT), Melbourne, VIC, Australia.10.1109/ICIT.2019.8755038 ↗
52. Chan, W.K. (2021). Vision-Based Hand Gesture Recognition for Human-Computer Interaction Using MobileNetV2. Proceedings of the 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain, 12–16 July 2021, IEEE.
53. Dong, K., Zhou, C., Ruan, Y., and Li, Y. (2020). MobileNetV2 Model for Image Classification. Proceedings of the 2020 2nd International Conference on Information Technology and Computer Application (ITCA), Guangzhou, China, 18–20 December 2020, IEEE.10.1109/ITCA52113.2020.00106 ↗
54. A Survey on Deep Learning in Medical Image Analysis10.1016/j.media.2017.07.005 ↗
55. Speaker Gender Recognition Based on Deep Neural Networks and ResNet5010.1155/2022/4444388 ↗
56. Pan, Y., Liu, J., Cai, Y., Yang, X., Zhang, Z., Long, H., Zhao, K., Yu, X., Zeng, C., and Duan, J. (2023). Fundus Image Classification Using Inception V3 and ResNet-50 for the Early Diagnostics of Fundus Diseases. Front. Physiol., 14.10.3389/fphys.2023.1126780 ↗
57. Research on the Practical Classification and Privacy Protection of CT Images of Parotid Tumors Based on ResNet50 Model10.1088/1742-6596/1576/1/012040 ↗
58. Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., and Zhong, N. Early Detection of Parkinson’s Disease from Micrographic Static Hand Drawings. Proceedings of the Brain Informatics 2021, Virtual Event, 17–19 September 2021.10.1007/978-3-030-86993-9 ↗
59. Afroze, T., Akther, S., Chowdhury, M.A., Hossain, E., Hossain, M.S., and Andersson, K. (2021, January 30–31). Glaucoma Detection Using Inception Convolutional Neural Network V3. Proceedings of the Applied Intelligence and Informatics 2021, Nottingham, UK.10.1007/978-3-030-82269-9_2 ↗
60. The Use of the Multi-Model Ensemble in Probabilistic Climate Projections10.1098/rsta.2007.2076 ↗
61. Lipton, Z.C., Elkan, C., and Naryanaswamy, B. (2014, January 15–19). Optimal Thresholding of Classifiers to Maximize F1 Measure. Proceedings of the Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2014, Nancy, France.10.1007/978-3-662-44851-9_15 ↗
62. Resolution Effect on the Stereological Estimation of Surface and Volume and its Interpretation in terms of Fractal Dimensions10.1111/j.1365-2818.1981.tb01198.x ↗
63. Vision-Based Human Grasp Reconstruction Inspired by Hand Postural Synergies10.1016/j.compeleceng.2017.10.018 ↗
64. A Face Detection Method Based on Skin Color Features and AdaBoost Algorithm10.1088/1742-6596/1748/4/042015 ↗
65. InceptionV3, ResNet50, ResNet18 and MobileNetV2 Performance Comparison on Face Recognition Classification
66. Zhang, L., Sun, M., and Pei, Y. (2023, January 23–28). Innovative Design of Aging-Friendly Household Cleaning Products from the Perspective of Ergonomics. Proceedings of the 25th International Conference on Human-Computer Interaction (HCII 2023), Copenhagen, Denmark.10.1007/978-3-031-48041-6_21 ↗
67. Low-frequency Background Estimation and Noise Separation from High-frequency for Background and Noise Subtraction10.1364/AO.507735 ↗
68. Kottha, B.R.T.N., Penumacha, N.K., and Gadde, H.V. (2024, January 18–19). Smart Traffic Management System using Background Subtraction. Proceedings of the 2024 International Conference on Expert Clouds and Applications (ICOECA), Bengaluru, India.10.1109/ICOECA62351.2024.00095 ↗