journal article·2024-02-22Open access

Usability Testing of Mobile Applications: A Methodological Framework

Paweł Weichbroth

Applied Sciences· Volume 14· Issue 5

Abstract

Less than five percent of all mobile applications have become successful throughout 2023. The success of a new mobile application depends on a variety of factors ranging from business understanding, customer value, and perceived quality of use. In this sense, the topic of usability testing of mobile applications is relevant from the point of view of user satisfaction and acceptance. However, the current knowledge seems to be fragmented, scattered across many papers and reports, and sometimes poorly documented. This paper attempts to fill this gap by investigating the current state of knowledge by reviewing the previous literature relevant to the research topic and developing a unified view. In particular, the methodological framework is outlined and discussed, including the discourse on settings for laboratory and field studies, data collection techniques, experimental designs for mobile usability testing, and a generic research framework. Therefore, the paper contributes to both the theory and practice of human–computer interaction by providing methodological foundations for usability testing of mobile applications, paving the way for further studies in this area. Moreover, the paper provides a better understanding of the related topics, in particular shedding light on methodological foundations, key concepts, challenges, and issues, equipping readers with a comprehensive knowledge base to navigate and contribute to the advancement of the field of mobile usability.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Usability and User Interface Design · Technology Adoption and User Behaviour · Software Engineering Techniques and Practices

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Software Engineering, Faculty of Electronics, Gdansk University of Technology, 80-233 Gdansk, Poland

Funding

internal resources of the Faculty of Electronics, Telecommunications and Informatics

References (100+)

Show source-provided references
1. Statista (2023, July 28). Forecast Number of Mobile Users Worldwide from 2020 to 2025. Available online: https://www.statista.com/statistics/218984/number-of-global-mobile-users-since-2010/.
2. Statista (2023, July 28). Time Spent with Nonvoice Activities on Mobile Phones Every Day in the United States from 2019 to 2024. Available online: https://www.statista.com/statistics/1045353/mobile-device-daily-usage-time-in-the-us/.
3. Elite Content Marketer (2023, July 28). Average Screen Time Statistics for 2023. Available online: https://elitecontentmarketer.com/screen-time-statistics/.
4. Statista (2023, July 28). Revenue from Smartphone Sales in the United States from 2013 to 2027. Available online: https://www.statista.com/statistics/619821/smartphone-sales-revenue-in-the-us/.
5. Admiral Media (2023, December 16). Why 99,5 percent of Consumer Apps Fail (And How To Keep Yours Alive). Available online: https://admiral.media/why-consumer-apps-fail-and-how-to-keep-yours-alive/.
6. Goyal, A. (2024, February 02). Top Reasons Why Mobile Apps Fail to Make a Mark in the Market. Available online: https://www.businessofapps.com/insights/top-reasons-why-mobile-apps-fail-to-make-a-mark-in-the-market/.
7. Swaid, S.I., and Suid, T.Z. (2019). Advances in Usability, User Experience and Assistive Technology, Proceedings of the AHFE 2018 International Conferences on Usability & User Experience and Human Factors and Assistive Technology, Orlando, FL, USA, 21–25 July 2018, Springer.
8. Tode, C. (2024, February 02). More than Half of Consumers Dissatisfied with Mobile Retail Experiences. Available online: https://www.retaildive.com/ex/mobilecommercedaily/more-than-half-of-shoppers-are-dissatisfied-with-mobile-retail-experiences-adobe.
9. Hedegaard, S., and Simonsen, J.G. (May, January 27). Extracting usability and user experience information from online user reviews. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.
10. An exploratory study of killer applications and critical success factors in m-commerce10.4018/jeco.2006070104 ↗
11. Usability dimensions for mobile applications-a review10.19026/rjaset.5.4776 ↗
12. Desak, G.F.P. (2020). Proceedings of the 2020 International Conference on Information Management and Technology (ICIMTech), Bandung, Indonesia, 13–14 August 2020, IEEE.
13. Au, F.T., Baker, S., Warren, I., and Dobbie, G. (2008, January 1). Automated usability testing framework. Proceedings of the Ninth Conference on Australasian User Interface, Darlinghurst, NSW, Australia.
14. Joshua, S.R., Abbas, W., Lee, J.H., and Kim, S.K. (2023). Trust Components: An Analysis in The Development of Type 2 Diabetic Mellitus Mobile Application. Appl. Sci., 13.10.3390/app13031251 ↗
15. Usability: Happier users mean greater profits10.1201/1078/43647.20.4.20030901/77295.10 ↗
16. Ahmad, W.F.W., Sulaiman, S., and Johari, F.S. (2010). Proceedings of the 2010 International Conference on User Science and Engineering (i-USEr), Shah Alam, Malaysia, 13–15 December 2010, IEEE.
17. Cliquet, G., Gonzalez, C., Huré, E., and Picot-Coupey, K. (2015). Ideas in Marketing: Finding the New and Polishing the Old, Proceedings of the 2013 Academy of Marketing Science (AMS) Annual Conference, Monterey Bay, CA, USA, 15–18 May 2013, Springer.
18. A meta-analytical review of empirical mobile usability studies
19. Thales (2023, December 17). The Evolution of the Smartphone. Available online: https://www.thalesgroup.com/en/worldwide-digital-identity-and-security/mobile/magazine/evolution-smartphone.
20. Standardization issues of mobile usability10.3991/ijim.v14i07.12129 ↗
21. The integrative review: Updated methodology10.1111/j.1365-2648.2005.03621.x ↗
22. Challenges, methodologies, and issues in the usability testing of mobile applications10.1207/s15327590ijhc1803_3 ↗
23. A usability checklist for the usability evaluation of mobile phone user interface10.1207/s15327590ijhc2003_3 ↗
24. A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors10.1016/j.intcom.2009.05.006 ↗
25. Hussain, A., and Kutar, M. (2009, January 22–23). Usability metric framework for mobile phone application. Proceedings of the PG Net’09: 10th Annual Conference on the Convergence of Telecommunications, Networking and Broadcasting, Liverpool, UK.10.1145/1497308.1497412 ↗
26. Detecting usability problems in mobile applications on the basis of dissimilarity in user behavior10.1016/j.ijhcs.2019.10.001 ↗
27. Usability of mobile applications: A systematic literature study10.1109/ACCESS.2020.2981892 ↗
28. (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018).
29. Owoc, M., Weichbroth, P., and Żuralski, K. (2017). Proceedings of the 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), Prague, Czech Republic, 3–6 September 2017, IEEE.
30. Yuan, H., Jin, T., and Ye, X. (2023). Establishment and Application of Crowd-Sensing-Based System for Bridge Structural Crack Detection. Appl. Sci., 13.10.3390/app13148281 ↗
31. Rafique, U., Khan, S., Ahmed, M.M., Kiani, S.H., Abbas, S.M., Saeed, S.I., Alibakhshikenari, M., and Dalarsson, M. (2022). Uni-planar MIMO antenna for sub-6 GHz 5G mobile phone applications. Appl. Sci., 12.10.3390/app12083746 ↗
32. Nakhimovsky, Y., Miller, A.T., Dimopoulos, T., and Siliski, M. (2010, January 10–15). Behind the scenes of google maps navigation: Enabling actionable user feedback at scale. Proceedings of the CHI’10 Extended Abstracts on Human Factors in Computing Systems, Atlanta, CA, USA.10.1145/1753846.1754052 ↗
33. Context awareness in mobile computing: A review
34. A survey of context simulation for testing mobile context-aware applications10.1145/3372788 ↗
35. (2024, January 20). Encyclopedia.com. Attribute. Available online: https://www.encyclopedia.com/science-and-technology/computers-and-electrical-engineering/computers-and-computing/attribute.
36. User testing on game usability, mobility, playability, and learning content of mobile game-based learning10.11113/jt.v77.6848 ↗
37. Beyond usability: The OoBE dynamics of mobile data services markets10.1007/s00779-004-0321-8 ↗
38. Experiencing visual usability and aesthetics in two mobile application contexts
39. Widyanti, A., and Ainizzamani, S.A.Q. (2017). Proceedings of the 2017 International Conference on Information Technology Systems and Innovation (ICITSI), Bandung, Indonesia, 23–24 October 2017, IEEE.
40. Usability of mobile applications: Literature review and rationale for a new usability model10.1186/2194-0827-1-1 ↗
41. Tracking the effectiveness of usability evaluation methods10.1080/014492997119789 ↗
42. Usability assessment of academic digital libraries: Effectiveness, efficiency, satisfaction, and learnability10.1515/LIBR.2005.96 ↗
43. Kabir, M.A., Salem, O.A., and Rehman, M.U. (2017). Proceedings of the 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 24–26 November 2017, IEEE.
44. Baumeister, R.F. (2007). Encyclopedia of Social Psychology, Sage.10.4135/9781412956253 ↗
45. Web credibility assessment: Conceptualization, operationalization, variability, and models10.1002/asi.23543 ↗
46. Carneiro, A. (2013). Handbook of Research on ICTs and Management Systems for Improving Efficiency in Healthcare and Social Care, IGI Global.
47. United States Agency for International Development (2024, January 20). Glossary of Evaluation Terms, Available online: https://pdf.usaid.gov/pdf_docs/PNADO820.pdf.
48. In defense of causal-formative indicators: A minority report10.1037/met0000056 ↗
49. On the nature and direction of relationships between constructs and measures10.1037/1082-989X.5.2.155 ↗
50. An empirical study on the impact of gender on mobile applications usability10.1109/ACCESS.2022.3219421 ↗
51. Cambridge Dictionary (2024, January 20). Satisfaction. Available online: https://dictionary.cambridge.org/dictionary/english/satisfaction.
52. To use or not to use, that is the question: Analysis of the determining factors for using NFC mobile payment systems in public transportation10.1016/j.techfore.2018.11.012 ↗
53. Exploring the influential factors in continuance usage of mobile social Apps: Satisfaction, habit, and customer value perspectives10.1016/j.tele.2015.08.014 ↗
54. Toward an understanding of university students’ continued intention to use MOOCs: When UTAUT model meets TTF model10.1177/2158244020941858 ↗
55. Usability heuristics as an assessment parameter: For performing usability testing
56. Usability testing: A case study10.5860/crl.60.6.552 ↗
57. Usability testing in 2000 and beyond10.1080/001401300409170 ↗
58. Lewis, J.R. (2012). Handbook of Human Factors and Ergonomics, John Wiley & Sons.
59. Usability testing10.1002/9781118976005.ch14 ↗
60. Mason, P., and Plimmer, B. (2005). A critical comparison of usability testing methodologies. NACCQ, 255–258. Available online: https://www.cs.auckland.ac.nz/~beryl/publications/NACCQ%202005%20Critical%20Comparison%20of%20Usability%20Testing%20Methodologies.pdf.
61. Gaffney, G. (2023, July 28). Information & Design Designing for Humans. Available online: https://infodesign.com.au/assets/UsabilityTesting.pdf.
62. Dumas, J.S., and Redish, J. (1999). A Practical Guide to Usability Testing, Intellect Books.
63. Extra-laboratorial usability tests: An empirical comparison of remote and classical field testing with lab testing10.1016/j.apergo.2018.08.011 ↗
64. Betiol, A.H., and de Abreu Cybis, W. (2005). Proceedings of the IFIP Conference on Human-Computer Interaction, Orlando, FL, USA, 9–14 July 2005, Springer.
65. Gawlik-Kobylińska, M., Kabashkin, I., Misnevs, B., and Maciejewski, P. (2023). Education Mobility as a Service: A Study of the Features of a Novel Mobility Platform. Appl. Sci., 13.10.3390/app13095245 ↗
66. Dehlinger, J., and Dixon, J. (2011, January 22–24). Mobile application software engineering: Challenges and research directions. Proceedings of the Workshop on Mobile Software Engineering, Honolulu, HI, USA.
67. Schusteritsch, R., Wei, C.Y., and LaRosa, M. (May, January 28). Towards the perfect infrastructure for usability testing on mobile devices. Proceedings of the CHI’07 extended abstracts on Human Factors in Computing Systems, San Jose, CA, USA.
68. The emerging field of health engineering10.1080/14639220500169531 ↗
69. The usability of passphrases for authentication: An empirical field study10.1016/j.ijhcs.2006.08.005 ↗
70. Comparison between field research and controlled laboratory research10.26502/acbr.50170011 ↗
71. Harbach, M., Von Zezschwitz, E., Fichtner, A., De Luca, A., and Smith, M. (2014, January 9–1). It’s a hard lock life: A field study of smartphone (Un)Locking behavior and risk perception. Proceedings of the 10th Symposium on Usable Privacy and Security (SOUPS 2014), Menlo Park, CA, USA.
72. Pousttchi, K., and Thurnher, B. (2006). Proceedings of the 2006 International Conference on Mobile Business, Copenhagen, Denmark, 26–27 June 2006, IEEE.
73. Field-based usability evaluation methodology for mobile geo-applications10.1179/174327708X305139 ↗
74. Evaluating the usability of a mobile guide: The influence of location, participants and resources10.1080/01449290512331319030 ↗
75. Context-aware mobile learning system: Usability assessment based on a field study10.1016/j.tele.2020.101346 ↗
76. Rowley, D.E. (1994, January 24–28). Usability testing in the field: Bringing the laboratory to the user. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.10.1145/191666.191758 ↗
77. Usability testing of mobile applications: A comparison between laboratory and field testing
78. Nielsen, C.M., Overgaard, M., Pedersen, M.B., Stage, J., and Stenild, S. (2006, January 14–18). It’s worth the hassle! the added value of evaluating the usability of mobile systems in the field. Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, Oslo, Norway.10.1145/1182475.1182504 ↗
79. Duh, H.B.L., Tan, G.C., and Chen, V.H.h. (2006, January 2–5). Usability evaluation for mobile device: A comparison of laboratory and field tests. Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services, Amsterdam, The Netherlands.10.1145/1152215.1152254 ↗
80. Nayebi, F., Desharnais, J.M., and Abran, A. (2012). Proceedings of the 2012 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), Montreal, QC, Canada, 29 April–2 May 2012, IEEE.
81. Kaya, A., Ozturk, R., and Altin Gumussoy, C. (2019). Proceedings of the Industrial Engineering in the Big Data Era: Selected Papers from the Global Joint Conference on Industrial Engineering and Its Application Areas, GJCIE 2018, Nevsehir, Turkey, 21–22 June 2018, Springer.
82. Suzuki, S., Bellotti, V., Yee, N., John, B.E., Nakao, Y., Asahi, T., and Fukuzumi, S. (2011, January 7–12). Variation in importance of time-on-task with familiarity with mobile phone models. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada.10.1145/1978942.1979314 ↗
83. Kugler, S., Czwick, C., and Anderl, R. (2019). Proceedings of the Product Lifecycle Management in the Digital Twin Era: 16th IFIP WG 5.1 International Conference, PLM 2019, Moscow, Russia, 8–12 July 2019, Springer. Revised Selected Papers 16.
84. Understanding the most influential user experiences in successful and unsuccessful technology adoptions10.1016/j.chb.2015.07.012 ↗
85. The influence of age in usability testing10.1016/j.apergo.2015.06.012 ↗
86. Usability testing of fitness mobile application: Methodology and quantitative results
87. Ahmad, N.A.N., and Hussaini, M. (2021). A Usability Testing of a Higher Education Mobile Application Among Postgraduate and Undergraduate Students. Int. J. Interact. Mob. Technol., 15.10.3991/ijim.v15i09.19943 ↗
88. Cambridge Dictionary (2023, August 06). Meaning of Questionnaire in English. Available online: https://dictionary.cambridge.org/dictionary/english-polish/questionnaire.
89. Cambridge Dictionary (2023, August 06). Meaning of Survey in English. Available online: https://dictionary.cambridge.org/dictionary/english-polish/survey.
90. Nielsen Norman Group (2023, August 06). Open-Ended vs. Closed-Ended Questions in User Research. Available online: https://www.nngroup.com/articles/open-ended-questions/.
91. Sauro, J., and Kindlund, E. (2005, January 2–7). A method to standardize usability metrics into a single score. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Portland, OR, USA.10.1145/1054972.1055028 ↗
92. Understanding and evaluating survey research
93. Preissle, J. (2010). The Corsini Encyclopedia of Psychology, University of Michigan.
94. Participant observation of a farmers-herders community in Anguwar Jaba Keffi Nigeria
95. Musante, K., and DeWalt, B.R. (2010). Participant Observation as a Data Collection Method, Rowman Altamira.
96. Research in context: Ethnographic usability10.2190/DPQF-VG74-1HQA-L2F9 ↗
97. Kawulich, B.B. (2005). Participant observation as a data collection method. Forum Qual. Sozialforschung Forum Qual. Soc. Res., 6.
98. Jorgensen, D.L. (1989). Participant Observation: A Methodology for Human Studies, Sage.10.4135/9781412985376 ↗
99. Tomlin, W.C., and Tomlin, W.C. (2018). UX Optimization: Combining Behavioral UX and Usability Testing Data to Optimize Websites, Apress.10.1007/978-1-4842-3867-7 ↗
100. Mohamad, U.H., Abdul Hakim, I.N., and Ali-Akbari, M. (2022). Usability of a gamified antibiotic resistance awareness mobile application: A qualitative evaluation. IET Netw.10.1049/ntw2.12075 ↗