Usability Textual Data Analysis: A Formulaic Coding Think-Aloud Protocol Method for Usability Evaluation
Source abstract
Abstract
Think-aloud protocols are among the most standard methods for usability evaluation, which help to discover usability problems and to examine improvements because they provide direct information on a user’s thinking and cognitive processes; however, it is often difficult to determine how to analyze the data to identify usability problems because there is no formulaic analysis procedure for textual data. Therefore, the analysis is time-consuming, and the quality of the results varies depending on an analyst’s skills. In the present study, the author proposes a formulaic analysis think-aloud protocol method that specifies the procedure for analyzing participants’ verbal responses during usability tests. The aim of the proposed think-aloud protocol method was to deliver an explicit procedure using step coding (SCAT) and 70 design items for textual data analysis, and then, the method was applied to a case study of usability evaluation to confirm that the method could extract the target system’s problems. By using step coding and 70 design items, the process of extracting usability problems from textual data was made explicit, and the problems were extracted analytically. In other words, the proposed method was less ambiguous. Once a formulaic analysis procedure was established, textual data analysis could be performed easily and efficiently. The analysis could be performed without hesitation after data acquisition, and there were fewer omissions. In addition, it is expected that the procedure would be easy to use, even for novice designers.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Usability and User Interface Design
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Intelligent Mechanical Systems, Okayama University, Okayama 700-8530, Japan
Funding
Japan Society for the Promotion of Science
References (46)
Show source-provided references
1. Nielsen, J. (1993). Usability Engineering, Academic Press, Inc.10.1016/B978-0-08-052029-2.50007-3 ↗
2. Lewis, C. (1982). Using the “Thinking-Aloud” Method in Cognitive Interface Design, IBM TJ Watson Research Center.
3. Dumas, J.S., and Redish, J. (1999). A Practical Guide to Usability Testing, Intellect Books.
4. Preece, J., Rogers, Y., and Sharp, H. (2015). Interaction Design: Beyond Human-Computer Interaction, Wiley.
5. Rubin, J., and Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests, John Wiley & Sons.
6. Exploring think-alouds in usability testing: An international survey10.1109/TPC.2011.2182569 ↗
7. Practices and challenges of using think-aloud protocols in industry: An international survey
8. Dogmas in the assessment of usability evaluation methods10.1080/01449290801939400 ↗
9. Verbal reports as data10.1037/0033-295X.87.3.215 ↗
10. Ericsson, K.A., and Simon, H. (1984). Protocol Analysis: Verbal Reports as Data, MIT Press.
11. Alhadreti, O., and Mayhew, P. (2018, January 21–26). Rethinking thinking aloud: A comparison of three think-aloud protocols. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI2018), Montreal, QC, Canada.10.1145/3173574.3173618 ↗
12. Ericsson, K.A., and Simon, H. (1993). Protocol Analysis: Verbal Reports as Data, MIT Press. [Revised ed.].10.7551/mitpress/5657.001.0001 ↗
13. Alekhya, P. (2012). Think Aloud: Can Eye Tracking Add Value in Detecting Usability Problems?. [Master’s Thesis, Rochester Institute of Technology].
14. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support10.1016/j.ijmedinf.2012.02.009 ↗
15. Thinking aloud: Reconciling theory and practice10.1109/47.867942 ↗
16. To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing
17. Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach10.1109/TPC.2010.2052859 ↗
18. Automatic detection of usability problem encounters in think-aloud sessions10.1145/3385732 ↗
19. Kawakita, J. (1967). Hasso ho (Methodology for Ideation), Chuko Shinsho. (In Japanese).
20. Glaser, B.G., and Strauss, A.L. (1967). The Discovery of Grounded Theory. Strategies for Qualitative Research, Routledge.10.1097/00006199-196807000-00014 ↗
21. Krippendorff, K. (1981). Content Analysis: An Introduction to Its Methodology, SAGE Publications.
22. Gilbert, N., and Mulkay, M.J. (1984). Opening Pandora’s Box: A Sociological Analysis of Scientists’ Discourse, Cambridge University Press.
23. Richardson, J.T.E. (1996). Discourse Analysis and Constructionist Approaches: Theoretical Background. Handbook of Qualitative Research Methods for Psychology and the Social Sciences, BPS Books.
24. Beyond the divide between cognition and discourse: Using interpretative phenomenological analysis in health psychology10.1080/08870449608400256 ↗
25. Šašinka, Č., Stachoň, Z., Sedlák, M., Chmelík, J., Herman, L., Kubíček, P., Šašinková, A., Doležal, M., Tejkl, H., and Urbánek, T. (2019). Collaborative Immersive Virtual Environments for Education in Geography. ISPRS Int. J. Geoinf., 8.10.3390/ijgi8010003 ↗
26. Kawakita, J. (1985). Hasso ho (KJ ho) to dezain (KJ Method and Design). Bull. JSSD, 2–8. (In Japanese).
27. Markopoulos, E., Goonetilleke, R.S., Ho, A.G., and Luximon, Y. (2021). Beyond IDEO’s Design Thinking: Combining KJ Method and Kansei Engineering for the Creation of Creativity. Advances in Creativity, Innovation, Entrepreneurship and Communication of Design, Springer. AHFE 2021; Lecture Notes in Networks and Systems.
28. User Design and Experience Preferences in a Novel Smartphone Application for Migraine Management: A Think Aloud Study of the RELAXaHEAD Application10.1093/pm/pny080 ↗
29. Villanueva, R.A. (2004). Think-Aloud Protocol and Heuristic Evaluation of Non-Immersive, Desktop Photo-Realistic Virtual Environments. [Master’s Thesis, University of Otago]. Available online: http://hdl.handle.net/10523/1324.
30. Medical students’ and residents’ conceptual structure of empathy: A qualitative study10.4103/1357-6283.112793 ↗
31. Japanese students’ perception of their learning from an interprofessional education program: A qualitative study10.5116/ijme.50e5.e29a ↗
32. “SCAT” A qualitative data analysis method by four-step coding: Easy startable and small data-applicable process of theorization
33. Goto, A., Rudd, R.E., Lai, A.Y., Yoshida, K., Suzuki, Y., Halstead, D.D., Yoshida-Komiya, H., and Reich, M.R. (2014). Leveraging public health nurses for disaster risk communication in Fukushima City: A qualitative analysis of nurses’ written records of parenting counseling and peer discussions. BMC Health Serv. Res., 14.10.1186/1472-6963-14-129 ↗
34. Karwowski, W., Soares, M.M., and Stanton, N.A. (2011). Manufacturing attractive products logically by using human design technology: A case of Japanese methodology. Human Factors and Ergonomics in Consumer Product Design: Methods and Techniques, CRC Press.
35. (2021, July 19). Google’s Design Guidelines. Available online: http://googlesystem.blogspot.com/2008/03/googles-design-guidelines.html.
36. (2021, July 19). Apple Human Interface Guidelines. Available online: https://developer.apple.com/design/human-interface-guidelines/.
37. (2021, July 19). IBM Design Language. Available online: https://www.ibm.com/design/language/philosophy/principles/.
38. Okada, K. (2002). Human Computer Interaction, Ohmsha.10.1145/512526.512528 ↗
39. Rubinstein, R., and Hersh, H.M. (1984). The Human Factor—Designing Computer System for People, Digital Press.
40. (2021, July 19). GNOME Human Interface Guidelines. Available online: https://developer.gnome.org/hig/stable/.
41. (2021, July 19). Windows User Experience Interaction Guidelines. Available online: http://www.glyfx.com/useruploads/files/UXGuide.pdf.
42. Saffer, D. (2008). Designing for Interaction, Mainichi Communications.
43. Research Group on Universal Design (2008). Human Engineering and Universal Design, Japan Industrial Publishing.
44. Lidwell, W., Holden, K., and Butler, J. (2003). Universal Principles of Design: 100 Ways to Enhance Usability, Influence Pereception, Increase Appeal, Make Better Design Decisions, and Teach Through Design, Rockport Pub.
45. A human interface design checklist and its effectiveness
46. Yamaoka, T. (2012, January 18–20). How to construct form logically based on human design technology and form construction principles. Proceedings of the 2nd International Conference on Design Creativity, Glasgow, UK.