Usability Evaluations Employing Online Panels Are Not Bias-Free
Source abstract
Abstract
A growing trend in UX research is the use of Online Panels (OPs), namely people enrolled in a web platform who have agreed to participate regularly in online studies and/or in the execution of simple and repetitive operations. The effect of the participation of such “professional respondents” on data quality has been questioned in a variety of fields (e.g., Psychology and Marketing). Notwithstanding the increasing use of OPs in UX research, there is a lack of studies investigating the bias affecting usability assessments provided by this type of respondents. In this paper we have addressed this issue by comparing the usability evaluations provided by professional respondents commonly involved in debugging activities, non-professional respondents, and naive people not belonging to any OP. In a set of three studies, we have addressed both the effect of expertise and type of task (debugging vs. browsing) on the usability assessments. A total of 138 individuals participated in these studies. Results showed that individuals who performed the debugging test provided more positive usability ratings regardless of their skills, conversely, professional respondents provided more severe and critical ratings of perceived usability than non-professionals. Finally, the comparison between the online panelists and naive users allowed us to better understand whether professional respondents can be involved in usability evaluations without jeopardizing them.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Survey Methodology and Nonresponse · Technology Adoption and User Behaviour · Focus Groups and Qualitative Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Ergoproject, 00146 Rome, Italy
Unguess, 20124 Milan, Italy
Department of Planning, Design, and Technology of Architecture, Sapienza University of Rome, 00196 Rome, Italy
References (41)
Show source-provided references
1. Batinic, B., Reips, U.-D., and Bosnjak, M. (2002). Online Panels. Online Social Sciences, Hogrefe & Huber.
2. Comley, P. (2007, January 24–26). Online panels are unsustainable: Online communities are the future–BBC World Service Case Study. Proceedings of the ESOMAR Panel Research Annual Conference, Orlando, FL, USA.
3. Web Panels: Replacement Technology for Market Research
4. Postoaca, A. (2006). The Anonymous Elect: Market Research through Online Access Panels, Springer.
5. Lynn, P. (2008). Recruitment and Retention for a Consumer Panel. Methodology of Longitudinal Surveys, John Wiley & Sons. Inc.
6. Joinson, A.N., McKenna, K.Y.A., Postmes, T., and Reips, U.-D. (2007). Using Online Panels in Psychological Research. The Oxford Handbook of Internet Psychology, Oxford University Press.
7. Issues of representation in eHealth research (with a focus on web surveys)10.1016/j.amepre.2007.01.017 ↗
8. MTurk research: Review and recommendations
9. The Use of Online Panel Data in Management Research: A Review and Recommendations
10. An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use10.1177/1745691617706516 ↗
11. All that is solid: Bench-building at the frontiers of two experimental sciences10.1177/0003122415607230 ↗
12. Flaherty, C. (2022, April 01). Faculty Home Work. Inside Higher Ed. 24 March 2020. Available online: https://www.insidehighered.com/news/2020/03/24/working-home-during-covid-19-proves-challenging-faculty-members.
13. Internet panels, professional respondents, and data quality
14. Incentives in web surveys: Methodological issues and a review
15. Whitsett, H.C. (2022, April 01). Understanding Frequent Survey Responders on Online Panels. Nera Economic Consulting. 2013. Available online: https://www.nera.com/content/dam/nera/publications/archive2/PUB_Frequent_Survey_Responders_0313.pdf.
16. Comley, P. (2005). Understanding the Online Panelist. Worldwide Panel Research: Developments and Progress, ESOMAR.
17. Conrad, F.G., Tourangeau, R., Couper, M.P., and Zhang, G. (2010, January 13–16). Professional web respondents and data quality. Proceedings of the 2010 AAPOR Conference, Chicago, IL, USA.
18. Gittelman, S., and Trimarchi, E. (2022, April 01). Variance between Purchasing Behaviour Profiles in a Wide Spectrum of Online Sample Sources. Available online: https://www.mktginc.com/pdf/Short_%20Variance.pdf.
19. Using non-probability samples for confusion surveys-mall intercepts and the Internet10.29115/SP-2009-0031 ↗
20. An evaluation of a cognitive theory of response order effects in survey measurement10.1086/269029 ↗
21. Response strategies for coping with the cognitive demands of attitude measures in surveys10.1002/acp.2350050305 ↗
22. Tourangeau, R., Rips, L.J., and Rasinski, K. (2000). The Psychology of Survey Response, Cambridge University Press.10.1017/CBO9780511819322 ↗
23. Callegaro, M., Baker, R., Bethlehem, J., Göritz, A.S., Krosnick, J.A., and Lavrakas, P.J. (2014). Professional Respondents in Nonprobability Online Panels. Online Panel Research: A Data Quality Perspective, Wiley.10.1002/9781118763520 ↗
24. Professional Respondents in Opt-in Online Panels: What Do We Really Know?10.1177/0894439319845102 ↗
25. Schirra, S., and Allison, C. (2018, January 21–26). “I know what the companies are wanting more of”: Professional Participants in Online Usability Studies. Proceedings of the Extended Abstracts of 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3170427.3188681 ↗
26. Law, E.L.-C., Roto, V., Hassenzahl, M., Vermeeren, A.P., and Kort, J. (2009, January 4–9). Understanding, scoping and defining user experience: A survey approach. ACM. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.10.1145/1518701.1518813 ↗
27. (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018).
28. Luther, L., Tiberius, V., and Brem, A. (2020). User Experience (UX) in Business, Management, and Psychology: A Bibliometric Mapping of the Current State of Research. Multimodal Technol. Interact., 4.10.3390/mti4020018 ↗
29. The one number you need to grow
30. Reichheld, F. (2006). The Ultimate Question: Driving Good Profits and True Growth, Harvard Business School Press.
31. Di Nocera, F. (2013). Usability Evaluation 2.0: Una Descrizione (S)Oggettiva Dell’Usabilità, Ergoproject.
32. Website complexity and usability: Is there a role for mental workload?10.1504/IJHFE.2022.122393 ↗
33. Brooke, J. (1986). System Usability Scale (SUS): A Quick-And-Dirty Method of System Evaluation User Information, Digital Equipment Co., Ltd.
34. The Usability Metric for User Experience10.1016/j.intcom.2010.04.004 ↗
35. An experimental evaluation of instantaneous self-assessment as a measure of workload10.1080/00140139608964495 ↗
36. American association of public opinion researchers report on online panels
37. Respondent conditioning in online panel surveys: Results of two field experiments10.1177/0894439315574022 ↗
38. Panel Conditioning in Difficult Attitudinal Questions10.1093/poq/nft030 ↗
39. Duncan, G., Kalton, G., Kasprzyk, D., and Singh, M.P. (1989). Evidence of Conditioning Effects in the British Attitudes Panel Survey. Panel Surveys, Wiley.
40. Data quality of platforms and panels for online behavioral research10.3758/s13428-021-01694-3 ↗
41. The role of expertise research and human factors in capturing, explaining, and producing superior performance10.1518/001872008X312206 ↗