journal article·2026-02-04Open access

DRIVE-T: A Methodology for Discriminative and Representative Items Selection for Design Quality Constructs and Assessments

Angela Locoro, Silvia Golia, Davide Falessi

Applied Sciences· Volume 16· Issue 3

Abstract

The lack of measurement constructs for both users’ literacy and artifacts difficulty in HCI hinders the quality of assessment tests. This paper proposes DRIVE-T (Discriminating and Representative Items for Validating Expressive Tests), a methodology designed to construct and evaluate items for measuring progressive levels of users’skills and artifacts usability. Given an artifact (a text, a data visualization, an interface prototype), DRIVE-T supports the identification of items discriminability and representativeness for measuring distinct traits of a quality property to be measured and its levels of progression in users and artifacts. DRIVE-T consists of three steps: (1) tagging task-based items associated with an artifact; (2) rating them by independent raters for their difficulty; (3) analyzing raters’ raw scores through a Many-Facet Rasch Measurement model. The emergence of difficulty levels of the measurement construct can be derived from the discriminability and representativeness of items for each artifact, ordered into Many-Facets construct levels. The DRIVE-T methodology operationalizes an inductive, practice-based measurement construct design. Based on the expertise of the authors in the data visualization domain, visualization literacy is the quality property of users that is exploited as a case scenario for applying DRIVE-T. Results from a pilot test show the validity of the approach.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Data Visualization and Analytics · Usability and User Interface Design · Persona Design and Applications

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Economics and Management, Universitá degli Studi di Brescia, Contrada Santa Chiara, 50, 25122 Brescia, Italy

Department of Informatics Engineering, Universitá di Roma Tor Vergata, Viale del Politecnico 1, 00133 Roma, Italy

Funding

European Union – Next-Generation, Mission 4 Component 1

References (47)

Show source-provided references
1. Barricelli, B.R., Fogli, D., and Locoro, A. (2023). EUDability: A new construct at the intersection of end-user development and computational thinking. J. Syst. Softw., 195.10.1016/j.jss.2022.111516 ↗
2. Islam, M.N., Khan, N.I., Inan, T.T., and Sarker, I.H. (2023). Designing user interfaces for illiterate and semi-literate users: A systematic review and future research agenda. Sage Open, 13.10.1177/21582440231172741 ↗
3. Ge, L.W., Cabouat, A.F., Bonilla, K., Cui, Y., Ding, Y., Rakotondravony, N., Creamer, M.M., Otto, J.T., Hedayati, M., and Kwon, B.C. (2025). An Autoethnography on Visualization Literacy: A Wicked Measurement Problem. IEEE Trans. Vis. Comput. Graph., 1–11.10.31219/osf.io/dfr4p_v1 ↗
4. Burns, A., Xiong, C., Franconeri, S., Cairo, A., and Mahyar, N. (2020). How to evaluate data visualizations across different levels of understanding. arXiv.10.1109/BELIV51497.2020.00010 ↗
5. A principled way of assessing visualization literacy10.1109/TVCG.2014.2346984 ↗
6. Characterizing Data Visualization Literacy for Standardization: A Systematic Literature Review10.1109/ACCESS.2025.3559298 ↗
7. Barricelli, B.R., Fogli, D., Gargioni, L., Locoro, A., and Valtolina, S. (2024, January 3–7). Towards the Unification of Computational Thinking and EUDability: Two Cases from Healthcare. Proceedings of the 2024 International Conference on Advanced Visual Interfaces, Arenzano, Italy.10.1145/3656650.3656671 ↗
8. Mari, L., Wilson, M., and Maul, A. (2023). Measurement Across the Sciences: Developing a Shared Concept System for Measurement, Springer Nature.10.1007/978-3-031-22448-5 ↗
9. Wilson, M. (2004). Constructing Measures: An Item Response Modeling Approach, Routledge.
10. Cabitza, F., and Locoro, A. (2014, January 21–24). “Made with Knowledge”—Disentangling the IT Knowledge Artifact by a Qualitative Literature Review. Proceedings of the International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, Rome, Italy. Available online: https://api.semanticscholar.org/CorpusID:28985187.10.5220/0005086100640075 ↗
11. Eco, U. (1979). A Theory of Semiotics, Indiana University Press. Available online: https://books.google.it/books?id=BoXO4ItsuaMC.
12. De Souza, C.S. (2005). The Semiotic Engineering of Human-Computer Interaction, MIT Press.10.7551/mitpress/6175.001.0001 ↗
13. The Digital Divide in Eye Health Care Delivery10.1001/jamaophthalmol.2024.0577 ↗
14. Institutional Ecology, ‘Translations’ and Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–3910.1177/030631289019003001 ↗
15. Otto, J.T., and Davidoff, S. (2024, January 14). Visualization Artifacts are Boundary Objects. Proceedings of the 2024 IEEE Evaluation and Beyond-Methodological Approaches for Visualization (BELIV), St. Pete Beach, FL, USA.10.1109/BELIV64461.2024.00014 ↗
16. Structural Validation for the Developmental Model of Computational Thinking10.1177/07356331211017794 ↗
17. Schneider, W.J., and McGrew, K.S. (2012). The Cattell-Horn-Carroll Model of Intelligence, Guilford Press. [3rd ed.].
18. Graph interpretation aspects of statistical literacy: A Japanese perspective10.1007/BF03217380 ↗
19. Graph Literacy: A Cross-Cultural Comparison10.1177/0272989X10373805 ↗
20. Ordinal scales and foundations of misinference
21. VLAT: Development of a Visualization Literacy Assessment Test10.1109/TVCG.2016.2598920 ↗
22. Krejci, S.E., Ramroop-Butts, S., Torres, H.N., and Isokpehi, R.D. (2020). Visual literacy intervention for improving undergraduate student critical thinking of global sustainability issues. Sustainability, 12.10.3390/su122310209 ↗
23. Visual information literacy: Definition, construct modeling and assessment10.1109/ACCESS.2021.3078429 ↗
24. Explaining with examples: Lessons learned from crowdsourced introductory description of information visualizations10.1109/TVCG.2021.3128157 ↗
25. Identifying deception as a critical component of visualization literacy10.1109/MCG.2021.3132004 ↗
26. P-lite: A study of parallel coordinate plot literacy10.1016/j.visinf.2022.05.002 ↗
27. Ge, L.W., Cui, Y., and Kay, M. (2023, January 23–28). CALVI: Critical Thinking Assessment for Literacy in Visualizations. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI’23), Hamburg, Germany.10.1145/3544548.3581406 ↗
28. The risks of ranking: Revisiting graphical perception to model individual differences in visualization performance10.1109/TVCG.2022.3226463 ↗
29. Promises and Pitfalls: Using Large Language Models to Generate Visualization Items10.1109/TVCG.2024.3456309 ↗
30. Making open data more personal through a social value perspective: A methodological approach10.1007/s10796-018-9854-7 ↗
31. Mini-VLAT: A Short and Effective Measure of Visualization Literacy10.1111/cgf.14809 ↗
32. Linacre, J.M. (1989). Many-Facet Rasch Measurement, MESA Press.
33. Eckes, T. (2009). Many-facet Rasch measurement. Reference Supplement to the Manual for Relating Language Examinations to the Common European Framework of Reference for Languages: Learning Teaching, Assessment, Council of Europe/Language Policy Division.
34. Detecting and Measuring Rater Effects Using Many-Facet Rasch Measurement: Part I
35. Eckes, T. (2015). Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments, Peter Lang Edition. [2nd rev. and updated ed.].
36. Assessing the inter-rater agreement for ordinal data through weighted indexes10.1177/0962280214529560 ↗
37. Andrich, D. (1988). Rasch Models for Measurement, Sage Publications.10.4135/9781412985598 ↗
38. Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests, University of Chicago Press.
39. A rating formulation for ordered response categories10.1007/BF02293814 ↗
40. A rasch model for partial credit scoring10.1007/BF02296272 ↗
41. Linacre, J.M. (2002). What do infit and outfit, mean-square and standardized means?. Rasch Meas. Trans., 16.
42. Construction of measures from many-facet data
43. Linacre, J.M. (2026, January 31). Facets Computer Program for Many-Facet Rasch Measurement. Available online: https://www.winsteps.com/facets.htm.
44. Linacre, J.M. (2026, January 31). Winsteps® Rasch Measurement Computer Program (Version 5.4.0). Available online: https://www.winsteps.com.
45. Bond, T.G., and Fox, C.M. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences, Routledge. [3rd ed.].
46. Nielsen, J. (1992, January 3–7). Finding usability problems through heuristic evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Monterey, CA, USA.10.1145/142750.142834 ↗
47. Nielsen, J. (1994). The Theory Behind Heuristic Evaluations, Nielsen Normal Group.