Examining the Usefulness of Quality Scores for Generating Learning Object Recommendations in Repositories of Open Educational Resources
Source abstract
Abstract
Open educational resources (OER) can contribute to democratize education by providing effective learning experiences with lower costs. Nevertheless, the massive amount of resources currently available in OER repositories makes it difficult for teachers and learners to find relevant and high-quality content, which is hindering OER use and adoption. Recommender systems that use data related to the pedagogical quality of the OER can help to overcome this problem. However, studies analyzing the usefulness of these data for generating OER recommendations are very limited and inconclusive. This article examines the usefulness of using pedagogical quality scores for generating OER recommendations in OER repositories by means of a user study that compares the following four different recommendation approaches: a traditional content-based recommendation technique, a quality-based non-personalized recommendation technique, a hybrid approach that combines the two previous techniques, and random recommendations. This user study involved 53 participants and 400 OER whose quality was evaluated by reviewers using the Learning Object Review Instrument (LORI). The main finding of this study is that pedagogical quality scores can enhance traditional content-based OER recommender systems by allowing them to recommend OER with more quality without detriment to relevance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Open Education and E-Learning · Recommender Systems and Techniques · Online Learning and Analytics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Departamento de Sistemas Informáticos, Universidad Politécnica de Madrid, 28031 Madrid, Spain
Department of Computer Science, KU Leuven, B-3001 Leuven, Belgium
References (71)
Show source-provided references
1. (2013). Open Educational Resources: Breaking the Lockbox on Education (White Paper), William and Flora Hewlett Foundation.
2. (2013). Opening Up Education: Innovative Teaching and Learning for all through new Technologies and Open Educational Resources, European Commission.
3. (2002). Forum on the Impact of Open Courseware for Higher Education in Developing Countries, UNESCO.
4. (2007). Giving Knowledge for Free: The Emergence of Open Educational Resources, OECD.
5. (2020, June 09). Wikimedia Commons. Available online: http://commons.wikimedia.org.
6. (2020, June 09). Europeana. Available online: http://www.europeana.eu.
7. (2020, June 09). OER Commons. Available online: http://www.oercommons.org.
8. (2020, June 09). MERLOT (Multimedia Educational Resource for Learning and Online Teaching). Available online: http://www.merlot.org.
9. (2020, June 09). Open Discovery Space Portal (ODS Portal). Available online: http://www.opendiscoveryspace.eu.
10. (2020, June 09). Learning Resource Exchange (LRE). Available online: http://lreforschools.eun.org.
11. The power of open: Benefits, barriers, and strategies for integration of open educational resources10.1080/02680513.2019.1677222 ↗
12. De los Arcos, B., Cirigottis, G.E., Egglestone, N., Farrow, R., McAndrew, P., Perryman, L.-A., Pitt, R., and Weller, M. (2015). OER Data Report 2013–2015: Building Understanding of Open Education, OER Research Hub.
13. Rauber, A., Christodoulakis, S., and Tjoa, A.M. (2005). Finding Appropriate Learning Objects: An Empirical Evaluation. Research and Advanced Technology for Digital Libraries: Proceedings of the 9th European Conference on Digital Libraries (ECDL 2005), Springer.10.1007/11551362 ↗
14. Comparative Analysis of Current Methods in Searching Open Education Content Repositories
15. Exploring Teachers Perceptions of Web-Based Learning Tools
16. Spector, J.M., Merrill, M.D., Elen, J., and Bishop, M.J. (2014). Open Educational Resources: A Review of the Literature. Handbook of Research on Educational Communications and Technology, Springer.10.1007/978-1-4614-3185-5 ↗
17. Open educational resources repositories literature review—Towards a comprehensive quality approaches framework10.1016/j.chb.2015.03.026 ↗
18. Kahin, B., and Foray, D. (2006). The impact of ICT on tertiary education: Advances and promises. Advancing Knowledge and the Knowledge Economy, MIT Press.10.7551/mitpress/1132.001.0001 ↗
19. Atkins, D.E., Brown, J.S., and Hammond, A.L. (2007). A Review of the Open Educational Resources (OER) Movement: Achievements, Challenges, and New Opportunities, Technical report to the William and Flora Hewlett Foundation.
20. User-oriented quality for OER: Understanding teachers’ views on re-use, quality, and trust10.1111/j.1365-2729.2011.00450.x ↗
21. Learning Object Evaluation: Computer-Mediated Collaboration And Inter-Rater Reliability
22. Taylor Northrup, P. (2007). An Overview of Learning Object Repositories. Learning Objects for Instruction: Design and Evaluation, IGI Global.10.4018/978-1-59904-334-0 ↗
23. A Framework for Evaluating the Quality of Multimedia Learning Resources
24. UNE 71362: Calidad de los materiales educativos digitales
25. Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT Evaluation Scale10.1016/j.chb.2011.04.007 ↗
26. Examining the Effectiveness of Web-Based Learning Tools in Middle and Secondary School Science Classrooms
27. Domínguez, E., de Armas, I., and Fernández-Pampillón, A. (2016). The COdA scoring rubric. Technology-Enhanced Language Learning for Specialized Domains: Practical Applications and Mobility, Routledge.
28. A multi-component model for assessing learning objects: The learning object evaluation metric (LOEM)10.14742/ajet.1192 ↗
29. Quality evaluation model for learning objects from pedagogical perspective. A case of study
30. Web-Based Tools for Collaborative Evaluation of Learning Resources
31. Luke, R., Mallory, D., Pinet, R., and Segiun, A. (2007, January 15). Evaluating Learning Objects with an Online Version of the Learning Object Review Instrument: Results of a Design Study. Proceedings of the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Montreal, QC, Canada.
32. Sarasa, A., Fernández-Pampillón, A., Rueda, A., and Riani, C. (2016, January 13–15). A web tool for assessment of the quality of digital educational materials. Proceedings of the 2016 International Symposium on Computers in Education (SIIE 2016), Salamanca, Spain.
33. Gordillo, A., Barra, E., and Quemada, J. (2014, January 22–25). A flexible open source web platform to facilitate learning object evaluation. Proceedings of the 44th Frontiers in Education Conference (FIE 2014), Madrid, Spain.10.1109/FIE.2014.7044498 ↗
34. Relevance Ranking Metrics for Learning Objects10.1109/TLT.2008.1 ↗
35. Ranking Metrics and Search Guidance for Learning Object Repository10.1109/TLT.2010.15 ↗
36. Ranking Learning Objects through Integration of Different Quality Indicators10.1109/TLT.2010.23 ↗
37. Ochoa, X., and Duval, E. (2006). Use of Contextualized Attention Metadata for Ranking and Recommending Learning Objects. Proceedings of the 1st International Workshop on Contextualized Attention Metadata: Collecting, Managing and Exploiting of Rich Usage Information, Association for Computing Machinery.10.1145/1183604.1183608 ↗
38. Ricci, F., Rokach, L., Shapira, B., and Kantor, P.B. (2011). Recommender Systems in Technology Enhanced Learning. Recommender Systems Handbook, Springer.10.1007/978-0-387-85820-3 ↗
39. Ricci, F., Rokach, L., and Shapira, B. (2015). Panorama of Recommender Systems to Support Learning. Recommender Systems Handbook, Springer.10.1007/978-1-4899-7637-6 ↗
40. Shakya, S., Prasad, P.W.C., Alsadoon, A., and Maag, A. (2019). A systematic review: Machine learning based recommendation systems for e-learning. Educ. Inf. Technol.
41. Ricci, F., Rokach, L., Shapira, B., and Kantor, P.B. (2011). Introduction to Recommender Systems Handbook. Recommender Systems Handbook, Springer.10.1007/978-0-387-85820-3 ↗
42. Recommender systems survey10.1016/j.knosys.2013.03.012 ↗
43. Gordillo, A. (2017). Contribution to the Authoring, Distribution, Evaluation and Integration of Learning Objects. [Ph.D. Thesis, Universidad Politécnica de Madrid].
44. Learning object repositories as knowledge management systems
45. A quantitative analysis of learning object repositories as knowledge management systems
46. Understanding and Using Context10.1007/s007790170019 ↗
47. Ricci, F., Rokach, L., Shapira, B., and Kantor, P.B. (2011). Context-Aware Recommender Systems. Recommender Systems Handbook, Springer.10.1007/978-0-387-85820-3 ↗
48. Context-aware recommender systems for learning: A survey and future challenges10.1109/TLT.2012.11 ↗
49. Use of Repositories of Digital Educational Resources. State of the Art Review
50. Clough, P., Otegi, A., Agirre, E., and Hall, M. (2014). Implementing Recommendations in the PATHS System. Theory and Practice of Digital Libraries—TPDL 2013 Selected Workshops, Springer International Publishing.10.1007/978-3-319-14226-5_17 ↗
51. Otegi, A., Agirre, E., and Clough, P. (2014, January 8–12). Personalised PageRank for Making Recommendations in Digital Cultural Heritage Collections. Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2014), London, UK.10.1109/JCDL.2014.6970149 ↗
52. Gordillo, A., Charleer, S., and Verbert, K. (2018, January 12–14). An open source hybrid learning object recommender system based on Europeana. Proceedings of the 11th International Conference of Education, Research and Innovation (ICERI 2018), Seville, Spain.10.21125/iceri.2018.0634 ↗
53. A hybrid recommendation model for learning object repositories10.1109/TLA.2017.7867596 ↗
54. Learning Object Recommendations for Teachers Based On Elicited ICT Competence Profiles10.1109/TLT.2015.2434824 ↗
55. Lemos, H., Cechinel, C., Giustozzi, F., Casali, A., and Deco, C. (2016, January 13–15). Users opinions about Learning Object Recommendations: A case study. Proceedings of the 2016 International Symposium on Computers in Education (SIIE), Salamanca, Spain.
56. A Framework of Semantic Recommender System for e-Learning10.17706/jsw.10.3.317-330 ↗
57. A framework for recommendation in learning object repositories: An example of application in civil engineering10.1016/j.advengsoft.2012.10.005 ↗
58. Evaluation and selection of group recommendation strategies for collaborative searching of learning objects10.1016/j.ijhcs.2014.12.002 ↗
59. A semantically enriched context-aware OER recommendation strategy and its application to a computer science OER repository10.1109/TE.2014.2309554 ↗
60. Evaluating collaborative filtering recommendations inside large learning object repositories10.1016/j.ipm.2012.07.004 ↗
61. Gallego, D., Barra, E., Aguirre, S., and Huecas, G. (2012, January 3–6). A Model for Generating Proactive Context-Aware Recommendations in e-Learning Systems. Proceedings of the 2012 Frontiers in Education Conference (FIE 2012), Seattle, WA, USA.10.1109/FIE.2012.6462246 ↗
62. Gallego, D., Barra, E., Rodríguez, P., and Huecas, G. (2013, January 22–24). Incorporating Proactivity to Context-Aware Recommender Systems for E-Learning. Proceedings of the 2013 World Congress on Computer and Information Technology (WCCIT 2013), Sousse, Tunisia.10.1109/WCCIT.2013.6618700 ↗
63. Gallego, D., Barra, E., Gordillo, A., and Huecas, G. (2013, January 23–26). Enhanced recommendations for e-learning authoring tools based on a proactive context-aware recommender. Proceedings of the 43rd Frontiers in Education Conference (FIE 2013), Oklahoma City, OK, USA.10.1109/FIE.2013.6685060 ↗
64. McNee, S.M., Riedl, J., and Konstan, J.A. (2006, January 22–27). Accurate is not always good: How Accuracy Metrics have hurt Recommender Systems. Proceedings of the Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006), Montreal, QC, Canada.
65. (2002). 1484.12.1-2002—IEEE Standard for Learning Object Metadata, The Institute of Electrical and Electronics Engineers.
66. (2020, June 09). ADL (Advanced Distributed Learning) Initiative SCORM, Available online: http://adlnet.gov/projects/scorm-2004-4th-edition.
67. Gordillo, A., Barra, E., and Quemada, J. (2014, January 22–25). Towards a learning object pedagogical quality metric based on the LORI evaluation model. Proceedings of the 44th Frontiers in Education Conference (FIE 2014), Madrid, Spain.10.1109/FIE.2014.7044499 ↗
68. Understanding inverse document frequency: On theoretical arguments for IDF10.1108/00220410410560582 ↗
69. Ricci, F., Rokach, L., Shapira, B., and Kantor, P.B. (2011). Evaluating Recommendation Systems. Recommender Systems Handbook, Springer.10.1007/978-0-387-85820-3 ↗
70. Mining Models for Automated Quality Assessment of Learning Objects
71. Estimación de calidad de objetos de aprendizaje en repositorios de recursos educativos abiertos basada en las interacciones de los estudiantes