A Social-Network-Based Crowd Selection Approach for Crowdsourcing Mobile Apps Requirements Engineering Tasks
Source abstract
Abstract
Mobile apps have revolutionized almost every aspect of our daily lives, shaping the way we shop, learn and work. The transformative and unprecedented impact they have made on our lifestyle and the convenience they have offered have increased their adoption in diverse domains. Therefore, it is of paramount importance to hear from the interested audience about their desires and requirements in mobile apps. This has stressed the need to employ crowdsourcing in requirements engineering (RE) activities to harness the scattered talent in the crowd. RE tasks require certain software domain knowledge, hence, selecting a suitable subset of the crowd is crucial to obtain high-quality contributions. For that, we propose a crowd selection approach for crowdsourcing mobile app requirements engineering tasks which leverages the untapped crowd available on the social network Twitter (recently changed to X). This article is an extension of our previous work, where we present the proposed social-network-based crowd selection approach design, continue to work on the remaining component of the approach and evaluate the approach through a controlled experiment. For evaluation, the approach was utilized to select a real crowd that were invited to contribute to crowdsourcing requirements elicitation tasks for a fitness mobile app. The quality of the crowdsourced requirements was assessed by experts and the results have provided encouraging and compelling insights about the effectiveness of the proposed approach. The obtained assessment scores for the five quality factors clarity, creativity, relatedness, feasibility and diversity were respectively 4.36, 4.01, 4.29, 4.45 and 4.43 out of 5. Overall, we believe that the proposed social-network-based crowd selection approach could help in eliciting mobile app requirements and features that could cater to the needs of a large audience.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Mobile Crowdsensing and Crowdsourcing · Open Source Software Innovations · Green IT and Sustainability
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Systems, College of Computer and Information Sciences, King Saud University, Riyadh 11421, Saudi Arabia
Department of Information Systems, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, Riyadh 11564, Saudi Arabia
References (96)
Show source-provided references
1. Towards crowd-based requirements engineering a research preview
2. Johann, T., and Maalej, W. (2015, January 24–28). Democratic mass participation of users in Requirements Engineering?. Proceedings of the 2015 IEEE 23rd International Requirements Engineering Conference (RE), Ottawa, ON, Canada.10.1109/RE.2015.7320433 ↗
3. Sharma, R., and Sureka, A. (2017, January 10–12). CRUISE: A platform for crowdsourcing Requirements Elicitation and evolution. Proceedings of the 2017 10th International Conference on Contemporary Computing (IC3), Noida, India.10.1109/IC3.2017.8284308 ↗
4. Evaluation on crowdsourcing research: Current status and future direction10.1007/s10796-012-9350-4 ↗
5. Srivastava, P.K., and Sharma, R. (2015, January 25). Crowdsourcing to elicit requirements for MyERP application. Proceedings of the 2015 IEEE 1st International Workshop on Crowd-Based Requirements Engineering (CrowdRE), Ottawa, ON, Canada.10.1109/CrowdRE.2015.7367586 ↗
6. Framework and Literature Analysis for Crowdsourcing’s Answer Aggregation
7. Wang, H., Wang, Y., and Wang, J. (2014, January 18–21). A participant recruitment framework for crowdsourcing based software requirement acquisition. Proceedings of the 2014 IEEE 9th International Conference on Global Software Engineering, Shanghai, China.10.1109/ICGSE.2014.26 ↗
8. Alamer, G., and Alyahya, S. (2022, January 4–6). A Proposed Approach to Crowd Selection in Crowdsourced Requirements Engineering for Mobile Apps. Proceedings of the ICISE 2022: 2022 7th International Conference on Information Systems Engineering, Charleston, SC, USA.10.1145/3573926.3573927 ↗
9. Alamer, G., Alyahya, S., and Al-Dossari, H. (2023). Identifying Users and Developers of Mobile Apps in Social Network Crowd. Electronics, 12.10.3390/electronics12163422 ↗
10. A Survey on Task Assignment in Crowdsourcing
11. Selection of Crowd in Crowdsourcing for Smart Intelligent Applications: A Systematic Mapping Study
12. An Efficient Decision Support System for the Selection of Appropriate Crowd in Crowdsourcing10.1155/2021/5518878 ↗
13. Collecting samples from online services: How to use screeners to improve data quality10.1016/j.ijresmar.2021.05.001 ↗
14. Multi-task crowdsourcing via an optimization framework10.1145/3310227 ↗
15. Increasing the Reliability of Crowdsourcing Evaluations Using Online Quality Assessment10.1109/TAFFC.2015.2493525 ↗
16. Kazai, G., Kamps, J., and Milic-Frayling, N. (November, January 29). The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. Proceedings of the CIKM’12: 21st ACM International Conference on Information and Knowledge Management, Maui, HI, USA.
17. Using Attention Testing to Select Crowdsourced Workers and Research Participants10.1177/0894439319848726 ↗
18. Difallah, D.E., Demartini, G., and Cudré-mauroux, P. (2012, January 17). Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms. Proceedings of the CrowdSearch 2012 Workshop at WWW 2012, Lyon, France.
19. From Crowd Ratings to Predictive Models of Newsworthiness to Support Science Journalism10.1145/3555542 ↗
20. Mok, R.K.P., Kawaguti, G., and Okamoto, J. (2020, January 16). Improving the Efficiency of QoE Crowdtesting. Proceedings of the 1st Workshop on Quality of Experience (QoE) in Visual Multimedia Applications (QoEVMA 2020), Seattle, WA, USA.10.1145/3423328.3423499 ↗
21. Reimann, M., Wegen, O., Pasewaldt, S., Semmo, A., Döllner, J., and Trapp, M. (2021, January 3–7). Teaching data-driven video processing via crowdsourced data collection. Proceedings of the 42nd Annual Conference of the European Association for Computer Graphics (Eurographics 2021), Vienna, Austria.
22. Argument Mining in Tweets: Comparing Crowd and Expert Annotations for Automated Claim and Evidence Detection10.1007/978-3-030-80599-9_25 ↗
23. Weidema, E.R.Q., López, C., Nayebaziz, S., Spanghero, F., and Van Der Hoek, A. (2016, January 16). Toward Microtask crowdsourcing software design work. Proceedings of the 3rd International Workshop on CrowdSourcing in Software Engineering, Austin, TX, USA.10.1145/2897659.2897664 ↗
24. Thierry, C., Dubois, J.C., Le Gall, Y., and Martin, A. (2019, January 4–6). Modeling uncertainty and inaccuracy on data from crowdsourcing platforms: MONITOR. Proceedings of the 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), Portland, OR, USA.10.1109/ICTAI.2019.00112 ↗
25. CrowdCog A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing10.1145/3415181 ↗
26. GRBMC: An effective crowdsourcing recommendation for workers groups10.1016/j.eswa.2021.115039 ↗
27. Characterizing Crowds to Better Optimize Worker Recommendation in Crowdsourced Testing10.1109/TSE.2019.2918520 ↗
28. Qiu, C., Squicciarini, A.C., Carminati, B., Caverlee, J., and Khare, D.R. (2016, January 24–28). CrowdSelect: Increasing Accuracy of Crowdsourcing Tasks through Behavior Prediction and User Selection. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, Indianapolis, IN, USA.10.1145/2983323.2983830 ↗
29. Awwad, T., Bennani, N., Ziegler, K., Sonigo, V.R., Brunie, L., Kosch, H., Awwad, T., Bennani, N., Ziegler, K., and Sonigo, V.R. (2017, January 4–8). Efficient Worker Selection Through History-based Learning in Crowdsourcing. Proceedings of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Turin, Italy.10.1109/COMPSAC.2017.275 ↗
30. Fang, X., Si, S., Sun, G., Sheng, Q.Z., Wu, W., Wang, K., and Lv, H. (2022). Selecting Workers Wisely for Crowdsourcing When Copiers and Domain Experts Co-exist. Future Internet, 14.10.3390/fi14020037 ↗
31. Ben Rjab, A., Kharoune, M., Miklos, Z., and Martin, A. (2016). Characterization of experts in crowdsourcing platforms. Belief Functions: Theory and Applications, Springer.10.1007/978-3-319-45559-4_10 ↗
32. A Framework for Optimal Worker Selection in Spatial Crowdsourcing Using Bayesian Network10.1109/ACCESS.2020.3005543 ↗
33. Jarrett, J., and Blake, M.B. (July, January 27). Using collaborative filtering to automate worker-job recommendations for crowdsourcing services. Proceedings of the 2016 IEEE International Conference on Web Services (ICWS), San Francisco, CA, USA.10.1109/ICWS.2016.89 ↗
34. Cui, Q., Wang, J., Yang, G., Xie, M., Wang, Q., and Li, M. (2017, January 4–8). Who Should Be Selected to Perform a Task in Crowdsourced Testing?. Proceedings of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Turin, Italy.10.1109/COMPSAC.2017.265 ↗
35. Hettiachchi, D., van Berkel, N., Hosio, S., Kostakos, V., and Goncalves, J. (2019, January 2–6). Effect of Cognitive Abilities on Crowdsourcing Task Performance. Proceedings of the IFIP Conference on Human-Computer Interaction, Paphos, Cyprus.10.1007/978-3-030-29381-9_28 ↗
36. Utilizing short version big five traits on crowdsouring10.1108/IJCS-11-2019-0031 ↗
37. Lykourentzou, I., Antoniou, A., and Naudet, Y. (2015). Matching or Crashing? Personality-based Team Formation in Crowdsourcing Environments. arXiv.
38. Ye, B., Wang, Y., and Liu, L. (July, January 27). Crowd Trust: A Context-Aware Trust Model for Worker Selection in Crowdsourcing Environments. Proceedings of the 2015 IEEE International Conference on Web Services (ICWS), New York, NY, USA.10.1109/ICWS.2015.26 ↗
39. Small Profits and Quick Returns: An Incentive Mechanism Design for Crowdsourcing under Continuous Platform Competition10.1109/JIOT.2019.2953278 ↗
40. An incentive mechanism with privacy protection in mobile crowdsourcing systems10.1016/j.comnet.2016.03.016 ↗
41. A Survey of General-Purpose Crowdsourcing Techniques10.1109/TKDE.2016.2555805 ↗
42. Mahmud, J., Zhou, M.X., Megiddo, N., Nichols, J., and Drews, C. (2013, January 19–22). Recommending targeted strangers from whom to solicit information on social media. Proceedings of the IUI’13: 18th International Conference on Intelligent User Interfaces, Santa Monica, CA, USA.
43. Zhao, Z., Yan, D., Ng, W., and Gao, S. (2013, January 11–14). A transfer learning based framework of crowd-selection on twitter. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487708 ↗
44. Nushi, B., Alonso, O., Hentschel, M., and Kandylas, V. (2015, January 23–26). CrowdSTAR: A Social Task Routing Framework for Online Communities. Proceedings of the 15th International Conference on Engineering the Web in the Big Data Era (ICWE 2015), Rotterdam, The Netherlands.10.1007/978-3-319-19890-3_15 ↗
45. Gil-solla, A., and Ramos-carber, M. (2020, January 11–13). Tasks Recommendation in Crowdsourcing based on Workers’ Implicit Profiles and Performance History. Proceedings of the ICSIE ’20: 9th International Conference on Software and Information Engineering, Cario, Egypt.
46. Difallah, D.E., Demartini, G., and Cudré-Mauroux, P. (2013, January 13–17). Pick-a-crowd: Tell me what you like, and I’ll tell you what to do: A crowdsourcing platform for personalized human intelligence task assignment based on social networks. Proceedings of the WWW’13: 22nd International Conference on World Wide Web, Rio de Janeiro, Brazil.10.1145/2488388.2488421 ↗
47. Hear the Whole Story: Towards the Diversity of Opinion in Crowdsourcing Markets10.14778/2735479.2735482 ↗
48. Incentivizing Social Media Users for Mobile Crowdsourcing10.1016/j.ijhcs.2016.09.007 ↗
49. Context-Aware Reliable Crowdsourcing in Social Networks10.1109/TSMC.2017.2777447 ↗
50. SocialRecruiter: Dynamic Incentive Mechanism for Mobile Crowdsourcing Worker Recruitment with Social Networks10.1109/TMC.2020.2973958 ↗
51. CAT: Context-aware trust-oriented worker selection in social crowd10.4149/cai_2017_3_517 ↗
52. Group-oriented task allocation for crowdsourcing in social networks10.1109/TSMC.2019.2933327 ↗
53. Snijders, R., Dalpiaz, F., Hosseini, M., Shahri, A., and Ali, R. (2014, January 8–11). Crowd-centric requirements engineering. Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC), London, UK.10.1109/UCC.2014.96 ↗
54. Crowdsourcing Requirements Engineering: A Taxonomy-based Review
55. Lim, S.L., Quercia, D., and Finkelstein, A. (2010, January 2–8). StakeSource: Harnessing the power of crowdsourcing and social networks in stakeholder analysis. Proceedings of the ICSE’10: 32nd International Conference on Software Engineering, Cape Town, South Africa.10.1145/1810295.1810340 ↗
56. StakeRare: Using social networks and collaborative filtering for large-scale requirements elicitation10.1109/TSE.2011.36 ↗
57. A social network based process to minimize in-group biasedness during requirement engineering10.1109/ACCESS.2018.2879385 ↗
58. Lim, S.L., Damian, D., and Finkelstein, A. (2011, January 21–28). StakeSource2.0: Using social networks of stakeholders to identify and prioritise requirements. Proceedings of the ICSE11: International Conference on Software Engineering, Honolulu, HI, USA.10.1145/1985793.1985983 ↗
59. Lim, S.L., Quercia, D., and Finkelstein, A. (2010, January 2–8). StakeNet: Using social networks to analyse the stakeholders of large-scale software projects. Proceedings of the ICSE’10: 32nd International Conference on Software Engineering, Cape Town, South Africa.10.1145/1806799.1806844 ↗
60. Alvertis, I., Papaspyros, D., Koussouris, S., Mouzakitis, S., and Askounis, D. (September, January 31). Using crowdsourced and anonymized Personas in the requirements elicitation and software development phases of software engineering. Proceedings of the 2016 11th International Conference on Availability, Reliability and Security (ARES), Salzburg, Austria.10.1109/ARES.2016.71 ↗
61. Lim, S.L., and Bentley, P.J. (2017, January 29–30). Using PseudoGravity to Attract People an Automated Approach to Engaging a Target Audience Using Twitter. Proceedings of the Future Technologies Conference 2017, Vancouver, BC, Canada.
62. Guzman, E., Alkadhi, R., and Seyff, N. (2016, January 12–16). A Needle in a Haystack: What Do Twitter Users Say about Software?. Proceedings of the 2016 IEEE 24th International Requirements Engineering Conference (RE), Beijing, China.10.1109/RE.2016.67 ↗
63. GARUSO: A gamification approach for involving stakeholders outside organizational reach in requirements engineering10.1007/s00766-019-00314-z ↗
64. Lim, S.L., Bentley, P.J., and Ishikawa, F. (September, January 31). Reaching the unreachable: A method for early stage software startups to reach inaccessible stakeholders within large corporation. Proceedings of the 2020 IEEE 28th International Requirements Engineering Conference (RE), Zurich, Switzerland.10.1109/RE48521.2020.00051 ↗
65. Snijders, R., Dalpiaz, F., Brinkkemper, S., Hosseini, M., Ali, R., and Özüm, A. (2015, January 25). REfine: A gamified platform for participatory requirements engineering. Proceedings of the 2015 IEEE 1st International Workshop on Crowd-Based Requirements Engineering (CrowdRE), Ottawa, ON, Canada.10.1109/CrowdRE.2015.7367581 ↗
66. UCFrame10.1145/2894784.2894795 ↗
67. Murukannaiah, P.K., Ajmeri, N., and Singh, M.P. (2016, January 12–16). Acquiring Creative Requirements from the Crowd: Understanding the Influences of Personality and Creative Potential in Crowd RE. Proceedings of the 2016 IEEE 24th International Requirements Engineering Conference (RE), Beijing, China.10.1109/RE.2016.68 ↗
68. Identifying and Classifying User Requirements in Online Feedback via Crowdsourcing
69. Eliciting activity requirements from crowd using genetic algorithm
70. Taj, S., Arain, Q., Memon, I., and Zubedi, A. (2019, January 9–12). To apply data mining for classification of crowd sourced software requirements. Proceedings of the ICSIE’19: 2019 8th International Conference on Software and Information Engineering, Cairo, Egypt.10.1145/3328833.3328837 ↗
71. Condori-Fernandez, N., Lago, P., Luaces, M., and Catala, A. (2019, January 29–31). A Nichesourcing Framework applied to Software Sustainability Requirements. Proceedings of the 2019 13th International Conference on Research Challenges in Information Science (RCIS), Brussels, Belgium.10.1109/RCIS.2019.8877000 ↗
72. Bozzon, A., Brambilla, M., Ceri, S., Silvestri, M., and Vesci, G. (2013, January 18–22). Choosing the right crowd: Expert finding in social networks. Proceedings of the DBT/ICDT’13: Joint 2013 EDBT/ICDT Conferences, Genoa, Italy.10.1145/2452376.2452451 ↗
73. Dalpiaz, F., Korenko, M., Salay, R., and Chechik, M. (2015, January 25). Using the crowds to satisfy unbounded requirements. Proceedings of the 2015 IEEE 1st International Workshop on Crowd-Based Requirements Engineering (CrowdRE), Ottawa, ON, Canada.10.1109/CrowdRE.2015.7367584 ↗
74. Expertise retrieval10.1561/1500000024 ↗
75. Reimers, N., and Gurevych, I. (2019, January 3–7). Sentence-BERT: Sentence embeddings using siamese BERT-networks. Proceedings of the EMNLP-IJCNLP 2019—2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1410 ↗
76. Skrlj, B., and Petkovic, M. (2021, January 7–10). Compressibility of Distributed Document Representations. Proceedings of the IEEE International Conference on Data Mining (ICDM), Auckland, New Zealand.10.1109/ICDM51629.2021.00166 ↗
77. Le, Q., and Mikolov, T. (2014, January 21–26). Distributed representations of sentences and documents. Proceedings of the 31st International Conference on Machine Learning, Beijing, China.
78. (2023, June 07). SBERT Pretrained Models. Available online: https://www.sbert.net/docs/pretrained_models.html.
79. (2023, July 01). Pretrained SBERT all-mpnet-base-v2. Available online: https://huggingface.co/sentence-transformers/all-mpnet-base-v2.
80. (2023, June 15). Doc2Vec. Available online: https://github.com/jhlau/doc2vec.
81. GitHub (2023, August 14). Fitness Application. Available online: https://github.com/rahulmaddineni/Stayfit.
82. GitHub (2023, August 14). Crossword Application. Available online: https://rschroll.github.io/crosswords/.
83. GitHub (2023, August 14). Matematica Vera Application. Available online: https://github.com/rafalambrozewicz/matematica_vera.
84. GitHub (2023, August 14). OpenCryptofolio Application. Available online: https://github.com/applebaum/open-cryptofolio.
85. GitHub (2023, August 14). CodeEdit Application. Available online: https://github.com/CodeEditApp/CodeEdit.
86. GitHub (2023, August 14). Photo Editor. Available online: https://github.com/bevy/photo-editor-android.
87. Personalized Book Recommendation System using Machine Learning Algorithm
88. (2024, May 04). Apps Descriptions. Available online: https://github.com/GhadaAl23/apps-descriptions.
89. Beglar, D., and Nemoto, T. (2014). Developing Likert-scale questionnaires. JALT2013 Conference Proceedings, JALT.
90. An Approach for Analyzing ISO/IEC 25010 Product Quality Requirements based on Fuzzy Logic and Likert Scale for Decision Support Systems
91. Alspaugh, T.A., and Richardson, D.J. (2006). The Importance of Clarity in Usable Requirements Specification Formats, Institute for Software Research, University of California.
92. Sandhu, G., and Sikka, S. (2015, January 15–16). State-of-art practices to detect inconsistencies and ambiguities from software requirements. Proceedings of the International Conference on Computing, Communication and Automation (ICCCA), Greater Noida, India.10.1109/CCAA.2015.7148485 ↗
93. Mechanisms to Engage an Online Community in Crowdsourcing: Insights from an Idea Contest in Training10.19245/25.05.pij.6.2.7 ↗
94. Anh Do, Q. (2020). Towards Generation of Creative Software Requirements. [Ph.D. Thesis, Mississippi State University].
95. Denger, C., and Olsson, T. (2005). Quality assurance in requirements engineering. Engineering and Managing Software Requirements, Springer.10.1007/3-540-28244-0_8 ↗
96. Vidal, V.S., Suriani, M.A.M., Braga, R.A.S., Santos, A.C.O., Silva, O.S., and Campos, R.J. (2021, January 10–14). Use of Crowdsourcing Questionnaires to Validate the Requirements of an Application for Pet Management. Proceedings of the ITNG 2021 18th International Conference on Information Technology-New Generations, Las Vegas, NV, USA.10.1007/978-3-030-70416-2_28 ↗