A Study of a Gain Based Approach for Query Aspects in Recall Oriented Tasks
Source abstract
Abstract
Evidence-based healthcare integrates the best research evidence with clinical expertise in order to make decisions based on the best practices available. In this context, the task of collecting all the relevant information, a recall oriented task, in order to take the right decision within a reasonable time frame has become an important issue. In this paper, we investigate the problem of building effective Consumer Health Search (CHS) systems that use query variations to achieve high recall and fulfill the information needs of health consumers. In particular, we study an intent-aware gain metric used to estimate the amount of missing information and make a prediction about the achievable recall for each query reformulation during a search session. We evaluate and propose alternative formulations of this metric using standard test collections of the CLEF 2018 eHealth Evaluation Lab CHS.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Recommender Systems and Techniques · Data Quality and Management · Data-Driven Disease Surveillance
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Engineering, University of Padova, Via Gradenigo 6/b, 35131 Padova, Italy
References (45)
Show source-provided references
1. Bailey, P., Moffat, A., Scholer, F., and Thomas, P. (2016, January 17–21). UQV100: A test collection with query variability. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy.10.1145/2911451.2914671 ↗
2. Bailey, P., Moffat, A., Scholer, F., and Thomas, P. (2017, January 7–11). Retrieval consistency in the presence of query variations. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Japan.10.1145/3077136.3080839 ↗
3. Boosting Search Performance Using Query Variations10.1145/3345001 ↗
4. Thomas, P., Scholer, F., Bailey, P., and Moffat, A. (2017, January 7–8). Tasks, queries, and rankers in pre-retrieval performance prediction. Proceedings of the 22nd Australasian Document Computing Symposium, Brisbane, QLD, Australia.10.1145/3166072.3166079 ↗
5. Zendel, O., Shtok, A., Raiber, F., Kurland, O., and Culpepper, J.S. (2019, January 21–25). Information needs, queries, and query performance prediction. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France.10.1145/3331184.3331253 ↗
6. Culpepper, J.S., Faggioli, G., Ferro, N., and Kurland, O. (2021). Topic Difficulty: Collection and Query Formulation Effects. ACM Trans. Inf. Syst., 40.10.1145/3470563 ↗
7. Culpepper, J.S., Faggioli, G., Ferro, N., and Kurland, O. (2021, January 13–15). Do hard topics exist? A statistical analysis. Proceedings of the 11th Italian Information Retrieval Workshop, Bari, Italy.
8. Variations in relevance judgments and the evaluation of retrieval performance10.1016/0306-4573(92)90031-T ↗
9. Sheldon, D., Shokouhi, M., Szummer, M., and Craswell, N. (2011, January 9–12). LambdaMerge: Merging the results of query reformulations. Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, Hong Kong, China.10.1145/1935826.1935930 ↗
10. Combining the evidence of multiple query representations for information retrieval10.1016/0306-4573(94)00057-A ↗
11. Liu, B., Craswell, N., Lu, X., Kurland, O., and Culpepper, J.S. (2019, January 2–5). A comparative analysis of human and automatic query variants. Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, Santa Clara, CA, USA.10.1145/3341981.3344223 ↗
12. Dang, V., Bendersky, M., and Croft, W.B. (2010, January 19–23). Learning to rank query reformulations. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva, Switzerland.10.1145/1835449.1835626 ↗
13. Scells, H., Azzopardi, L., Zuccon, G., and Koopman, B. (2018, January 8–12). Query variation performance prediction for systematic reviews. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.10.1145/3209978.3210078 ↗
14. Faggioli, G., and Marchesin, S. (2021, January 11–15). What makes a query semantically hard?. Proceedings of the 2nd International Conference on Design of Experimental Search & Information REtrieval Systems, 2021, DESIRES ’21, Padua, Italy.
15. Shtok, A., Kurland, O., and Carmel, D. (2016). Query Performance Prediction Using Reference Lists. ACM Trans. Inf. Syst., 34.10.1145/2926790 ↗
16. Roitman, H. (2017, January 7–11). An enhanced approach to query performance prediction using reference lists. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Tokyo, Japan.10.1145/3077136.3080665 ↗
17. Estimating the horizon of articles to decide when to stop searching in systematic reviews: An example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools
18. Roegiest, A., Cormack, G.V., Grossman, M.R., and Clarke, C.L. (2015, January 17–20). TREC 2015 total recall track overview. Proceedings of the Twenty-Fourth TREC 2015, Gaithersburg, MD, USA.10.6028/NIST.SP.500-319.recall-overview ↗
19. Grossman, M.R., Cormack, G.V., and Roegiest, A. (2016, January 15–18). TREC 2016 total recall track overview. Proceedings of the Twenty-Fifth TREC 2016, Gaithersburg, MD, USA.10.6028/NIST.SP.500-321.recall-overview ↗
20. Cappellato, L., Ferro, N., Nie, J., and Soulier, L. Overview of the CLEF 2018 consumer health search task. Proceedings of the Working Notes of CLEF 2018—Conference and Labs of the Evaluation Forum, Avignon, France, 10–14 September 2018, Available online: http://repository.ubaya.ac.id/37282/.
21. Umemoto, K., Yamamoto, T., and Tanaka, K. (2016, January 17–21). ScentBar: A Query suggestion interface visualizing the amount of missed relevant information for intrinsically diverse search. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy.10.1145/2911451.2911546 ↗
22. Banchs, R.E., Silvestri, F., Liu, T.Y., Zhang, M., Gao, S., and Lang, J. (2013). Estimating intent types for search result diversification. Information Retrieval Technology, Springer.10.1007/978-3-642-45068-6 ↗
23. Cronen-Townsend, S., Zhou, Y., and Croft, W.B. (2002, January 11–15). Predicting query performance. Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland.10.1145/564376.564429 ↗
24. Hauff, C., Hiemstra, D., and de Jong, F. (2008, January 26–30). A survey of pre-retrieval query performance predictors. Proceedings of the 17th ACM Conference on Information and Knowledge Management, Napa Valley, CA, USA.10.1145/1458082.1458311 ↗
25. He, B., and Ounis, I. (2004, January 5–8). Inferring query performance using pre-retrieval predictors. Proceedings of the International Symposium on String Processing and Information Retrieval, Padova, Italy.10.1007/978-3-540-30213-1_5 ↗
26. Mothe, J., and Tanguy, L. (2005, January 15–19). Linguistic features to predict query difficulty. Proceedings of the ACM Conference on Research and Development in Information Retrieval, SIGIR, Predicting Query Difficulty-Methods and Applications Workshop, Salvador, Brazil.
27. Query Association Surrogates for Web Search10.1002/asi.20011 ↗
28. Zhao, Y., Scholer, F., and Tsegay, Y. (April, January 30). Effective pre-retrieval query performance prediction using similarity and variability evidence. Proceedings of the European Conference on Information Retrieval, Glasgow, UK.
29. Aslam, J.A., and Pavlu, V. (2007, January 14–17). Query hardness estimation using jensen-shannon divergence among multiple scoring functions. Proceedings of the European Conference on Information Retrieval, Lisbon, Portugal.
30. Roitman, H. (2018, January 8–12). Query performance prediction using passage information. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.10.1145/3209978.3210070 ↗
31. Zamani, H., Croft, W.B., and Culpepper, J.S. (2018, January 8–12). Neural query performance prediction using weak supervision from multiple signals. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.10.1145/3209978.3210041 ↗
32. Zhou, Y., and Croft, W.B. (2006, January 6–11). Ranking robustness: A novel framework to predict query performance. Proceedings of the 15th ACM International Conference on Information and Knowledge Management, Kansas City, VA, USA.10.1145/1183614.1183696 ↗
33. Carmel, D., Yom-Tov, E., Darlow, A., and Pelleg, D. (2006, January 6–11). What makes a query difficult?. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, DC, USA.10.1145/1148170.1148238 ↗
34. Document Score Distribution Models for Query Performance Inference and Prediction10.1145/2559170 ↗
35. Diaz, F. (2007, January 23–27). Performance prediction using spatial autocorrelation. Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands.10.1145/1277741.1277841 ↗
36. Amati, G., Carpineto, C., and Romano, G. (2004, January 14–17). Query difficulty, robustness, and selective application of query expansion. Proceedings of the European Conference on Information Retrieval, Lisbon, Portugal.10.1007/978-3-540-24752-4_10 ↗
37. Cronen-Townsend, S., Zhou, Y., and Croft, W.B. (2004). A Language Modeling Framework for Selective Query Expansion, Center for Intelligent Information Retrieval, University of Massachusetts. Technical Report.10.21236/ADA477592 ↗
38. Zhou, Y., and Croft, W.B. (2007, January 23–27). Query performance prediction in web search environments. Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands.10.1145/1277741.1277835 ↗
39. Predicting Query Performance by Query-Drift Estimation10.1145/2180868.2180873 ↗
40. Tao, Y., and Wu, S. (2014, January 3–7). Query performance prediction by considering score magnitude and variance together. Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, Shanghai, China.10.1145/2661829.2661906 ↗
41. Faggioli, G. (2021, January 11–15). Enabling performance prediction in information retrieval evaluation. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event Canada.10.1145/3404835.3463265 ↗
42. Chifu, A.G., Laporte, L., Mothe, J., and Ullah, M.Z. (2018, January 8–12). Query performance prediction focused on summarized letor features. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.10.1145/3209978.3210121 ↗
43. Shtok, A., Kurland, O., and Carmel, D. (2010, January 19–23). Using statistical decision theory and relevance models for query-performance prediction. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva, Switzerland.10.1145/1835449.1835494 ↗
44. Hiemstra, D., Moens, M.F., Mothe, J., Perego, R., Potthast, M., and Sebastiani, F. (2021). An enhanced evaluation framework for query performance prediction. Advances in Information Retrieval, Springer International Publishing.10.1007/978-3-030-72113-8 ↗
45. Zendel, O., Culpepper, J.S., and Scholer, F. (2021, January 11–15). Is query performance prediction with multiple query variations harder than topic performance prediction?. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event Canada.10.1145/3404835.3463039 ↗