Evaluating the Utilities of Foundation Models in Single‐Cell Data Analysis
Source abstract
Abstract
ABSTRACT Foundation Models (FMs) have made significant strides in both industrial and scientific domains. In this paper, we evaluate the performance of FMs for single‐cell sequencing data analysis through comprehensive experiments across eight downstream tasks pertinent to single‐cell data. Overall, the top FMs include scGPT, Geneformer, and CellFM by considering model performances and user accessibility among ten single‐cell FMs. However, by comparing these FMs with task‐specific methods, we found that single‐cell FMs may not consistently excel than task‐specific methods in all tasks, which challenges the necessity of developing foundation models for single‐cell analysis. In addition, we evaluated the effects of hyperparameters, initial settings, and stability for training single‐cell FMs based on a proposed scEval framework, and provide guidelines for pre‐training and fine‐tuning to enhance the performances of single‐cell FMs. Our work summarizes the current state of single‐cell FMs, points to their constraints and avenues for future development, and offers a freely available evaluation pipeline to benchmark new models and improve method development.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Single-cell and spatial transcriptomics · Cell Image Analysis Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Interdepartmental Program in Computational Biology & Bioinformatics Yale University Connecticut USA
Department of Biostatistics Yale University Connecticut USA
Funding
National Institutes of Health
References (100+)
Show source-provided references
1. 10.1038/s41586-020-2157-410.1038/s41586-020-2157-4 ↗
2. 10.1093/nar/gku55510.1093/nar/gku555 ↗
3. 10.1126/science.aan682810.1126/science.aan6828 ↗
4. 10.1038/s41586-019-1195-210.1038/s41586-019-1195-2 ↗
5. L.Zhang C. H.He S.Coffey et al. “Single‐Cell Transcriptomic Atlas of Alzheimer's Disease Middle Temporal Gyrus Reveals Region Cell Type and Sex Specificity of Gene Expression With Novel Genetic Risk for Mertk in Female ”medRxiv(2023): 2023–02.10.1101/2023.02.18.23286037 ↗
6. 10.1146/annurev-genom-111320-09043610.1146/annurev-genom-111320-090436 ↗
7. 10.1038/s12276-018-0071-810.1038/s12276-018-0071-8 ↗
8. 10.1038/ncomms1404910.1038/ncomms14049 ↗
9. 10.1038/nmeth.438010.1038/nmeth.4380 ↗
10. 10.1126/science.aab160110.1126/science.aab1601 ↗
11. 10.1038/s41467-018-07771-010.1038/s41467-018-07771-0 ↗
12. 10.1126/science.aan335110.1126/science.aan3351 ↗
13. 10.1038/nbt.411210.1038/nbt.4112 ↗
14. Method of the Year 2019: Single‐Cell Multimodal Omics
15. 10.1146/annurev-biodatasci-020422-05064510.1146/annurev-biodatasci-020422-050645 ↗
16. W. X.Zhao K.Zhou J.Li et al. “A Survey of Large Language Models ”arXiv preprint arXiv:2303.18223(2023):1–124.
17. J.Achiam S.Adler S.Agarwal et al. “GPT‐4 technical report ”arXiv preprint arXiv:2303.08774 (2023).
18. H.Touvron T.Lavril G.Izacard et al. “Llama: Open and efficient foundation language models ” arXiv preprint arXiv:2302.13971 (2023).
19. 10.1038/s41591-023-02448-810.1038/s41591-023-02448-8 ↗
20. 10.18653/v1/2023.acl-industry.410.18653/v1/2023.acl-industry.4 ↗
21. D.Driess F.Xia M. S.Sajjadi et al. “Palm‐e: An Embodied Multimodal Language Model ” in Proceedings of the 40th International Conference on Machine Learning (ICML'23) arXiv preprint arXiv:2303.03378(2023):8469–8488.
22. 10.1016/j.cell.2019.05.03110.1016/j.cell.2019.05.031 ↗
23. 10.1038/s41467-023-35923-410.1038/s41467-023-35923-4 ↗
24. Driess D. · 84698488
25. T.Kuzman N.Ljubešić andI.Mozetič “Chatgpt: Beginning of an End of Manual Annotation? Use Case of Automatic Genre Identification ”arXiv preprint arXiv:2303.03953(2023).
26. 10.1038/s41576-020-00292-x10.1038/s41576-020-00292-x ↗
27. 10.1093/bib/bbr01210.1093/bib/bbr012 ↗
28. A Review on Word Embedding Techniques for Text Classification10.1007/978-981-15-9651-3_23 ↗
29. 10.1038/s41576-023-00586-w10.1038/s41576-023-00586-w ↗
30. 10.1038/s41592-021-01252-x10.1038/s41592-021-01252-x ↗
31. 10.1038/s41467-023-36476-210.1038/s41467-023-36476-2 ↗
32. 10.1038/s41586-023-06291-210.1038/s41586-023-06291-2 ↗
33. 10.1038/s42256-022-00534-z10.1038/s42256-022-00534-z ↗
34. S.Zhao J.Zhang andZ.Nie “Large‐Scale Cell Representation Learning via Divide‐and‐Conquer Contrastive Learning ”arXiv preprint arXiv:2306.04371(2023).
35. 10.1038/s41586-023-06139-910.1038/s41586-023-06139-9 ↗
36. 10.1038/s41592-024-02201-010.1038/s41592-024-02201-0 ↗
37. 10.1038/s41592-024-02305-710.1038/s41592-024-02305-7 ↗
38. 10.1016/j.isci.2023.10653610.1016/j.isci.2023.106536 ↗
39. 10.1038/s41422-024-01034-y10.1038/s41422-024-01034-y ↗
40. 10.1038/s41586-024-08411-y10.1038/s41586-024-08411-y ↗
41. Y.Rosen Y.Roohani A.Agrawal et al. “Universal cell embeddings: A foundation model for cell biology ”bioRxiv(2023): 2023–11.10.1101/2023.11.28.568918 ↗
42. 10.1038/s41467-025-59926-510.1038/s41467-025-59926-5 ↗
43. H.Wen W.Tang X.Dai et al. “CellPLM: Pre‐Training of Cell Language Model Beyond Single Cells ” inThe Twelfth International Conference on Learning Representations(2023).10.1101/2023.10.03.560734 ↗
44. Q.Ye H.Xu G.Xu et al. “mplug‐owl: Modularization Empowers Large Language Models With Multimodality ”arXiv preprint arXiv:2304.14178(2023).
45. 10.1038/s41586-019-1876-x10.1038/s41586-019-1876-x ↗
46. 10.1093/database/baz04610.1093/database/baz046 ↗
47. 10.1038/s41587-019-0071-910.1038/s41587-019-0071-9 ↗
48. 10.1038/s41592-019-0619-010.1038/s41592-019-0619-0 ↗
49. 10.1093/bioinformatics/btac42710.1093/bioinformatics/btac427 ↗
50. 10.21105/joss.0086110.21105/joss.00861 ↗
51. F.Xue Y.Fu W.Zhou Z.Zheng andY.You “To Repeat or not to Repeat: Insights From Scaling LLM Under Token‐Crisis ” inProceedings of theThirty‐seventh Conference on Neural Information Processing Systems(2023):59304–59322 https://openreview.net/forum?id=Af5GvIj3T5.10.52202/075280-2590 ↗
52. D.KingmaandJ.Ba “Adam: A Method for Stochastic Optimization ” inInternational Conference on Learning Representations San Diega CA USA(ICLR 2015).
53. S.Ruder “An Overview of Gradient Descent Optimization Algorithms ”arXiv preprint arXiv:1609.04747(2016).
54. H.Liu Z.Li D. L. W.Hall P.Liang andT.Ma “Sophia: A Scalable Stochastic Second‐Order Optimizer for Language Model Pre‐Training ” inThe Twelfth International Conference on Learning Representations(2024) https://openreview.net/forum?id=3xHDeA8Noi.
55. Symbolic Discovery of Optimization Algorithms
56. 10.1038/s41587-021-01161-610.1038/s41587-021-01161-6 ↗
57. 10.1038/s41587-022-01284-410.1038/s41587-022-01284-4 ↗
58. Scikit‐Learn: Machine Learning in Python
59. 10.1186/s13059-019-1795-z10.1186/s13059-019-1795-z ↗
60. 10.1186/s12864-018-5370-x10.1186/s12864-018-5370-x ↗
61. 10.1038/s41587-023-01905-610.1038/s41587-023-01905-6 ↗
62. 10.1038/s41592-021-01336-810.1038/s41592-021-01336-8 ↗
63. 10.1093/nar/gkx113210.1093/nar/gkx1132 ↗
64. 10.1002/dvdy.38410.1002/dvdy.384 ↗
65. 10.1002/pro.421810.1002/pro.4218 ↗
66. 10.1038/s41576-023-00618-510.1038/s41576-023-00618-5 ↗
67. 10.1093/bib/bbac48210.1093/bib/bbac482 ↗
68. 10.1038/s41598-019-41695-z10.1038/s41598-019-41695-z ↗
69. 10.1038/s41588-023-01586-610.1038/s41588-023-01586-6 ↗
70. 10.1007/s00251-017-1017-310.1007/s00251-017-1017-3 ↗
71. 10.1534/g3.114.01578410.1534/g3.114.015784 ↗
72. 10.1038/s41596-023-00853-410.1038/s41596-023-00853-4 ↗
73. 10.1016/S1672-6529(08)60101-810.1016/S1672-6529(08)60101-8 ↗
74. 10.1038/ni.376810.1038/ni.3768 ↗
75. 10.1038/s41592-021-01264-710.1038/s41592-021-01264-7 ↗
76. 10.1038/s41467-023-37168-710.1038/s41467-023-37168-7 ↗
77. A Step‐by‐Step Workflow for Low‐Level Analysis of Single‐Cell RNA‐seq Data With Bioconductor
78. 10.1038/nbt.319210.1038/nbt.3192 ↗
79. 10.1186/s13059-017-1382-010.1186/s13059-017-1382-0 ↗
80. 10.1186/s13059-017-1305-010.1186/s13059-017-1305-0 ↗
81. 10.1038/s41587-023-01772-110.1038/s41587-023-01772-1 ↗
82. J.Kaplan S.McCandlish T.Henighan et al. “Scaling Laws for Neural Language Models ”arXiv preprint arXiv:2001.08361(2020).
83. B.Zhang Z.Liu C.Cherry andO.Firat “When Scaling Meets LLM Finetuning: The Effect of data Model and Finetuning Method ” inThe Twelfth International Conference on Learning Representations(2024) https://openreview.net/forum?id=5HCnKDeTws.
84. D.Hernandez J.Kaplan T.Henighan andS.McCandlish “Scaling laws for transfer ”arXiv preprint arXiv:2102.01293(2021).
85. 10.1038/s41592-024-02191-z10.1038/s41592-024-02191-z ↗
86. 10.1038/s41592-025-02899-610.1038/s41592-025-02899-6 ↗
87. 10.15252/msb.2020962010.15252/msb.20209620 ↗
88. 10.1093/nar/gkae107110.1093/nar/gkae1071 ↗
89. 10.1038/s41467-023-41855-w10.1038/s41467-023-41855-w ↗
90. 10.1038/s41592-018-0229-210.1038/s41592-018-0229-2 ↗
91. 10.1038/s41592-020-01033-y10.1038/s41592-020-01033-y ↗
92. 10.1016/j.cell.2021.04.04810.1016/j.cell.2021.04.048 ↗
93. E. J.Hu yelong shen P.Wallis et al. “LoRA: Low‐Rank Adaptation of Large Language Models ” inInternational Conference on Learning Representations(2022) https://openreview.net/forum?id=nZeVKeeFYf9.
94. Z.Hu L.Wang Y.Lan et al. “LLM‐Adapters: An Adapter Family for Parameter‐Efficient Fine‐Tuning of Large Language Models ” inThe 2023 Conference on Empirical Methods in Natural Language Processing(2023).10.18653/v1/2023.emnlp-main.319 ↗
95. 10.1038/s42256-023-00626-410.1038/s42256-023-00626-4 ↗
96. Y.Wang S.Si D.Li et al. “Preserving In‐Context Learning Ability in Large Language Model Fine‐Tuning ”arXiv preprint arXiv:2211.00635(2022).
97. R.Xu F.Luo Z.Zhang et al. “Raise a Child in Large Language Model: Towards Effective and Generalizable Fine‐Tuning ” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing M. F.Moens X.Huang L.Specia andS. W. T.Yih Eds. (Online and Punta Cana Dominican Republic: Association for Computational Linguistics 2021) 9514–9528 https://aclanthology.org/2021.emnlp‐main.749.10.18653/v1/2021.emnlp-main.749 ↗
98. L.Chen M.Zaharia andJ.Zou “How is chatgpt's behavior changing over time?”arXiv preprint arXiv:2307.09009(2023).10.1162/99608f92.5317da47 ↗
99. Y.Ji Y.Deng Y.Gong et al. “Exploring the impact of instruction data scaling on large language models: An empirical study on real‐world use cases ”arXiv preprint arXiv:2303.14742(2023).
100. T.Carta C.Romac T.Wolf S.Lamprier O.Sigaud andP. Y.Oudeyer “Grounding large language models in interactive environments with online reinforcement learning ” inInternational Conference on Machine Learning(PMLR 2023) 3676–3713.