A Critical Review of Methods and Challenges in Large Language Models
Source abstract
Abstract
This critical review provides an in-depth analysis of Large Language Models (LLMs), encompassing their foundational principles, diverse applications, and advanced training methodologies. We critically examine the evolution fr... | Find, read and cite all the research you need on Tech Science Press
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (72)
Show source-provided references
1. The potential of generative artificial intelligence across disciplines: perspectives and future directions
2. Generative artificial intelligence10.1007/s12525-023-00680-1 ↗
3. Large language models in medicine10.1038/s41591-023-02448-8 ↗
4. ChatGPT for good? On opportunities and challenges of large language models for education10.1016/j.lindif.2023.102274 ↗
5. BC4LLM: a perspective of trusted artificial intelligence when blockchain meets large language models10.1016/j.neucom.2024.128089 ↗
6. Benchmarking medical large language models10.1038/s44222-023-00097-7 ↗
7. Generating text with recurrent neural networks
8. A review of recurrent neural networks: LSTM cells and network architectures10.1162/neco_a_01199 ↗
9. Attention is all you need
10. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2018.
11. Overview of the transformer-based models for NLP tasks
12. Transformers: state-of-the-art natural language processing
13. Can large language models reason about medical questions?10.1016/j.patter.2024.100943 ↗
14. Wu S, Irsoy O, Lu S, Dabravolski V, Dredze M, Gehrmann S, et al. Bloomberggpt: a large language model for finance. arXiv:2303.17564. 2023.
15. Large language models in healthcare and medical domain: a review10.3390/informatics11030057 ↗
16. Lin Z, Gong Y, Shen Y, Wu T, Fan Z, Lin C, et al. Text generation with diffusion language models: a pre-training approach with continuous paragraph denoise. Paper presented at: Proceedings of the 40th International Conference on Machine Learning; 2023. Vol. 202, p. 21051–64.
17. Gholami S, Omar M. Do generative large language models need billions of parameters? arXiv:2309.06589. 2023.10.4018/979-8-3693-6517-5.ch003 ↗
18. Self-supervised representation learning: introduction, advances, and challenges10.1109/MSP.2021.3134634 ↗
19. w2v-BERT: combining contrastive learning and masked language modeling for self-supervised speech pre-training
20. Exploring the limits of transfer learning with a unified text-to-text transformer
21. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, et al. Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv:1910.13461. 2019.10.18653/v1/2020.acl-main.703 ↗
22. A comprehensive survey on design and application of autoencoder in deep learning10.1016/j.asoc.2023.110176 ↗
23. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. Roberta: a robustly optimized bert pretraining approach. arXiv:1907.11692. 2019.
24. Xlnet: generalized autoregressive pretraining for language understanding
25. Language models are few-shot learners
26. An empirical analysis of compute-optimal large language model training
27. Workshop B, Scao TL, Fan A, Akiki C, Pavlick E, Ilić S, et al. Bloom: a 176b-parameter open-access multilingual language model. arXiv:2211.05100. 2022.
28. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M-A, Lacroix T, et al. LLaMA: open and efficient foundation language models. arXiv:2302.13971. 2023.
29. Efficient hierarchical domain adaptation for pretrained language models
30. Wang X, Zhu W, Wang WY. Large language models are implicitly topic models: explaining and finding good demonstrations for in-context learning. arXiv:2301.11916. 2023.
31. A review of generalized zero-shot learning methods
32. One-shot learning approach for unknown malware classification
33. Zero- and few-shot NLP with pretrained language models
34. A comprehensive survey of few-shot learning: evolution, applications, challenges, and opportunities10.1145/3582688 ↗
35. Wei J, Bosma M, Zhao VY, Guu K, Yu AW, Lester B, et al. Finetuned language models are zero-shot learners. Paper presented at: The Tenth International Conference on Learning Representations; 2022.
36. Zhang S, Dong L, Li X, Zhang S, Sun X, Wang S, et al. Instruction tuning for large language models: a survey. arXiv:2308.10792. 2023.
37. Measuring catastrophic forgetting in neural networks
38. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks
39. On the effectiveness of parameter-efficient fine-tuning
40. Parameter-efficient fine-tuning of large-scale pre-trained language models10.1038/s42256-023-00626-4 ↗
41. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning
42. Lialin V, Deshpande V, Rumshisky A. Scaling down to scale up: a guide to parameter-efficient fine-tuning. arXiv:2303.15647. 2023.
43. Cross-attention is all you need: adapting pretrained transformers for machine translation
44. AdapterHub: a framework for adapting transformers
45. SPoT: better frozen model adaptation through soft prompt transfer
46. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. Low-rank adaptation of large language models. arXiv:2106.09685. 2021.
47. Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. Qlora: efficient finetuning of quantized llms. arXiv:2305.14314. 2023.
48. Bias and fairness in large language models: a survey10.1162/coli_a_00524 ↗
49. Probing toxic content in large pre-trained language models
50. A review on interactive reinforcement learning from human social feedback10.1109/ACCESS.2020.3006254 ↗
51. Training language models to follow instructions with human feedback
52. Fine-tuning language models to find agreement among humans with diverse preferences
53. Rafailov R, Sharma A, Mitchell E, Ermon S, Manning CD, Finn C. Direct preference optimization: your language model is secretly a reward model. Paper presented at: 37th Conference on Neural Information Processing Systems (NeurIPS); 2023.
54. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv:1707.06347. 2017.
55. Casper S, Davies X, Shi C, Gilbert TK, Scheurer J, Rando J, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv:2307.15217. 2023.
56. Defining and characterizing reward gaming
57. Bai Y, Jones A, Ndousse K, Askell A, Chen A, DasSarma N, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. 2022.
58. Coste T, Anwar U, Kirk R, Krueger D. Reward model ensembles help mitigate overoptimization. arXiv:2310.02743. 2023.
59. HaluEval: a large-scale hallucination evaluation benchmark for large language models
60. Retrieval-augmented generation for knowledge-intensive nlp tasks
61. Han Y, Liu C, Wang P. A comprehensive survey on vector database: storage and retrieval technique, challenge. arXiv:2310.11703. 2023.
62. Palma DD. Retrieval-augmented recommender system: enhancing recommender systems with large language models. Paper presented at: Proceedings of the 17th ACM Conference on Recommender Systems; 2023; Singapore.
63. Brown H, Lee K, Mireshghallah F, Shokri R, Tramèr F. What does it mean for a language model to preserve privacy? Paper presented at: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency; 2022; Seoul, Republic of Korea.10.1145/3531146.3534642 ↗
64. Countering misinformation through semantic-aware multilingual models
65. Transparency helps reveal when language models learn meaning10.1162/tacl_a_00565 ↗
66. Estimating the carbon footprint of bloom, a 176b parameter language model
67. Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, et al. Scaling laws for neural language models. arXiv:2001.08361. 2020.
68. Transformers on sarcasm detection with context
69. Data curation alone can stabilize in-context learning
70. Yin S, Fu C, Zhao S, Li K, Sun X, Xu T, et al. A survey on multimodal large language models. arXiv:2306.13549. 2023.10.1093/nsr/nwae403 ↗
71. Explainability for large language models: a survey10.1145/3639372 ↗
72. Evaluating the robustness of neural language models to input perturbations