AdaptForever: Elastic and Mutual Learning for Continuous NLP Task Mastery
Source abstract
Abstract
In natural language processing (NLP), managing multiple downstream tasks through fine-tuning pre-trained models often requires maintaining separate task-specific models, leading to practical inefficiencies. To address this ch... | Find, read and cite all the research you need on Tech Science Press
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Intelligent Tutoring Systems and Adaptive Learning · Topic Modeling · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (31)
Show source-provided references
1. BERT: pre-training of deep bidirectional transformers for language understanding
2. A robustly optimized BERT pre-training approach with post-training
3. Energy and policy considerations for deep learning in NLP
4. Language models are few-shot learners
5. He J, Zhou C, Ma X, Berg-Kirkpatrick T, Neubig G. Towards a unified view of parameter-efficient transfer learning. arXiv:2110.04366. 2021.
6. Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, et al. Parameter-efficient transfer learning for NLP. arXiv:1902.00751. 2019.
7. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. Lora: low-rank adaptation of large language models. arXiv:2106.09685. 2021.
8. AdapterFusion: non-destructive task composition for transfer learning
9. BERT and PALs: projected attention layers for efficient adaptation in multi-task learning
10. AdaMix: mixture-of-adaptations for parameter-efficient model tuning
11. Adaptive budget allocation for parameter-efficient fine-tuning
12. Zuo S, Liu X, Jiao J, Kim YJ, Hassan H, Zhang R, et al. Taming sparsely activated transformer with stochastic experts. arXiv:2110.04260. 2021.
13. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer
14. Deep mutual learning
15. MuiDial: improving dialogue disentanglement with intent-based mutual learning
16. Overcoming catastrophic forgetting in neural networks10.1073/pnas.1611835114 ↗
17. Vaswani · Advances in neural information processing systems · 2017 · 6000
18. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity
19. AdapterDrop: on the efficiency of adapters in transformers
20. On the effectiveness of adapter-based tuning for pretrained language model adaptation
21. Li D, Ma Y, Wang N, Ye Z, Cheng Z, Tang Y, et al. MixLoRA: enhancing large language models fine-tuning with lora based mixture of experts. arXiv:2404.15159. 2024.
22. Schulhoff S, Ilie M, Balepur N, Kahadze K, Liu A, Si C, et al. The prompt report: a systematic survey of prompting techniques. arXiv:2406.06608. 2024.
23. Kepel D, Valogianni K. Autonomous prompt engineering in large language models. arXiv:2407.11000. 2024.
24. Li Y, Luo F, Tan C, Wang M, Huang S, Li S, et al. Parameter-efficient sparsity for large language models fine-tuning. arXiv:2205.11005. 2022.10.24963/ijcai.2022/586 ↗
25. BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models
26. Unsupervised cross-lingual representation learning at scale
27. Misinformation detection using multitask learning with mutual learning for novelty detection and emotion recognition10.1016/j.ipm.2021.102631 ↗
28. Lepikhin D, Lee H, Xu Y, Chen D, Firat O, Huang Y, et al. GShard: scaling giant models with conditional computation and automatic sharding. arXiv:2006.16668. 2020.
29. Note on the quadratic penalties in elastic weight consolidation10.1073/pnas.1717042115 ↗
30. Batra H, Clark R. EVCL: elastic variational continual learning with weight consolidation. arXiv:2406.15972. 2024.
31. Continual lifelong learning in natural language processing: a survey