UniTrans: Unified Parameter-Efficient Transfer Learning and Multimodal Alignment for Large Multimodal Foundation Model
Source abstract
Abstract
With the advancements in parameter-efficient transfer learning techniques, it has become feasible to leverage large pre-trained language models for downstream tasks under low-cost and low-resource conditions. However, applyin... | Find, read and cite all the research you need on Tech Science Press
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Text and Document Classification Technologies · Seismology and Earthquake Studies · Neural Networks and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (64)
Show source-provided references
1. Rajič F, Ke L, Tai YW, Tang CK, Danelljan M, Yu F. Segment anything meets point tracking. arXiv:230701197. 2023. doi:10.48550/arXiv.2307.01197.
2. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al. LLaMA: open and efficient foundation language models. arXiv:230213971. 2023. doi:10.48550/arXiv.2302.13971.
3. Segment anything
4. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation
5. Brown TB. Language models are few-shot learners. arXiv:2005.14165. 2020. doi:10.48550/arXiv.2005.14165.
6. Deep transfer learning for automatic speech recognition: towards better generalization10.1016/j.knosys.2023.110851 ↗
7. A review of deep transfer learning and recent advancements10.3390/technologies11020040 ↗
8. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. LoRA: low-rank adaptation of large language models. arXiv:2106.09685. 2021. doi:10.48550/arXiv.2106.09685.
9. Parameter-efficient transfer learning for NLP
10. Li XL, Liang P. Prefix-tuning: optimizing continuous prompts for generation. arXiv:2101.00190. doi:10.48550/arXiv.2101.00190.
11. Cost aggregation for open-vocabulary semantic segmentation
12. DenseCLIP: language-guided dense prediction with context-aware prompting
13. Vl-adapter: parameter-efficient transfer learning for vision-and-language tasks
14. Lu H, Huo Y, Yang G, Lu Z, Zhan W, Tomizuka M, et al. UniAdapter: unified parameter-efficient transfer learning for cross-modal modeling. arXiv:2302.06605. 2023. doi:10.48550/arXiv.2302.06605.
15. Learning transferable visual models from natural language supervision
16. Segment anything in 3D with NeRFs
17. Kheddar H. Transformers and large language models for efficient intrusion detection systems: a comprehensive survey. arXiv:2408.07583. 2024. doi:10.48550/arXiv.2408.07583.10.1016/j.inffus.2025.103347 ↗
18. Devlin J. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2018. doi:10.48550/arXiv.1810.04805.
19. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models
20. Visual instruction tuning
21. IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval
22. Dynamic modality interaction modeling for image-text retrieval
23. Stacked attention networks for image question answering
24. Unicoder-VL: a universal encoder for vision and language by cross-modal pre-training
25. Context-I2W: mapping images to context-dependent words for accurate zero-shot composed image retrieval
26. LocVTP: video-text pre-training for temporal localization
27. Towards fast and accurate image-text retrieval with self-supervised fine-grained alignment10.1109/TMM.2023.3280734 ↗
28. Dense contrastive learning for self-supervised visual pre-training
29. Tang Y, Yu J, Gai K, Wang Y, Hu Y, Xiong G, et al. Align before Search: aligning ads image to text for accurate cross-modal sponsored search. arXiv:2309.16141. 2023. doi:10.48550/arXiv.2309.16141.
30. Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. arXiv:2104.08691. 2021. doi:10.48550/arXiv.2104.08691.10.18653/v1/2021.emnlp-main.243 ↗
31. GPT understands, too10.1016/j.aiopen.2023.08.012 ↗
32. Pfeiffer J, Kamath A, Rücklé A, Cho K, Gurevych I. Adapterfusion: non-destructive task composition for transfer learning. arXiv:2005.00247. 2020. doi:10.48550/arXiv.2005.00247.10.18653/v1/2021.eacl-main.39 ↗
33. Rücklé A, Geigle G, Glockner M, Beck T, Pfeiffer J, Reimers N, et al. AdapterDrop: on the efficiency of adapters in transformers. arXiv:2010.11918. 2020. doi:10.48550/arXiv.2010.11918.10.18653/v1/2021.emnlp-main.626 ↗
34. Zaken EB, Ravfogel S, Goldberg Y. BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv:2106.10199. 2021. doi:10.48550/arXiv.2106.10199.
35. Guo D, Rush AM, Kim Y. Parameter-efficient transfer learning with diff pruning. arXiv:2012.07463. 2020. doi:10.48550/arXiv.2012.07463.10.18653/v1/2021.acl-long.378 ↗
36. Zhang Q, Chen M, Bukharin A, Karampatziakis N, He P, Cheng Y, et al. AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. arXiv:2303.10512. 2023. doi:10.48550/arXiv.2303.10512.
37. Renduchintala A, Konuk T, Kuchaiev O. Tied-LoRA: enhacing parameter efficiency of lora with weight tying. arXiv:2311.09578. 2023. doi:10.48550/arXiv.2311.09578.10.18653/v1/2024.naacl-long.481 ↗
38. Liu SY, Wang CY, Yin H, Molchanov P, Wang YCF, Cheng KT, et al. DoRA: weight-decomposed low-rank adaptation. arXiv:2402.09353. 2024. doi:10.48550/arXiv.2402.09353.
39. Hyeon-Woo N, Ye-Bin M, Oh TH. Fedpara: low-rank hadamard product for communication-efficient federated learning. arXiv:2108.06098. 2021. doi:10.48550/arXiv.2108.06098.
40. Dosovitskiy A. An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929. 2020. doi:10.48550/arXiv.2010.11929.
41. Microsoft COCO: common objects in context
42. Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models
43. Localizing moments in video with natural language
44. Making the V in VQA matter: elevating the role of image understanding in visual question answering
45. Video question answering via gradually refined attention over appearance and motion
46. Uniter: universal image-text representation learning
47. Li W, Gao C, Niu G, Xiao X, Liu H, Liu J, et al. UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learning. arXiv:2012.15409. 2020. doi:10.18653/v1/2021.acl-long.202.10.18653/v1/2021.acl-long.202 ↗
48. Scaling up visual and vision-language representation learning with noisy text supervision
49. Align before fuse: vision and language representation learning with momentum distillation
50. Oscar: object-semantics aligned pre-training for vision-language tasks
51. Unifying vision-and-language tasks via text generation
52. Seeing out of the box: end-to-end pre-training for vision-language representation learning
53. Conceptual 12m: pushing web-scale image-text pre-training to recognize long-tail visual concepts
54. Zhang P, Li X, Hu X, Yang J, Zhang L, Wang L, et al. VinVL: making visual representations matter in vision-language models. arXiv:2101.00529. 2021. doi:10.48550/arXiv.2101.00529.10.1109/CVPR46437.2021.00553 ↗
55. Scaling up vision-language pre-training for image captioning
56. Less is more: CLIPBERT for video-and-language learning via sparse sampling
57. Frozen in time: a joint video and image encoder for end-to-end retrieval
58. Look before you speak: visually contextualized utterances
59. Align and prompt: video-and-language pre-training with entity prompts
60. Fu TJ, Li L, Gan Z, Lin K, Wang WY, Wang L, et al. VIOLET: end-to-end video-language transformers with masked visual-token modeling. arXiv:2111.12681. 2021. doi:10.48550/arXiv.2111.12681.
61. Just ask: learning to answer questions from millions of narrated videos
62. All in one: exploring unified video-language pre-training
63. Clip4Clip: an empirical study of clip for end to end video clip retrieval and captioning10.1016/j.neucom.2022.07.028 ↗
64. MERLOT: multimodal neural script knowledge models