BOLT: Building Open-Source LLMs for Your Target Domain via Automated Hierarchical Knowledge Distillation
Source abstract
Abstract
Adapting open-source large language models (LLMs) to specialized domains remains a critical challenge due to domain knowledge gaps, data scarcity, and reference hallucination. Existing approaches often neglect the structural characteristics of domain knowledge and fail to provide principled estimations of knowledge scope, resulting in data homogenization and suboptimal adaptation, while leaving reference hallucination unmitigated. This paper introduces BOLT(Building Open-source LLMs for your Target domain), a modular end-to-end framework that tailors open-source LLMs for domain-specific scenarios. BOLT systematically estimates domain scope, constructs structured hierarchical knowledge trees, distills diverse and semantically aligned training data from advanced teacher LLMs, and employs curriculum learning for progressive model optimization. To address reference hallucination, BOLT substitutes generative methods, which are susceptible to hallucinations, with a matching-based strategy, thereby alleviating the problem and significantly improving reference recommendation accuracy. Extensive experiments across diverse domains and models demonstrate that BOLT enables the efficient modeling of structured hierarchical domain knowledge and effectively enhances reference recommendation accuracy while preserving both training efficiency and robustness throughout the adaptation process.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Advanced Graph Neural Networks · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Tongji University, Shanghai 201804, China
Funding
Fundamental Research Funds for the Central Universities
References (43)
Show source-provided references
1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
2. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., and Ruan, C. (2024). Deepseek-v3 technical report. arXiv.
3. Zhang, H., Chen, J., Jiang, F., Yu, F., Chen, Z., Li, J., Chen, G., Wu, X., Zhang, Z., and Xiao, Q. (2023). Huatuogpt, towards taming language model to be a doctor. arXiv.10.18653/v1/2023.findings-emnlp.725 ↗
4. Cui, J., Li, Z., Yan, Y., Chen, B., and Yuan, L. (2023). Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv.
5. Zhang, X., and Yang, Q. (2023, January 21–25). Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, Birmingham, UK.10.1145/3583780.3615285 ↗
6. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv.
7. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv.
8. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen technical report. arXiv.
9. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., and Hajishirzi, H. (2022). Self-instruct: Aligning language models with self-generated instructions. arXiv.10.18653/v1/2023.acl-long.754 ↗
10. Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. (2023). Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. arXiv.10.18653/v1/2023.emnlp-main.183 ↗
11. Survey of hallucination in natural language generation10.1145/3571730 ↗
12. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT10.1080/00330124.2023.2190373 ↗
13. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv.
14. Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. (2023). Bloomberggpt: A large language model for finance. arXiv.
15. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y.T. (2023). Textbooks are all you need ii: Phi-1.5 technical report. arXiv.
16. Large language model as attributed training data generator: A tale of diversity and bias
17. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T.B. (2025, September 23). Stanford Alpaca: An iNstruction-Following Llama Model. Available online: https://www.kaggle.com/discussions/general/394598.
18. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. (2023). Wizardlm: Empowering large language models to follow complex instructions. arXiv.
19. Chen, H., Quan, X., Chen, H., Yan, M., and Zhang, J. (2024). Knowledge distillation for closed-source language models. arXiv.
20. Liang, C., Zuo, S., Zhang, Q., He, P., Chen, W., and Zhao, T. (2023, January 23–29). Less is more: Task-aware layer-wise distillation for language model compression. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA. PMLR.10.3390/s23031483 ↗
21. Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. (2023). Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv.
22. Direct preference optimization: Your language model is secretly a reward model
23. Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, T. (2024). A survey on knowledge distillation of large language models. arXiv.
24. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information10.1177/08465371231171125 ↗
25. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions
26. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., and Nabeshima, N. (2020). The pile: An 800gb dataset of diverse text for language modeling. arXiv.
27. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C.C.T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., and Saarikivi, O. (2023). Textbooks are all you need. arXiv.
28. Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., and Babenko, A. (2020). Editable neural networks. arXiv.
29. Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., and Ni, Y. (2024). A comprehensive study of knowledge editing for large language models. arXiv.
30. Exposing attention glitches with flip-flop language modeling
31. Training language models to follow instructions with human feedback
32. Factuality enhanced language models for open-ended text generation
33. Inference-time intervention: Eliciting truthful answers from a language model
34. Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., and Yih, S.W.t. (2023). Trusting your evidence: Hallucinate less with context-aware decoding. arXiv.10.18653/v1/2024.naacl-short.69 ↗
35. Lima: Less is more for alignment
36. Retrieval-augmented generation for knowledge-intensive NLP tasks
37. Étude comparative de la distribution florale dans une portion des Alpes et des Jura
38. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. (2024). The Llama 3 herd of models. arXiv.
39. Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., and Yan, D. (2023). Baichuan 2: Open large-scale language models. arXiv.
40. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., and Huang, F. (2024). Qwen2 Technical Report. arXiv.
41. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv.
42. A mathematical theory of communication10.1002/j.1538-7305.1948.tb01338.x ↗
43. GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., and Zhao, H. (2024). Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv.