Memory-Efficient Adaptation and Unanswerable-Aware Supervision for Reliable Private Email Question Answering
Source abstract
Abstract
Private document question answering (QA) requires language models to answer from local evidence while abstaining when the supplied context is insufficient. This study evaluates quantized parameter-efficient adaptation for private email QA using EnronQA. A Qwen2.5-7B-Instruct backbone is adapted with QLoRA and QPiSSA under matched rank, target modules, quantization, and training budget. QPiSSA combines PiSSA initialization with residual quantization, whereas QLoRA quantizes the original frozen weight matrix and trains standard low-rank adapters. In addition to answerable-only fine-tuning, an unanswerable-aware supervision variant augments training with synthetic missing-evidence examples whose target response is “I don’t know.” Experiments are conducted on a fixed EnronQA subset with 20,000 training, 2000 development, and 2000 test examples. Fine-tuning substantially improves oracle-email answer quality over zero-shot prompting, but answerable-only tuning sharply reduces abstention under missing evidence. Adding unanswerable examples restores abstention accuracy to approximately 0.998 for both QLoRA and QPiSSA while preserving most answer-generation performance. Bootstrap confidence intervals over the fixed test set, multi-seed evaluation, BERTScore evaluation, leakage diagnostics, strict filtered unanswerable evaluation, retrieval-augmented testing, quantization-error diagnostics, and a small diagnostic ablation support a cautious interpretation: QLoRA and QPiSSA perform similarly under this matched configuration, and the main reliability gain comes from explicit missing-evidence supervision rather than from the adapter choice alone.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Expert finding and Q&A systems · Personal Information Management and User Behavior
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Siirt University, Siirt 56100, Türkiye
References (18)
Show source-provided references
1. Wang, S., Liu, J., Song, S., Cheng, J., Fu, Y., Guo, P., Fang, K., Zhu, Y., and Dou, Z. (2024). DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation. arXiv.
2. Ryan, M.J., Xu, D., Nivera, C., and Campos, D. (2025). EnronQA: Towards Personalized RAG over Private Documents. arXiv.
3. Li, J., Yuan, Y., and Zhang, Z. (2024). Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases. arXiv.
4. Wang, Y., Qu, W., Zhai, S., Jiang, Y., Liu, Z., Liu, Y., Dong, Y., and Zhang, J. (2025). Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries. arXiv.
5. Hounie, I., Kanatsoulis, C., Tandon, A., and Ribeiro, A. (2025). LoRTA: Low Rank Tensor Adaptation of Large Language Models. arXiv.
6. Al-Onaizan, Y., Bansal, M., and Chen, Y.N. (2024). ApiQ: Finetuning of 2-Bit Quantized Large Language Model. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics.
7. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S.Q. (2024). Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey. arXiv.
8. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023, January 10–16). QLORA: Efficient finetuning of quantized LLMs. Proceedings of the 37th International Conference on Neural Information Processing Systems, Red Hook, NY, USA.
9. PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models10.52202/079017-3846 ↗
10. Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (2025). HD-PiSSA: High-Rank Distributed Orthogonal Adaptation. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics.
11. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
12. Biderman, D., Portes, J., Ortiz, J.J.G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., and Frankle, J. (2024). LoRA Learns Less and Forgets Less. arXiv.
13. Guo, H., Greengard, P., Xing, E., and Kim, Y. (2024, January 7–11). LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
14. Lawton, N., Padmakumar, A., Gaspers, J., FitzGerald, J., Kumar, A., Steeg, G.V., and Galstyan, A. (2024). QuAILoRA: Quantization-Aware Initialization for LoRA. arXiv.
15. Che, W., Nabende, J., Shutova, E., and Pilehvar, M.T. (2025). L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.
16. Duh, K., Gomez, H., and Bethard, S. (2024). LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics.
17. Hu, X., Cheng, X., Liu, P., Liu, W., Luan, J., Wang, B., and Liu, Y. (2025). DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models. Advances in Knowledge Discovery and Data Mining, Springer Nature.10.1007/978-981-96-8186-0_2 ↗
18. Chong, Y.H., and Qu, P. (2025). Singular Value Decomposition on Kronecker Adaptation for Large Language Model. arXiv.