RUS  ENG
Full version
JOURNALS // Zapiski Nauchnykh Seminarov POMI // Archive

Zap. Nauchn. Sem. POMI, 2024 Volume 540, Pages 162–177 (Mi znsl7549)

Improving RAG with LoRA finetuning for persona text generation

V. Pavliukevichab, A. Zherdevaab, O. Makhnytkinaab, D. Dyrmovskiyab

a Speech Technology Center, Saint Petersburg, Russia
b ITMO University, Saint Petersburg, Russia

Abstract: We address the challenge of maintaining consistency in Retrieval-Augmented Generation (RAG) systems for persona text generation when databases are subject to rapid updates and conventional large language model (LLM) fine-tuning is inadequate. We propose an approach that enhances an existing RAG system used for persona-based information retrieval in dialogue agents through the application of Low-Rank Adaptation fine-tuning on synthetic data. We find that this method improves the system's logic and correctness by 5% on SSA scores and ensures that generated content remains more coherent and contextually relevant.

Key words and phrases: retrieval-augmented generation, large language models, fine-tuning.

Received: 15.11.2024

Language: English



© Steklov Math. Inst. of RAS, 2025