Comparative Analysis of Knowledge Internalization through Supervised Fine-Tuning and Contextual Reasoning through Retrieval-Augmented Fine-Tuning in Large Language Models for Indonesian Village Regulations

Abstract

Large language models (LLMs) are prone to hallucination, making their reliability in Indonesian-language legal question answering uncertain, particularly for Village Regulations, which are predominantly available as scanned PDF documents. This study provides a controlled comparison between knowledge internalization through Supervised Fine-Tuning (SFT) and contextual reasoning through Retrieval-Augmented Fine-Tuning (RAFT), using both LoRA and full fine-tuning from the same Llama-3.1-8B-Instruct checkpoint. The study also develops a traceable document-processing pipeline and a question-answer dataset derived from 55 Village Regulations from Badung Regency, consisting of 50 scanned and five born-digital documents. Model performance was evaluated using ROUGE-L, METEOR, BERTScore, and RAGAS. At the fourth epoch, RAFT-LoRA achieved the highest aggregate score (0.883), followed by RAFT-Full (0.881), while the best-performing SFT configuration reached 0.749. RAFT improved faithfulness from 0.640 to 0.872 and context recall from 0.637 to 0.907. The results indicate that training data structure had a greater impact on performance than the parameter-update strategy, while LoRA maintained RAFT performance with lower adaptation costs. These findings suggest that legal knowledge can be maintained in updatable external documents without requiring model retraining, while the model can be trained to generate evidence-grounded responses. However, the findings are limited to a single regency and automated evaluation; therefore, validation by legal experts is required before deployment.