Mitigating Hallucinations in Third-Grade Primary Mathematics Education: An Empirical Analysis of Local RAG and LLMs

Ocak A., Diri S., Kilimci Z. H.

Süleyman Demirel Üniversitesi Fen Bilimleri Enstitüsü Dergisi, cilt.30, sa.2, ss.140-153, 2026 (TRDizin)

Özet

This study investigates an optimized Retrieval-Augmented Generation (RAG) architecture running entirely on local hardware to address the issues of recency and hallucination encountered in Turkish primary school mathematics education. Within the scope of the research, Llama3, Mistral, Gemma, and Phi3 language models were tested end-to-end using fixed and semantic chunking strategies alongside multilingual MiniLM and Turkish-specific BERTurk embedding models. The primary dataset comprises texts extracted from 25 educational videos, totaling approximately 20 hours of content, which were curated into a 1,270-row "Question-Answer-Context" ground truth set by two domain experts. The reliability of the dataset was validated with a Cohen’s Kappa coefficient of 0.85. Evaluation was conducted using Google Gemini as an LLM-as-a-Judge on a structured 1–5 Likert scale to measure relevance and faithfulness. Experimental findings indicate that the RAG architecture significantly improves reliability by reducing the hallucination rate in the Mistral model from 60% to 35%. The Gemma-7B model emerged as the most successful system, achieving a response time of ~2.4 seconds and an F1 score of 0.28. In conclusion, optimized local RAG architectures provide a robust, privacy-preserving alternative for Turkish educational technologies.