Qlora Fine-tuning of Mistral-7b for Serbian High School Mathematics Competition Tasks

Keywords:
Large Language Models, Qlora Fine-Tuning, Mathematics Competitions, LLM-As-A-Judge, Multi-Criteria Evaluation.

Abstract:
This paper examines the extent to which QLoRA fine-tuning can improve the performance of the large language model Mistral-7B-Instruct-v0.3 on Serbian high school mathematics competition tasks. Based on a dataset of tasks in Serbian, a fine-tuned model, Math-SRB-Mistral-7B, was developed and compared with the base model. The responses were evaluated using Claude 3.7 Sonnet as a judge, according to multiple criteria, including final answer accuracy, logical coherence, explanation quality, and an aggregate score. The results suggest that the applied fine-tuning did not lead to improved performance; instead, the fine-tuned model achieved slightly lower scores across all evaluated dimensions. This finding suggests that parameter-efficient adaptation of general-purpose LLMs on small and challenging mathematical datasets does not necessarily result in better generalization to new tasks. At the same time, the results highlight the importance of multi-criteria evaluation in the analysis of mathematical reasoning generated by LLMs.

CITATION:

IEEE format

Copy to clipboard

M. Pavković, M. Svičević, A. Milutinović, N. Vučićević, A. Milenković, “Qlora Fine-tuning of Mistral-7b for Serbian High School Mathematics Competition Tasks,” in Sinteza 2025 - International Scientific Conference on Information Technology, Computer Science, and Data Science, Belgrade, Singidunum University, Serbia, 2026, pp. 129-136. doi:10.15308/Sinteza-2026-129-136

APA format

Copy to clipboard

Pavković, M., Svičević, M., Milutinović, A., Vučićević, N., Milenković, A. (2026). Qlora Fine-tuning of Mistral-7b for Serbian High School Mathematics Competition Tasks. Paper presented at Sinteza 2025 - International Scientific Conference on Information Technology, Computer Science, and Data Science. doi:10.15308/Sinteza-2026-129-136

BibTeX format
Download

RefWorks Tagged format
Download