Analisis Groundedness Model LLM dalam Pembentukan Soal Cerita Matematika dari Teks Bacaan Berbahasa Indonesia
Abstract
Soal cerita matematika (Math Word Problem/MWP) merupakan instrumen evaluasi penting yang mengintegrasikan kemampuan literasi dan numerasi. Perkembangan Large Language Model (LLM) membuka peluang otomatisasi pembuatan MWP dari teks bacaan. Namun, tantangan utama adalah fenomena hallucination yang menurunkan kualitas pedagogis soal. Penelitian ini bertujuan: (1) mengembangkan kerangka evaluasi groundedness berbasis enam aspek sebagai produk utama, dan (2) mengevaluasi tingkat groundedness MWP yang dihasilkan model gemma3:1b dari teks bacaan berbahasa Indonesia. Pendekatan Research and Development (R&D) dengan model ADDIE digunakan. Enam aspek groundedness dikembangkan: Entity Consistency (EC), Event Consistency (EV), Context Alignment (CA), Semantic Alignment (SA), Numerical Groundedness (QN), dan Question Relevance (QR), masing-masing dinilai menggunakan skala biner 0–1. Evaluasi dilakukan terhadap 460 soal cerita matematika valid dari empat teks bacaan BSE SD Kurikulum Merdeka menggunakan model gemma3:1b dengan lima konfigurasi parameter (E1–E5). Hasil menunjukkan gemma3:1b grounded pada teks sederhana–menengah (Teks 1: 73,25%; Teks 3: 75,91%) namun gagal pada teks kompleks (Teks 4: 47,22%). Konfigurasi E3 (temperature=0,3; top_p=0,9) memberikan keseimbangan terbaik. Kerangka evaluasi terbukti reliabel dengan Cohen's Kappa 0,68–0,81.
Downloads
References
OECD, PISA 2022 Results (Volume I): The State of Learning and Equity in Education, OECD Publishing, 2023. https://doi.org/10.1787/53f23881-en
L. Verschaffel, W. Van Dooren, B. Greer, and S. Mukhopadhyay, "Reconceptualising word problems as exercises in mathematical modelling," J. für Mathematik-Didaktik, vol. 31, no. 1, pp. 9–29, 2010. https://doi.org/10.1007/s13138-010-0007-x
Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi, Panduan Pembelajaran dan Asesmen Kurikulum Merdeka Jenjang Pendidikan Dasar dan Menengah, Kemendikbudristek, 2022. https://kurikulum.kemdikbud.go.id
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Kumar, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, ... D. Amodei, "Language models are few-shot learners," Adv. Neural Inf. Process. Syst., vol. 33, pp. 1877–1901, 2020. https://arxiv.org/abs/2005.14165
Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung, "Survey of hallucination in natural language generation," ACM Comput. Surv., vol. 55, no. 12, pp. 1–38, 2023. https://doi.org/10.1145/3571730
J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, "On faithfulness and factuality in abstractive summarization," in Proc. 58th Annu. Meet. Assoc. Comput. Linguist., 2020, pp. 1906–1919. https://doi.org/10.18653/v1/2020.acl-main.173
P. Laban, T. Schneider, C.-S. Wu, and M. A. Hearst, "SummaC: Re-visiting NLI-based models for inconsistency detection in summarization," Trans. Assoc. Comput. Linguist., vol. 10, pp. 163–177, 2022. https://doi.org/10.1162/tacl_a_00453
H. Rashkin, E. M. Smith, M. Li, and Y.-L. Boureau, "Increasing faithfulness in knowledge-grounded dialogue with controllable features," in Proc. 59th Annu. Meet. Assoc. Comput. Linguist. 11th Int. Jt. Conf. Nat. Lang. Process., 2021, pp. 763–777. https://doi.org/10.18653/v1/2021.acl-long.58
N. Dziri, H. Rashkin, T. Shi, and K. McKeown, "Faithfulness in natural language generation: A systematic survey of definitions, evaluation, and methodologies," arXiv, 2022. https://arxiv.org/abs/2203.05227
R. M. Branch, Instructional Design: The ADDIE Approach, Springer, 2009. https://doi.org/10.1007/978-0-387-09506-6
S. Mishra, D. Khashabi, C. Baral, Y. Choi, and H. Hajishirzi, "Reframing instructional prompts to GPTk's language," in Findings Assoc. Comput. Linguist.: ACL 2022, 2022, pp. 589–612. https://doi.org/10.18653/v1/2022.findings-acl.50
P. Wang, L. Pang, M. Lan, J. Shen, S. Cheng, and X. Cheng, "Math-NLP: Mathematical reasoning in large language models," arXiv, 2023. https://arxiv.org/abs/2304.01399
Y. Nie, Y. Tian, M. Bansal, and R. Rudinger, "Do large language models know what they don't know? A survey on numerical reasoning capabilities and limitations," arXiv, 2023. https://arxiv.org/abs/2305.09898
A. Ushio, L. Espinosa-Anke, S. Schockaert, and J. Camacho-Collados, "BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies?" in Proc. 59th Annu. Meet. Assoc. Comput. Linguist., 2021, pp. 3609–3624. https://doi.org/10.18653/v1/2021.acl-long.279
J. R. Landis and G. G. Koch, "The measurement of observer agreement for categorical data," Biometrics, vol. 33, no. 1, pp. 159–174, 1977. https://doi.org/10.2307/2529310
M. L. McHugh, "Interrater reliability: The kappa statistic," Biochemia Medica, vol. 22, no. 3, pp. 276–282, 2012. https://doi.org/10.11613/BM.2012.031
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Adv. Neural Inf. Process. Syst., vol. 30, 2017. https://doi.org/10.48550/arXiv.1706.03762
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, "Training language models to follow instructions with human feedback," Adv. Neural Inf. Process. Syst., vol. 35, pp. 27730–27744, 2022. https://arxiv.org/abs/2203.02155
A. Prasetya and M. D. Aprianto, "Structural classification of Indonesian arithmetic word problems using hierarchical agglomerative clustering," Jurnal Komputer Teknologi Informasi Sistem Komputer (JUKTISI), vol. 5, no. 1, pp. 393–400, 2026. https://doi.org/10.62712/juktisi.v5i1.1026
Copyright (c) 2026 Feri Andriawan Saputra, Agung Prasetya, Mohamad Khoirul Ansor
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
.png){kind=logo}
