Developing Curriculum-Aligned Math Assessments Using Generative AI in Malaysia

TLDR: This research explores using Generative AI, specifically OpenAI’s GPT-4o, to create curriculum-aligned multiple-choice questions for Malaysian secondary mathematics in Bahasa Melayu. It compares four methods, finding that Retrieval-Augmented Generation (RAG) pipelines significantly outperform non-grounded approaches in terms of curriculum alignment and factual validity. The study also introduces a novel dual evaluation framework and discusses the trade-offs between different RAG implementations, highlighting the critical role of grounding AI models with official curriculum documents for educational content generation in low-resource languages.

The Malaysian education system, guided by the Kurikulum Standard Sekolah Menengah (KSSM), faces a significant challenge: creating a large volume of diverse, consistent, and high-quality assessment materials. This task places a considerable burden on educators, who must develop questions for various assessments, homework, and revision exercises. The rise of digital learning platforms further amplifies this need, requiring vast repositories of questions to support personalized and adaptive learning.

Generative AI (GenAI), powered by advanced models like OpenAI’s GPT-4o, offers a promising solution to automate question generation. This automation can dramatically reduce teacher workload and facilitate the creation of extensive question banks. However, using AI in education comes with risks, primarily factual inaccuracies, contextual irrelevance, and the generation of misleading information, often referred to as “hallucinations.” An incorrectly formulated question can undermine the learning process and reinforce misconceptions.

The Grounding Problem and Low-Resource Languages

The core of these risks lies in what is known as the “grounding problem.” Large Language Models (LLMs) are typically trained on vast, general, and predominantly English-language internet data. Their internal knowledge, while extensive, doesn’t inherently align with the specific, localized, and structured knowledge of a national curriculum like Malaysia’s KSSM. This misalignment is particularly challenging in low-resource language contexts. Bahasa Melayu, despite being Malaysia’s national language, has a significantly smaller digital footprint compared to English, making it a lower-resourced language for global AI development. This data disparity means an AI model’s understanding of the Malaysian Form 1 Mathematics curriculum—its specific terminology, pedagogical sequencing, and preferred methods—is likely to be sparse or unreliable. Without a mechanism to anchor the model’s generation process to authoritative curriculum documents, it is highly prone to producing questions that are contextually invalid.

Retrieval-Augmented Generation (RAG) as a Solution

Retrieval-Augmented Generation (RAG) has emerged as a leading technique to address this grounding problem. RAG combines the strengths of a pre-trained LLM with an external, retrievable knowledge source. When given a prompt, the system first retrieves relevant documents from this external source and then uses this retrieved context to inform and constrain its generated output. This approach significantly reduces hallucinations and improves the factual accuracy of LLM responses. In educational technology, RAG is invaluable because it allows AI systems to provide factually consistent, curriculum-aligned answers and content. Moreover, since the knowledge base is external, it can be easily updated with new information, such as a revised syllabus, without the need for costly LLM retraining.

Research Approach and Methodology

This study systematically evaluated four distinct GenAI pipelines for generating multiple-choice questions for Form 1 Mathematics in Bahasa Melayu. All pipelines used OpenAI’s GPT-4o as the core generative engine and were grounded in official Malaysian curriculum documents: teacher-prepared notes (Nota Bab 1.pdf) and the Yearly Teaching Plan (RPT Bab 1.pdf).

The four generation methods explored were:

1. Structured Prompting: Used GPT-4o’s function calling to ensure a syntactically valid output structure.
2. Basic Prompting: A simpler, non-grounded baseline that instructed the model to generate JSON output.
3. Framework-based RAG (LangChain): A standard RAG implementation using the LangChain framework, involving document ingestion, chunking, embedding, and retrieval from a FAISS vector store.
4. Manual RAG: A more controlled, domain-aware RAG pipeline with custom chunking logic to preserve the semantic structure of the educational text, using PyMuPDF for parsing and scikit-learn for similarity ranking.

A dual-pronged automated evaluation framework assessed each generated question:

1. Curriculum Alignment via Semantic Textual Similarity (STS): Measured how closely a generated question aligned with official learning standards by comparing its semantic similarity to chunks from the RPT Bab 1.pdf.
2. Contextual Validity via RAG-based Question-Answering (RAG-QA): A novel functional test where the question was fed as a query to a separate RAG-QA pipeline, with the RPT Bab 1.pdf as its sole knowledge base. If the QA system could answer the question using the RPT, it was deemed “Valid.”

Also Read:

• Assessing AI’s Role in Guiding Interdisciplinary STEM Learning
• Enhancing Mathematical Problem-Solving with Knowledge Graphs and Executable Code

Key Findings and Implications

The results provided clear evidence supporting the necessity of grounding. All methods produced linguistically fluent questions in Bahasa Melayu. However, a stark difference in factual and contextual accuracy was observed:

• Superiority of RAG: RAG-based methods (LangChain RAG and Manual RAG) significantly outperformed non-grounded prompting methods. Their average STS scores were much higher (0.86 and 0.89 vs. 0.58 and 0.55), indicating stronger semantic alignment with official learning standards.
• Compelling Validity: The RAG-QA Validity metric showed a dramatic difference. RAG-generated questions had validity rates of 92% and 96%, meaning they were overwhelmingly answerable using the RPT. In contrast, non-grounded questions had validity rates of only 15% and 12%, confirming they frequently strayed from core learning objectives.
• RAG Implementation Trade-offs: Manual RAG achieved slightly higher scores than LangChain RAG, suggesting that domain-aware chunking can yield measurable improvements in question quality by providing higher-quality context to the LLM. However, this comes at the cost of significantly increased development complexity compared to the more efficient framework-based approach.

The study confirmed GPT-4o’s proficiency in handling specific Bahasa Melayu mathematical terminology. The RAG methods were particularly effective at adopting the precise phrasing and style of the teacher-prepared notes. This research underscores that AI should augment, not replace, professional educators. While automated generation and evaluation can create high-quality initial drafts, the final assessment of pedagogical value still requires human expert review to assess higher-order thinking skills.

This work presents a validated methodology for generating curriculum-specific educational content in a low-resource language, introduces a symbiotic RAG-QA evaluation technique, and provides actionable insights for the development and deployment of practical EdTech solutions in Malaysia and similar regions. For more details, you can refer to the full research paper here.