Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO
Large language models (LLMs) often struggle to generate explanatory text that simultaneously satisfies semantic accuracy and pedagogical soundness. Method: We propose a lightweight semantic reward modeling approach: a small encoder-only Transformer serves as the reward model, computing fine-grained, dense rewards via cosine similarity between generated and reference explanations—replacing brittle keyword-based metrics (e.g., ROUGE) or costly LLM-based evaluators. This reward model is integrated into the Group Relative Policy Optimisation (GRPO) framework, augmented with domain-adaptive continual pretraining (CPT) and supervised fine-tuning (SFT). Results: Evaluated on the Italian medical school entrance exam explanation generation task, our method significantly improves explanation faithfulness and clarity over strong SFT baselines. It demonstrates that effective semantic alignment can be achieved without large-scale judgment models, validating both feasibility and efficacy of our lightweight, semantics-driven optimization strategy.