KGCE: Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models

2025-10-05

IEEE International Conference on Systems, Man and Cybernetics

AI Summary

This work addresses the lack of fine-grained task execution evaluation in existing educational agent benchmarking frameworks, particularly across heterogeneous platforms such as private school software environments. To bridge this gap, the authors construct a cross-platform dataset comprising 104 educational tasks and propose the first dual-graph evaluation framework tailored for educational settings. The framework enhances agents’ understanding of proprietary software through knowledge base augmentation and enables fine-grained assessment of sub-goal-level execution trajectories by integrating structured task decomposition with multimodal large language models. Experimental results demonstrate that this approach significantly improves task execution efficiency of agents in private school environments while delivering interpretable and platform-agnostic performance metrics.

Technology Category

• Multiagent Systems: Multiagent Planning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

With the rapid adoption of multimodal large language models (MLMs) in autonomous agents, cross-platform task execution capabilities in educational settings have garnered significant attention. However, existing benchmark frameworks still exhibit notable deficiencies in supporting cross-platform tasks in educational contexts, especially when dealing with school-specific software (such as XiaoYa Intelligent Assistant, HuaShi XiaZi, etc.), where the efficiency of agents often significantly decreases due to a lack of understanding of the structural specifics of these private-domain software. Additionally, current evaluation methods heavily rely on coarse-grained metrics like goal orientation or trajectory matching, making it challenging to capture the detailed execution and efficiency of agents in complex tasks. To address these issues, we propose KGCE (Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models), a novel benchmarking platform that integrates knowledge base enhancement and a dual-graph evaluation framework. We first constructed a dataset comprising 104 education-related tasks, covering Windows, Android, and cross-platform collaborative tasks. KGCE introduces a dual-graph evaluation framework that decomposes tasks into multiple sub-goals and verifies their completion status, providing fine-grained evaluation metrics. To overcome the execution bottlenecks of existing agents in private-domain tasks, we developed an enhanced agent system incorporating a knowledge base specific to school-specific software. The code can be found at https://github.com/Kinginlife/KGCE.

Problem

• cross-platform educational agent
• benchmarking
• multimodal language models
• private-domain software
• fine-grained evaluation

Innovation

• knowledge-augmented agent
• dual-graph evaluation
• cross-platform benchmarking
• multimodal language models
• educational AI agent