MotivGraph-SoIQ: Integrating Motivational Knowledge Graphs and Socratic Dialogue for Enhanced LLM Ideation

2025-09-26

AI Summary

Large language models (LLMs) exhibit confirmation bias in academic idea generation, compromising the foundational rigor and novelty of generated concepts. To address this, we propose MotivGraph-SoIQ: a framework that explicitly models three core academic motivation elements—problem, challenge, and solution—via a structured Motivation Knowledge Graph (MotivGraph); and introduces a query-driven, dual-agent Socratic dialogue mechanism that refines ideas through iterative, structured questioning and critical reasoning. This design significantly enhances the reliability and depth of ideation. Evaluated on an ICLR’25 paper-topic dataset, MotivGraph-SoIQ outperforms existing state-of-the-art methods across automated LLM scoring, ELO ranking, and human evaluation—demonstrating the efficacy of jointly modeling motivation-aware guidance and critical dialogue for rigorous, innovative idea generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) hold substantial potential for accelerating academic ideation but face critical challenges in grounding ideas and mitigating confirmation bias for further refinement. We propose integrating motivational knowledge graphs and socratic dialogue to address these limitations in enhanced LLM ideation (MotivGraph-SoIQ). This novel framework provides essential grounding and practical idea improvement steps for LLM ideation by integrating a Motivational Knowledge Graph (MotivGraph) with a Q-Driven Socratic Ideator. The MotivGraph structurally stores three key node types(problem, challenge and solution) to offer motivation grounding for the LLM ideation process. The Ideator is a dual-agent system utilizing Socratic questioning, which facilitates a rigorous refinement process that mitigates confirmation bias and improves idea quality across novelty, experimental rigor, and motivational rationality dimensions. On the ICLR25 paper topics dataset, MotivGraph-SoIQ exhibits clear advantages over existing state-of-the-art approaches across LLM-based scoring, ELO ranking, and human evaluation metrics.

Problem

• Grounding ideas in LLM academic ideation process
• Mitigating confirmation bias for idea refinement
• Improving idea quality across multiple evaluation dimensions

Innovation

• Integrates motivational knowledge graphs for idea grounding
• Uses dual-agent Socratic questioning to reduce bias
• Combines problem-challenge-solution nodes for motivation