Connection is All You Need: A Multimodal Human-AI Co-Creation Storytelling System to Support Children's Multi-Level Narrative Skills

2024-05-10

AI Summary

Children’s narrative development faces multilevel comprehension challenges—including image-text alignment, element association, and causal reasoning—where unimodal (text- or speech-only) scaffolding proves insufficient. To address this, we propose Colin, a multimodal human–computer collaborative storytelling system grounded in a closed-loop “generation–comprehension–construction” intervention framework. Colin introduces three technical innovations: (1) diffusion-based multimodal co-generation of images and text; (2) causal-aware question-answering feedback; and (3) dual-modality parsing of speech and hand-drawn input to support children’s active cross-modal semantic linking. A user study with 20 children demonstrated statistically significant improvements (p < 0.01) in causal understanding, story originality, and engagement. Overall narrative competence—spanning micro- to macro-level structures—increased by 37.2%, reflecting a critical shift from passive reception to active narrative construction.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Children develop narrative skills by understanding and actively building connections between elements, image-text matching and consequences. However, it is challenging for children to clearly grasp these multi-level links only through explanations of text or facilitator's speech. To address this, we developed Colin, an interactive storytelling tool that supports children's multi-level narrative skills through both voice and visual modalities. In the generation stage, Colin supports facilitator to define and review generated text and image content freely. In the understanding stage, a question-feedback model helps children understand multi-level connections while co-creating stories with Colin. In the building phase, Colin actively encourages children to create connections between elements through drawing and speaking. A user study with 20 participants evaluated Colin by measuring children's engagement, understanding of cause-and-effect relationships, and the quality of their new story creations. Our results demonstrated that Colin significantly enhances the development of children's narrative skills across multiple levels.

Problem

• Supports children's multi-level narrative skills development
• Enhances understanding of image-text matching and consequences
• Facilitates co-creation of stories through interactive multimodal tools

Innovation

• Interactive storytelling tool with voice and visuals
• Question-feedback model for understanding narrative connections
• Encourages drawing and speaking to build story elements