Syllables to Scenes: Literary-Guided Free-Viewpoint 3D Scene Synthesis from Japanese Haiku

2025-02-17

AI Summary

Mapping literary abstractions—such as haiku—to navigable 3D spaces in the metaverse suffers from semantic distortion and emotional attenuation. To address this, we propose a Hierarchical Literary Criticism Theory-Guided Parsing method (H-LCTGP) and a Progressive 3D Synthesis framework (PDS), enabling, for the first time, joint modeling and staged generation of haiku’s implicit affect and explicit imagery. Our approach integrates poetic analysis, multi-stage diffusion modeling, geometric optimization, and real-time rendering into an end-to-end, literature-guided 3D generation pipeline. Experiments demonstrate that our method significantly outperforms state-of-the-art text-to-3D approaches in both literary fidelity and visual quality, producing high-fidelity, immersive scenes adhering to haiku’s aesthetic principles. This work establishes a novel paradigm for metaverse-based activation of intangible cultural heritage.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

In the era of the metaverse, where immersive technologies redefine human experiences, translating abstract literary concepts into navigable 3D environments presents a fundamental challenge in preserving semantic and emotional fidelity. This research introduces HaikuVerse, a novel framework for transforming poetic abstraction into spatial representation, with Japanese Haiku serving as an ideal test case due to its sophisticated encapsulation of profound emotions and imagery within minimal text. While existing text-to-3D methods struggle with nuanced interpretations, we present a literary-guided approach that synergizes traditional poetry analysis with advanced generative technologies. Our framework centers on two key innovations: (1) Hierarchical Literary-Criticism Theory Grounded Parsing (H-LCTGP), which captures both explicit imagery and implicit emotional resonance through structured semantic decomposition, and (2) Progressive Dimensional Synthesis (PDS), a multi-stage pipeline that systematically transforms poetic elements into coherent 3D scenes through sequential diffusion processes, geometric optimization, and real-time enhancement. Extensive experiments demonstrate that HaikuVerse significantly outperforms conventional text-to-3D approaches in both literary fidelity and visual quality, establishing a new paradigm for preserving cultural heritage in immersive digital spaces. Project website at: https://syllables-to-scenes.github.io/

Problem

• Translate literary concepts into 3D environments
• Preserve semantic and emotional fidelity
• Enhance text-to-3D with advanced generative technologies

Innovation

• Hierarchical Literary-Criticism Theory Grounded Parsing
• Progressive Dimensional Synthesis
• Literary-guided 3D scene generation