STRinGS: Selective Text Refinement in Gaussian Splatting

2025-12-08

AI Summary

3D Gaussian Splatting (3DGS) struggles to preserve fine-grained text details in high-fidelity reconstruction, where even minor character distortions cause severe semantic degradation. To address this, we propose a text-aware region-wise optimization framework: (1) text segmentation masks are employed to explicitly separate text and non-text regions; (2) a selective refinement mechanism prioritizes geometric and appearance optimization within text regions; and (3) we introduce OCR-based Character Error Rate (CER) as a novel, differentiable readability metric for 3D text reconstruction—marking the first use of CER in this context. To support rigorous evaluation, we curate STRinGs-360, the first benchmark dataset dedicated to 3D text reconstruction. Experiments demonstrate that our method achieves a 63.6% CER improvement over standard 3DGS within only 7K iterations, significantly enhancing text legibility and semantic fidelity—especially in complex, multi-line, or curved text layouts.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Text as signs, labels, or instructions is a critical element of real-world scenes as they can convey important contextual information. 3D representations such as 3D Gaussian Splatting (3DGS) struggle to preserve fine-grained text details, while achieving high visual fidelity. Small errors in textual element reconstruction can lead to significant semantic loss. We propose STRinGS, a text-aware, selective refinement framework to address this issue for 3DGS reconstruction. Our method treats text and non-text regions separately, refining text regions first and merging them with non-text regions later for full-scene optimization. STRinGS produces sharp, readable text even in challenging configurations. We introduce a text readability measure OCR Character Error Rate (CER) to evaluate the efficacy on text regions. STRinGS results in a 63.6% relative improvement over 3DGS at just 7K iterations. We also introduce a curated dataset STRinGS-360 with diverse text scenarios to evaluate text readability in 3D reconstruction. Our method and dataset together push the boundaries of 3D scene understanding in text-rich environments, paving the way for more robust text-aware reconstruction methods.

Problem

• Improves text readability in 3D Gaussian Splatting reconstructions
• Separately refines text and non-text regions for optimization
• Introduces a dataset to evaluate text in 3D scenes

Innovation

• Selective refinement framework for text in 3D Gaussian Splatting
• Separates text and non-text regions for independent optimization
• Introduces OCR-based metric and dataset for text readability evaluation