Meshtryoshka: Differentiable Rendering of Real-World Scenes via Mesh Rasterization

2026-06-26

AI Summary

Existing grid-based differentiable rendering methods struggle to scale to large, unbounded scenes and rely on custom renderers. This work proposes a novel differentiable mesh rendering framework that dynamically extracts geometry from signed distance functions using nested mesh shells and synthesizes images via standard triangle rasterization combined with alpha compositing. Notably, this approach is the first to achieve compatibility with off-the-shelf, non-differentiable mesh renderers without requiring explicit gradients with respect to vertex positions, thereby overcoming the scene-scale limitations inherent in conventional mesh-based methods. Experiments demonstrate that the method matches the performance of state-of-the-art surface rendering techniques on object-centric scenes and achieves near state-of-the-art novel view synthesis quality on unbounded real-world scenes compared to current non-mesh-based approaches.

Abstract

Differentiable rendering has emerged as a powerful approach for 3D reconstruction and novel view synthesis. State-of-the-art differentiable rendering methods combine a variety of custom representations of 3D geometry and appearance with specialized renderers. However, most downstream tasks in computer graphics rely on 3D meshes. While prior work has attempted differentiable rendering with mesh representations, these approaches are limited to object-centric scenes and fail to reconstruct large-scale, unbounded scenes. In this work, we introduce Meshtryoshka, a novel mesh differentiable rendering framework that combines an off-the-shelf triangle rasterizer with a 3D representation that consists of nested mesh shells which resemble a matryoshka doll. In every forward pass, the mesh shells are extracted anew from a 3D signed distance function via iso-surface extraction, and the opacities for each vertex are computed as a function of signed distance. Each mesh shell is then rasterized independently, and the final image is created via alpha compositing. Crucially, mesh vertex positions are updated only indirectly via gradients that flow through the opacity values into the signed distance function, and hence, our method is compatible with off-the-shelf mesh renderers that need not be differentiable with respect to vertex positions. On object-centric scenes, our method performs competitively with surface-based differentiable rendering techniques. Our differentiable mesh rendering method scales to unbounded, real-world 3D scenes, where it yields high-quality novel view synthesis results approaching those of state-of-the-art, non-mesh methods. Our method suggests that it may be possible to solve the differentiable rendering problem without relying on specialized renderers, only using conventional tools from the computer graphics toolbox.

Problem

• differentiable rendering
• mesh representation
• unbounded scenes
• novel view synthesis
• 3D reconstruction

Innovation

• differentiable rendering
• mesh rasterization
• signed distance function
• nested mesh shells
• novel view synthesis