MeshFIM: Local Low-Poly Mesh Editing via Fill-in-the-Middle Autoregressive Generation

2026-05-09

AI Summary

Traditional autoregressive mesh generation methods suffer from inefficiency and structural disruption during local editing, as they require full regeneration of the mesh. This work proposes MeshFIM, a novel framework that introduces, for the first time, a fill-in-the-middle autoregressive mechanism to low-poly mesh editing, enabling efficient and controllable generation within user-specified regions. MeshFIM innovatively integrates boundary vertex tokens, contextual positional encoding, expanded context width, context enhancement strategies, and a gated subtraction mechanism within its geometric encoder to precisely govern generated content while preserving topological consistency. Experiments demonstrate that MeshFIM significantly outperforms existing baselines in tasks such as mesh refinement, defect repair, and localized editing, and further supports practical applications including interactive brush-based editing and automatic inpainting.

Abstract

Autoregressive (AR) models can generate high-quality low-poly meshes from point clouds, but they still operate in an all-or-nothing manner: when a local region is unsatisfactory, the entire mesh must be regenerated, wasting computation and destroying satisfactory mesh structure elsewhere. We introduce MeshFIM, a Fill-in-the-Middle (FIM) framework that regenerates a target region of a low-poly mesh conditioned on the surrounding context. MeshFIM addresses three mesh-specific challenges: enforcing exact attachment along the exposed boundary, preserving topological order in the context, and suppressing overflow beyond the intended region. It does so with five complementary design choices: boundary vertex markers, context positional embeddings, expanded context width, context augmentation, and a low-poly geometry encoder whose gated subtraction mechanism focuses generation on the missing region by leveraging the difference between the reference surface and the existing mesh. Detailed ablation studies are presented to show the effectiveness of every introduced component. Based on MeshFIM, we demonstrate two applications: interactive brush-based editing and automatic defect repair on low-poly mesh (see Figure 1). Last but not least, experiments show that MeshFIM outperforms a range of baselines in mesh refinement, mesh repair and whole mesh generation plus stitch-back scheme.

Problem

• low-poly mesh
• local editing
• autoregressive generation
• mesh refinement
• Fill-in-the-Middle

Innovation

• Fill-in-the-Middle
• autoregressive generation
• low-poly mesh editing
• local mesh regeneration
• boundary-aware generation