Seeing Through the Weights: Privacy Leakage in Scene Coordinate Regression

2026-06-30

AI Summary

Although scene coordinate regression (SCR) methods are commonly regarded as privacy-preserving due to their implicit encoding of scene information within network parameters, their actual privacy guarantees remain unverified. This work is the first to demonstrate that SCR poses severe privacy risks: by feeding unrelated proxy images into the model and analyzing the perturbation stability of its output coordinates, high-fidelity 3D geometry of the original training scenes can be reconstructed under both white-box and black-box settings. Moreover, integrating feature inversion techniques enables the synthesis of novel-view images that faithfully recover recognizable layouts and sensitive scene elements. The study establishes a new paradigm for reconstructing both scene geometry and appearance directly from model weights without access to the original training data, empirically validating the real-world privacy threats of deploying SCR in privacy-sensitive scenarios across multiple indoor and outdoor datasets.

Abstract

Scene Coordinate Regression (SCR) methods are increasingly adopted for visual localization. In these approaches, the scene is implicitly encoded within a neural network that regresses a 3D world coordinate for each image pixel. Because the scene is represented only through the network parameters and not stored explicitly as images or maps, such methods are often assumed to be privacy-preserving. In this work, we show that this assumption is incorrect in practice. Specifically, we introduce a query-based attack that reconstructs the 3D geometry of the training environment from an SCR model under different levels of model access. To do so, we repeatedly query the model with batches of proxy images unrelated to the target scene to obtain dense pixel-wise 3D coordinates. Reliable points are identified through their stability under small input perturbations and can be further refined in a white-box setting. These stable points are accumulated across independent query batches to recover the scene geometry. From the recovered 3D representation, we also invert the network features to synthesize images from arbitrary viewpoints, revealing additional appearance information. Experiments on indoor and outdoor datasets demonstrate that substantial portions of training environments can be reconstructed with high geometric fidelity. Beyond geometry, we also recover an approximate color appearance, which exposes recognizable layout and potentially sensitive scene elements. This directly contradicts claims in the literature that SCR representations are privacy-preserving by design, and reveals a real risk when such systems are deployed in private or security-critical spaces. The project page is available at https://jaeminch0.github.io/seeing-through-the-weights-privacy-leakage-in-scene-coordinate-regression.

Problem

• privacy leakage
• scene coordinate regression
• 3D reconstruction
• visual localization
• model inversion

Innovation

• Scene Coordinate Regression
• Privacy Leakage
• 3D Reconstruction
• Model Inversion
• Query-based Attack