Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

2026-08-02

AI Summary

Current systems struggle to deliver real-time, centimeter-accurate street-level flood depth estimation, hindering effective urban emergency response. This work proposes FloodLlama-MI6, the first approach to leverage mechanistic interpretability analysis for guiding sparse fine-tuning of a vision-language model. By identifying cross-attention layers with causal influence on depth prediction and fine-tuning only those, the method reveals a two-stage internal adaptation mechanism. Integrating QLoRA, linear probing, Logit Lens, CKA, and attention entropy analyses, and trained on synthetic data generated with Unreal Engine 5, FloodLlama-MI6 achieves 98.62% accuracy within 5 cm on real-world scenes while reducing trainable parameters by 86–88%. Its densely fine-tuned counterpart, FloodLlama-Dense, attains a remarkably low mean absolute error of 0.40 cm.

Abstract

Urban flooding poses an escalating threat to transportation infrastructure, yet no operational system provides real-time, street-level flood-depth estimates at centimeter resolution. This paper presents three vision-language models fine-tuned for continuous flood-depth estimation from street-level imagery: FloodLlama-Dense, a fully fine-tuned QLoRA baseline, and FloodLlama-MI5 and FloodLlama-MI6, interpretability-guided sparse variants that fine-tune only the top five and six causally relevant cross-attention layers identified through mechanistic interpretability analysis, respectively. Training uses an approximately 610,000-image subset of a 2.81-million-image synthetic corpus generated in Unreal Engine 5. The dataset combines single-vehicle subsets with 5 cm depth increments and mixed-vehicle subsets with 1 cm depth increments, spanning seven vehicle types, four weather conditions, and flood depths from 0 to 40 cm. FloodLlama-Dense achieves an MAE of 0.40 cm, an RMSE of 1.97 cm, and an Acc@5cm of 97.59%. Mechanistic interpretability analysis combining linear probing, logit lens, centered kernel alignment (CKA), and cross-attention entropy reveals a two-stage adaptation pattern: layers L13-L22 restructure visual representations, while depth first becomes linearly decodable at layer L23. FloodLlama-MI5 and FloodLlama-MI6 leverage this insight by fine-tuning only five or six of the eight cross-attention layers, achieving an 86-88% reduction in trainable parameters (6.55-7.86 million versus 54.4 million) with minimal accuracy loss. On a real-world benchmark, FloodLlama-MI6 achieves 98.62% accuracy, compared with 86.61% for the published STURM-FloodDepth baseline.

Problem

• urban flooding
• flood depth estimation
• centimeter-level resolution
• street-level imagery
• vision-language models

Innovation

• mechanistic interpretability
• selective fine-tuning
• vision-language models
• flood depth estimation
• sparse adaptation