When a Zero-Shooter Cheats: Improving Age Estimation via Activation Steering

2026-05-17

AI Summary

This work addresses a critical shortcut in current vision-language models for zero-shot age estimation: rather than inferring age from visual cues, these models often rely on recognizing celebrity identities and retrieving memorized age information, leading to inflated benchmark performance and poor generalization to non-celebrity images. The study is the first to systematically expose this identity-based shortcut and proposes an activation steering method that requires no retraining. By intervening in the model’s hidden states to suppress identity-related signals, the approach redirects attention toward genuine visual age indicators. Experiments demonstrate that this technique reduces mean absolute error by up to 25% on standard benchmarks, substantially improving age estimation accuracy for both known and unknown individuals.

Abstract

Different age-related regulations have been proposed to protect minors from harmful content and interactions online. Automated age estimation is central to enforcing such regulations, and vision-language models (VLMs) achieve state-of-the-art performance on this task. However, we find that the zero-shot nature of VLM-based age estimation produces an unexpected side effect we call the identity shortcut: Instead of estimating age from visual features, VLMs tend to identify the depicted person and infer their age from memorized knowledge. This phenomenon leads to substantially incorrect predictions when non-celebrities are misidentified as celebrities. It also produces deceptively high robustness to noise and adversarial perturbations on celebrity images, which dominate popular benchmarks. To mitigate this, we propose an activation steering method that suppresses the shortcut by intervening on the hidden states of the VLM. This method improves age estimation accuracy for both memorized and unseen identities, reducing mean absolute error by up to 25% across popular benchmarks.

Problem

• age estimation
• vision-language models
• identity shortcut
• zero-shot learning
• bias mitigation

Innovation

• activation steering
• identity shortcut
• zero-shot age estimation
• vision-language models
• age estimation