See & Sniff: Learning Visuo-Olfactory Representations

2026-06-25

AI Summary

Olfaction has long been overlooked in multimodal learning due to the scarcity of vision–smell paired data. This work introduces SmellNet-V, a large-scale cross-modal dataset synthesized by leveraging web images and unimodal olfactory data under the assumption of odor invariance within semantic categories, thereby circumventing the need for costly synchronized acquisition. Furthermore, the authors propose See & Sniff, a self-supervised framework that learns joint visual–olfactory representations through local dense alignment and generates odor saliency maps for pixel-level odor source localization. Evaluated solely on odor inputs, the method surpasses baseline models by 7% in classification accuracy and demonstrates strong generalization in cross-modal retrieval and odor localization tasks. This study establishes the first benchmark for such evaluations and opens a new avenue in multimodal perception research.

Abstract

While modern multimodal models integrate vision with language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V, a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark without costly co-collection. Building on this dataset, we propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via dense local alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduce pixel-level smell localization task and a benchmark for evaluation. Our method surpasses smell-only baselines by 7% in smell classification from smell alone and generalizes to cross-modal retrieval and smell localization, establishing visuo-olfactory learning as a new direction in multimodal perception.

Problem

• visuo-olfactory
• multimodal perception
• smell localization
• olfactory data
• cross-modal representation

Innovation

• visuo-olfactory representation
• self-supervised learning
• smell localization
• multimodal perception
• synthetic data pairing