Dual-Imbalance Continual Learning for Real-World Food Recognition

2026-03-30

AI Summary

This work addresses the dual imbalance challenge in continual food recognition from real-world dietary records, where samples within categories exhibit a long-tailed distribution and the number of new classes introduced across incremental phases varies significantly. To tackle this problem, the paper proposes DIME, the first framework explicitly modeling and resolving these imbalances. DIME employs parameter-efficient fine-tuning by learning lightweight task-specific adapters and introduces a class-count-guided spectral merging strategy coupled with a rank-aware threshold modulation mechanism to enable stable knowledge integration and adaptive retention. Evaluated on realistic long-tailed food datasets, DIME outperforms the strongest baseline by over 3% on average while requiring only a single merged adapter during inference, thus achieving both computational efficiency and deployment friendliness.

Abstract

Visual food recognition in real-world dietary logging scenarios naturally exhibits severe data imbalance, where a small number of food categories appear frequently while many others occur rarely, resulting in long-tailed class distributions. In practice, food recognition systems often operate in a continual learning setting, where new categories are introduced sequentially over time. However, existing studies typically assume that each incremental step introduces a similar number of new food classes, which rarely happens in real world where the number of newly observed categories can vary significantly across steps, leading to highly uneven learning dynamics. As a result, continual food recognition exhibits a dual imbalance: imbalanced samples within each food class and imbalanced numbers of new food classes to learn at each incremental learning step. In this work, we introduce DIME, a Dual-Imbalance-aware Adapter Merging framework for continual food recognition. DIME learns lightweight adapters for each task using parameter-efficient fine-tuning and progressively integrates them through a class-count guided spectral merging strategy. A rank-wise threshold modulation mechanism further stabilizes the merging process by preserving dominant knowledge while allowing adaptive updates. The resulting model maintains a single merged adapter for inference, enabling efficient deployment without accumulating task-specific modules. Experiments on realistic long-tailed food benchmarks under our step-imbalanced setup show that the proposed method consistently improves by more than 3% over the strongest existing continual learning baselines. Code is available at https://github.com/xiaoyanzhang1/DIME.

Problem

• dual imbalance
• continual learning
• food recognition
• long-tailed distribution
• class imbalance

Innovation

• dual-imbalance
• continual learning
• adapter merging
• long-tailed recognition
• parameter-efficient fine-tuning