MobileFetalCLIP: Selective Repulsive Knowledge Distillation for Mobile Fetal Ultrasound Analysis

2026-03-05

AI Summary

This work addresses the challenge of deploying large fetal ultrasound foundation models (>300M parameters) on portable devices, where conventional knowledge distillation struggles under extreme teacher-student capacity gaps. The authors propose a selective repulsion knowledge distillation method that decomposes contrastive distillation into diagonal and non-diagonal components: while preserving alignment of positive sample pairs, it assigns negative weights to non-diagonal terms, guiding a lightweight student Vision Transformer (11.4M parameters) to avoid inter-class confusion exhibited by the teacher and instead leverage its native representational capacity. This mechanism not only circumvents redundant structural mimicry under extreme compression but also exploits teacher errors for inverse guidance. The resulting student model surpasses the teacher in both zero-shot HC18 biometric measurement validity (88.6% vs. 83.5%) and brain subplane F1 score (0.784 vs. 0.702), achieving a 1.6ms inference latency on an iPhone 16 Pro.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Fetal ultrasound AI could transform prenatal care in low-resource settings, yet current foundation models exceed 300M visual parameters, precluding deployment on point-of-care devices. Standard knowledge distillation fails under such extreme capacity gaps (~26x), as compact students waste capacity mimicking architectural artifacts of oversized teachers. We introduce Selective Repulsive Knowledge Distillation, which decomposes contrastive KD into diagonal and off-diagonal components: matched pair alignment is preserved while the off-diagonal weight decays into negative values, repelling the student from the teacher's inter-class confusions and forcing discovery of architecturally native features. Our 11.4M parameter student surpasses the 304M-parameter FetalCLIP teacher on zero-shot HC18 biometry validity (88.6% vs. 83.5%) and brain sub-plane F1 (0.784 vs. 0.702), while running at 1.6 ms on iPhone 16 Pro, enabling real-time assistive AI on handheld ultrasound devices. Our code, models, and app are publicly available at https://github.com/numanai/MobileFetalCLIP.

Problem

• mobile fetal ultrasound
• knowledge distillation
• model compression
• point-of-care devices
• foundation models

Innovation

• Selective Repulsive Knowledge Distillation
• Mobile Fetal Ultrasound
• Contrastive Knowledge Distillation
• Model Compression
• Zero-shot Biometry