🤖 AI Summary
This work addresses the high deployment cost of monolithic models in referring expression segmentation by proposing a resource-aware, two-stage framework. It first employs a lightweight vision-language model (Florence-2 or Moondream2) for text-guided object localization and then leverages MobileSAM to convert the predicted bounding box into a pixel-level mask, with both stages fine-tuned independently. Evaluated on the RefCOCO validation set, the method achieves 73.64 mIoU and 84.60 P@0.5, requiring only 2.20 GB of GPU memory per query and attaining a throughput of 22.8 FPS on an RTX 6000 Ada GPU. These results demonstrate that the proposed output token compression and ground-truth box adaptation strategies effectively maintain high accuracy while substantially reducing computational overhead.
📝 Abstract
Referring expression segmentation requires language conditioned localization and pixel-accurate masks, but monolithic models can be costly to deploy. We present VespaSeg, a modular pipeline that grounds a text query with a compact vision-language model and converts the predicted box to a mask with MobileSAM. We study Florence-2-base, Florence-2-large, and Moondream2 grounders together with targeted adaptation of the grounding and segmentation stages. Under a repository-specific RefCOCO validation protocol containing the first expression for each of 3,811 referenced-object records, the adapted Florence-2-base pipeline obtains 73.64 mean intersection over union (mIoU) and 84.60 precision at IoU 0.5. On an NVIDIA RTX 6000 Ada GPU it processes 22.8 cached-image queries per second with 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for Florence-2-base and 72.82 for Florence-2-large, while the base model is 1.70 times faster and uses 1.17 GB less allocated memory. Ablations show that ground-truth-box adaptation raises MobileSAM mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. These results support compact, modular grounding and segmentation, while also exposing the need for evaluation on the complete standard RefCOCO expression splits and deployment hardware.