Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation

2025-06-10

AI Summary

To address the trade-off between efficiency and accuracy in point-prompted image segmentation, this paper proposes a foveation-based efficient segmentation framework. Methodologically, it dynamically crops the input image centered on the prompt point and introduces a distance-adaptive radial multi-resolution tokenization scheme, concentrating computational resources on the region of interest; it further adapts the ViT architecture and incorporates a lightweight decoder. The core innovation lies in the first-ever point-driven progressive downsampling tokenization, which significantly reduces computational load without compressing model parameters. Experiments demonstrate that the method maintains competitive benchmark performance while reducing input tokens by over 60% and achieving inference speeds exceeding 30 FPS on consumer-grade GPUs. This enables real-time interaction in applications such as augmented reality and robotic vision.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Hardware-aware ML

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper presents Segment This Thing (STT), a new efficient image segmentation model designed to produce a single segment given a single point prompt. Instead of following prior work and increasing efficiency by decreasing model size, we gain efficiency by foveating input images. Given an image and a point prompt, we extract a crop centered on the prompt and apply a novel variable-resolution patch tokenization in which patches are downsampled at a rate that increases with increased distance from the prompt. This approach yields far fewer image tokens than uniform patch tokenization. As a result we can drastically reduce the computational cost of segmentation without reducing model size. Furthermore, the foveation focuses the model on the region of interest, a potentially useful inductive bias. We show that our Segment This Thing model is more efficient than prior work while remaining competitive on segmentation benchmarks. It can easily run at interactive frame rates on consumer hardware and is thus a promising tool for augmented reality or robotics applications.

Problem

• Efficient image segmentation using point prompts
• Variable-resolution patch tokenization for reduced computation
• Interactive frame rates for augmented reality applications

Innovation

• Foveated input images for efficiency
• Variable-resolution patch tokenization technique
• Reduces tokens without shrinking model