Panoramic Distortion-Aware Tokenization for Person Detection and Localization Using Transformers in Overhead Fisheye Images

2025-03-18

AI Summary

Detecting and localizing persons in top-down fisheye images remains challenging due to severe rotational distortion and extremely small object scales. To address this, we propose a distortion-aware panoramic Transformer: first, fisheye images are remapped to equidistant cylindrical projection (ECP) to mitigate geometric distortion; second, a self-similarity-driven, non-overlapping tokenization mechanism adaptively partitions the image while preserving structural features of small objects; third, a saliency-weighted feature aggregation strategy is introduced to enhance discriminative representations for tiny, highly rotated persons—particularly in the top region. This work pioneers joint optimization of panoramic distortion modeling and tokenization. Evaluated on a large-scale top-down fisheye dataset, our method achieves significant improvements in mAP (+5.2%) and small-object recall, notably enhancing localization robustness in the top region.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Search and Optimization: Distributed Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements

Abstract

Person detection methods are used widely in applications including visual surveillance, pedestrian detection, and robotics. However, accurate detection of persons from overhead fisheye images remains an open challenge because of factors including person rotation and small-sized persons. To address the person rotation problem, we convert the fisheye images into panoramic images. For smaller people, we focused on the geometry of the panoramas. Conventional detection methods tend to focus on larger people because these larger people yield large significant areas for feature maps. In equirectangular panoramic images, we find that a person's height decreases linearly near the top of the images. Using this finding, we leverage the significance values and aggregate tokens that are sorted based on these values to balance the significant areas. In this leveraging process, we introduce panoramic distortion-aware tokenization. This tokenization procedure divides a panoramic image using self-similarity figures that enable determination of optimal divisions without gaps, and we leverage the maximum significant values in each tile of token groups to preserve the significant areas of smaller people. To achieve higher detection accuracy, we propose a person detection and localization method that combines panoramic-image remapping and the tokenization procedure. Extensive experiments demonstrated that our method outperforms conventional methods when applied to large-scale datasets.

Problem

• Addresses person detection in overhead fisheye images
• Solves issues of person rotation and small-sized persons
• Improves detection accuracy using panoramic distortion-aware tokenization

Innovation

• Convert fisheye images to panoramic images
• Use panoramic distortion-aware tokenization
• Aggregate tokens based on significance values