GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking

2025-08-05

AI Summary

GUI prototyping heavily relies on manual effort, and existing natural-language-driven GUI retrieval methods suffer from poor generalization and low precision. To address this, we propose GUI-ReRank, the first framework to integrate multimodal large language models (MLLMs) into the re-ranking stage of GUI retrieval. It combines lightweight embedding-based constrained retrieval with a customizable GUI repository processing pipeline, supporting user-defined annotation, indexing, and RAG integration. Our framework significantly improves cross-dataset generalization and deployment flexibility, outperforming state-of-the-art models on standard benchmarks in both retrieval accuracy and efficiency. Furthermore, we provide a cost-performance trade-off analysis, empirically validating the feasibility and practicality of MLLM-based re-ranking in real-world GUI reuse scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Humans and AI: Intelligent User Interfaces

Application Category

• Search and Retrieval-Augmented AI: Web search models and ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

GUI prototyping is a fundamental component in the development of modern interactive systems, which are now ubiquitous across diverse application domains. GUI prototypes play a critical role in requirements elicitation by enabling stakeholders to visualize, assess, and refine system concepts collaboratively. Moreover, prototypes serve as effective tools for early testing, iterative evaluation, and validation of design ideas with both end users and development teams. Despite these advantages, the process of constructing GUI prototypes remains resource-intensive and time-consuming, frequently demanding substantial effort and expertise. Recent research has sought to alleviate this burden through NL-based GUI retrieval approaches, which typically rely on embedding-based retrieval or tailored ranking models for specific GUI repositories. However, these methods often suffer from limited retrieval performance and struggle to generalize across arbitrary GUI datasets. In this work, we present GUI-ReRank, a novel framework that integrates rapid embedding-based constrained retrieval models with highly effective MLLM-based reranking techniques. GUI-ReRank further introduces a fully customizable GUI repository annotation and embedding pipeline, enabling users to effortlessly make their own GUI repositories searchable, which allows for rapid discovery of relevant GUIs for inspiration or seamless integration into customized LLM-based RAG workflows. We evaluated our approach on an established NL-based GUI retrieval benchmark, demonstrating that GUI-ReRank significantly outperforms SOTA tailored LTR models in both retrieval accuracy and generalizability. Additionally, we conducted a comprehensive cost and efficiency analysis of employing MLLMs for reranking, providing valuable insights regarding the trade-offs between retrieval effectiveness and computational resources. Video: https://youtu.be/_7x9UCh82ug

Problem

• Improving GUI retrieval performance and generalizability
• Reducing resource-intensive GUI prototyping efforts
• Enabling customizable GUI repository search and integration

Innovation

• Multi-modal LLM-based reranking enhances GUI retrieval
• Customizable GUI repository annotation and embedding pipeline
• Combines embedding-based retrieval with MLLM reranking