Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

2026-08-22

AI Summary

研究解决GUI元素语义接地评估中混淆可见标签恢复与语义接地的问题，通过对比句子嵌入和词汇基线方法，提出改善评估策略。

Abstract

GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible-label recovery. Lexical baselines remain competitive at top-1, label-poor targets remain weak for text-only methods, and encoder top-1 hits are predictable from lexical rank, candidate-pool size, and label type. We evaluate each action as a same-screen ranking task, comparing five off-the-shelf single-vector encoders with lexical baselines. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target-aware oracle gains. These findings show that embedding-based evaluations can conflate visible-label recovery with semantic GUI grounding. Embedding-based evaluations should therefore report lexical baselines, label-type stratification, and deployable-fusion diagnostics. Our released repository provides analysis scripts and detexted per-step panels: https://github.com/qijia123/lexical-coupling-release.

Problem

• GUI grounding
• sentence embeddings
• visible-label recovery
• semantic grounding

Innovation

• Lexical Coupling
• GUI Grounding
• Embedding Similarity
• Visible-label Recovery
• Semantic Understanding