UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing

2026-01-26

AI Summary

This work proposes an interactive UI design system that addresses the challenges non-expert users face in expressing design intent and trusting retrieved examples, which often lead to design fixation or disorientation. The system uniquely integrates multimodal retrieval-augmented generation (MMRAG) with source transparency, enabling fine-grained retrieval, remixing, and adaptation of UI examples at both interface and component levels. By transparently presenting metadata such as example ratings, download counts, and developer information, the system enhances users’ confidence in their design decisions. User studies demonstrate that this approach significantly improves users’ ability to achieve their design goals, facilitates efficient iteration, and encourages diverse exploration, effectively balancing controllability with creative discovery.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

Designing user interfaces (UIs) is a critical step when launching products, building portfolios, or personalizing projects, yet end users without design expertise often struggle to articulate their intent and to trust design choices. Existing example-based tools either promote broad exploration, which can cause overwhelm and design drift, or require adapting a single example, risking design fixation. We present UI Remix, an interactive system that supports mobile UI design through an example-driven design workflow. Powered by a multimodal retrieval-augmented generation (MMRAG) model, UI Remix enables iterative search, selection, and adaptation of examples at both the global (whole interface) and local (component) level. To foster trust, it presents source transparency cues such as ratings, download counts, and developer information. In an empirical study with 24 end users, UI Remix significantly improved participants'ability to achieve their design goals, facilitated effective iteration, and encouraged exploration of alternative designs. Participants also reported that source transparency cues enhanced their confidence in adapting examples. Our findings suggest new directions for AI-assisted, example-driven systems that empower end users to design with greater control, trust, and openness to exploration.

Problem

• UI design
• example-based design
• design fixation
• end-user design
• design trust

Innovation

• multimodal retrieval-augmented generation
• example-driven design
• UI remixing
• source transparency
• interactive design system