Does mapping elites illuminate search spaces? A large-scale user study of MAP-Elites applied to human-AI collaborative design
🤖 AI Summary
Current AI collaboration evaluation methods overemphasize behavioral and outcome metrics, neglecting multidimensional user experience—particularly affective, cognitive, and behavioral engagement—in human-AI co-design. Method: We conducted a large-scale online experiment (808 sessions) and a double-blind lab study (12 participants), integrating virtual automotive physics simulation with mixed methods—including behavioral logging, semi-structured interviews, and quantitative analysis—to rigorously assess the role of the MAP-Elites evolutionary algorithm in co-design. Contribution/Results: We first demonstrate that MAP-Elites’ visual diversity significantly enhances user engagement and final design quality, outperforming single-point quality-based recommendations. A hybrid strategy (MAP-Elites + random sampling) yields optimal outcomes, whereas purely algorithmic recommendations fail to meaningfully improve the design process itself. Our findings expose fundamental limitations in prevailing AI collaboration evaluation paradigms and propose a novel assessment framework centered on “engagement experience.”
Application Category
📝 Abstract
Two studies of a human-AI collaborative design tool were carried out in order to understand the influence design recommendations have on the design process. The tool investigated is based on an evolutionary algorithm attempting to design a virtual car to travel as far as possible in a fixed time. Participants were able to design their own cars, make recommendations to the algorithm and view sets of recommendations from the algorithm. The algorithm-recommended sets were designs which had been previously tested; some sets were simply randomly picked and other sets were picked using MAP-Elites. In the first study 808 design sessions were recorded as part of a science outreach program, each with analytical data of how each participant used the tool. To provide context to this quantitative data, a smaller double-blind lab study was also carried out with 12 participants. In the lab study the same quantitative data from the large scale study was collected alongside responses to interview questions. Although there is some evidence that the MAP-Elites provide higher-quality individual recommendations, neither study provides convincing evidence that these recommendations have a more positive influence on the design process than simply a random selection of designs. In fact, it seems that providing a combination of MAP-Elites and randomly selected recommendations is beneficial to the process. Furthermore, simply viewing recommendations from the MAP-Elites had a positive influence on engagement in the design task and the quality of the final design produced. Our findings are significant both for researchers designing new mixed-initiative tools, and those who wish to evaluate existing tools. Most significantly, we found that metrics researchers currently use to evaluate the success of human-AI collaborative algorithms do not measure the full influence these algorithms have on the design process.
Problem
Research questions and friction points this paper is trying to address.
Rethinking evaluation methods for human-AI collaborative design systems
Challenging conventional metrics that focus solely on behavioral outcomes
Proposing holistic assessment of emotional, behavioral and cognitive engagement
Innovation
Methods, ideas, or system contributions that make the work stand out.
Interactive evolutionary algorithm for co-creative design
MAP-Elites gallery exposure enhances engagement
Holistic evaluation beyond conventional metrics
🔎 Similar Papers
No similar papers found.
