An experimental approach: The graph of graphs

2025-08-24

AI Summary

This study addresses the optimal design of pairwise comparison (PC) structures in decision-making and preference modeling: specifically, how to select the most informative comparison pairs and their presentation sequence to minimize cognitive load on decision-makers while maximizing the accuracy of weight estimation. We propose a “graph-of-graphs” framework to systematically construct and analyze the taxonomy of PC graphs. Empirical validation involves 301 participants in color calibration and sensory testing tasks; results confirm strong alignment between simulated optimal designs—particularly balanced incomplete block designs—and observed preference responses, with empirically derived near-optimal sequences matching simulation predictions exactly. Weight estimation via logarithmic least squares reveals high similarity between the empirical distribution of PC matrix entries and theoretical distributions from simulation. Contributions include a practical recommendation table of optimal PC patterns and an open-source Java toolkit, substantially enhancing both the efficiency and deployability of preference modeling.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Models for Web evolution

Abstract

One of the essential issues in decision problems and preference modeling is the number of comparisons and their pattern to ask from the decision maker. We focus on the optimal patterns of pairwise comparisons and the sequence including the most (close to) optimal cases based on the results of a color selection experiment. In the test, six colors (red, green, blue, magenta, turquoise, yellow) were evaluated with pairwise comparisons as well as in a direct manner, on color-calibrated tablets in ISO standardized sensory test booths of a sensory laboratory. All the possible patterns of comparisons resulting in a connected representing graph were evaluated against the complete data based on 301 individual's pairwise comparison matrices (PCMs) using the logarithmic least squares weight calculation technique. It is shown that the empirical results, i.e., the empirical distributions of the elements of PCMs, are quite similar to the former simulated outcomes from the literature. The obtained empirically optimal patterns of comparisons were the best or the second best in the former simulations as well, while the sequence of comparisons that contains the most (close to) optimal patterns is exactly the same. In order to enhance the applicability of the results, besides the presentation of graph of graphs, and the representing graphs of the patterns that describe the proposed sequence of comparisons themselves, the recommendations are also detailed in a table format as well as in a Java application.

Problem

• Optimizing pairwise comparison patterns for decision making
• Evaluating color preference using empirical distribution analysis
• Identifying optimal comparison sequences via graph representation

Innovation

• Optimal pairwise comparison patterns
• Graph of graphs representation
• Java application implementation