Evaluating Perceptual Distance Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data

2024-03-15

AI Summary

In crowdsourced 2AFC (two-alternative forced choice) triplet data, images are not shared across triplets, preventing global perceptual distance estimation and reliable model evaluation. Method: We propose a discriminative modeling framework based on a location-dependent binomial distribution, estimating local discrimination probabilities on the reference–distortion distance plane and constructing a global log-likelihood evaluation metric. For the first time, we model 2AFC responses as a spatially varying binomial process conditioned on sampling positions—enabling robust estimation under non-uniform sampling and variable response counts per triplet. The metric is derived via maximum likelihood estimation with parameterization of the perceptual distance plane, yielding an interpretable, differentiable, and scale-invariant score. Results: Experiments demonstrate substantial improvements in reliability and discriminability for model evaluation in the no-image-sharing setting, overcoming fundamental limitations of conventional accuracy-based metrics.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

The two-alternative forced choice (2AFC) experimental method is popular in the visual perception literature, where practitioners aim to understand how human observers perceive distances within triplets made of a reference image and two distorted versions. In the past, this had been conducted in controlled environments, with triplets sharing images, so it was possible to rank the perceived quality. This ranking would then be used to evaluate perceptual distance models against the experimental data. Recently, crowd-sourced perceptual datasets have emerged, with no images shared between triplets, making ranking infeasible. Evaluating perceptual distance models using this data reduces the judgements on a triplet to a binary decision, namely, whether the distance model agrees with the human decision - which is suboptimal and prone to misleading conclusions. Instead, we statistically model the underlying decision-making process during 2AFC experiments using a binomial distribution. Having enough empirical data, we estimate a smooth and consistent distribution of the judgements on the reference-distorted distance plane, according to each distance model. By applying maximum likelihood, we estimate the parameter of the local binomial distribution, and a global measurement of the expected log-likelihood of the measured responses. We calculate meaningful and well-founded metrics for the distance model, beyond the mere prediction accuracy as percentage agreement, even with variable numbers of judgements per triplet -- key advantages over both classical and neural network methods.

Problem

• Evaluate perceptual distance models using 2AFC data.
• Address limitations of crowd-sourced datasets without shared images.
• Propose binomial distribution modeling for accurate judgment analysis.

Innovation

• Uses binomial distribution for 2AFC data analysis
• Estimates smooth judgement distribution on distance plane
• Compares models using likelihood-based metrics