Optimization of Scoring Rules

๐Ÿ“… 2020-07-06
๐Ÿ›๏ธ ACM Conference on Economics and Computation
๐Ÿ“ˆ Citations: 43
โœจ Influential: 6
๐Ÿ“„ PDF
๐Ÿค– AI Summary
This paper addresses the design of proper scoring rules for multidimensional forecasting settings, aiming to incentivize forecasters to exert effort and truthfully report their beliefs. Methodologically, it introduces the first optimization framework explicitly targeting *effort incentives*, integrating game-theoretic modeling with convex optimization. For simple settings, it derives closed-form characterizations of optimal rules; for general cases, it develops an efficient and exact algorithm; and it identifies several structurally simple approximate rules with near-optimal performance. Theoretical analysis reveals that classical proper scoring rulesโ€”such as the quadratic scoreโ€”can substantially deviate from optimality under multidimensional effort. In contrast, the proposed algorithm computes exact optimal rules, while the simple approximations achieve over 95% of the optimal incentive efficiency. These results establish a new paradigm for information design and prediction market mechanisms, bridging incentive alignment with practical implementability.
๐Ÿ“ Abstract
This paper introduces an objective for optimizing proper scoring rules. The objective is to maximize the increase in payoff of a forecaster who exerts a binary level of effort to refine a posterior belief from a prior belief. In this framework we characterize optimal scoring rules in simple settings, give efficient algorithms for computing optimal scoring rules in complex settings, and identify simple scoring rules that are approximately optimal. In comparison, standard scoring rules in theory and practice -- for example the quadratic rule, scoring rules for the expectation, and scoring rules for multiple tasks that are averages of single-task scoring rules -- can be very far from optimal.
Problem

Research questions and friction points this paper is trying to address.

Optimizing scoring rules for truthful information reporting
Designing incentives for multi-dimensional information acquisition
Comparing optimal scoring rules with standard alternatives
Innovation

Methods, ideas, or system contributions that make the work stand out.

Binary bets optimize single-dimensional scoring rules
Multi-dimensional scoring selects most surprising signal dimension
Approximate optimality maintained for asymmetric distributions
Yale University | Northwestern University | Toyota Technological Institute at Chicago
Yingkai Li
Yingkai Li
National University of Singapore
Mechanism DesignAlgorithmic Game TheoryOnline Algorithms
J
Jason D. Hartline
Computer Science Department, Northwestern University, Evanston, IL, United States, 60208
Liren Shan
Liren Shan
Research Assistant Professor, TTIC
Approximation AlgorithmClusteringMachine Learning
Y
Yifan Wu
Computer Science Department, Northwestern University, Evanston, IL, United States, 60208