A Fair Objective for Human-Empowerment-Preserving AI: Desiderata, Design, and Likely Behavioral Consequences

2026-08-08

AI Summary

This work proposes a parametric and decomposable AI objective function designed to safeguard human well-being and safety while maintaining an equitable balance of power in human–AI interaction. The function aggregates human utilities with explicit consideration for long-term outcomes, risk aversion, and disparities in human capabilities, integrating models of bounded rationality and social norms to accommodate diverse human goals. Grounded in axiomatic design principles, it explicitly enshrines human empowerment and power balance as core desiderata, yielding a functional form and associated parameter constraints that satisfy desirable theoretical properties. Theoretical analysis and case studies demonstrate that the proposed objective effectively achieves soft maximization of human utility and reveals emergent instrumental subgoals and behavioral implications inherent to its structure.

Abstract

This paper explores the idea of promoting well-being and safety in human-AI interactions by forcing AI agents explicitly to empower humans and to manage the power balance between humans and AI agents in a desirable way. Using a principled, partially axiomatic approach based on desirable properties, we design a parametrizable and decomposable objective function for AI systems that represents an inequality- and risk-averse long-term aggregate of human power. It can take into account models of human bounded rationality and social norms, and crucially, considers a wide variety of possible human goals. We prove how certain desiderata enforce particular functional forms and restrict parameter ranges. We exemplify the consequences of softly maximizing this metric in several paradigmatic situations and describe what instrumental sub-goals it will likely imply.

Problem

• human empowerment
• AI fairness
• power balance
• well-being
• human-AI interaction

Innovation

• human empowerment
• AI objective design
• power balance
• bounded rationality
• axiomatic approach