Surrogate Representation Inference for Noisy Text and Image Annotations

2025-09-15

AI Summary

Existing text/image annotation methods rely on high-quality human labels, yet real-world annotations often contain non-negligible noise, leading to substantial bias and inflated standard errors in downstream statistical inference. Method: We propose the Surrogate Representation Inference (SRI) framework—the first theoretical framework jointly optimizing low-dimensional representation learning and statistical inference—under the assumption that unstructured data fully mediates the relationship between annotations and structural variables. SRI corrects non-differential measurement error without requiring error-free gold-standard labels, ensuring identifiability and valid inference. Leveraging neural representation learning and semiparametric efficient estimation, SRI enables end-to-end joint optimization for text-output settings. Contribution/Results: Experiments demonstrate that SRI reduces standard errors by over 50% compared to state-of-the-art methods and maintains robust inference performance even under substantial annotation noise.

Technology Category

• Machine Learning: Representation Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

As researchers increasingly rely on machine learning models and LLMs to annotate unstructured data, such as texts or images, various approaches have been proposed to correct bias in downstream statistical analysis. However, existing methods tend to yield large standard errors and require some error-free human annotation. In this paper, I introduce Surrogate Representation Inference (SRI), which assumes that unstructured data fully mediate the relationship between human annotations and structured variables. The assumption is guaranteed by design provided that human coders rely only on unstructured data for annotation. Under this setting, I propose a neural network architecture that learns a low-dimensional representation of unstructured data such that the surrogate assumption remains to be satisfied. When multiple human annotations are available, SRI can further correct non-differential measurement errors that may exist in human annotations. Focusing on text-as-outcome settings, I formally establish the identification conditions and semiparametric efficient estimation strategies that enable learning and leveraging such a low-dimensional representation. Simulation studies and a real-world application demonstrate that SRI reduces standard errors by over 50% when machine learning prediction accuracy is moderate and provides valid inference even when human annotations contain non-differential measurement errors.

Problem

• Correcting bias in noisy machine-generated annotations
• Reducing standard errors in statistical analysis with surrogates
• Handling non-differential measurement errors in human annotations

Innovation

• SRI assumes unstructured data mediate annotations
• Neural network learns low-dimensional surrogate representation
• Corrects non-differential measurement errors in annotations