Aligning Human and Machine Attention for Enhanced Supervised Learning

2025-02-04

AI Summary

To address the weak generalization of supervised learning under imbalanced and sparsely labeled data, this paper proposes Human-Machine Attention Learning (HuMAL), the first framework to systematically investigate alignment between human self-reported attention and machine-derived attention. Methodologically, HuMAL leverages pre-trained Transformer models—including BERT, GPT-2, and XLNet—and integrates human attention annotations from the Yelp and myPersonality datasets. It introduces cognitive-inspired attention distillation and loss-weighting fusion strategies to explicitly incorporate human attention priors into supervised learning. The core contribution lies in formalizing human attention as an interpretable, cognitively grounded inductive bias within the training pipeline. Experiments demonstrate that HuMAL significantly improves performance under few-shot (<1k samples) and long-tailed label distribution settings, achieving up to a 5.7-percentage-point gain in accuracy. These results validate the effectiveness and practicality of synergistic human–machine attention modeling for robust generalization in low-resource, imbalanced learning scenarios.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Unsupervised & Self-Supervised Learning
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Attention, or prioritization of certain information items over others, is a critical element of any learning process, for both humans and machines. Given that humans continue to outperform machines in certain learning tasks, it seems plausible that machine performance could be enriched by aligning machine attention with human attention mechanisms -- yet research on this topic is sparse and has achieved only limited success. This paper proposes a new approach to address this gap, called Human-Machine Attention Learning (HuMAL). This approach involves reliance on data annotated by humans to reflect their self-perceived attention during specific tasks. We evaluate several alternative strategies for integrating such human attention data into machine learning (ML) algorithms, using a sentiment analysis task (review data from Yelp) and a personality-type classification task (data from myPersonality). The best-performing HuMAL strategy significantly enhances the task performance of fine-tuned transformer models (BERT, as well as GPT-2 and XLNET), and the benefit is particularly pronounced under challenging conditions of imbalanced or sparse labeled data. This research contributes to a deeper understanding of strategies for integrating human attention into ML models and highlights the potential of leveraging human cognition to augment ML in real-world applications.

Problem

• Aligning machine and human attention mechanisms
• Enhancing supervised learning with human attention data
• Improving ML models under imbalanced or sparse data

Innovation

• Human-Machine Attention Learning
• Human-annotated attention data
• Transformer models enhancement