"I want to be pushed, I want to grow": Enabling social workers to design evaluations of LLM augmentation in their work

2026-08-23

AI Summary

研究通过让19名社会工作者参与设计AI评估标准，提出一种自下而上的方法来定义和衡量AI在工作中的有效辅助。

Abstract

Workers are increasingly asked to adopt AI systems to assist their work, yet are rarely given a voice in defining what meaningful AI augmentation should look like or how to evaluate for it. In this paper, we propose worker-driven AI measurement---a bottom-up approach to AI evaluation where workers collaboratively shape decisions about which tasks AI should augment, what "successful" augmentation looks like, and how it should be measured. We explore how to support this through a case study with 19 workers from a local school social work organization. Through a series of eight workshops, workers iteratively develop their own measurement goals for AI evaluation, systematize these goals, and then design a benchmark to capture how effectively an LLM can "challenge" them to reflect on their own assumptions and biases in the context of their day-to-day work. Workers collaboratively design and refine an LLM-as-a-judge rubric based on their professional and lived expertise. In validations of the worker-created benchmark, we find that there is strong agreement between worker and LLM judge ratings and that the resulting benchmark can differentiate performance across six state-of-the-art LLMs. Based on our case study, we discuss opportunities for future work to support worker-driven AI measurement as a complementary approach to existing top-down AI evaluation approaches.

Problem

• worker-driven AI measurement
• AI augmentation
• evaluation
• collaborative design
• LLM

Innovation

• worker-driven AI measurement
• bottom-up approach
• LLM augmentation
• benchmark design
• reflective practice