Promptimizer: User-Led Prompt Optimization for Personal Content Classification

2025-10-10

AI Summary

Current LLM-driven content classifiers support natural language prompting but lack automated optimization methods capable of modeling users’ dynamic adjustments to filtering intent during initialization and iterative refinement. To address this, we propose a user-centric prompt optimization framework that enables progressive, personalized classifier construction via human-AI collaboration: users articulate intent at each stage and receive interpretable, executable prompts, while the system integrates natural language instructions, LLM-based reasoning, and closed-loop human feedback. The method has been deployed in YouTube’s creator tool Puffin. A three-week field study with 10 creators demonstrated successful construction of diverse comment filters, yielding significant improvements in intent controllability, transparency, and alignment with user preferences. Results validate an effective trade-off between practical usability and classification performance.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Intelligent User Interfaces
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While LLMs now enable users to create content classifiers easily through natural language, automatic prompt optimization techniques are often necessary to create performant classifiers. However, such techniques can fail to consider how social media users want to evolve their filters over the course of usage, including desiring to steer them in different ways during initialization and iteration. We introduce a user-centered prompt optimization technique, Promptimizer, that maintains high performance and ease-of-use but additionally (1) allows for user input into the optimization process and (2) produces final prompts that are interpretable. A lab experiment (n=16) found that users significantly preferred Promptimizer's human-in-the-loop optimization over a fully automatic approach. We further implement Promptimizer into Puffin, a tool to support YouTube content creators in creating and maintaining personal classifiers to manage their comments. Over a 3-week deployment with 10 creators, participants successfully created diverse filters to better understand their audiences and protect their communities.

Problem

• Optimizing prompts for personalized content classification
• Incorporating user input during prompt iteration process
• Creating interpretable prompts for social media filtering

Innovation

• User-centered prompt optimization for content classification
• Human-in-the-loop approach maintains high performance
• Produces interpretable prompts for personal filters