Retrospective Learning from Interactions

2024-10-17

arXiv.org

AI Summary

Large language models (LLMs) struggle to leverage implicit user feedback—such as rephrasing, emotional cues, or task switching—in multi-turn interactions, often relying on costly manual annotations. Method: We propose ReSpect, an unsupervised retrospective learning framework that automatically detects correction signals from task-agnostic implicit behaviors without any external supervision. ReSpect integrates implicit feedback recognition and response refinement within a multimodal LLM architecture, combining interaction trajectory modeling with online policy updating for zero-annotation continual optimization. Contribution/Results: Evaluated on abstract reasoning tasks, ReSpect boosts task completion rate from 31% to 82%, demonstrating strong effectiveness and generalization. This work establishes the first systematic approach to implicit-feedback-driven unsupervised retrospective learning, introducing a novel paradigm for online self-adaptation of LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.

Problem

• Learning from implicit feedback in multi-turn LLM-user interactions
• Improving task completion via retrospective signal analysis
• Enhancing multimodal LLM performance without external annotations

Innovation

• Utilizes implicit feedback from multi-turn interactions
• Introduces ReSpect for learning without additional annotations
• Improves task completion rate via retrospective learning