PaperRouter-Agent: A Content-Grounded LLM Agent for Personalized Hierarchical Paper Routing

2026-07-13

AI Summary

This work addresses the challenge of organizing user-defined, dynamically evolving folder hierarchies in reference management systems, which are poorly handled by conventional hierarchical classification methods. The authors propose a training-free LLM-based agent that routes papers by analyzing their full content within target folders—rather than relying solely on folder names—to capture both topical anchors and personalized semantics. The approach integrates candidate hierarchy pruning, evidence retrieval, membership verification, and a similarity gating mechanism informed by historical rejection feedback, enabling adaptation to diverse organizational schemes such as topics, conferences, or years without fine-tuning. Evaluated on real user data, the method achieves a Recall@1 of 0.61 (+22%) and Recall@3 of 0.83; on the LaMP-2 benchmark, it attains 51.5% accuracy and a 9.0-point gain in macro F1, significantly outperforming baselines while maintaining low deployment overhead.

Abstract

Researchers organize the papers they collect into personal folder hierarchies in reference managers, and route each new paper into the folder where it belongs. This task differs from standard hierarchical text classification. A user's folder hierarchy is not a fixed, shared taxonomy but a private and evolving folksonomy whose folder meanings may be topical, shorthand, venue-based, or process-oriented, and are often defined by the papers already stored inside them. We formalize this setting as personalized hierarchical paper routing (PHPR): assigning an incoming paper to folders in a user-specific hierarchy without per-user training. We propose PaperRouter-Agent, a training-free LLM agent that grounds routing decisions in folder members rather than folder names alone. The agent first narrows the candidate hierarchy, retrieves folder-specific evidence, verifies fit by inspecting member papers, and incorporates similarity-gated feedback from past user rejections. A formative study on real personal libraries shows that PaperRouter-Agent raises overall Recall@1 from 0.39 to 0.61 and Recall@3 from 0.57 to 0.83, with the largest gains on organizational folders defined by metadata such as venue or year, where single-shot methods collapses (Recall@1 0.09 to 0.50). On the public LaMP-2 benchmark, the same approach improves accuracy from 44.5% to 51.5% (+9.0 macro-F1) over a single-shot baseline, while remaining low-cost for practical use.

Problem

• personalized hierarchical paper routing
• folder hierarchy
• reference management
• folksonomy
• paper classification

Innovation

• content-grounded LLM agent
• personalized hierarchical paper routing
• folder folksonomy
• training-free routing
• evidence-based retrieval