A Multistakeholder Approach to Value-Driven Co-Design of Recommender System Evaluation Metrics in Digital Archives

2025-07-04

AI Summary

Existing evaluation of digital archive recommendation systems (RecSys) overrelies on commercial engagement metrics, neglecting the diverse value claims of multiple stakeholders. Method: We propose a multi-stakeholder co-designed evaluation framework, introducing for the first time a four-stage research funnel model—“Discovery–Interaction–Integration–Impact”—to ground novel, transferable metrics including research pathway quality, contextual appropriateness, and metadata-weighted relevance. Through five high-fidelity focus groups (n=25), we qualitatively identified core value dimensions—visibility, professional adaptability, and transparency—and aligned each metric to its corresponding stage. Contribution/Results: The framework establishes a value-driven evaluation paradigm that balances archival professionalism with public accessibility, mitigating collection representation bias and stakeholder value conflicts. It constitutes the first scalable, context-adaptive RecSys evaluation framework tailored to cultural heritage institutions.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

This paper presents the first multistakeholder approach for translating diverse stakeholder values into an evaluation metric setup for Recommender Systems (RecSys) in digital archives. While commercial platforms mainly rely on engagement metrics, cultural heritage domains require frameworks that balance competing priorities among archivists, platform owners, researchers, and other stakeholders. To address this challenge, we conducted high-profile focus groups (5 groups x 5 persons) with upstream, provider, system, consumer, and downstream stakeholders, identifying value priorities across critical dimensions: visibility/representation, expertise adaptation, and transparency/trust. Our analysis shows that stakeholder concerns naturally align with four sequential research funnel stages: discovery, interaction, integration, and impact. The resulting framework addresses domain-specific challenges including collection representation imbalances, non-linear research patterns, and tensions between specialized expertise and broader accessibility. We propose tailored metrics for each stage in this research journey, such as research path quality for discovery, contextual appropriateness for interaction, metadata-weighted relevance for integration, and cross-stakeholder value alignment for impact assessment. Our contributions extend beyond digital archives to the broader RecSys community, offering transferable evaluation approaches for domains where value emerges through sustained engagement rather than immediate consumption.

Problem

• Balancing stakeholder values in RecSys for digital archives
• Addressing collection representation imbalances and research patterns
• Developing tailored metrics for diverse research funnel stages

Innovation

• Multistakeholder focus groups identify value priorities
• Sequential research funnel stages guide metric design
• Tailored metrics address domain-specific challenges