DDL2PropBank Agent: Benchmarking Multi-Agent Frameworks' Developer Experience Through a Novel Relational Schema Mapping Task

2026-02-03

arXiv.org

AI Summary

This study addresses the lack of systematic evaluation metrics for assessing how amenable multi-agent frameworks are to AI programming assistants. To this end, the authors propose a composite metric termed AI-assistability (𝒜ℐ) and introduce the DDL2PropBank benchmark task. They uniformly implement an Agent-as-a-Tool paradigm across ten mainstream frameworks and evaluate each based on structural alignment and functional correctness (pass@1). Experimental results demonstrate that alignment between framework APIs and established programming conventions contributes more significantly to AI-assistability than declarative design alone: Agno achieves the highest 𝒜ℐ score (0.55) due to its strong convention alignment, whereas the most declarative framework, DSPy, performs worst (0.07). Correlation analysis further confirms that convention alignment is a key driver of AI-assistability (r = 0.576).

Abstract

Multi-agent frameworks promise to simplify LLM-driven software development, yet there is no principled way to evaluate their developer experience in a controlled setting. We introduce DDL2PropBank, a novel benchmark task that maps relational database schemas to PropBank rolesets, requiring autonomous retrieval of candidate frames and fine-grained linguistic reasoning over table names, columns, and relations. Using the Agent-as-a-Tool pattern, we implement identical agent logic across 10 frameworks and evaluate along two dimensions: (i) code complexity via static analysis, and (ii) AI-assistability -- the extent to which LLMs can autonomously generate correct, framework-specific code. Our results reveal a threefold complexity spectrum, with Pydantic AI and Agno requiring the least implementation overhead. For AI-assistability, structural alignment scores reliably proxy runtime success for frameworks with single canonical patterns, but overestimate correctness for multi-pattern frameworks. Agno emerges as the strongest overall performer, combining lowest complexity with highest structural alignment and 83% pass@1.

Problem

• AI-assistability
• multi-agent frameworks
• declarative design
• convention alignment
• LLM-driven software development

Innovation

• AI-assistability
• multi-agent frameworks
• declarative design
• convention alignment
• benchmarking