CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation

2026-06-14

AI Summary

Existing diffusion models struggle to simultaneously preserve identity consistency, bind subject-specific attributes such as personal items and clothing, and maintain background fidelity in multi-subject reference-based image generation. To address this, this work introduces a benchmark dataset comprising 1,952 curated images featuring 100 celebrities, 115 unique items or garments, and 29 realistic backgrounds, along with 1,361 compositional prompts supporting multi-subject generation, text-to-image synthesis, and reference retrieval tasks. The study further proposes the first unified six-dimensional evaluation protocol, introducing two novel metrics: BG-Sim (based on DINOv3) for background fidelity and Attr-VQA (leveraging multimodal large models) for attribute binding accuracy, supplemented by structured interaction and positional maps as supervisory signals. Evaluations of five state-of-the-art methods reveal a significant performance drop as subject count increases, with attribute binding nearly failing for scenes involving more than three subjects.

Abstract

Multi-subject reference-based image generation requires jointly preserving multiple human identities, binding per-person objects and fashion items, and respecting a specified background scene, a regime where current diffusion models remain brittle. Existing benchmarks evaluate only one axis at a time and none jointly captures multi-identity composition with human-object interaction, background grounding, and spatial plausibility. We introduce CogCanvas, a benchmark of 1,952 curated reference images spanning 100 celebrity identities, 115 distinctive objects and fashion items, and 29 real-world background scenes including landmarks, from which we construct 1,361 compositional prompts covering 2-5 person group sizes. The curation pipeline combines DINOv2-based deduplication, two-stage aesthetic filtering, and automated derivation of structured interaction and position graphs that serve as ground-truth supervision. CogCanvas supports three tasks, reference-based multi-human-object generation (primary), text-to-image compositional generation, and reference retrieval, under a unified six-axis evaluation protocol. We introduce two metrics tailored to the multi-reference setting: BG-Sim, which scores background fidelity on SAM 3-masked regions via DINOv3 feature similarity, and Attr-VQA, which uses a multimodal LLM to verify per-subject attribute binding and inter-person interactions against the structured graphs. Benchmarking five SOTA methods reveals that every model degrades substantially as group size grows from 2 to 5, with near-complete failure on object/fashion binding beyond three subjects.

Problem

• multi-subject image generation
• reference-based generation
• identity preservation
• human-object interaction
• background grounding

Innovation

• multi-subject image generation
• reference-based generation
• compositional benchmark
• structured interaction graph
• attribute binding evaluation