RAVA: Retrieval-Augmented Viewpoint Alignment for Subject-Driven Image Generation

2026-06-16

AI Summary

This work addresses the challenges of unreliable cross-subject viewpoint control in multi-reference image generation, which often leads to viewpoint drift and structural misalignment. To mitigate these issues, the authors propose a retrieval-augmented geometry-guided framework that, for the first time, integrates a retrieval mechanism into cross-subject viewpoint alignment. The method learns cross-instance viewpoint embeddings to explicitly retrieve reference images with consistent viewpoints and employs a LogDet-based subset selection strategy to construct a compact, viewpoint-coherent, and structurally complementary reference set for a fine-tuned generator. Experimental results demonstrate that the proposed approach significantly outperforms zero-shot baselines and alternative retrieval strategies, confirming the effectiveness of combining retrieval augmentation with geometric priors in enhancing both viewpoint alignment accuracy and structural consistency in generated images.

Abstract

Reference-driven image generation has made rapid progress on identity preservation, but reliable viewpoint control across different subjects remains poorly understood. The difficulty is not merely generating a new image of the target subject: the model must infer the implicit viewpoint of one subject and transfer it to another subject using only image-level evidence, without camera poses, depth, or ray-based conditions. In this setting, existing generators conditioned on multiple image references often rely on spurious semantic correlations, which lead to viewpoint drift, part-level structural mismatches, and missing or unsupported target-specific content. We formulate this challenge as cross-subject viewpoint alignment and propose RAVA, a retrieval-augmented framework that supplies explicit geometric evidence before generation. RAVA first learns a cross-instance viewpoint embedding that retrieves target-subject images aligned with the anchor viewpoint, then applies a LogDet-based subset selection strategy to retain a compact reference set that is both view-consistent and structurally complementary. The selected references are finally consumed by a fine-tuned multi-reference image generator. Experiments show that generic semantic embeddings are nearly random for this task, while the proposed retriever substantially improves viewpoint retrieval quality. On cross-subject generation, RAVA consistently outperforms zero-shot baselines and stronger retrieval alternatives under the same generation backbone. These results indicate that cross-subject viewpoint alignment benefits from retrieval-augmented geometric grounding rather than relying on end-to-end generation alone.

Problem

• viewpoint alignment
• subject-driven generation
• cross-subject transfer
• reference-driven image generation
• geometric grounding

Innovation

• retrieval-augmented generation
• viewpoint alignment
• cross-subject image generation
• geometric grounding
• LogDet subset selection