Navigating User Behavior toward Personalized Multimodal Generation

2026-06-23

AI Summary

This work addresses the challenge that users often struggle to precisely articulate visual generation requests, leading to AI-generated content misaligned with their intent. To bridge this gap, the authors propose a personalized multimodal generation approach grounded in modeling user interaction history. Their method employs a dual-token behavior representation mechanism—coupling collaborative and textual encodings—to transform behavioral signals into structured instructions. A two-stage training framework is introduced: first, supervised fine-tuning via evolutionary search-guided distillation, followed by hierarchical self-consistent reinforcement learning for alignment optimization. This strategy effectively mitigates insufficient behavioral semanticization and weak instruction generation capabilities, significantly improving output quality, instruction specificity and feasibility, and next-item prediction accuracy across multiple tasks.

Abstract

Modern AIGC pipelines deliver high-fidelity images and videos but presuppose a well-formed creation instruction, while end users rarely articulate visual details, leaving generators misaligned with user demand. We study personalized content generation, which turns a user's interaction history into an executable instruction for downstream synthesis, and identify two obstacles: behavior must be encoded in a form legible to language reasoning, and the model must acquire instruction-writing skill absent from both pretraining and behavior data. We propose NaviGen, which represents each item with a dual identifier coupling a collaborative code and a textual code as a behavioral substrate and a semantic bridge in one token stream. On this representation, a two-stage SFT+RL pipeline first distills preference reasoning and instruction writing from evolutionarily searched supervision, then aligns generation with user intent through hierarchical and self-consistent rewards. Experiments across product, game, and short-video domains show that NaviGen improves personalized image and video generation, strengthens next-item prediction, and yields more specific, relevant, and visually generatable instructions. Our code is anonymously released at: https://github.com/iLearn-Lab/NaviGen.

Problem

• personalized generation
• user behavior
• multimodal generation
• instruction alignment
• AIGC

Innovation

• personalized multimodal generation
• dual identifier representation
• behavior-to-instruction distillation
• SFT+RL alignment
• user intent modeling