Preference-conditioned image generation aims to produce images that reflect a user's aesthetics in addition to prompt semantics. PrefGen leverages a fine-tuned multimodal LLM to extract preference-aware embeddings from a few liked/disliked images, decomposes user signals into identity and semantic components, aligns semantic component distributionally to diffusion text embeddings via an MMD loss, and injects a fused user representation into a diffusion backbone to guide generation. The pipeline yields improved fidelity and stronger alignment to user preferences across automatic metrics and human studies.

Figure 1: Overview of our MLLM-based preference learning framework.

1. MLLM training: Fine-tune an MLLM on a preference-oriented VQA dataset — this helps the model learn multi-image preference reasoning.
2. Layer probing & embedding extraction:
   • esem: extracted from the top 4 layers (last-token pooling) — captures like/dislike semantics and styles used for aligning to text space.
   • ecore: middle-to-upper layer embeddings — capture stable user identity signals that generalize across prompts.
3. Distribution alignment: Map esem through a small MLP and minimize MMD against paired CLIP text embeddings (attributes). Distribution-level loss preserves diversity and avoids collapse that point-wise losses can cause.
4. Fusion & conditioning: Form final user vector e_u = [ĕsem; ecore; eimg], where eimg is a CLIP image embedding from a liked example. Inject into the diffusion UNet via a parallel user cross-attention branch.

Agent dataset: ~990,998 generated images from 50,153 simulated users (each agent has ~50 attributes; we sample like/dislike pairs). This large-scale synthetic dataset provides controlled and diverse preference signals, enabling robust training of our preference extraction and alignment modules.

Figure 2: Examples from the agent (synthetic) dataset.

Processed Pick-a-Pic (Real users): We additionally build a cleaned and standardized split from the Pick-a-Pic dataset, which contains preference choices from real human users over images generated by multiple diffusion models. Raw Pick-a-Pic contains noisy, single-step preferences, so we perform a multi-stage processing pipeline:

• remove low-quality or inconsistent entries,
• group samples by real user ID to construct reliable preference histories,
• aggregate multiple pairwise choices into like/dislike sets,
• ensure each user has enough images to form a meaningful preference profile.

Figure 3: Examples from the processed Pick-a-Pic dataset (real users).