stable-diffusion-image-generation

Stable Diffusion 图像生成

使用 HuggingFace Diffusers 库生成 Stable Diffusion 图像的综合指南。

何时使用 Stable Diffusion

在以下情况下使用 Stable Diffusion：

• 根据文本描述生成图像
• 执行图像到图像转换（风格迁移、增强）
• 图像修复（填充蒙版区域）
• 图像扩展（将图像扩展到边界之外）
• 为现有图像创建变体
• 构建自定义图像生成工作流

主要功能：

• 文本到图像：根据自然语言提示词生成图像
• 图像到图像：通过文本引导转换现有图像
• 图像修复：使用上下文感知内容填充蒙版区域
• ControlNet：添加空间条件（边缘、姿势、深度）
• LoRA 支持：高效微调和风格适配
• 多种模型：支持 SD 1.5、SDXL、SD 3.0、Flux

改用其他方案：

• DALL-E 3：无需 GPU 的 API 生成
• Midjourney：生成具有艺术风格的图像
• Imagen：与 Google Cloud 集成
• Leonardo.ai：基于网页的创意工作流

快速开始

安装

pip install diffusers transformers accelerate torch
pip install xformers  # Optional: memory-efficient attention

基本文本到图像生成

from diffusers import DiffusionPipeline
import torch

# Load pipeline (auto-detects model type)
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
)
pipe.to("cuda")

# Generate image
image = pipe(
    "A serene mountain landscape at sunset, highly detailed",
    num_inference_steps=50,
    guidance_scale=7.5
).images[0]

image.save("output.png")

使用 SDXL（质量更高）

from diffusers import AutoPipelineForText2Image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")

# Enable memory optimization
pipe.enable_model_cpu_offload()

image = pipe(
    prompt="A futuristic city with flying cars, cinematic lighting",
    height=1024,
    width=1024,
    num_inference_steps=30
).images[0]

架构概览

三大支柱设计

Diffusers 围绕三个核心组件构建：

Pipeline (orchestration)
├── Model (neural networks)
│   ├── UNet / Transformer (noise prediction)
│   ├── VAE (latent encoding/decoding)
│   └── Text Encoder (CLIP/T5)
└── Scheduler (denoising algorithm)

Pipeline 推理流程

Text Prompt → Text Encoder → Text Embeddings
                                    ↓
Random Noise → [Denoising Loop] ← Scheduler
                      ↓
               Predicted Noise
                      ↓
              VAE Decoder → Final Image

核心概念

Pipelines

Pipelines 用于编排完整工作流：

Pipeline	用途
StableDiffusionPipeline	文本到图像（SD 1.x/2.x）
StableDiffusionXLPipeline	文本到图像（SDXL）
StableDiffusion3Pipeline	文本到图像（SD 3.0）
FluxPipeline	文本到图像（Flux 模型）
StableDiffusionImg2ImgPipeline	图像到图像
StableDiffusionInpaintPipeline	图像修复

调度器

调度器控制去噪过程：

调度器	步数	质量	用途
EulerDiscreteScheduler	20-50	良好	默认选择
EulerAncestralDiscreteScheduler	20-50	良好	生成更多变体
DPMSolverMultistepScheduler	15-25	优秀	快速且高质量
DDIMScheduler	50-100	良好	确定性生成
LCMScheduler	4-8	良好	速度非常快
UniPCMultistepScheduler	15-25	优秀	快速收敛

切换调度器

from diffusers import DPMSolverMultistepScheduler

# Swap for faster generation
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config
)

# Now generate with fewer steps
image = pipe(prompt, num_inference_steps=20).images[0]

生成参数

关键参数

参数	默认值	描述
prompt	必填	对所需图像的文本描述
negative_prompt	无	图像中要避免的内容
num_inference_steps	50	去噪步数（越多 = 质量越高）
guidance_scale	7.5	对提示词的遵循程度（通常为 7-12）
height、width	512/1024	输出尺寸（8 的倍数）
generator	无	用于确保结果可复现的 Torch 生成器
num_images_per_prompt	1	批次大小

可复现的生成

import torch

generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt="A cat wearing a top hat",
    generator=generator,
    num_inference_steps=50
).images[0]

负面提示词

image = pipe(
    prompt="Professional photo of a dog in a garden",
    negative_prompt="blurry, low quality, distorted, ugly, bad anatomy",
    guidance_scale=7.5
).images[0]

图像到图像

通过文本引导转换现有图像：

from diffusers import AutoPipelineForImage2Image
from PIL import Image

pipe = AutoPipelineForImage2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

init_image = Image.open("input.jpg").resize((512, 512))

image = pipe(
    prompt="A watercolor painting of the scene",
    image=init_image,
    strength=0.75,  # How much to transform (0-1)
    num_inference_steps=50
).images[0]

图像修复

填充蒙版区域：

from diffusers import AutoPipelineForInpainting
from PIL import Image

pipe = AutoPipelineForInpainting.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16
).to("cuda")

image = Image.open("photo.jpg")
mask = Image.open("mask.png")  # White = inpaint region

result = pipe(
    prompt="A red car parked on the street",
    image=image,
    mask_image=mask,
    num_inference_steps=50
).images[0]

ControlNet

添加空间条件以实现精准控制：

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch

# Load ControlNet for edge conditioning
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_canny",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# Use Canny edge image as control
control_image = get_canny_image(input_image)

image = pipe(
    prompt="A beautiful house in the style of Van Gogh",
    image=control_image,
    num_inference_steps=30
).images[0]

可用的 ControlNet

ControlNet	输入类型	用途
canny	边缘图	保留结构
openpose	姿势骨架	人体姿势
depth	深度图	具备 3D 感知的生成
normal	法线图	表面细节
mlsd	线段	建筑线条
scribble	粗略草图	从草图生成图像

LoRA 适配器

加载微调后的风格适配器：

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# Load LoRA weights
pipe.load_lora_weights("path/to/lora", weight_name="style.safetensors")

# Generate with LoRA style
image = pipe("A portrait in the trained style").images[0]

# Adjust LoRA strength
pipe.fuse_lora(lora_scale=0.8)

# Unload LoRA
pipe.unload_lora_weights()

多个 LoRA

# Load multiple LoRAs
pipe.load_lora_weights("lora1", adapter_name="style")
pipe.load_lora_weights("lora2", adapter_name="character")

# Set weights for each
pipe.set_adapters(["style", "character"], adapter_weights=[0.7, 0.5])

image = pipe("A portrait").images[0]

内存优化

启用 CPU 卸载

# Model CPU offload - moves models to CPU when not in use
pipe.enable_model_cpu_offload()

# Sequential CPU offload - more aggressive, slower
pipe.enable_sequential_cpu_offload()

注意力切片

# Reduce memory by computing attention in chunks
pipe.enable_attention_slicing()

# Or specific chunk size
pipe.enable_attention_slicing("max")

xFormers 高效内存注意力

# Requires xformers package
pipe.enable_xformers_memory_efficient_attention()

面向大图像的 VAE 切片

# Decode latents in tiles for large images
pipe.enable_vae_slicing()
pipe.enable_vae_tiling()

模型变体

加载不同精度的模型

# FP16 (recommended for GPU)
pipe = DiffusionPipeline.from_pretrained(
    "model-id",
    torch_dtype=torch.float16,
    variant="fp16"
)

# BF16 (better precision, requires Ampere+ GPU)
pipe = DiffusionPipeline.from_pretrained(
    "model-id",
    torch_dtype=torch.bfloat16
)

加载指定组件

from diffusers import UNet2DConditionModel, AutoencoderKL

# Load custom VAE
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")

# Use with pipeline
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    vae=vae,
    torch_dtype=torch.float16
)

批量生成

高效生成多张图像：

# Multiple prompts
prompts = [
    "A cat playing piano",
    "A dog reading a book",
    "A bird painting a picture"
]

images = pipe(prompts, num_inference_steps=30).images

# Multiple images per prompt
images = pipe(
    "A beautiful sunset",
    num_images_per_prompt=4,
    num_inference_steps=30
).images

常见工作流

工作流 1：高质量生成

from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler
import torch

# 1. Load SDXL with optimizations
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

# 2. Generate with quality settings
image = pipe(
    prompt="A majestic lion in the savanna, golden hour lighting, 8k, detailed fur",
    negative_prompt="blurry, low quality, cartoon, anime, sketch",
    num_inference_steps=30,
    guidance_scale=7.5,
    height=1024,
    width=1024
).images[0]

工作流 2：快速原型设计

from diffusers import AutoPipelineForText2Image, LCMScheduler
import torch

# Use LCM for 4-8 step generation
pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load LCM LoRA for fast generation
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.fuse_lora()

# Generate in ~1 second
image = pipe(
    "A beautiful landscape",
    num_inference_steps=4,
    guidance_scale=1.0
).images[0]

常见问题

CUDA 内存不足：

# Enable memory optimizations
pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing()
pipe.enable_vae_slicing()

# Or use lower precision
pipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)

黑屏／噪点图像：

# Check VAE configuration
# Use safety checker bypass if needed
pipe.safety_checker = None

# Ensure proper dtype consistency
pipe = pipe.to(dtype=torch.float16)

生成速度慢：

# Use faster scheduler
from diffusers import DPMSolverMultistepScheduler
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

# Reduce steps
image = pipe(prompt, num_inference_steps=20).images[0]

参考资料

• 高级用法 - 自定义管线、微调、部署
• 故障排除 - 常见问题及解决方案

资源

• 文档：https://huggingface.co/docs/diffusers
• 代码库：https://github.com/huggingface/diffusers
• 模型中心：https://huggingface.co/models?library=diffusers
• Discord：https://discord.gg/diffusers