From the immediate to the image, Secure Diffusion is a pipeline with many parts and parameters. All these parts working collectively creates the output. If a element behave otherwise, the output will change. Due to this fact, a nasty setting can simply damage your image. On this submit, you will note:

• How the completely different parts of the Secure Diffusion pipeline impacts your output
• Find out how to discover the very best configuration that will help you generate a top quality image

Let’s get began.

Find out how to Use Secure Diffusion Successfully.
Photograph by Kam Idris. Some rights reserved.

Overview

This submit is in three components; they’re:

• Significance of a Mannequin
• Choosing a Sampler and Scheduler
• Measurement and the CFG Scale

Significance of a Mannequin

If there’s one element within the pipeline that has probably the most influence, it have to be the mannequin. Within the Internet UI, it’s known as the “checkpoint”, named after how we saved the mannequin after we educated a deep studying mannequin.

The Internet UI helps a number of Secure Diffusion mannequin architectures. The most typical structure these days is the model 1.5 (SD 1.5). Certainly, all model 1.x share the same structure (every mannequin has 860M parameters) however are educated or fine-tuned underneath completely different methods.

Structure of Secure Diffusion 1.x. Determine from Rombach et al (2022)

There’s additionally Secure Diffusion 2.0 (SD 2.0), and its up to date model 2.1. This isn’t a “revision” from model 1.5, however a mannequin educated from scratch. It makes use of a special textual content encoder (OpenCLIP as an alternative of CLIP); due to this fact, they might perceive key phrases otherwise. One noticeable distinction is that OpenCLIP is aware of fewer names of celebrities and artists. Therefore, the immediate from Secure Diffusion 1.5 could also be out of date in 2.1. As a result of the encoder is completely different, SD2.x and SD1.x are incompatible, whereas they share the same structure.

Subsequent comes the Secure Diffusion XL (SDXL). Whereas model 1.5 has a local decision of 512×512 and model 2.0 elevated it to 768×768, SDXL is at 1024×1024. You aren’t advised to make use of a vastly completely different dimension than their native decision. SDXL is a special structure, with a a lot bigger 6.6B parameters pipeline. Most notably, the fashions have two components: the Base mannequin and the Refiner mannequin. They arrive in pairs, however you possibly can swap out considered one of them for a appropriate counterpart, or skip the refiner if you want. The textual content encoder used combines CLIP and OpenCLIP. Therefore, it ought to perceive your immediate higher than any older structure. Working SDXL is slower and requires far more reminiscence, however normally in higher high quality.

Structure of SDXL. Determine from Podell et al (2023)

What issues to you is that you must classify your fashions into three incompatible households: SD1.5, SD2.x, and SDXL. They behave otherwise along with your immediate. Additionally, you will discover that SD1.5 and SD2.x would want a adverse immediate for a very good image, however it’s much less necessary in SDXL. In case you’re utilizing SD2.x fashions, additionally, you will discover which you could choose your refiner within the Internet UI.

Photographs generated with the immediate, ‘A quick meals restaurant in a desert with identify “Sandy Burger”’, utilizing SD 1.5 with completely different random seed. Observe that none of them spelled the identify appropriately.

Photographs generated with the immediate, ‘A quick meals restaurant in a desert with identify “Sandy Burger”’, utilizing SD 2.0 with completely different random seed. Observe that not all of them spelled the identify appropriately.

Photographs generated with the immediate, ‘A quick meals restaurant in a desert with identify “Sandy Burger”’, utilizing SDXL with completely different random seed. Observe that each one of them spelled the identify appropriately.

One attribute of Secure Diffusion is that the unique fashions are much less succesful however adaptable. Due to this fact, a whole lot of third-party fine-tuned fashions are produced. Most important are the fashions specializing in sure types, corresponding to Japanese anime, western cartoons, Pixar-style 2.5D graphics, or photorealistic footage.

You will discover fashions on Civitai.com or Hugging Face Hub. Search with key phrases corresponding to “photorealistic” or “2D” and sorting by score would normally assist.

Choosing a Sampler and Scheduler

Picture diffusion is to begin with noise and replaces the noise strategically with pixels till the ultimate image is produced. It’s later discovered that this course of might be represented as a stochastic differential equation. Fixing the equation numerically is feasible, and there are completely different algorithms of various accuracy.

Essentially the most generally used sampler is Euler. It’s conventional however nonetheless helpful. Then, there’s a household of DPM samplers. Some new samplers, corresponding to UniPC and LCM, have been launched lately. Every sampler is an algorithm. It’s to run for a number of steps, and completely different parameters are utilized in every step. The parameters are set utilizing a scheduler, corresponding to Karras or exponential. Some samplers have an alternate “ancestral” mode, which provides randomness to every step. That is helpful in order for you extra artistic output. These samplers normally bear a suffix “a” of their identify, corresponding to “Euler a” as an alternative of “Euler”. The non-ancestral samplers converge, i.e., they may stop altering the output after sure steps. Ancestral samplers would give a special output for those who enhance the step dimension.

Choosing sampler, scheduler, steps, and different parameters within the Secure Diffusion Internet UI

As a person, you possibly can assume Karras is the scheduler for all circumstances. Nonetheless, the scheduler and step dimension would want some experimentation. Both Euler or DPM++2M needs to be chosen as a result of they stability high quality and pace finest. You can begin with a step dimension of round 20 to 30; the extra steps you select, the higher the output high quality when it comes to particulars and accuracy, however proportionally slower.

Measurement and CFG Scale

Recall that the picture diffusion course of begins from a loud image, step by step putting pixels conditioned by the immediate. How a lot the conditioning can influence the diffusion course of is managed by the parameter CFG scale (classifier-free steering scale).

Sadly, the optimum worth of CFG scale is determined by the mannequin. Some fashions work finest with a CFG scale of 1 to 2, whereas others are optimized for 7 to 9. The default worth is 7.5 within the Internet UI. However as a normal rule, the upper the CFG scale, the stronger the output picture conforms to your immediate.

In case your CFG scale is simply too low, the output picture is probably not what you anticipated. Nonetheless, there’s another excuse you don’t get what you anticipated: The output dimension. For instance, for those who immediate for an image of a person standing, chances are you’ll get a headshot of a half-body shot as an alternative until you set the picture dimension to a top considerably higher than the width. The diffusion course of units the image composition within the early steps. It’s simpler to plan a standing man on a taller canvas.

Producing a half-body shot if offered a sq. canvas.

Producing a full physique shot with the identical immediate, similar seed, and solely the canvas dimension is modified.

Equally, for those who give an excessive amount of element to one thing that occupies a small a part of the picture, these particulars can be ignored as a result of there usually are not sufficient pixels to render these particulars. That’s the reason SDXL, for instance, is mostly higher than SD 1.5 because you normally use a bigger pixel dimension.

As a last comment, producing footage utilizing picture diffusion fashions includes randomness. All the time begin with a batch of a number of footage to verify the dangerous output isn’t merely because of the random seed.

Additional Readings

This part supplies extra assets on the subject if you wish to go deeper.

Abstract

On this submit, you discovered about some refined particulars that impacts the picture technology in Secure Diffusion. Particularly, you discovered:

• The distinction between completely different variations of Secure Diffusion
• How the scheduler and sampler impacts the picture diffusion course of
• How the canvas dimension could have an effect on the output