Turning Any Stable Diffusion Model Into an Inpainting Model
In the spirit of full disclosure, the content of this post is heavily cribbed from this post on Reddit. However, as we've seen, the Internet is not forever. It is entirely possible that a wealth of knowledge could be lost at any time due to any number of reasons. Because I have found this particular post so helpful and find myself coming back to it over and over, I thought it would be appropriate to share this method of creating an inpainting model from any custom stable diffusion model.
Inpainting models, like the name suggests, are specialized models that excel at "filling in" or replacing sections of an image. They're especially good at decoding what a section of an image should look like based on the section of image that already exists. This is useful when you're generating images and only a small section needs to be corrected, or if you're trying to add something specific to an image that exists.
So how is this done? With a model merge. Automatic1111 has an excellent model merging tool that we'll use. Let's assume that you have a custom model called my-awesome-model.ckpt that is based on stable-diffusion-1.5.
In the A1111 Checkpoint Merger interface, follow these steps:
- Set "Primary model (A)" to
stable-diffusion-1.5-inpainting.ckpt. - Set "Secondary model (B)" to
my-awesome-model.ckpt. - Set "Tertiary model (C)" to
stable-diffusion-1.5.ckpt. - Set "Multiplier (M)" to
1. - Set "Interpolation Method" to
Add difference. - Give your model a name in the "Custom Name" field, such as
my-awesome-model-inpainting.ckpt.- Adding "-inpainting" will signal to A1111 that the model is an inpainting model. This is useful for extensions such as openOutpaint. Also, it's just a good idea to properly label your models. Because we know you're a degenerate that has hundreds of custom waifu models downloaded from CivitAI.
- Click
Merge.
And bazinga! You have your own custom inpainting stable diffusion model. Thanks again to /u/MindInTheDigits for sharing the process.