Documenting my current process with NovelAI and Stable Diffusion.
I'm starting off with a very early version of my Quicky Sanders character many years ago, which I found again. This is very obviously an inexperienced and crude drawing, prior to my development in digital art.
Popping this straight into SD as an image reference is tough. The AI often doesn't recognise the features that make this a human in a mudpit, partly because of how badly is drawn, but also because of the style. I'm trying to create a photorealistic output, and a flat 2D drawing doesn't transition well. It might be easier to turn this into another, higher quality 2D art style, but I want the photorealistic effect. I need to turn this into a form that SD can work with. This is where my experience with NovelAI comes in.
NovelAI's model, though based on anime, can generate some decent photorealistic styles with the right prompts. The best part is that it allows very easy editing. You could otherwise make the edits yourself using a Photoshop-style tool (and if you're really good, you can skip this step entirely and just create a photomanipulation and run that through SD). The crucial part is that you want to get the mud texture looking enough like a mud texture. A few ripples with NovelAI's editing tool and I get something like this.
This is workable, though I cannot emphasise enough how the mud texture will influence how SD will perceive the setting. Depending on the texture you want, you may want to edit the image to have the right appearance. The quality of the character and background isn't important, as SD will work with that.
You can really see how easily Stable Diffusion works with the output from NovelAI. It is a closer approximation to what I want, thereby removing the randomness. SD has done a decent job with the mud texture, being close to the NovelAI output, though the background, trees and mud could do some more work. I use a mask to work with the mud and background and run that a few times.
This is already looking miles better. I try to run it a few more times as a whole image to see if I can squeeze more photorealism. The returns diminish at this point, but I ended up with this.
It's not the most photorealistic output I've attained. I actually had far more realistic ones, though it ended up turning the character too much into the Asian side and it didn't match the mud texture. Ultimately, considering I started from a basic digital drawing, the transformation is mind-boggling good - and if I wanted to, I could have made it better.
You do not have the required permissions to view the files attached to this post.