Viridian's AI Experiments

[Viridian]

Very cool stuff, I went to that site and tinkered around most of the time my images the AI would lift them out of the quicksand lol

That one happened to me 99% of the time. The lack of control of what it generates is a problem. There aren't tools that effectively allow the use of key frames.

[MDLambert]

I did take one of the vids and ran it through a reverse tool. Its not horrible.

[Viridian]

Now for today's updates. It's been a pretty busy day trying to upskill. This is what we have so far.

I've refined my process for voiceovers and lip sync. I currently use D-ID for AI voices, though instead of generating the still image with lip sync, I just record the audio. Instead, I run the video through Wav2Lip, which is a free lip-sync encoder. This results in "saf_runway", the same Safari GIrl image made with Stable Diffusion), voiceover with D-ID, animated with Runway, and synced with Wav2Lip. That's quite a workflow for 4 seconds - and that's not even a polished product.

I've also learned that Runway is, at this point, not very good at using image to video. As MDLambert has tried, it very often morphs quicksand images because it doesn't recognise them. Either the character is jumping out of the quicksand, or the AI thinks it's a plus-sized model. Very specific images work - the shoulder-deep ones like I demonstrated earlier have the most consistent results. You otherwise end up burning through a lot of credits gambling for a good generation.

However, the TEXT to video is perhaps the best AI tool right now, edging out Pika. It is exceptionally good at creating an AI-generated dynamic scene. "tombtest" shows both an environment and a character pan. "saf_runwaytest" is my attempt at rendering Safari Girl in this style, with Wav2Lip sync.

It's still in its infancy, but after seeing animators create animated trailers, I had to try it myself. Safari Girl ends up looking ghoulish when rendered in Runway through text, but Runway has the function of training AI portraits. All it takes is 15-30 samples. I created around 20 AI portraits of Quicky Sanders, which allows Runway to generate a closer version of my OC whenever I type her name in the prompt. That includes the videos.

That brings me to my great work for today: "Into The Woods", a proof of concept for a quicksand film trailer.

[bogbud]

Into the woods is the film we want to see!
Already now it's better than that movie "Quicksand"

[MDLambert]

I have been trying the text to vid with an image base and all I get is nightmare fuel. I am going to try without the image and see what I get. I signed up for a months sub and I am beginning to think that was a mistake lol. I thin the D-ID is probably better at this point. The AI just doesnt understand.

[Viridian]

I have beeen trying the text to vid with an image base and all I get is nightmare fuel. I am going to try without the image and see what I get

I read that the prompt flat out ignores the image. It's supposedly meant to apply the style of text prompt to the image, not direct what happens to the image. As it stands now, the limitation is that we can't feed images into it and get predictable results since it doesn't know what it's working with. It's better at generating it's own content, but without the negative prompts that image generators have, the results can be nightmare fuel. When it does something it's good at - especially on a trained portrait - it's very good.

[Viridian]

I'm having trouble uploading this file. Here's an extended quicksand dialogue scene with the "into the Woods" concept:
https://www.deviantart.com/viridianqs/a ... -978919317

Only the mud sound is from an existing sound effect file. EVERYTHING else is AI generated.

Visuals and animations are made with RunwayML, with Sanders created with Stable Diffusion.

Voiceovers made with D-ID.

Lip sync done with Wav2Lip.

Soundtrack made with Soundraw.

Bird ambience made with AudioLDM.

[MDLambert]

That was pretty cool, well done. Im rooting for quicky!

[Viridian]

Another set of sets.

[Theo]

...That brings me to my great work for today: "Into The Woods", a proof of concept for a quicksand film trailer.

When's the premier? I've got the popcorn ready to go!