Prompt-led AI video tools - how do you iterate?

[MartynFoster]

Hi all - this is a bit off-topic for this board, but I know a few people here work with AI generation, so I figured this was the right place to ask.

I have been looking at prompt-to-video tools for short previz clips: you describe the shot in plain text, get a few seconds of video back, and then rework the prompt rather than the keyframes. The one that got me curious is muse-video.app, which is built around that text-prompt-first workflow for short clips.

Two things I keep wondering about:

- How much of the improvement actually comes from rewriting the prompt, versus just re-running the same one?
- Do you treat the output as a finished 3-5 second shot, or is it always previz that you finish elsewhere?

Curious what everyone's iteration loop looks like - whether you use that tool or something completely different.

[Viridian]

Working with video, half the battle is using the right source image, the other half is in writing a good prompt. If you're comparing quality creators with casual beginners, the difference is in animation model first, but if given the same model, then the difference is in the image. If this image is the same, then the difference is on how we prompt.

Most people who get into AI animation use prompts that are FAR too vague, which leaves more room for the AI create whatever it wants. For example, someone might simply prompt "struggling and sinking in quicksand" and the whole thing is a plop and sink in 10 seconds in a puddle. A good prompt is far more specific - the exact motion (head, arms, legs, hips), the behaviour of the substance, the camera angle and movement, etc.

The AI will make what you tell it to make (assuming it can). If you don't specify what you want, it makes what it thinks you want based on its predictive algorithms.

This is where text-first is so important. If you don't specifically care about the details in the image or the video, then you don't need to engineer a good prompt.

Running AI prompts is more or less like gambling. You want to improve the odds with a good prompt, not re-roll the same prompt to get the same result. If my output doesn't match what I want, I have to reevaluate the text prompt to see why it's giving that result.

For example, I might have an output that results in a fairly quiet scene with heavy breathing rather than grunting. If I look at my prompt, it might say both "grunting and breathing heavily" but the AI can't do both, so it picks the one that is more likely in that output (the breathing), so I would have to rewrite the prompt to enable the AI to correctly render the scene.