• Popcorn Points determine how popular a video is. You can click the popcorn bucket or simply react (Like, Love, etc.) and it will register a vote.

Layering

Nate North

Nate North


Just a quick example of some basic layering, a foundational element of our system that makes many shots interchangeable, allowing set and art design to happen quickly at an extremely low cost. This is identical to the techniques of many studios, except that we're creating an internal set of specific standards that allow enhanced modularity, and using AI as a force multiplier in conjunction with that modularity.

Layman's terms, we use a robot dolly to film a tree at 35 mm, then repeat the same camera move with a replay function in front of a canyon, a city, and a river. Now that tree or any other asset filmed on "SP camera track X" can be used together seamlessly in cinematic motion. To avoid confusion, I'd add that this video does not show off that cinematic motion, just the layering concept, for people new to animation.
 
Upvote 1
Looks amazing! I'm curious to learn more about your this technique. Would you mind breaking down how does the use of AI enhance the modularity? Which AI tools and prompts applied ? Can you tell more about that, if you don't mind me asking?
 
It gets pretty complicated to answer in one post, but I'll try to give some quick answers.

How does the AI enhance modularity? That's not exactly what's going on. Basically, the modularity aspect was achieved at the very beginning, and it just has to do with breaking down all the physical aspects of cinematography so as to make a lot of work reusable or interchangeable. A 35mm flat horizon shot of a tree at noon will composite well with a 35mm flat horizon shot of a forest at noon, and so on. Understanding the key rules of this inherent modularity is great, but we had to build a system that took advantage of the possibility.

First came control. UE5 is used to create the original images. This is because for that modularity to exist, lego 68 and lego 783 must have EXACTLY the same aspects in several metrics. Then finished UE5 takes are fed through the pipeline into the AI stages for recomposition. So our AI images can't be "random" like the grand majority of what you see out there is, more like style transfer that creates detail and composition on the fly.

I'm using a long chain of AIs with scripts and automation, but the core AI platform we're using right now is Stable Diffusion, albeit a HEAVILY modified version with many extra models and controls connected. Just an off the shelf will make you plenty of good pictures, but keeping animation coherent, mass processing roto masks, automating ai batch upscaling, and so on requires a good bit of effort to learn.

Basically, when I talk about the AI multiplying the inherent advantages of the modularity, I mean that it provides us with a near infinite set of content options for a given camera and scene scenario we create. So, having a bottle of coke that never looses it's fizz would be great, but if you had a bottle of coke that never lost it's fizz and had the same amount in it no matter how much you drink, that's on a whole different level.

So now that it's set up for modular, and we can create content based on rulesets that make everything uniformly modular out of the gate, it's a powerful combination.

TLDR Movie element lego factory. Tell the robot to only make parts that fit, then suddenly all the parts fit.

Here's a more sophisticated shot, showing the beta stage limitations. The leaves were a test to see if I could get away with using certain layers without running them through the entire pipe, but people noticed, so I can't use that.

 
Last edited:
Wow cool! Nate, thank you very much for such a detailed explanation. That's very kind of you. I suspected that the AI was not so easy to apply here, because my attempts to do something in Dall-E or Midjourney so far have been crappy to say the least. :bang: I realized that I need to delve into prompt engineering. But I did not expect prompts to be as complex as scripts. ...
And in the example with leaves - I think the main difficulty is that the car crosses the stream. That is, in reality, part of the leaves should stick to it and be carried away with it. The layers should then seem to intersect... Perhaps, several additional layers-frames with leaves stuck on the machine are needed...It's a real challenge
 
Basically I just tossed an uncalibrated layer on top of the reel. It will all work fine if I just ran the leaves through the pipe like everything else. I tried this particular experiment because of course there are layers that are natively modular like that, such as some rain and snow clips, fog, etc.
 
excellent work. i want to learn ue but it means to buy a much stronger computer. at this point i have to stay with my mechanical knowledge. here is what mechanics can do.
rumble.com

soul, the guardian of dreams

a yuppie finds a magic sword which makes the dream come true he didn't know he had. he first desperately tries to get back to reality but eventually he accepts his dream and follows it.
rumble.com rumble.com
 
You must log in or register to reply here.
Back
Top