• Popcorn Points determine how popular a video is. You can click the popcorn bucket or simply react (Like, Love, etc.) and it will register a vote.

AI Layer Test 3


Here are some random clips from the third stage of AI post layer testing. It's a long way from working correctly, but you can see a lot of improvement from the first version, with this one being far more stable. This final one will of course need to be nearly 100% stable.

These tests are run at 1/8 of final resolution, for the sake of speed, since it's an iterative process requiring many experiments. It's a very complex problem, with many possible solutions (we're testing various combinations of code from hundreds of competing research papers). Right now I'm thinking that using optical flow to "remember" stylization solves from frame to frame might be the key.

Some people ask why I'm doing this layer at all. It's extremely important. Basically, this functions as a consolidating layer that will ultimately auto correct all compositing errors and asset mismatches across all frames project wide. I can buy a stock footage clip, drop it into the background of a UE5 project, and use this to combine them seamlessly with one click. That means that this layer, once complete, will function as a universal adapter, seamlessly joining the output of not only different programs, but different media sources. In example, since this is a 100% refabrication of every pixel, I can use any source photo without copywrite infringement. So I can take any frame from google maps street view, and use that as a background for an animated scene. Same with any footage. If I need a car to break down on a road in front of Everest, I can build the car and road in UE5, grab any photo of Everest, and use this layer to combine them seamlessly, reconstituted into a completely original work.

In one of the first shots, a crowd of people can be seen walking along a spaceport causeway. Those people come from archvis packs, a different, poorly matched style that doesn't quite fit the look of the spaceport, which is from a different creator using a different style. In the version above with the primitive alpha layer, you can see that everything on the screen now looks like it was drawn by the same artist. It's automated tonal matching. This won't shave hours off of production, it will shave years.

Here is the second test, which I never bothered to post. They are all still terrible, but you can start to see where I'm going with all this.


For reference, or anyone who never saw it, this was the first test, a few months back. You can see that we've improved the coherency a lot since this one.

 
Upvote 1
Just curious. Are you waiting for the technology to fully develop or is this something separate from your main project? (sorry, I can never remember the name,,, the one with the cat)
 
The cat project "The Labyrinth" is actually the side project, and this is the main project (or rather a component of it). Save Point is the main story, with human characters and a broad scope. The Labyrinth, is basically a "training wheels" project to gain experience working in a new way, and field test design aspects and production pipelines for use in the master project.

I had been experimenting with what we internally call "the McJarkanizer filter" in one way or another since about 2011. The first idea for hybrid AI choice based film came about 2005, and was initially called "Cartoon Shaman". We tried again in 2011, and made a brief story that was.... not great. At that point I was just using stuff like roto and outline filters, subtractive masking, etc.

Save Point actually launched about the time I realized that these technologies would be ready in a few years, or at least good enough that I could Frankenstein a solution for the master style layer. I knew that there was an enormous amount of setup work to be done before I'd even be ready to use that layer effectively, so that's what the last few years have been, building up the infrastructure and knowledge I would need to be able to communicate to the AI layer effectively.

So basically, I'm not aiming where my target is, I'm aiming at where it's going to be, understanding that it will take years for the bullet to reach the target. About the time I get this layer solved, I should be prepared to supply the input with enough content to validate the layer's purpose.

The idea is to crash budgetary restrictions on marketable animation, and allow high volume milling that facilitates a new type of story that simply wasn't practical at established rates of speed and cost. If you've ever started mapping out what an exponentially expanding plot looks like, it's something that just isn't possible when one has to spend weeks of time making 3 minutes of film. So in answer to your question, the knowledge that this AI layer technology was coming farther along in the timeline was the genesis of the current version project, rather than just an aspect. I knew that everyone would try to jump on board the minute it was possible, and anyone who had been preparing for it for years would have a huge advantage.
 
Pretty cool!

Except for 56-60 when it puts them in blackface for some reason.
wtf ai!
Yeah, I saw that. What's happening there is that it's trying to guess race by skin tone based on pixel luminance, and in darker scenes, it's not able to differentiate between a 72 luma value from darker skin, and a 72 from lighter skin in a darker area.

The issue is that it has no memory, so It does not realize that it painted the same character differently in a previous frame. It's something I have to get fixed at some point in development, but will probably be sort of easy since even a single second of memory would get rid of this issue.

"I have assigned shape x attribute x for 15 out of the last 24 frames, should I suddenly reassign it? Goto line 4000, run luminance shift window check, if check positive then return to line and set value negative"

Also the fog in this scene probably gave the AI a lot of trouble. We know what fog is, but to the AI, it's just noise on top of the frame.
 
Last edited:
and here's the next test. Doubled the resolution (this one took 8 hours) and tried to dial it back a bit to get a more cohesive look.

Near the end there is a section where for a moment, the whole thing is almost working perfectly, but it's still got a long way to go.

it's still processing as of this post, but should be viewable about 15 minutes from when I post this. I timestamped it to the part where it actually works for a minute.

 
Last edited:
Top