news Deepfaking Lips - No more Bad Dubs!

[sfoster]

It's been a running gag my whole life... how poorly dubbed asian movies lips don't match up to the english.
Even some comics like i think jim carrey have done jokes based on this technical issue.. which is about to be a thing of the past!

Scroll down in this article - there is a forest gump video where he is speaking in spanish!

Deepfake Lips Are Coming to Dubbed Films

For those who find reading a foreign film’s subtitles too distracting, movies are often dubbed into different languages. But that can be equally

gizmodo.com

Pretty awesome.
I actually had this realization after speaking briefly with Mara that it would be possible.. did a google search, and sure enough, it's being worked on for a couple years now. This article is like two years old. I don't think I ever heard of it though.

With the newer technology I bet they could replicate the original actors voice for the dub and really make it authentic.

 

[Nate North]

It's going to be everything. Right now people are spinning in their tracks thinking about all these individual capabilities, but where this is headed is kind of beyond what people are imagining. We're less than a decade away from just being able to take a MP4 file of star wars, type in "make this tron plus krull" and then just watch a version of photoreal Star wars with the two other films mixed in.

You'll be able to take any movie, and just say, "Modernize" and it will be the same movie with current actors, some pulled from your net history the way artists who's videos you watch get suggested to you on youtube. Back to the Future starring Tom Holland and Zendaya.

They'll charge advertisers the rights to place products in the blur of foveated rendering, and there will be ads that fabricate themselves on the fly based on search history. Do you buy lawncare equipment and listen to the music of David Hasselhoff? Suddenly an ad comes on with a young David Hasselhoff pushing a shiny new lawnmower up to your front door, because the AI knows your GPS coordinates, and can refabricate your neighborhood in a split second at photorealism.

They will build it into glasses and let you modify what you look like in the mirror, and what others look like on the street. One of the projects I talked about building years back is an AR thing, called Skin, which just lets you have stylistics you can apply to all flat surfaces in the world. Like attach a color scheme or art style to all physical objects, a simple example is you could just ask it to make all buildings except the one you looking for black and white. So you're trying to find a restaurant to meet your friend, and just speak a command and that one is the only one in color until you switch skins. Something like an instagram filter, but for sidewalks and buildings and the like. You could drive through town and every building had a 40 ft Matisse painted on it.

There's going to be a day at some point, where people use this technology as a weapon of war, far beyond the primitive examples we've already seen, and I cannot even conjecture as to how #$%^$ a situation that is going to be. It's technically possible right now for people like Putin to create videos where opposing politicians commit crimes, admit treason, or whatever they need to blackmail and control, or turn the public against person x. Right now an expert, or another AI, could tell if a video was fake, but even that will go away.

 

[sfoster]

I think is a serious misnomer to represent something like this as "spinning tracks" thinking about individual capabilities.
I'm talking about foreign films having major releases across all countries, in theatres, as native films.

This is ENORMOUS like when big chinese films come out in china, they could also come out in america, with english, and the original voice actors voice. I'm talking about billions and billions of dollars.

One click of a button, your film is in chinese ready to sell to china.
Fucking awesome - and totally possible this year or next if someone had the money and motivation to build such software.

Google is working on that thousand language AI Model as we speak.

 

[Nate North]

Oh, I'm not diminishing it, it's a huge leap forward for cinema. For that matter it's a huge leap forward for youtube. Ultimately though, I doubt it's more than a decade before everything in the film can be generated from an animatic. That's what I've been working on, and since control net came out, 500 new guys are all doing the same thing all the sudden. No question that one of us will create what will become synthfilm. That's a big win for indies when it happens. I don't know what the studios will do. It will more than decimate their financial advantage in terms of production quality.

That still leaves the advertising though, and that's a problem. Right now smaller shows still have a chance because not everyone could always find something they liked on tv. What's already started to happen though is that the pool is exponentially more diluted than before, meaning people are starting to have more options than they can handle. They will seek simplicity, sign up to a streaming service, and watch what disneycorp tells them to watch. Once AI does democratize film, and it will, sooner rather than later, we'll have solved one problem and created another.

For example, if you are a fan of standard fare anime, this has kind of already happened. There are so many nearly identical shows following the same template, that even someone very enthusiastic about the genre (which I am not) probably has a hard time deciding which of the 10,000 spiky black haired protagonists they want to watch swing a sword at a demon next to a computer.

As far as my saying that people were getting blown away by smaller things, that wasn't really directed at you personally. As you know I study this exact area every day of the week, and this last week, probably several hundred videos with titles like "NEW MIND BLOWING AI DOES WHAT????" and then goes on to describe some new mod that builds of some adjunct of existing software.

My point was that we're all missing the forest for the trees. Within a decade or two at the most, the entire system of creative value will collapse completely. It's basically started happening over the last few years, and when the full scale AI arrives, that will be like throwing gasoline on a fire.

An interesting question is, do we actually need stars, or to be stars? Entertainment won't go away, it will get drastically better. What will likely go away is the idea that there is only one city in the world that can make a good movie. It's probably for the best. I just hope that some of the other AI scenarios don't have such terrible effects that it nullifies the good it does. Ultimately, this is a way for the rich to buy intellect, one 22,000 nvidia card at a time. This could quickly accelerate the wealth gap to a breaking point (again, kind of already happening) and result in the collapse of multiple empires that currently seem etched in stone.

That Silicon Valley bank collapse was scary to watch. The news report started out like this "on wednsday evening, it was just another day at the office, but by monday, the entire empire had collapsed" You just don't think of things that big, that were that stable for that long collapsing in a matter of days, but there it was, a tiny scale model of what could happen to America. I bought a 2 liter of pinapple soda at the store, and it was 3.79. That's up from 99c 5 years ago.

 

[mlesemann]

That Silicon Valley bank collapse was scary to watch. The news report started out like this "on wednsday evening, it was just another day at the office, but by monday, the entire empire had collapsed" You just don't think of things that big, that were that stable for that long collapsing in a matter of days, but there it was, a tiny scale model of what could happen to America.

Having worked in the financial industry during the 2008 crisis - meaning that I watched the collapse of Lehman Brothers et al in real time, and watched the European bank where I worked suddenly be unable to fund itself - I wasn't at all surprised.

 

[Nate North]

What was particularly disturbing about this crash at SV bank was the role that panic spreading through social media caused it to a significant degree. From what I understand, they were not in great shape, but it was a recoverable situation, and facebook, twitter, etc, spread panic so fast that it degenerated into an unrecoverable position virtually overnight. The news called it an incident of "shouting fire in a crowded theater" that took down this gargantuan organization.

Basically, without going on about the algorithm, "if it bleeds it leads" meaning that a negative news story will reach many multiples of the number a positive story, or even a positive side of a situation. I haven't counted how that particular set of posts spread, but I'm guessing that 10 people tweeted "lets help save SV bank", and another 10 tweeted "run for your lives, pull out all your money and make it crash" and the dangerous sounding, alarming headline got pushed by the algorithm 1000x as much. Obviously people click on disaster warnings faster than they do on fundraising drives. I'd love to see the actual numbers. How many ads were sold by pushing the fearmongering angle. I bet they tore down that huge bank to make 2700 dollars from .002 cent banner ads. Not intentionally, but just because they are set up to monetize everything automatically regardless of who's hurt or what's destroyed in the process.

Anyway, didn't mean to hijack the thread. Perfecting deepfake lip-syncing will be a really positive thing, that greatly improves the reach of filmmakers, and the quality of choices available to audiences worldwide. Especially since there are countries where they speak languages that nobody currently translates into.

I wanted to do translations into maybe the top 5 languages, but by combining this tech with advancing translation software, It should become possible to automate refitting a film for full global distro.