• Wondering which camera, gear, computer, or software to buy? Ask in our Gear Guide.

Roto-effing-scoping, here's my number, save me maybe?

All jokes aside, I'm in a world of pain. 2 minutes of rotoscoping outside of the green screen area. Well, 2 minutes left, I've done 20 seconds already.

Here's what the job looks like. The last photo is what I need to get. Luckily, resolution in the final comp is greatly downscaled so I don't need to be THAT precise. It looks like this because we needed the green screen on the left more than on the right. http://tinypic.com/view.php?pic=2q33z85&s=8#.VabhFPkpV8E

So, I tried tracking Luma Matte, bad results. I tried tracking the mask shape, horrible results. I tried Mocha, too difficult because the shape is too complex and I need to switch tools and fine tweak the hell out of it with every greater movement.

(Please don't read this: 2*60seconds*24fps=2880 frames. On average, I need one keyframe every 7 frames, resulting in roughly 411 keyframes left to go. It'll take me roughly 40 seconds to 1 minute per keyframe, so let's say 50sec on average. 411*50=almost 6 hours of work. This degenerates into 2, maybe 3 days of work considering how many breaks I'm taking because of miserably running away from it, procrastinating, or staying on forums, or learning other things. Oh, and there's also the green patch above his feet that I need to get back to when all that is done.)

I haven't tried Alpha Matte Masking by using the R,G and B channels yet but I find it difficult to tweak, while not knowing if I will get the results.

Is there ANY method I'm missing that would give me the desired result?
 
Last edited:
All jokes aside, I'm in a world of pain. 2 minutes of rotoscoping outside of the green screen area. Well, 2 minutes left, I've done 20 seconds already.

Here's what the job looks like. The last photo is what I need to get. Luckily, resolution in the final comp is greatly downscaled so I don't need to be THAT precise. It looks like this because we needed the green screen on the left more than on the right. http://tinypic.com/view.php?pic=2q33z85&s=8#.VabhFPkpV8E

So, I tried tracking Luma Matte, bad results. I tried tracking the mask shape, horrible results. I tried Mocha, too difficult because the shape is too complex and I need to switch tools and fine tweak the hell out of it with every greater movement.

(Please don't read this: 2*60seconds*24fps=2880 frames. On average, I need one keyframe every 7 frames, resulting in roughly 411 keyframes left to go. It'll take me roughly 40 seconds to 1 minute per keyframe, so let's say 50sec on average. 411*50=almost 6 hours of work. This degenerates into 2, maybe 3 days of work considering how many breaks I'm taking because of miserably running away from it, procrastinating, or staying on forums, or learning other things. Oh, and there's also the green patch above his feet that I need to get back to when all that is done.)

I haven't tried Alpha Matte Masking by using the R,G and B channels yet but I find it difficult to tweak, while not knowing if I will get the results.

Is there ANY method I'm missing that would give me the desired result?

I can't see your pictures anymore, it keeps taking me to some ad.

How did you create your luma matte. If you're able to create a proper alpha or luma (which might be difficult), tracking shouldn't be a problem. You're still going to be tracking over the original footage right? you're only going to use the matte to reveal the areas you want. Or am I missing something?
 
How did you create your luma matte.

I squeezed the levels together in what seemed to be the sweet spot so that I get the whitest highlights and the blackest shadows I could. I assume for the alpha it should be the same wouldn't it?
EDIT: The problem is that the leaves in the background have quite some leeway between shadows and highlights.

On my machine, the link still works but let's try with some other uploads:
here: http://postimg.org/image/aeni50yzh/
or here: http://imageshack.com/a/img905/9283/GpRodE.jpg
or here: http://tinypic.com/view.php?pic=2q33z85&s=8#.Vab4nvkpV8E

I did continue with the traditional roto in the meantime and I advanced another 20 seconds, I was lucky the actor was very still for about 10 seconds heheee !
 
Last edited:
I squeezed the levels together in what seemed to be the sweet spot so that I get the whitest highlights and the blackest shadows I could. I assume for the alpha it should be the same wouldn't it?
EDIT: The problem is that the leaves in the background have quite some leeway between shadows and highlights.

On my machine, the link still works but let's try with some other uploads:
here: http://postimg.org/image/aeni50yzh/
or here: http://imageshack.com/a/img905/9283/GpRodE.jpg
or here: http://tinypic.com/view.php?pic=2q33z85&s=8#.Vab4nvkpV8E

I did continue with the traditional roto in the meantime and I advanced another 20 seconds, I was lucky the actor was very still for about 10 seconds heheee !

I just had a better look at the picture. His background is actually black, so creating a pretty good luma or an alpha key shouldn't be very difficult. I'm sure you're using hue/saturation to take down the saturation and using both the curves adjustment and levels to get the key. So if you have a good key, then to hell with rotoscoping. You might have to do a little bit around the chin and neck/beard area, but even that you may get away with.

Key it man. good luck.

Edit: You've got a little greenscreen showing above his knee through the wooden railings.
 
Last edited:
As an interesting aside, I would never use "Rotoscoping" to describe what you are doing here (aka Keying/Masking)... has the term come to describe this more than it's original meaning? Just curious.
 
I'm sure you're using hue/saturation to take down the saturation and using both the curves adjustment and levels to get the key.
I only tinted it to black and white beforehand, haven't done saturation and curves alongside though. I'll try it. It's the highlights in the leaves causing me issues.

Have a look at what I'm getting. I'll show how much of the rotoscoping has been done so far too:
1: http://postimg.org/image/6m8v0kh3j/
2: https://imageshack.us/i/ip6OSYmMj
3: http://tinypic.com/view.php?pic=xm2v7n&s=8#.VacGVvkpV8F
(by the way, which of the links worked?)

I don't think it will work, these are the best results I (personally) can get. His eyes, the shadows on his shirt are in the darks (If I adjust it the other way around, I'm getting back the highlights in the leaves so I can't). And also the legs (which I could roto) but especially his tie is another tough one.


have you tried Rotobrush?
Yeah, but I don't like it because it adds too many vertexes and will make the project even harder to work with.
 
have you tried Rotobrush?

Yeah. Why aren't you using Rotobrush?
Cause like, you should never use a mask outline if the shape you're trying to isolate is as complex as an entire human body. Too many points to think about. And if you mess up on some key frames, then when you go back to fix it, you have to move every single point that's wrong for every key frame that you made. Whereas Rotobrush will always update itself once you go back and make a change. It's still irritating as hell, but at least its biggest perk is that it's an intelligent effect.

I commend you for trying a Luma Matte by adjusting the levels because that can often work. But I'm also surprised you haven't tried a combination of different matte strategies to get the cleanest result. You aren't limited to only using one in each case.

I just recently worked through 75 different shots from my 10 minute short film. Nearly every single shot is an effect shot, and there were green screen walls throughout almost the entire project.

I got about 1-3 shots cleaned up and keyed every day the past month and a half, but even though I filmed on a nearly perfect green screen sound stage, I still ended up with green shadows and hotspots that were different than the solid green walls, which introduced problems when I only used Keylight. I also had a few reflective areas on my cast with their buttons, their hats, and their boots, which resulted in empty spaces all over their body that I had to cover up later with white solids on the Matte reference layer.

The single best thing you can do, first of all, is to start with your original footage unaltered, duplicate it, precompose the duplicate, and create an entire composition just focused on making your actor white, and everything behind him black. Then once everything is perfect, you can place a "Set Matte" effect on your original layer to reference this black and white layer, and your colors and your actors won't be affected other than what will be removed by the black areas of the matte.

Doing this also allows you to get very precise with removing your unwanted elements. You can even combine things by working on some portions of your actor with a Keylight "Combined matte" effect, placing a luma matte on top of this from a separate precomposition, and then you can animate black and white solids with masks on them (not your video layer) to match the rest of your actor for whatever is still causing your trouble.

I made a few tutorial videos all about my own process on these different approaches.

https://www.youtube.com/watch?v=kTMxS4JEhWA

But the biggest thing to do first, if you've had trouble with other approaches, is to just grind through Rotobrush. Indeed I can imagine rotoscoping an entire 2 minutes would be quite the task, as I've never had to rotoscope that long of a sequence. I'm actually curious why you've ended up having to rotoscope such a long portion, and why you didn't end up placing the green screen behind your actor entirely. Is there a specific reason that prevented you from doing that?

Nonetheless, Rotobrush will become a tedious frame-by-frame process, but it may end up being the best shot you have to get a clean break with this. My main suggestion for it, though, would be to split your 2:20 minute project into 20 second intervals or something like that, and work with the Rotobrush on each portion individually. This will allow your Rotobrush cache to not grow too large for one single clip, and if something goes wrong, then you won't lose everything. You can also make sure to freeze the frames of your work every 20 seconds that you complete.
 
I only tinted it to black and white beforehand, haven't done saturation and curves alongside though. I'll try it. It's the highlights in the leaves causing me issues.

Have a look at what I'm getting. I'll show how much of the rotoscoping has been done so far too:
1: http://postimg.org/image/6m8v0kh3j/
2: https://imageshack.us/i/ip6OSYmMj
3: http://tinypic.com/view.php?pic=xm2v7n&s=8#.VacGVvkpV8F
(by the way, which of the links worked?)

I don't think it will work, these are the best results I (personally) can get. His eyes, the shadows on his shirt are in the darks (If I adjust it the other way around, I'm getting back the highlights in the leaves so I can't). And also the legs (which I could roto) but especially his tie is another tough one.



Yeah, but I don't like it because it adds too many vertexes and will make the project even harder to work with.

What you have at these picture links is a perfect start. Don't give up on it if it IS actually removing a good portion, and keeping a good portion of what you want. Because once you get the majority of your footage worked out, you can always go back and tweak more with white solids, black solids, and other mattes that you blend with this one to get the cleanest removal possible. Any portion you can get the way you want is a portion you don't have to work on later.

But again, this only works if you don't do your matte work on the original layer, and you use precompositions and a set matte effect.

Everything like this will take a crap load of time. It doesn't matter how hard we try to avoid it. What matters is what course of action will you take to limit the amount of work you have to do in the end.
 
Last edited:
FilmmakerJ, a great contribution, thanks a lot ! I was actually avoiding combining what I got so far with all sorts of mattes, solids, and what have you because I'm still a beginner in AE. It's my first project in fact so it intimidates me to make these complex combinations. I am also very much afraid of my laptop not living up to the task of a much more complex project (the only machine I actually have for this project, isn't money a bitch?). I'm doing things myself because it is my own project (writer,director, amongst the many other hats worn) and I want certain things a certain way, plus I love learning new things like AE.

I will try though (with the project backed up in case my laptop won't handle it). Ah, the answer to why there isn't green screen there was mentioned in the first post, second row.
 
Last edited:
That looks pretty workable to me. Where you have blacks around the eyes you can actually put a white solid on top and just mask it around the eyes to make it completely white.

Also try to put the curves in between the tint and the levels to see if you get any difference.
 
FilmmakerJ, a great contribution, thanks a lot ! I was actually avoiding combining what I got so far with all sorts of mattes, solids, and what have you because I'm still a beginner in AE. It's my first project in fact so it intimidates me to make these complex combinations. I am also very much afraid of my laptop not living up to the task of a much more complex project (the only machine I actually have for this project, isn't money a bitch?). I'm doing things myself because it is my own project (writer,director, amongst the many other hats worn) and I want certain things a certain way, plus I love learning new things like AE.

I will try though (with the project backed up in case my laptop won't handle it). Ah, the answer to why there isn't green screen there was mentioned in the first post, second row.

Glad to be of help. Working on my current project was the first time I ever wanted to be as perfect as possible with my chroma keying process. So I tried every trick that I could think of to make each shot as easy as possible and to get a good result. But ironically, the very last few shots that I worked on were the ones that took multiple days to complete because of all the Rotobrush work I ended up doing. lol

I can completely understand the concern about your computer's work load and the bottle-neck issue. In which case I would also agree with you that Rotobrush might not be the wisest choice at the moment. So just trying to tweak around with Keylight on Combined Matte, a Luma Matte inside of another precomposed layer, and then black and white solids should clean this up quite nicely and leave you with a relatively small amount of key frames and timeline information.

I've also toyed around once with a Keylight on top of another keylight by taking two copies of a video, precomposing the second inside the first, and putting keylight on the outer composition and one on the inner composition.

The reason for this is that if you put keylight on a layer, you can't also mask out that layer, because it will create a black box boarder around it when Keylight is activated. So you have to mask that layer outside of a precomposition, and then it will work.

But once you do this, it will allow you to change the threshold of your clip white and clip black two separate ways, then you can mix them together to get just the look you want. This is especially helpful if you need to bump your clip white all the way up to get a clean matte on your actor's bodies, but then tone it down on a second version if you also need to get a softer matte for their fluffy or stringy hair.
 
Guys, after all, the only question I'm left wondering is: Isn't path masking the tie and the feet on top of having done mattes just as much work as if I were to "simply" mask path everything from beginning to end?

Speaking about irony with the last shots needing the most work, that is the irony in my case too.

I have enough info from you people to make a career out of AE now I guess.
Oh, I'm very happy for all you guys contributing, I'll just type in some phrases and "tags" now for people who will need this insightful thread (if you are one of them, know the entire thread from the beginning is useful): masking outside of the green screen area, rotoscoping outside of the green screen area, mask path, green screen and mattes, green screen and masks, green screen and mask paths.

Oh and don't forget to check my film when editing will be finished. It will be released with the title It Goes Both Ways, I will have BTS too and one of the actors (only voice) is Morgan Freeman's French dubbing artist, Benoit Allemane.
 
Last edited:
Guys, after all, the only question I'm left wondering is: Isn't path masking the tie and the feet on top of having done mattes just as much work as if I were to "simply" mask path everything from beginning to end?

Speaking about irony with the last shots needing the most work, that is the irony in my case too.

No. Mask the tie with a white solid on top. It doesn't have to be perfect, so there's no time wasting making things perfect. Also while you're path masking the tie, keep the solid semi transparent so you can easily see the tie as it moves around. What you want is a completely white area. So after the path masking is done around the tie and the eyes, you're good. Just turn off the transparency after you're done. you don't have to concern yourself with rotoing/masking the head as it moves, perfectly.

good luck. i'm going to bid you adieu for now. I'm pretty sure filmmakerj will solve your issues. He looks good for it :)
 
Last edited:
Guys, after all, the only question I'm left wondering is: Isn't path masking the tie and the feet on top of having done mattes just as much work as if I were to "simply" mask path everything from beginning to end?

Speaking about irony with the last shots needing the most work, that is the irony in my case too.

Not at all.
Like I said, every portion of your video that you can turn white and turn black is a portion you don't have to do anything with. Anything still left black you obviously do have to keyframe for the entire clip, but you don't have to use nearly as many masking points, and because your actors eyes are set inside a larger white area, you can probably get away with keyframing and not having to go frame by frame for larger sections. Though, I suppose I'd have to see the entire 2:20 clip your working with to determine how severe your situation actually is.

If you're trying to work out the keyframes and how often to make them, from my experience its best to make them at perfectly regular intervals rather than placing a keyframe on extreme movements, because there may be too many to try and find.

If you look at your timeline, as you zoom in and out to go from seconds to frames, look for the points marked by a number, and place a keyframe for each of your white solid masks every 20 frames or so. Then once you have a keyframe for every 20 frames, zoom in further and get one for every 10 frames. Then every 5. And then go frame by frame where-ever you need to.

I haven't needed to try this yet. But you might even be able to motion track your actors eye and other dark spots on his body, to then motion track a mask to that area. Then you wouldn't even have to mess with the points except to fix where the mask goes out of bounds. Just an extra possible answer.
 
Also while you're path masking the tie, keep the solid semi transparent so you can easily mask the tie as it moves around. What you want is a completely white area.

I usually just turn the solid off entirely, and even turn the keylight or levels adjustment off so I can see exactly where certain areas are that I may still be missing.
 
roto is tedious, but it's not really difficult -- that is, manually masking and animating those masks.. it's been about 5hrs since this post was opened, if the OP had been properly rotoing (with masks) since then, this job would've been done by now. :)
 
roto is tedious, but it's not really difficult -- that is, manually masking and animating those masks.. it's been about 5hrs since this post was opened, if the OP had been properly rotoing (with masks) since then, this job would've been done by now. :)

I disagree with that estimate. Rotoscoping just 30 seconds could end up taking days depending on how many things need to be rotobrushed in the frame. It depends on how many different effects one needs to achieve a clean matte.

In ~Claude's case, it doesn't quite sound like even this shot of this actor sitting at a desk could get done in one day, depending on how much the actor actually moves, and whether or not he gets up and sits back down somewhere in the timeline.
 
Do you have a frame without the actor?
If so: try difference matte.
It will more or less succesfully compare the empty frame with the frame with actor.
There is a possibility you still need to fill in some blanks with an extra layer with a mask, but it should get you started. :)
(I can't tell whether it will work because of the red overlay.)
 
Back
Top