In AE:
- select layer you want to make mask.
- press [G] - the pen tool for masks is selected
- everytime you click the mouse you add a point to the mask. If you move the mouse whill doing this the point will have a curve instead of a straight corner.
- the mask is compleet when you click the first point again: everything within the mask is now visible, the rest is not.
- in layer properties a new 'tab' has popped up: Mask
- open it
- you can now set keyframes 'on' in shape. (click the stopwatch-icon)
- now you can move the points, and change the curves in every frame
- once finished you can set the mask feather
Other things to know:
- you can change the color of the mask-line by clicking the yellow box. Can be usefull to make the maskline and points stand out more
- you can use severa masks together to achieve what you want
- you can set mask to :add, subtract,nothing and intersect.
Tips:
- dublicate the layer before you start: this way can always see the complete picture when masking the toplayer
- add a solid to the composition with a very saturated color that stands out against the scene: place it beneath the original layer. When disabling the original layer, you now get to see the masked part against a contrast background.
- when masking movement I useally start with keyframing the mask every 4 or 8 frames (depending on movements and my educated guess). You will see the mask will gradually change in the in between frames. Now I check the frame halfway between the keyframes and change it to fit.
After that I move to the other frames and change the mask when needed.
- to make a mask ineffective while the shot still lasts you can do 2 things:
A) keyframe the mask opactity: from 100% to 0%
B) move it with the shape keyframe out of the frame
Have fun!
This may take you several days to do.
The rotobrush needs clear lines and contrast to make the 'guess', but you also need to point the program where the lines are.