The blinking you're talking about is due to the mask shifting drastically from frame to frame. You'll need to spend more time on the rotoscope matte. It's an annoying, time consuming thing.
Then, you'll need to feather the mask a little bit to let a little of the background through so the edge doesn't look so hard.
Yes, it has to be on a separate layer, you may end up with 5-10 layers just for this one effect due to all of the little adjustments you may end up doing to it. Often, a complex mask like this will be 5-10 separate masks (hand, forearm, upper arm, torso, head core, hair) just for the actual composite. This allow you to have really fine control over the individual parts of the matte and avoid the points along the edge jumping around too much.
Rotoscoping sucks, but if you take alot of time with it, can get you really cool results. All of the BTS show "we cut them out of the frame and moved them"... what they mean to say is, "we paid an intern minimum wage to site for weeks moving hundreds of little points around the screen to cut them out, so our compositor would be able to take an afternoon to move the actor around."