"Training Day", at least half of the equation.
At the start of the movie, Denzel's character is definitely a good guy. We like him. By the end of the movie, we've learned how dispicable he really is. The great part about it is that over the course of the movie, we see behind his actions - his motivations, and how he actually has twisted himself into thinking he's one of the good guys.
I think this example is right along line with what wheat recommended. It's not that the character changes sides, but that the audience learns more about them. This one was always good. That one was always bad. We (the audience) just didn't know enough about them in the beginning, and as we learn more, our opinions change. I think that's probably a pretty good model to follow.