Since audio post is my bailiwick, I find mlesemann's comment most distressing; just hand it over to me, my dear, and I'll get it done.
For me the "painful" part is my prep work; doing my cue sheets (DX, Foley, SFX) and setting up my bulletin board. I'm impatient to get down to the fun parts. Besidesindietalk's delegation comment, I would say that may newbies find preproduction (the in depth prep work) to be the most painful part.
In my experience, for newbies especially, I’d say post is the the most painful part specifically because they blow past prepro.
But yes, prepro can be the most “painful”, which is a bit of a misnomer anyway. None of filmmaking is actually “painful”, but some parts take a little more elbow grease than others. And the honest truth is that prepro done right actually makes the rest of the operation relatively smooth.
Food for thought: I would say that at the end of the day, editing is the most painful for new filmmakers. Why? When you sit down to edit, you can see and hear everything that went wrong on the shoot....poorly recorded audio, poor camera white balance, out-of-focus shots, no cutaways, continuity issues, things in the frame that are not supposed to be there (like a light stand), missing shots, etc. Moral of the story: try to surround yourself with talented people. They can prevent a lot of problems from happening.
For me, it's definitely finding locations and actors. I enjoy every other part of the process, but on all my movies (shorts and features) casting and getting locations secured is always the most frustrating part. I'm in the middle of my fourth feature right now. We shot the majority of the film back in June/July, but still have three days of shooting left due to casting and locations that I'm trying to get locked down. Aside from this, everything else is gravy to me.