Budget aside, it's question of dedication. Way back when I was a struggling musician. I pumped gas in a service station from 7am to 2pm (this was long before there was such a thing as self-serve; that gives you an idea of how long ago it was). Three afternoons a week and Saturday mornings I played piano and was audio tech for a dance school. I was also organist for a small church on Sunday mornings. In the evenings - four or five a week - I either rehearsed or performed with the obligatory rock band in my quest to be a "rock star." I somehow managed at least an hour of practice every day as well. I got precious little sleep. I also restricted myself to a couple of beers a week - I was so tired one beer usually knocked me on my butt anyway - and had nothing in the way of a social life. Over the years the jobs and the bands changed, but not the long hours. It took me almost ten years, but I was eventually able to support myself and my family as a working musician
In my 40's I was a full time Mr. Mom while getting my audio post business off the ground. Trust me, it's a lot harder when you're older, plus having responsibilities like a wife, kids, a mortgage, etc.! Even now I work crazy hours marketing/looking for work and doing the other donkey work of running a small business when I'm not doing paying work for clients.
How badly do you want to be a filmmaker? Is it worth sacrificing almost everything in the pursuit of your goal? That's what it takes, and there are no guarantees. And guess what? It doesn't get much better once you achieved some measure of success; it's just a little more profitable.