As G mentioned, you probably have some quiet bits before and after they speak, right? It's a bit of a pain, but you could grab a few of those, careful join them up (overlapping and crossfading if needed) to produce a usable roomtone, certainly better than none.
And even with music, you'd still...