I have a feeling this is a networking issue rather than a software configuration issue. My money would be on a "keep-alive" issue. It would be rather hard to diagnos without knowing your complete external network structure.
Is there a VPN in use? I had a very similiar problem with two remote offices that VPN together and one uses the exchange server of the other. The VPN connection itself would time out and then disconnect the exchange connection.
Try this for sh1ts and giggles. Setup a WAN machine and have it ping nonstop, the TS server and see if the session drops while you're idle. I'll be it doesn't.