Look at VNC, its halfway what you are looking for...and it's source is out there for all to see..