i've had this problem before.

I had a problem between two routers like this once. One router was on 100Mb/full duplex, while the other one on 100Mb/half duplex mode. Which would explain some packetloss, cause one interface does look only at data over two wires while the full-duplex one sends and receives on 4 wires.

Hope this helps.

<edit>
So i'd check the interfaces when vpn is setup.
</edit>