What your gonna need to do on the the computers is set up a static IP on all 3 computers.

start-->control panel-->network connections.

For windows 98 and ME scroll down till you see something that says

TCP/IP (your ethernet adapter name here)

FOr windows XP, just select your Local Area Connection and hit propeties and scroll down on that first page till you see TCP/IP Then hit properties.

highlight it and select properties.

From there, select Use the following IP address

For one computer put in something like 192.168.1.10 and a Subnet mask of 255.255.255.0

Then for your second one put in 192.168.1.11 and so fort using the same subnet mask.

Are you also connected to the internet through that router?

Also if your using any firewalls, your gonna have to either add your IP range on your network to the trusted zone, or you'll have to completely disable the firewall.