Originally Posted by
petershene
If you are on apache server, or any other server for that matter. You should have a log avaible to you in your pontrol panel that lists all of your website visits. From there you can look at all the ips and user agent names and block them via you htacess file.
I must also mention that a bot can call itself whatever it wants ( even googlebot ) so blocking a bot via user agent string issometimes a waste.
Ip is a little more effective but most have dynamic ips so it will change you can try blocking a ip range though.
The best is to identify write a script of a bot that does not request or adhere to your robot.txt file , is it passes that parameter block it since google bot and all "good bots" will request your rbt.txt file.
You can also look at something called a honeypot trap in php for your website.