Blocking Crawlers/Bots With .htaccess
-
Trying to get a site running phpBB to not get crawled as much.
We had to blow away a huge list of IPs recently, which has upped traffic. But I think the issue is that IPs that were once "bad" are now good, and doing this by IP is a moving target at best.
Has anyone ever done this using .htaccess, and if so, how are you doing it?
I saw a lot of article on Google about using a list of user-agents or know IPs, but figured I'd start here with the experts first.
