It seems like some others my sites are being bombarded with hits from a mob called AllResearch. Apparently one of the things they do is hit RSS feeds and suck down every page referenced, for some kind of indexing. Judging from the amount of traffic they’re burning up, they suck big-time, in fact. I mean, indexers usually put in a lot of hits on web sites, but these guys are hitting more than 10 times as much as the next one down the list, MSN.
These are the top hitters over at toxiccustard.com:
- 45541 sp1.allresearch.com
- 3448 msnbot.msn.com
- 3110 index.atomz.com
- 1328 crawl25-public.alexa.com
Time for a little .htaccess magic:
order allow,deny
deny from 38.144.36.
allow from all
BEING EVIL:
OK, I have a solution of sorts to referrer spam – I bounce it back to them. I have almost eliminated it, except for one from h**hprof*tclub who refuses to go away. To be really EVIL, I’ve redirected that spammers output alone to the signup page at http://www.webclipping.com/signup.html. Probably not entirely ethical but then we’re dealing with two unethical orgs – why not play them off against each other? (Yeah I know, two wrongs…)
It does mean more crap in my log file (the way I have it set up) but it’s fun. Now, my log file sees an attempt to access a page on my site, which is redirected. However, because of the way it is redirected the bot still thinks it is on MY site – so I also get to see the cgi request. For example:
207.248.240.119 - - [04/Mar/2005:06:19:37 -0600] "GET /cgi-bin/signup.cgi?c_name=web+poker&c_address=+%3Ch1%3EYou+can+also+check+out+the+sites+about%3CA ...So now webclippings are being auto-spammed by a spammer! We hate them both – now they can hate each other. And I *think* I am effectively hidden from this loop, thanks to how these redirects work. It just looks like h**hprofit is spamming them direct!
Ha ha ha! Death to the spammer…