Welcome to Resource Zone.

DMOZ/ODP crawlers/bots

Thread starter #1

Keef

Member
Joined
Dec 2, 2010
Location
Debatable Lands, Cumbria UK
Hi,

Is there a list of DMOZ/ODP site crawlers or bots?

I ask because I am keen to avoid accidentally blocking them in my .htaccess

Many thanks,
Keef


PS. I hope I'm asking this in the right place - apologies if I'm not.
 
Moderator #2

pvgool

DMOZ Meta
Curlie Meta
Joined
Oct 8, 2002
As far as I know thare is not such a list.
There are a few cralwers who check for websites that have gone down.
Blocking them would cause the website to be seen as not available anymore. But such websites will always be checked by a human to be sure they are gpne. I a human sees the site is still available the crwaler will be overwritten.
Best not to block anything coming dmoz.org or any of its subdomains xxx.dmoz.org
 
Thread starter #3

Keef

Member
Joined
Dec 2, 2010
Location
Debatable Lands, Cumbria UK
Thanks for getting back to me about that.

To be on the safe side, I've had a good trawl through my .htaccess file and there's nothing there to cause any problem.

Oddly enough, there was a bot I saw mentioned elsewhere a little while ago which had added "dmoz" into its user agent title (if that's the correct term). I gather it had nothing to do with dmoz, but had presumably included/hijacked the name to appear more legitimate than it actually was - it didn't contain anything explicitly referring to dmoz.org though.
I'd post a reference but I can't remember where I read it.