Spam Whackers Blog

Exposing Various Types of Spam – Offering SEO & Webmaster Tips

August 31, 2008

Blocked User Agents

Filed under: SEO General — Connie @ 6:12 pm

I just added a page that list over 400 user agents that many web masters block for one reason or another. Every user agent listed is not necessarily a bad user agent. Some are from from small legitimate SEs. I think the bulk are from scrappers, and email harvesters.

I think I have explained on the page why I have included the different user agents.  As time allows I will do research on each of the user agents and update the page.

You are the only one who can determine if you should block any user agent.

If you have comments on the list you can make them here.

You can view the list of user agents here. 

4 Comments

  1. As a webmaster, you definitely should use user-agent headers to manager server traffic. But understand that this is purely a pragmatic tactic and not a serious security measure.

    I wrote more about this here:

    Webmaster Tips: Blocking Selected User-Agents
    http://faseidl.com/public/item/213126

    Comment by F. Andy Seidl — September 20, 2008 @ 9:08 pm

  2. It all depends on the user agent as to how great the security risk is.

    It might depend on your definition of security risk. Mine includes bots that scrape my content and post it on some other website.

    Comment by Connie — September 21, 2008 @ 7:42 am

  3. >>It might depend on your definition of security risk.<<
    I see what you mean, but what I was really getting at is that it is trivially easy to to forge a user-agent header. So, if someone really wants to exploit a security hole (regardless how that might be defined), you really can not prevent the exploit by simply examinging the user agent string. You may reduce some unwanted traffic by looking at user agent strings (and, in practice, you should), but if you really want to *completely* plug the security hole, something more is needed.

    Comment by F. Andy Seidl — September 21, 2008 @ 12:02 pm

  4. great post..It all depends on the user agent as to how great the security risk is. thanks

    Comment by rusli zainal sang visioner — June 28, 2009 @ 3:29 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress