What do I mean by Opt-in? In short rather than always adding user agents, or IP addresses to your .htaccess file to blacklist them, only allow certain bots, and browsers to access your site. So rather than using the a black list method to prevent unwanted bots, you just white list certain bots. All others will get a 403 error page.
August 28, 2007
August 27, 2007
There are a lot of different things a webmaster needs to do to prevent their content from being scrapped. Unfortunately at this time there is no simple solution. One thing I can tell you is this.
August 22, 2007
All of these bots have got caught in one of my bot trap at least once. Bots like Panscient and Cyveillance have got caught on at least two of my sites. A few of the bots (spiders) listed are legitimate bots, who claim to obey Robots.txt.
August 20, 2007
Some times you just can’t win.
A husband and wife are sitting quietly in bed reading when the wife looks over at him and asks the question….
August 18, 2007
Webmaster Tools has an excellent way for any webmaster to see how Google will respond to their robots.txt file. I was using the robots.txt checker today and discovered a couple of things I did not know. (more…)
August 17, 2007
Several months ago Doug Heil (aka ihelpyou) launched a new website for his business. The new site which I think is still in beta, added a couple of features beyond site design. There was an article section, and a blog added. (more…)
August 9, 2007
Panscient Data Services Pty Ltd is a Company in Australia. In the last few days their bot hit my bot trap on two different sites. The IP both times was 22.214.171.124 and the user agent was panscient.com. They are operating from this IP Range. 126.96.36.199/30. (more…)
August 6, 2007
Supposedly Cyveillance is a security company roaming the net looking for illegal content which they sell the information to their customers. According to this article about Cyveillance their clients are those from the music & movie industry, thus the RIAA, and MPAA. If the article is correct, and you look at the end of the article you will see some customers listed who have nothing to do with RIAA or the MPAA industries. I do not know how accurate the authors information is, but he seems to have spent a lot of time researching this. Read the article and draw your own conclusions. (more…)