Archived; click post to view.
Excerpt: I decided that I would test what I think is an inconsistency in the interpretation of the robots.txt specification by various implementors cited here.I created a robots.txt file for this site that is contrived to test how various spiders interpret the specification. Here it is:User-agent: *Disallow: /blog/seo/automatically-highlighting-internal-links-p51.htmlDisallow: /blog/seo/msn-search-p5.htmlUser-agent: googlebotDisallow: /blog/seo/msn-search-p5.htmlUser-agent: msnbotDisallow: /blog/seo/using-referers-http_referer-to-increase-conversions-and-perceived-relevance-p9.htmlUser-agent: slurpDisallow: /blog/seo/yahoo-hostings-lack-of-htaccess-support-p8.htmlSince I'd like to make sure all these pages are indexed well in the first place (and because I'm a link-whore), please link to this post for me :)We already know that Google will only exclude the page on MSN search, but I'm curious how Yahoo and…
Archived; click post to view.
Excerpt: Some people may know about this already, but it's worth discussing since it has been pertinent to me a few times:In theory, according to my interpretation of the robots.txt specification, if a Disallow: under User-agent: "*" exists, as well a Disallow: under a specific robot's User-agent:, and that robot accesses the web site, both rules should be applied, and both should be excluded. However, Google does not interpret it this way, and only applies the rules for the specific robot User-agent:, "googlebot." For Google, it is necessary to repeat all rules in "*" under googlebot’s User-agent: as well to get…
Archived; click post to view.
Excerpt: The following is a quote straight from Google's Investor Relations web page. "If an opportunity arises to take advantage of competitors' confidential information, remember: don't be evil. We compete, but we don't cheat." Unfortunately, I'm not so sure this speaks for Google's most recent actions that are, in my opinion, aimed at swiping Microsoft employees. One of them, Kai-Fu Lee, was apparently an integral part of Microsoft's search engine team. The result was an ugly law suit and a later undisclosed settlement. He is currently president of Google in China. Microsoft argued that Lee…
