- Jul. 21st, 2006
- 1 comments
I created a robots.txt file for this site that is contrived to test how various spiders interpret the specification. Here it is:
Since I'd like to make sure all these pages are indexed well in the first place (and because I'm a link-whore), please link to this post for me
We already know that Google will only exclude the page on MSN search, but I'm curious how Yahoo and MSN will interpret the file. In theory, according, at least, to Google's interpretation, the "*" section should be completely ignored by all — Google, Yahoo, and MSN; and effectively (unless you care about the other search engines), completely without effect here.
We'll know soon.