NEED A GREAT WEB SITE? NEED IT TO BE SEARCH-ENGINE-FRIENDLY? SEO Egghead is a web development firm dedicated to creating custom, search engine optimized web site applications. We specialize in eCommerce and content management web sites that not only render information beautifully to the human, but also satisfy the "third browser" — the search engine. To us, search engines are people too. Click here to talk to us. We'd love to help!

X

We finally have a conclusion on how exactly to interpret a robot.txt file for the edge cases mentioned here.  Someone started a WebmasterWorld thread on the subject of contention.

Indeed, according to the specification, the rules for a specific matching user agent entirely override the "User-agent: *" rules.  Therefore, any rule under "User-agent: *" that should also be applied to a specific bot must be repeated under the "User-agent:" for that specific bot.  In other words, the more specific set of directives takes precedence over the default, and only one set is applied.

Googleguy says in the thread that he " … believes most/all search engines interpret robots.txt this way … "  This is also consistent with my testing.

However, some recommend placing the "*" rule last just in case, because some bots may not follow this specification, and take the first match — even if it's a "*."  Doing so achieves the intended result regardless.