- Aug. 16th, 2006
- 2 comments
Indeed, according to the specification, the rules for a specific matching user agent entirely override the "User-agent: *" rules. Therefore, any rule under "User-agent: *" that should also be applied to a specific bot must be repeated under the "User-agent:" for that specific bot. In other words, the more specific set of directives takes precedence over the default, and only one set is applied.
However, some recommend placing the "*" rule last just in case, because some bots may not follow this specification, and take the first match — even if it's a "*." Doing so achieves the intended result regardless.
"2 Wise Comments Banged Out Somewhere On The Internet ..."