Tutorial :robots.txt: Disallow bots to access a given “url depth”


I have links with this structure:

I want google & co to spider all links that have ONE tag in the URL, but NOT the URLs that have two or more tags.

Currently I use the html meta tag "robots" -> "noindex, nofollow" to solve the problem.

Is there a robots.txt solution (that works at least for some search bots) or do I need to continue with "noindex, nofollow" and live with the additional traffic?


I don't think you can do it using robots.txt. The standard is pretty narrow (no wildcards, must be at the top level, etc.).

What about disallowing them based on user-agent in your server?

Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Next Post »