Thanks for the reply!

Blocking all of /filtered/ is definitely not what I want.
I don’t want to block ALL Google access to filters. In fact, that would be Very Bad. I definitely want Google to index things like “everything that’s happening at Brasils Nightclub.”

What I *don’t* want Google to do is index combinations of two or more filters. End users have a pretty good idea which combinations are useful, but of course search engines try to explore the whole space… combinatorial explosion! Kaboom!

I separate filters with semicolons, so my original thought was to do this in robots.txt, following the globbing syntax that robots.txt resembles (but doesn’t quite follow):

Disallow: *;*

But everything I read up to that point led me to believe that this was not supported. So I wound up taking a different approach.

Now that I’ve read the article you linked to, I’m aware that I can in fact write just:

Disallow *;

And I don’t need a trailing * because the expanded Robot Exclusion Protocol doesn’t really work like a shell “glob” pattern: you don’t have to match the whole thing. In that sense it works more like a Perl-style regular expression, although it is confusing because the use of * strongly suggests it will act like globbing. But this is a pre-existing issue in the REP since:

Disallow: /foo

Has been around for a while.

I also see that you’ve added $, which is a logical thing to offer if the REP is going to match substrings beginning from the root by default and not complete URLs. Nice.

I will try Disallow: *; and see how that plays out. Though I’m not sure it will matter since there is already a rule on Google’s side. I see that MSN is grinding through my site too, although not a CPU-pinning fashion, so I’ll watch and see if they respond well to the new rule.

I did not find the Google Webmaster Blog article because I was searching for robots.txt and not Robot Exclusion Protocol. Ironically, Google didn’t offer me that blog article as a result for the former. It did offer me several pages that fail to mention the new features and don’t use the term Robot Exclusion Protocol. They use “Robot Exclusion Standard” and other not-quite-the-same terminology. Perhaps Google could reach out to the webmasters of those pages. It would also be more than acceptable to give a bump to your own webmaster blog article on this, IMHO, as it’s clearly what people need to know.

The Wikipedia article doesn’t cover your improvements to the Robot Exclusion Protocol either, although it does at least reference the term:

http://en.wikipedia.org/wiki/Robots.txt

* * *

My hack certainly cut down on the Googlebot traffic to the site. But so far the Google results for the filters are still pretty messy. A search for:

salsadelphia brasils

Yields this as the first result:

http://salsadelphia.com/filtered/genre,bachata;venue,brasils

Which narrows things too much by also filtering for bachata, and gives a weird impression of what is available there.

When what I want to see here is just:

http://salsadelphia.com/filtered/venue,brasils

I am hoping that the multiple-filter results will age off your search indexes soon?

In the meantime, I’ve been told that a regexp was added on Google’s end, and sure enough Googlebot has calmed down. Today’s Googlebot traffic looks like this:

/filtered/instructor,joe_figueroa
/filtered/studio,take_the_lead
/filtered/dj,rumbero

… And so on. Lots of single filters, no multiple filters. This is exactly what I want.

I’ll be completely happy when the results for multiple filters stop showing up in results.

Thanks again for the response!

First off, yes, we – and I believe most of the other big search engines – support wildcards in robots.txt files. You can find a blog post about this at http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html . That said, you don’t need wildcards that to block the /filtered/ subdirectory, just use “disallow: /filtered/” in your robots.txt file and search engine crawlers will automatically know to ignore everything that starts with /filtered/ . To block everything with /filters/ in the URL, you could use “disallow: /*/filters/”.

Give it a shot & remove that redirect. I’m pretty sure it’ll work how you want it to work .
John