Who needs indices?

A couple of weeks ago I wanted to make a recipe I knew I had written about on my site — Linguine with Sausage and Peppers. I hopped over to my search page and looked up “recipe copyright” since I remembered I had shared that recipe in the post I wrote on that topic. My search revealed two results:

Hmm. That 2007 index isn’t very useful if you were looking for a particular post, but it’s seemingly relevant to search engines since it contains the same keywords I searched for. Presumably you’d only visit the 2007 index if (a) you’re me, and you’re feeling nostalgic, or (b) you’re a search engine and crawling my whole site. So, how do you tell search engines not to care about that post index?

Later, pointless pages

You might be familiar with robots.txt, where you specify in one place files/directories for search engines to exclude. But, that wasn’t really working for me. Luckily, you can specify some robots directives with special meta tags, and I just told search engines not to index the pages but to follow the links on them.

<META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>

(The follow isn’t actually necessary.)

Does it work?

Google seems to have removed the index pages from its search (but probably not from its awesome, evil, secret database) as shown by this search: recipe copyright site:www.paragiraffe.com - Google Search. Yahoo!, which I use for my site search, seemed to have picked up on the change, but as of this writing the index is back: recipe copyright site:www.paragiraffe.com - Yahoo! Search Results. So, it worked a little, and maybe I just need to give it more time.

Note: Here’s a Wordpress plugin for Robots Meta Tags.