User-agent: *Honestly, who searches for search results? When I hear about a new website, the first thing I do, is a site: search. The second thing I do is check the robots.txt. Isn't that what all normal people do?
Disallow: /search
July 28, 2008
Hey Cuil, Use a Robots.txt
Cuil.com is flooding my newsfeeds this morning. Reid already reported that their home page links are only periodically working. I, too, don't see my site on the first three pages of a vanity search. Also, looks like they need to add a robots.txt file to block other search engines from indexing their search results. This should work:
Subscribe to:
Post Comments (Atom)
3 comments:
Yeah, I find myself checking site searches and robots.txt files in my spare time as well. The worst is when I get some random comment on my blog, and have to investigate to make sure it's not some subtle spammer.
ha ha, those are exactly the same things I end up looking at,after the vanity search ofcourse. :)
You guys are nerds.
Post a Comment