The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. The information specifying the ...
http://en.wikipedia.org/wiki/Robots.txt

User-agent: * Allow: /searchhistory/ Disallow: /news?output=xhtml& Allow: /news?output=xhtml. Disallow: /search. Disallow: /groups. Disallow: /images. Disallow: /catalogs
http://www.google.com/robots.txt

A robots.txt file provides restrictions to search engine robots (known as "bots") that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots ...
http://www.google.com/support/webmasters/bin/answer.py?answer=40360&topic=8846

This file must be accessible via HTTP on the local URL " /robots.txt ". The contents of this file are specified below . This approach was chosen because it can be easily implemented on any existing WWW ...
http://www.robotstxt.org/wc/norobots.html

Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
http://www.robotstxt.org/

robots.txt, www.nytimes.com 6/29/2006 # User-agent: * Disallow: /pages/college/ Disallow: /college/ Disallow: /library/ Disallow: /learning/ Disallow: /aponline/
http://www.nytimes.com/robots.txt

Brett Tabke experiments with writing a weblog in a text file usually read only by robots. Commentary on the world of search engine marketing.
http://www.webmasterworld.com/robots.txt

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt ... robots.txt: Forum Charter , Library , Moderated by: ThomasB & goodroi: Forum Options: Reset Last Read ...
http://www.webmasterworld.com/robots_txt/

robots.txt for http://www.w3.org/ # # $Id: robots.txt,v 1.45 2006/06/05 01:11:19 ted Exp $ # # For use by search.w3.org. User-agent: W3C-gsa. Disallow: /Out-Of-Date
http://www.w3.org/robots.txt

Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on ... Introduction to "robots.txt" There is a hidden, relentless force that permeates the web and its ...
http://www.javascriptkit.com/howto/robots.shtml

Everybody Loves Vegas Baby! Wow - what a conference Pubcon 2006 Las Vegas was! Lets do it again next year! Same time, same channel - so save the date! Watch the conference blog and this space for ...
http://pubcon.com/

Information on the use of robots.txt as well as a robots.txt generator and a list of search engine robots.
http://www.robotstxt.ca/

To remove your site from the Wayback Machine, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt ...
http://www.archive.org/about/exclude.php

Information on using the robots.txt file to keep web crawlers, spiders and robots from indexing ... Search engine robots will check a special file in the root of each server called robots.txt ...
http://www.searchtools.com/robots/robots-txt.html

Disallow all crawlers access to certain pages. User-agent: * Disallow: /exec/obidos/account-access-login. Disallow: /exec/obidos/change-style. Disallow: /exec/obidos/flex-sign-in
http://www.amazon.com/robots.txt

Offers a single source to search the Web, images, audio, video, news from Google, Yahoo!, MSN, Ask and many more search engines.
http://info.webcrawler.com/mak/projects/robots/exclusion.html

Offers a single source to search the Web, images, audio, video, news from Google, Yahoo!, MSN, Ask and many more search engines.
http://info.webcrawler.com/mak/projects/robots/faq.html

Find help with setting up your robots.txt file on your web site. ... New posts: Hot thread with new posts: No new posts: Hot thread with no new posts: Thread is closed
http://forums.digitalpoint.com/forumdisplay.php?f=50

Slurp will obey the first entry in the robots.txt file with a User-agent containing "Slurp". If there is no such record, it will obey the first entry with a User-agent of "*" . If it is not able to ...
http://help.yahoo.com/help/us/ysearch/slurp/slurp-02.html

Analyzes a robots.txt file searching for syntax and "logical" errors, and shows a summary of what effect it will have.
http://tool.motoricerca.info/robots-checker.phtml