Site Navigation
Vector Space Search Engine Building a Spider Indexing the Internet Vector Space Postgres Programming What is a robots.txt file Stop List
Some Other Sites
Harry Jackson Job Site The Banana Tree HR-XML Builder -->

Robot text file uses that I have noticed.

While I have been searching the internet for links I have noticed some weird and wonderful uses for the robots.txt file. It would seem that this file is extremely misunderstood. The most common thing that I have seen is the use of the directive. 

          Allow *

There is no such directive supported as yet in the Robots protocol although it may be in the future it is not at the moment.

One of my favourite one is actually at http://otn.oracle.com/robots.txt this consist of a full page of html with the centre content containing 

          User-agent Disallow: /jscripts/

This is probably an automatically generated page which has been overlooked by someone. The following resource list some of the most common errors in using the robots.txt file. Common Mistakes in the robot.txt File