Hitfirm.com

An example of a robots.txt file can be taken directly from a popular auction site Ebay. You can see the file at: http://www.ebay.com/robots.txt. The content of this file is as follows:

### BEGIN FILE ###

#

# allow-all

#

#

# The use of robots or other automated means to access the eBay site

# without the express permission of eBay is strictly prohibited.

# Notwithstanding the foregoing, eBay may permit automated access to

# access certain eBay pages but soley for the limited purpose of

# including content in publicly available search engines. Any other

# use of robots or failure to obey the robots exclusion standards set

# forth at <http://www.robotstxt.org/ wc/ exclusion.html> is strictly

# prohibited.

# v3

#

User-agent: *

Disallow: /help/confidence/

Disallow: /help/policies/

Disallow: /disney/

### END FILE ###

This is the standard robots.txt that most sites put up in that it allows everything to be indexed except the cgi-bin directory. The beginning 15 lines are remarks and is a good guide for those not used to writing these files. If you are not sure what to have in your robots.txt file the last four lines without the pound sign are what are really read. You can create or edit these files with a simple text editor like the included notepad in Windows. Then just save the file as “robots.txt”. Then upload it to the main directory of your site. This is the directory that has the index.html.

I then recommend trying to go directly to your new file to make sure it is online. You can check this at: http://www.yoursite.com/robots.txt. It will take you directly to the text that you saved hopefully. Few people will actually travel to this page of yours, but search engine spiders will do this the first thing they do when they get to your site.