Well Yes you should use a robots.txt file for every website that you can.
This is important for it gives you control over what is indexed on your site and what isn’t. It also helps your site seem more normal to the search engines and more professional.
I have worked on some sites that were not being indexed by some search engines for some unknown reason. Once we created a robots.txt file and uploaded it and did some other tweaks their site began to get listed by that search engine. What change was the cause for the indexing was not clear.
But for the amount of time it takes to create a robots.txt file and upload it is definitely worth it.
Many sites do not have one and are indexed well and rank well for their keywords. It is not absolutely necessary but is good practice to create one. It also allows you to tell them what pages you do not want indexed. Not that this is absolute; as many web designers have had search engines ignore the details of the robots.txt file and index the whole site. If you have info you do not want indexed by a search engine like employee names and details, social security numbers and such you should consult with a security consultant about how to make this information available to your employees but hidden from the search engines. Like much of Internet marketing it is an attempt to control the search engines, which continue to be quite mysterious, but guidable with solid proven practices.
Tags: google, robots.txt, search engines
Powered by FireStats