robot.txt file is your invitation to the search engines when they send a spider to crawl through the pages of your web-site. These spiders are out and about on the web gathering information that contributes to your search engine ranking.
There are different kinds of robots/spiders/crawlers, some are just taking inventory, some are looking for what's new, but whatever kind of robot they are, when they hit your site, you have some control over what they do, and you need to give them some instructions.
You can block bots, you can tell them which pages they can and can't look at and you can tell them when you'd like them to return.
There are probably a thousand free robot.txt generators out there and here is one you can use: 1-hit.com
If you want to generate your own file read on...
