The robots.txt is a very important file that is placed in your root directory, the example would be www.yourdomain.com/robots.txt. This Robots.txt file Generator Help the search engines and other robots which areas of your site they are allowed to visit and index. When search engine spiders bots start crawling a website, they start by identifying a robots.txt file at the root domain level. You can add only one robots.txt on your site and ONLY in the root directory (where your home page is):
BAD - Won't work: www.yourdomain.com/subdirectory/robots.txt
It is a very useful tool that has made the lives of many webmasters easier by helping them making their websites Googlebot friendly. It is a robot.txt file generator tool that can generate the required file by performing the difficult task within no time and for absolutely free. Our tool comes with a user-friendly interface that offers you the options to include or exclude the things in the robots.txt file.
Online Robots.txt file Generator is an easy-to-use tool to create proper Robots.txt directives for your site. Our Robots.txt file Generator allows for the specification of several different search engines, including Google. You can set your crawl time, allow robots or deny, add sitemap of the website. use this tool to create Xml sitemap, Restrict Directories etc
Some people confused between "crawler", "robot" and "spider":
You Can Try Our Free XML Sitemap Generator