The robots.txt is a very important file that is placed on your root directory, example would be www.yourdomain.com/robots.txt. This file Help the search engines and other robots which areas of your site they are allowed to visit and index. When search engine spidersbots start crawling a website, they start by identifying a robots.txt file at the root domain level. You can add only one robots.txt on your site and ONLY in the root directory (where your home page is):
BAD - Won't work: www.yourdomain.com/subdirectory/robots.txt
Online Robots.txt Generator is an easy-to-use tool to create proper Robots.txt directives for your site. Our robots.txt generator allows for the specification of several different search engines, including Google. You can set your crawl time, allow robots or deny, add sitemap of website. use this tool to create Xml sitemap, Restrict Directories etc
Some people confused between "crawler", "robot" and "spider":
Robot: Any program that goes out onto the web to do job. Its includes search engine crawlers, but also many other programs, site testers, like email scrapers and so on.
Crawler: This is the term for the kind of robot that search engines use.
Spider: This is a term that many professional SEO's use - it's synonymous with crawler.
You Can Try Our Free XML Sitemap Generator