Online Robots.txt Generator

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Online Robots.txt Generator

The robots.txt is a very important file that is placed on your root directory, example would be www.yourdomain.com/robots.txt. This file Help the search engines and other robots which areas of your site they are allowed to visit and index. When search engine spidersbots start crawling a website, they start by identifying a robots.txt file at the root domain level. You can add only one robots.txt on your site and ONLY in the root directory (where your home page is):

OK: www.yourdomain.com/robots.txt

BAD - Won't work:  www.yourdomain.com/subdirectory/robots.txt

Online Robots.txt Generator is an easy-to-use tool to create proper Robots.txt directives for your site. Our robots.txt generator allows for the specification of several different search engines, including Google. You can set your crawl time, allow robots or deny, add sitemap of website. use this tool to create Xml sitemapRestrict Directories etc

Some people confused between "crawler", "robot" and "spider":

  • Robot: Any program that goes out onto the web to do job. Its includes search engine crawlers, but also many other programs, site testers, like email scrapers and so on.

  • Crawler: This is the term for the kind of robot that search engines use.

  • Spider:  This is a term that many professional SEO's use - it's synonymous with crawler.

You Can Try Our Free XML Sitemap Generator