Search Engine Optimization

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots.txt Generator

Robots.txt is probably the most miss understood that a site can get. Many people believe that by using a Robots.txt file on their site they are saving folders and pages from hackers and thieves. Actually it is fully the opposite! Robots.txt opens up a huge security hole that thieves and hackers will use to simply get access to the parts of your site that you don’t want them to.

What is Robots.txt?

Robots.txt is a file that you make and upload to your sites root directory that is used by search engine spider to check which parts of your site they should index and which pages/folder that you, the site owner, don’t want listed in search engine indexes.

Why would not want pages indexed?

There are many factors why you might not want search engines to index pages on your site, such as exclusive training pages, or private membership pages and such like.

If you are an online marketer selling your own ebook or online digital product, you definitely not want to index your pages or books online.

Many internet marketers who offer eBooks or other digital products for quick download or will list their download thank you pages in the Robots.txt file because they clearly don’t want those pages indexed in search engines.

By using Robots.txt this way though, you will be launching your product to anyone who has a pretty bit of knowledge and how the file jobs.

If you are making the robots.txt file for the first time and are surprising what you need to disallow then you can reject the blow-mentioned things.

  • Privacy page
  • Internal structure
  • The content page of your site
  • All media folder which you don’t want to view in search results
  • All images files which you don’t want to view in the search results
  • Now in order to produce the robots.txt you can take help from Robots.txt generator. Robot.txt. This tool is a free internet tool which helps you in making Robots.txt file for your site. If you have already made Robots.txt file and want to make it perfect them you can upload it and modify it using this instrument. Using this tool is very simple as the directories which you don’t want the search engines to index will be included in the disallow section whereas the directories which you want the search engines to index will be show in the permit section.

    If you want to create a Robots.txt file for your site.