Robots.txt Generator

Search Engine Optimization

Robots.txt Generator

Default - All Robots are:  
Sitemap: (leave blank if you don't have) 
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo MM
  Yahoo Blogs
  DMOZ Checker
  MSN PicSearch
Restricted Directories: The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

What are robots.txt files

As the name indicates, these are files that are not meant to be read by humans. Instead, we create these files and add them to our website in order to let "robots" or in this case search engines crawlers know which pages on our website they can access. The reason we want to let the search engine crawlers know exactly which pages to crawl, is to avoid overcrowding our server with unnecessary requests to pages that we don't want to be indexed by search engines.

How to use this tool

1. Fill the necessary fiels

2. Generate the robots.txt file

3. Copy the robots.txt file to your site