Robots and Robots.txt Files

Find it all in one place. Use this directory to find tools and information about RSS Feeds.  This listing is updated frequently, so be sure to Bookmark this Page for future reference.

Tools

Robots Txt Generator
Provided by 123promotion.co.uk
Use this tool to generate a simple robots file for your website, allowing you to hide files or directories that you dont wish to be spidered.

Robots.txt Validator
Provided by Search Engine World

Eye of the Spider
Provided by 123promotion.co.uk
Use this tool to show you how a search engine spider will view your web page.

Search Engine Spider Simulator
Provided by SeoPapers

Tutorials

Robots.txt Tutorial
Provided by Search Engine World

All About Search Indexing Robots and Spiders
Provided by Search Engine World

The Web Robots FAQ...
These frequently asked questions about Web robots.
Send suggestions and comments to Martijn Koster. This information is in the public domain.

The Robots Exclusion Protocol
Sometimes people find they have been indexed by an indexing robot, or that a resource discovery robot has visited part of a site that for some reason shouldn't be visited by robots.

The Robots META tag
The Robots META tag allows HTML authors to indicate to visiting robots if a document may be indexed, or used to harvest more links. No server administrator action is required.

The Web Robots Database
The List of Active Robots has been changed to a new format, called The Web Robots Database. This format will allow more information to be stored, updates to happen faster, and the information to be more clearly presented.

Spider Spotting Chart
Provided by SearchEngineWatch

Search engine robots and others
Provided by JafSoft Limited
The following table lists the search engines that spider the web, the IP addresses that they use, and the robot names they send out to visit your site.

Browsers
Provided by JafSoft Limited
Most browsers identify themselves with a string that begins "Mozilla...".

Link Checkers, Link monitors and bookmark managers
Provided by JafSoft Limited
Link checkers and bookmark managers are run by people wanting to keep their pages and bookmarks up to date.

Validators
Provided by JafSoft Limited
Validators check your web pages for HTML correctness and standards compliance.

Articles

Using a Robots Meta Tag
Provided by Search Engine World

Guidelines for Robot Writers
This document contains some suggestions for people who are thinking about developing Web Wanderers (Robots), programs that traverse the Web.

How to Set Up a robots.txt to Control Search Engine Spiders

Search Engine Robots - How They Work, What They Do (Part I)
Provided by SeoPapers

Search Engine Robots - How They Work, What They Do (Part II)
Provided by SeoPapers

Working with robots.txt file
Provided by SeoPapers

Creating a robots.txt file
Provided by SeoPapers
How you can create a robots.txt file in order to prevent your site from being penalized for spamming by the search engines

Article Index     Tutorial Index      Tools Index   
Top