Tutorials

Robots and Robots.txt Files

Find it all in one place. Use this directory to find tools and information about RSS Feeds. This listing is updated frequently, so be sure to Bookmark this Page for future reference.

Tools

Robot Control Code Generation Tool
By McAnerin Networks Inc.

Search Engine Spider Simulator
Provided by Webconfs.com

All About Search Indexing Robots and Spiders
Provided by SearchTools.com

The Web Robots Pages
Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

Bots vs Browsers : Public Bot / User Agent Database & Commentary
Database of 299,191 user agents and growing...

Spider Spotting Chart
Provided by SearchEngineWatch
Robot Agent and Host Names.

Search engine robots and others
Provided by JafSoft Limited
The following table lists the search engines that spider the web, the IP addresses that they use, and the robot names they send out to visit your site.

Browsers
Provided by JafSoft Limited
Most browsers identify themselves with a string that begins "Mozilla...".

Link Checkers, Link monitors and bookmark managers
Provided by JafSoft Limited
Link checkers and bookmark managers are run by people wanting to keep their pages and bookmarks up to date.

Validators
Provided by JafSoft Limited
Validators check your web pages for HTML correctness and standards compliance.

Articles

Guidelines for Robot Writers
This document contains some suggestions for people who are thinking about developing Web Wanderers (Robots), programs that traverse the Web.

How to Set Up a robots.txt to Control Search Engine Spiders
by Christopher Heng, thesitewizard.com
This article explains why you might also want to include a Robots.txt file on your sites, how you can do so, and notes some common mistakes made by new webmasters with regards the ROBOTS.TXT file.

Search Engine Robots - How They Work, What They Do (Part I)
Provided by SeoPapers
Automated search engine robots, sometimes called "spiders" or "crawlers", are the seekers of web pages. How do they work? What is it they really do? Why are they important?....

Search Engine Robots - How They Work, What They Do (Part II)
Provided by SeoPapers
If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Working with robots.txt file
Provided by SeoPapers
Learn all about working with robots.txt file. A useful guide that talks about what robots.txt file is, its advantages & disadvantages, how to optimize & use robots.txt file to define the content you want excluded from indexing, thus saving the crawler's indexing time…

Creating a robots.txt file
Provided by SeoPapers
How you can create a robots.txt file in order to prevent your site from being penalized for spamming by the search engines

Article Index Tutorial Index Tools Index
Top

Article Index Tutorial Index Tools Index