In basic, web robot is simply a program. Robot is used to automatically and recursively traverses a Web site to retrieve document content and information. Search engine spiders is the most common types of Web robots. These robots visit Web sites and follow the links to add more information to the search engine database.
Web robots often go by different names. You may hear them called: