What Is a Web Crawler?

Also known as a bot or a spider, a crawler is a program that crawls the web via links to add websites to the search engine’s index.