What Is Robots.txt?

A test file that tells web robots (usually search engines) which pages on your website to and to not crawl.