The Robots.txt file is a simple text file placed on your web server which tells web crawlers like Googlebot if they should access a file or not. The Robots Exclusion Protocol (REP) is a group of web standards that regulate web robot behavior and search engine indexing. The REP consists of extensions, tags, and other elements.