If you haven’t heard of robots.txt, it’s a text file that informs Googlebot which areas of a domain the search engine may crawl. It also includes a reference to the website’s XML sitemap, which contains a list of all its important pages. Before Google starts indexing a site, it first locates the robots.txt file in the root directory.