SEO & Web
What Is Robots.txt?
Robots.txt is a text file at a site’s root that tells search engine crawlers which parts of the site they may or may not crawl.
Robots.txt is a plain-text file placed at the root of a domain (example.com/robots.txt) that gives crawling instructions to search engine bots. It can disallow entire sections, block specific bots, and point to the XML sitemap.
Robots.txt controls crawling, not indexing — a disallowed URL can still be indexed if it is linked from elsewhere. It is best used to keep bots out of infinite spaces like faceted filters and admin areas, not to hide pages from search results (use noindex for that).