What Is robots.txt and How to Create One

robots.txt is a plain text file at the root of your site that tells well-behaved crawlers which parts of the site they're allowed to visit. It's a request, not a lock — it relies on crawlers choosing to respect it, which major search engines do, but it doesn't prevent a page from being accessed directly or block bad actors who ignore it.

What it can and can't do

• Can — tell crawlers not to waste time crawling admin areas, internal search results, duplicate parameter URLs, or staging content.
• Can — point crawlers to your sitemap so they discover your real content faster.
• Cannot — reliably keep a page out of search results. If a disallowed page is linked to from elsewhere, it can still be indexed with just the URL and no description, since the crawler was never allowed in to see the page's own noindex tag. Use a noindex meta tag directly on the page for that instead.
• Cannot — password-protect or truly hide anything. The file itself is publicly readable at yoursite.com/robots.txt, and listing sensitive paths in it can actually draw attention to them.

A basic, correct robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://example.com/sitemap.xml

User-agent: * applies the rules to every crawler. Disallow blocks a path; Allow explicitly permits one, useful for carving out an exception inside a blocked folder. The Sitemap line isn't a rule — it's a pointer, and it's one of the most reliable ways to make sure crawlers find your sitemap without needing to submit it manually everywhere.

Generating yours

1. List the paths you actually want to block (if any) in the Robots.txt Generator.
2. Download the file and upload it to your site's root directory — it must live at exactly /robots.txt, not in a subfolder, or crawlers won't find it.
3. Test it in Google Search Console's robots.txt tester to confirm it's being read correctly before relying on it.