Free OnlineRobots.txt GeneratorNo Sign-Up Needed

robots.txt is a plain-text file that tells search engine crawlers which parts of your website they may or may not crawl. It uses simple directives — User-agent, Allow, Disallow, Crawl-delay, and Sitemap — grouped by crawler. It is a crawl directive, not a security mechanism: well-behaved bots follow it, but it does not password-protect anything.

At the root of your domain, so it is reachable at https://www.example.com/robots.txt. A robots.txt file in a subfolder is ignored — crawlers only ever request it from the domain root. Each subdomain (and each protocol) needs its own file.

Strictly, no — if the file is missing, crawlers assume they may crawl everything. But having one is best practice: it lets you point crawlers at your XML sitemap, keep low-value pages (cart, search results, admin paths) out of the crawl, and manage AI crawler access. For anything beyond a tiny brochure site, you should have one.

No, and this is the most common misconception. Disallow stops crawling, not indexing. A disallowed URL can still appear in results (usually without a description) if other sites link to it. To keep a page out of Google, use a noindex meta tag or proper authentication — and note the page must be crawlable for Google to see the noindex tag.

Disallow (robots.txt) tells crawlers not to fetch a URL. noindex (a meta tag or X-Robots-Tag header) tells search engines not to show the page in results after crawling it. To de-index a page, use noindex and let it be crawled. Disallowing a page you want de-indexed is counterproductive — Google can never see the noindex instruction.

Create a group starting with User-agent: Googlebot and add your Allow or Disallow rules under it. Crawlers follow the most specific group that matches them, so a Googlebot group overrides a * group entirely for Googlebot. This generator lets you add any number of user-agent groups with their own rules.

Add a group for each AI crawler with Disallow: / under it. Common AI user agents include GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Gemini training), CCBot (Common Crawl), and OAI-SearchBot. All are available as one-click presets in this generator. Blocking Google-Extended does not affect your normal Google Search rankings.

No. Google renders pages like a browser, and if it cannot fetch your CSS and JS it may misjudge mobile-friendliness, layout, and content — hurting rankings. Blocking /wp-includes/ or asset folders was an old practice that now does more harm than good. This tool warns you if a rule looks like it blocks CSS or JS.

Yes. Add one Sitemap: line per sitemap, each with a full absolute URL. This is common for large sites that split sitemaps by section (pages, posts, products) or use a sitemap index file. Sitemap lines are independent of user-agent groups and can appear anywhere in the file.

Crawl-delay asks crawlers to wait a number of seconds between requests. Bing, Yandex, and some others respect it; Google ignores it entirely — you manage Google's crawl rate in Search Console instead. Only set a crawl delay if a specific non-Google bot is genuinely overloading your server.

Use a Disallow for the folder and a more specific Allow for the file, in the same group: Disallow: /private/ plus Allow: /private/public-file.html. When rules conflict, Google follows the most specific (longest) matching rule, so the Allow wins for that one file.

The simplest valid file is User-agent: * followed by Disallow: with nothing after the colon — an empty Disallow means nothing is blocked. This generator's 'Allow all' preset produces exactly that, and it is a perfectly good robots.txt for most small sites, ideally with a Sitemap line added.

Use the robots.txt report in Google Search Console (Settings > robots.txt) — it shows the fetched file, parse errors, and lets you check specific URLs. Test after every change: a single mistyped Disallow: / can de-facto remove your site from search. This generator also warns you live about the most dangerous patterns.

WordPress serves a minimal virtual robots.txt if no physical file exists, and SEO plugins like Yoast or Rank Math let you edit it. A physical file at the domain root always takes precedence over the virtual one. If you generate a file here, upload it via FTP or your host's file manager to the site root.

Yes, for the major engines. * matches any sequence of characters (Disallow: /*?* blocks every URL containing a query string) and $ anchors the end of a URL (Disallow: /*.pdf$ blocks all PDFs). Google, Bing, and most modern crawlers support both, though the original 1994 standard did not — very old bots may ignore them.