PBNLinksHub
Get Started
Technical SEO ToolsRuns in your browserFree · no sign-up

Robots.txt Generator

Build a robots.txt file from a few simple choices — a default policy, specific rules, optional AI crawler blocks and your sitemap — with each line explained, so you know exactly what you're publishing.

Robots.txt Generator
Default for all crawlers
Custom rules

Example: Disallow /cart/ or /*?sessionid=. Paths are case-sensitive.

Block AI crawlers (optional)

One absolute URL per line.

robots.txt















How to use the Robots.txt Generator


  1. Choose the default
    Most public sites should allow crawling. Blocking everything is for staging or private sites.
  2. Add specific rules
    Disallow sections that shouldn't be crawled, such as internal search results or cart pages. Use Allow for exceptions.
  3. Add your sitemap
    Enter its full URL so crawlers can find it.
  4. Download and test
    Upload the file as robots.txt at your domain root, then check it with the Robots.txt Tester.




What it checks




Path format
Rules must start with / or *; missing slashes are added and flagged.
Unsupported characters
Spaces in paths are encoded as %20 and flagged.
Sitemap URLs
Each Sitemap line must be an absolute URL.
Crawl-delay
Must be a number; a note explains that Google ignores it.
Risky choices
Blocking the whole site, and the difference between blocking crawling and preventing indexing.




Interpreting the results


The generated file is made of groups. Each starts with a User-agent line naming which crawler it applies to (* means all), followed by rules:


    

  • Disallow: /path/ — asks the crawler not to request URLs starting with that path.
    • 

  • Allow: /path/ — permits a path inside a broader disallowed section.
    • 

  • Sitemap: https://… — tells crawlers where your XML sitemap is.
    • 

  • Crawl-delay — asks some crawlers to wait between requests; Google ignores it.
    • 



A crawler obeys only the most specific group that matches its name. If you add a group for Googlebot, Googlebot ignores the * group entirely — so repeat any shared rules in both.


Detected marks facts read directly from what you provided. Guidance marks general recommendations based on those facts — rules of thumb, not guarantees or search engine requirements.


Practical guidance


Keep robots.txt short and deliberate. Typical candidates to block are internal search results, filtered or sorted URL variations that create near-infinite combinations, cart and checkout steps, and admin areas. Don't block CSS or JavaScript files that pages need to render.


Never block pages you want to rank or pages you're building links to. A crawler that can't fetch a page can't read its content or follow its links, so a blocked target page gets little from new PBN backlinks or guest posts. And remember: robots.txt controls crawling, not indexing. To keep a page out of results, allow crawling and add noindex — check it with the Meta Tag Analyzer.


After publishing, test real URLs against the file with the Robots.txt Tester, and make sure your sitemap exists — the XML Sitemap Generator can create one.


Limitations


  • It generates a starting point. No generator knows your site's URL structure — review the rules against your own paths.
  • robots.txt is a voluntary standard. Reputable crawlers follow it; it isn't a security measure.
  • AI crawler user-agent names change over time; check each operator's current documentation.


Frequently asked questions




Where do I put robots.txt?
At the root of your domain: https://example.com/robots.txt. Each subdomain and protocol needs its own file.
Will Disallow remove a page from Google?
Not reliably. It stops crawling, but a URL can still be indexed from links. Use noindex (and allow crawling) to keep a page out of results.
Should I block AI crawlers?
That's a business decision. Blocking relies on each operator honouring robots.txt.
Is anything uploaded?
No. The file is generated in your browser.




More in Technical SEO Tools · All 25 free SEO tools