Configure a crawl policy and review the output

Better Robots.txt Pro adds advanced governance and protection modules to a guided WordPress robots.txt workflow. Choose a preset, adjust settings explained in plain language, and inspect the generated file in Review & Save before publishing. Core WordPress protection rules remain visible and editable, and you can return to settings to regenerate the file later.

Set rules by crawler type

Manage search engine crawlers, AI and LLM crawlers, SEO utilities, bad bots, archive services, feed crawlers, social preview bots, and other automated agents. AI-related agents are grouped as training crawlers, search and discovery crawlers, and user-action fetchers.

• Pro and Premium add per-bot AI crawler rules and custom user agents.
• Set AI usage signals such as search, ai-input, and ai-train, and generate llms.txt and other machine-readable governance files.
• Manage archive and Wayback access, feeds, crawl traps, social preview crawlers, and ads.txt and app-ads.txt allowance.
• Review WooCommerce crawl cleanup settings, along with rules for CSS, JavaScript, and image crawling.
• Use advanced directives such as crawl-delay and custom rules.

The Free edition offers a basic allow or block policy for AI search and discovery crawlers, separately from its policy for training crawlers. Pro and Premium add more detailed controls. User-action fetchers are triggered by user requests, and robots.txt may not apply to them the same way it applies to automated crawling. Where available, server logs, WAF rules, provider verification, or IP-based controls can support stronger enforcement.

Choose a starting preset

Essential includes a clean starting point for general sites. AI-First is intended for publishers and content websites that want AI-related governance while keeping discovery open. Fortress is for sites seeking stronger protection against scraping, archive capture, and unnecessary crawling; it is included with Premium. Custom lets customers configure modules manually. After choosing a preset, revisit settings, adjust individual modules, and regenerate the file.

Pro and Premium controls

Pro adds advanced governance and protection options, including additional AI, crawler, and cleanup controls. Premium unlocks the most restrictive protection options and additional high-control modules. The interface labels options by edition so you can see which require a paid plan.

Install and save

Install and activate Better Robots.txt in WordPress, then open its settings. Choose a preset, configure the available crawler and governance controls, and follow the workflow to Review & Save. Inspect the generated robots.txt and save when it matches your intended policy.