Skip to content

Automatic robots.txt from the bundled agent registry - #135

Merged
cport1 merged 2 commits into
mainfrom
feat/automatic-robots-txt
Nov 16, 2026
Merged

cport1 merged 2 commits into
mainfrom
feat/automatic-robots-txt

Conversation

@cport1

@cport1 cport1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What

Automatic robots.txt. The site owner picks bot categories (AI training crawlers, AI search crawlers, SEO tools, scrapers, ...) and the plugin adds one robots.txt group naming every agent in those categories that has a known robots.txt token, ending in Disallow: /.

It uses the agent registry already bundled with the plugin, so it works with no account and no network request, and the agent list is kept current with each plugin update.

# WebDecoy: AI and bot rules (32 agents, updated with plugin version 2.12.0)
User-agent: Reflectionbot
User-agent: GPTBot
User-agent: ClaudeBot
...
Disallow: /
# End of WebDecoy rules

How

  • includes/class-webdecoy-robots-txt.php (WebDecoy_Robots_Txt): pure generation and sanitizing. Registry order, case-insensitive dedupe (first spelling kept, also across categories), agents with no token skipped, and any token that could break out of its line refused.
  • Hooked on robots_txt at priority 100: appends after whatever WordPress, sitemaps and other plugins produced, which is kept as is. Nothing is added when no category is selected, under WEBDECOY_DISABLE, or if the block is already present. The rules are added even when the site discourages search engines: the owner chose these categories, and the noindex tag that setting adds does not keep training crawlers away.
  • Setting robots_disallow_categories (list of category ids), empty by default: no site's robots.txt changes on upgrade. Sanitized against the registry's categories; none and categories with no robots tokens (headless browsers) are not offered.
  • Settings, Good Bots tab, new "Automatic robots.txt" section: a checkbox per category with its label (reuses WebDecoy_Crawler_Visits::category_label()) and agent count, a warning on Search engine, a read-only preview of the saved block, and a link to the live /robots.txt. Notices when a physical robots.txt exists (WordPress's virtual file is never served then; the file is never touched, the preview is there to copy), when the site lives in a subfolder.
  • wp webdecoy config set robots_disallow_categories training_crawler,seo_crawler (or none). An unknown id refuses the whole value.
  • Copy says plainly that robots.txt is a request and points to the block options for enforcement. Blocking behaviour is unchanged.

Tests

tests/RobotsTxtTest.php: generation (dedupe, empty tokens, unsafe tokens, registry order, unknown categories, empty selection), choices and sanitizing, append behaviour (existing output kept byte for byte, no double block, a discouraged site still gets the rules), register() wiring, physical-file and subfolder detection, and the CLI parse/display. Each guard was mutated to confirm a test fails without it.

Also run locally: phpcs (security + PHP 7.4 compat) and PHPStan clean for the changed files, and tests/wp-compat/smoke.sh on WordPress 7.1 / PHP 8.3 (SMOKE OK, empty debug.log), plus a manual pass: live /robots.txt before and after selection, discouraged site, settings form round trip keeps the selection, physical file notice.

No version bump, readme or changelog changes; those go in the release PR.

Site owners pick bot categories (AI training crawlers, SEO tools, ...) and
the plugin appends one robots.txt group naming every agent in those
categories that has a known robots.txt token, ending in Disallow: /. Uses
the agent registry bundled with the plugin: no account, no network request,
kept current with each plugin update.

- WebDecoy_Robots_Txt: pure generation (registry order, case-insensitive
  dedupe, unsafe or missing tokens skipped), sanitizing, and a late
  robots_txt filter that appends after existing output and adds nothing
  when the site discourages search engines.
- Setting robots_disallow_categories, empty by default, so no site's
  robots.txt changes on upgrade.
- Settings: Automatic robots.txt section in Good Bots with per-category
  counts, a search-engine warning, a preview, a link to the live file, and
  notices for a physical robots.txt or a subfolder install.
- wp webdecoy config accepts robots_disallow_categories.
- Blocking behaviour is unchanged.
The owner chose these categories, and the noindex tag WordPress adds in
that mode does not keep training crawlers away. The filter now takes only
the output, so the public flag cannot decide it, and the settings notice
that said rules were skipped is gone.
@cport1
cport1 merged commit 8828f4b into main Nov 16, 2026
4 checks passed
@cport1 cport1 mentioned this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant