Repository navigation
Automatic robots.txt from the bundled agent registry - #135
Merged
Merged
Conversation
Site owners pick bot categories (AI training crawlers, SEO tools, ...) and the plugin appends one robots.txt group naming every agent in those categories that has a known robots.txt token, ending in Disallow: /. Uses the agent registry bundled with the plugin: no account, no network request, kept current with each plugin update. - WebDecoy_Robots_Txt: pure generation (registry order, case-insensitive dedupe, unsafe or missing tokens skipped), sanitizing, and a late robots_txt filter that appends after existing output and adds nothing when the site discourages search engines. - Setting robots_disallow_categories, empty by default, so no site's robots.txt changes on upgrade. - Settings: Automatic robots.txt section in Good Bots with per-category counts, a search-engine warning, a preview, a link to the live file, and notices for a physical robots.txt or a subfolder install. - wp webdecoy config accepts robots_disallow_categories. - Blocking behaviour is unchanged.
The owner chose these categories, and the noindex tag WordPress adds in that mode does not keep training crawlers away. The filter now takes only the output, so the public flag cannot decide it, and the settings notice that said rules were skipped is gone.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Automatic robots.txt. The site owner picks bot categories (AI training crawlers, AI search crawlers, SEO tools, scrapers, ...) and the plugin adds one robots.txt group naming every agent in those categories that has a known robots.txt token, ending in
Disallow: /.It uses the agent registry already bundled with the plugin, so it works with no account and no network request, and the agent list is kept current with each plugin update.
How
includes/class-webdecoy-robots-txt.php(WebDecoy_Robots_Txt): pure generation and sanitizing. Registry order, case-insensitive dedupe (first spelling kept, also across categories), agents with no token skipped, and any token that could break out of its line refused.robots_txtat priority 100: appends after whatever WordPress, sitemaps and other plugins produced, which is kept as is. Nothing is added when no category is selected, underWEBDECOY_DISABLE, or if the block is already present. The rules are added even when the site discourages search engines: the owner chose these categories, and the noindex tag that setting adds does not keep training crawlers away.robots_disallow_categories(list of category ids), empty by default: no site's robots.txt changes on upgrade. Sanitized against the registry's categories;noneand categories with no robots tokens (headless browsers) are not offered.WebDecoy_Crawler_Visits::category_label()) and agent count, a warning on Search engine, a read-only preview of the saved block, and a link to the live/robots.txt. Notices when a physicalrobots.txtexists (WordPress's virtual file is never served then; the file is never touched, the preview is there to copy), when the site lives in a subfolder.wp webdecoy config set robots_disallow_categories training_crawler,seo_crawler(ornone). An unknown id refuses the whole value.Tests
tests/RobotsTxtTest.php: generation (dedupe, empty tokens, unsafe tokens, registry order, unknown categories, empty selection), choices and sanitizing, append behaviour (existing output kept byte for byte, no double block, a discouraged site still gets the rules),register()wiring, physical-file and subfolder detection, and the CLI parse/display. Each guard was mutated to confirm a test fails without it.Also run locally: phpcs (security + PHP 7.4 compat) and PHPStan clean for the changed files, and
tests/wp-compat/smoke.shon WordPress 7.1 / PHP 8.3 (SMOKE OK, empty debug.log), plus a manual pass: live/robots.txtbefore and after selection, discouraged site, settings form round trip keeps the selection, physical file notice.No version bump, readme or changelog changes; those go in the release PR.