feat(adf): compile preserves retained sections (#318 5/6)

CHANGELOG.md

@@ -10,6 +10,9 @@ The format is based on Keep a Changelog and follows Semantic Versioning.
 
 ### Changed
 
+- **Compiled files that carry your own retained sections change bytes on the next `adf compile --write`** (`#303`, `#318` part 5) — a compiled `CLAUDE.md`, `AGENTS.md`, `.cursorrules` or `GEMINI.md` with a hand-added `## Environment` (or another section `adf tidy` retains) no longer loses it. The next `--write` moves it into a `<!-- charter:retained:start -->` … `<!-- charter:retained:end -->` block at the end of the file, and every later `--write` copies that block as it stands. A compiled file with nothing to carry gets no block and compiles byte-identically to before. The block uses HTML markers in every target, `.cursorrules` included. All-caps headings the compiler could have produced (a module since removed from `.ai/`) and `## Module Index` are never carried. `adf compile` to stdout and `charter score` still use the plain render.
+- **`adf compile --check` reads such a file as stale until one `--write`** (`#303`) — the expected bytes are now the render plus the file's own retained block, so an edit inside the block is never stale. A compiled file whose retained sections sit outside the block reads `[stale]` with `staleReason: "retained-unfenced"`, and the remedy (`compile --target <t> --write`) keeps them. Each `--check` JSON row adds `staleReason` (`content`, `retained-unfenced`, or null).
+- **`refused-retained` is retired from `adf compile --write`, and `retained-divergent` from `adf vendor restore`** (`#303`) — JSON consumers no longer see either value. A pointer converted with `--mode compiled` now carries the retained sections that differ from its template into the block, instead of refusing. It still refuses, exit 1 with nothing written and `VENDORS` unchanged, over content the output would not carry: `refused-bloated` (items `adf tidy` would extract; the first remedy is `adf tidy --source <file>`, then `--force`, which discards them) and `refused-would-drop` (other lines of yours). `--mode compiled --force` over a hand-authored file carries its retained sections and discards the rest. `--force` never drops retained sections. Write JSON rows add `retainedSections`, `retainedSource`, `retainedSkipped` and `retainedAmbiguous`. `retainedSkipped` lists retained-looking sections whose heading is still in a `.ai/` module, so they are rendered `.ai/` text left behind by an edit; they are not carried, and each one is named on stderr. A section left inside a generated section that `.ai/` no longer renders, with no `.ai/` match, may be stale output or your own. `--write` refuses that target as `refused-ambiguous` (new value), with "wrap it in the retained block" as the first remedy and `--force` (which discards it) as the second. `--check` reports it with `staleReason: "retained-ambiguous"`. Parsing is code-fence aware, and marker lines match as loosely as `stripCharterSentinels`. `vendor restore` of a compiled file carries its retained sections into the pointer and lists them in `keptSections`, and the block is not counted as a `compiled-differs` difference. A block holding a non-retained heading, or text before its first heading, is refused as `would-drop` (new value) with `vendor set <file> compiled` as the first remedy, because a pointer cannot keep it.
 - **The pre-commit hook tidies only the staged vendor files and re-stages only the files tidy wrote** (`#320`, `#318` part 4) — re-run `charter hook install --pre-commit` to pick up staged-only tidy. An installed hook keeps its old behaviour until you do, and `charter doctor` reports it with an INFO `pre-commit hook` row (INFO, so `--ci` does not fail). Reinstalling overwrites a Charter-managed hook without `--force`, as before. The old hook ran a whole-tree `adf tidy` whenever any vendor file was staged, then ran `git add` on every staged vendor file and on all of `.ai/*.adf`. That swept unrelated `.ai/` edits into the commit and rewrote vendor files that were not part of it. The new hook passes each staged vendor file to `adf tidy` with `--source` and stages only the paths that both the dry run and the real run report. A staged vendor file that also has unstaged changes is skipped with a message, because tidy works on the working tree and staging its result would commit those hunks. If tidy would write into a module that has unstaged changes or is untracked, nothing is tidied for that commit and a message says why. A gitignored module (for example an `.ai/` kept out of git) is not treated as work in progress: tidy writes it, and the hook says it was left unstaged. Any file the hook cannot stage, and any file the real run wrote that the dry run did not plan, is named and left unstaged rather than aborting the hook. A staged lowercase `agents.md` is now tidied (`#302`), and a commit staging only compiled output still makes no CLI call. The tidy block itself never fails a commit, but the ADF evidence gate after it can, and it is unchanged. In the generated hook (no `verify:adf` script), vendor bloat that a skipped tidy leaves in the working tree still fails `charter doctor --adf-only --ci`. The skip message now says so and names the ways out: stage or stash the unstaged work; run `charter adf tidy --source <file>` and `git add` the result; or `git commit --no-verify`. When the gate then fails, one line says which skipped file still carries how many items. A new hook with an older local CLI that does not know `--print-paths` reports the files as clean and tidies nothing, so upgrade the CLI too.
 - **BREAKING: `charter adf compile --write` no longer converts pointer files without `--mode compiled`** (`#318`, part 2) — compile used to overwrite any Charter thin pointer with compiled output, silently changing its mode. Each target's mode now comes from the `VENDORS` record, or from its markers when there is no entry, and `--write` only writes targets that are compiled or absent. Under `--target all`, pointer and unmanaged targets are skipped (exit 0), and each `[skip]` line names the command that converts it. A single such `--target` is refused (exit 1). `--force` alone never changes a mode. `--mode compiled` records `compiled` before writing. If that cannot be recorded, nothing is written (`refused-record`). If the write then fails, the record is restored and the run continues. A converted file is reported as `Written CLAUDE.md (mode: pointer -> compiled)`. JSON output adds `skipped` and per-target `targets`. A compiled-only repo with no `VENDORS` section compiles as before.
 
@@ -28,6 +31,7 @@ The format is based on Keep a Changelog and follows Semantic Versioning.
 
 ### Fixed
 
+- **`adf compile --write` no longer deletes your `## Environment` section from a compiled file** (`#303`) — the `#295` guard only covered pointer files. Re-compiling a file that already carried the compile banner silently dropped a user-added `## Environment` (or protocol) section with exit 0, and git was the only way back. Those sections are now kept in the retained block.
 - **`charter bootstrap --force` no longer destroys a hand-authored `CLAUDE.md` before it can be migrated** (`#318`, part 3) — phase 3 overwrote every existing pointer-file name with the template under `--force`, before phase 4 (migrate) ran. Vendor files are not backed up (only `.ai/*.adf` go to `.ai/.backup/`), so phase 4 then found a pointer, migrated nothing, and the file's rules were gone with no `.pre-adf-migrate.bak`. The file is now skipped in phase 3, and with `--yes` phase 4 migrates it into `.ai/` and writes the backup (when it has no `VENDORS` entry).
 - **`charter adf compile --check` and `charter adf tidy` read each vendor file's mode** (`#318`, part 2; `#302`) — `--check` compared every target byte-for-byte against compiled output, so a thin pointer was reported `[stale]` forever. Each target is now checked by its mode. A compiled file is `fresh` or `stale` as before. A pointer is `pointer-ok` when `adf tidy` would extract nothing from it, and `pointer-bloated` (exit 1, remedy `charter adf tidy --source <file>`) when it would. An unmanaged file is `skipped-unmanaged` (exit 0). A recorded `compiled` or `pointer` file that looks like another mode on disk is `mode-drift` (exit 1), with both fixes printed: `charter adf vendor set <file> <mode>` to keep the file, or the command that restores it. A missing file is `missing`, and a recorded pointer that is missing names `git checkout -- <file>` or `adf vendor set <file> unmanaged`. No remedy suggests `compile --write` for a pointer or unmanaged file. JSON keeps `stale` (which now also lists bloated and drifted files), `current`, `missing`, `upToDate` and `nextActions`, and adds `skipped` and `targets`. `adf tidy` processes a file only when it is a pointer: no compile banner, at least one pointer marker, and not recorded otherwise. The banner still wins whatever the record says (`#296`). A file recorded `compiled` is skipped with status `compiled`. Its warning lists the content-preserving fix first (`adf vendor set <file> pointer`, then `adf tidy --source <file>`) and says how many items regenerating would discard; `--check` mode-drift does the same. An `unmanaged` file (recorded, or hand-authored with no marker) now has status `unmanaged` instead of `not-found`, and its warning appears when it is recorded or named with `--source`. A recorded pointer with no marker is skipped rather than guessed at. File rows gain `vendorFile`, `mode`, `source` and `skipReason`. A `--source` path that is not a repo-root vendor name keeps the old marker logic. Tidy now finds files through the shared discovery helper, so `AGENTS.md` and `agents.md` that are one file on a case-insensitive filesystem are walked once, not twice. With no `SYNC` entries, `charter adf sync` now points to `charter adf vendor status` in text and in JSON `nextActions`, rather than ending at "Wrote empty lock file". Record entry problems are still reported only by `adf vendor status`. `doctor`, `score`, `bootstrap` and both pre-commit hooks are unchanged and are later parts of `#318`.
 - **`charter doctor` now names the fix when `core.adf` or `state.adf` is missing from `DEFAULT_LOAD`** (`#241`) — the `adf required wiring` warning said only `Missing from DEFAULT_LOAD: core.adf`. It now says to move the module from `ON_DEMAND` to `DEFAULT_LOAD` in `.ai/manifest.adf` when it is registered there, or to add it (or run `charter adf create <module> --load default`) when it is not registered at all. The rule itself is unchanged and is filename-only — `core.adf` and `state.adf` are required in `DEFAULT_LOAD`, and module content is not consulted — but it now lives in one shared helper that `doctor` and `bootstrap`'s orphan registration both call, so `adf migrate` can adopt the same classification when it starts registering the modules it creates (`#239`).
@@ -42,7 +46,7 @@ The format is based on Keep a Changelog and follows Semantic Versioning.
 - **`charter <command> --help` now reaches the command's own docs** (`#240`) — the top-level dispatcher matched `--help`/`-h` anywhere in the raw arguments, so `charter adf migrate --help` printed the global help and the ADF subcommand reference was only reachable by running `charter adf` bare. `adf`, `hook`, `score` and `telemetry` now render their own help. The remaining commands have no `--help` handler and deliberately keep the root-help fallback, so a stray `--help` can never execute one for real.
 - **`adf populate` no longer invents an ESM extension rule for bundler projects** (`#294`) — the constraint "Use .js extensions for all ESM imports" was emitted whenever any `package.json` carried `"type": "module"`, which is true of every Vite, esbuild and modern Workers project. Under `moduleResolution: "bundler"` extensionless relative imports are the correct convention, so bootstrap was writing a wrong rule into `CONSTRAINTS [load-bearing]` and inviting agents to "fix" a correct codebase. The rule is now gated on TypeScript's `moduleResolution` — emitted for `node16`/`nodenext`, withheld for `bundler`, `node` and anything unrecognised — and a single bundler package vetoes it for the repository. Ambiguity (an unparseable JSONC `tsconfig.json`, a solution-style config, an `extends` chain this resolver does not follow) withholds the rule rather than guessing.
 - **`adf compile --write` no longer refuses the files `bootstrap` just generated** (`#295`) — the overwrite guard keyed on the compile banner alone, so a clean `bootstrap` followed by the documented `adf compile --target all --write` produced a wall of refusals and trained users to reach for `--force`. The guard now recognises Charter thin pointers as well as compiler output. Hand-authored files carrying no Charter marker are still refused, and a file converted from pointer to compiled output is reported as `(replaced charter pointer stub)` so the transition is visible.
-- **`adf compile --write` no longer discards retained sections** (`#295`) — it refuses, rather than silently overwriting, a pointer file carrying your own content under `## Environment` or another retained heading. Compiled output has no such section and `.ai/` does not hold that content, so the previous behavior lost it with git as the only recovery. Pass `--force` to discard deliberately.
+- **`adf compile --write` no longer discards retained sections** (`#295`) — it refuses, rather than silently overwriting, a pointer file carrying your own content under `## Environment` or another retained heading. Compiled output has no such section and `.ai/` does not hold that content, so the previous behavior lost it with git as the only recovery. Pass `--force` to discard deliberately. Since `#303`, the file is written with that content carried into a retained block instead (see Changed).
 - **`charter score` no longer counts directory-tree filenames as broken references** (`#293`) — a fenced tree block expresses the path prefix structurally, through indentation, so `random.ts` and `simulation/random.ts` were reported broken although `src/simulation/random.ts` exists. Grounding now falls back to matching a reference against the repository's real files before classifying it as broken. Two conditions must both hold: the basename is unique, and every path segment the author actually wrote agrees with the real file's trailing segments. The second is checked against the authored text rather than the resolved path, so a reference written inside `.ai/` still resolves, while a stale `src/utils/helper.ts` pointing at a file since moved to `packages/core/lib/helper.ts` stays broken — documentation rot is precisely what this check exists to surface. An ambiguous basename such as `index.ts` under several directories, and a filename that exists nowhere, also stay broken. On this repository the broken count falls from 79 to 48 and Grounding rises from 13/20 to 15/20, while genuinely absent files (`Cargo.toml`, `wrangler.toml`, `schema.sql`, `src/worker.ts`) are all still reported.
 
 ### Changed

README.md

@@ -255,7 +255,7 @@ For live session continuity snapshots, use `charter context-refresh` to produce
 | I want to compile task-specific agent context | `charter adf bundle --task "Fix the login flow"` |
 | I want to enforce metric/file-size ceilings | `charter adf evidence --auto-measure --ci` |
 | I want to migrate existing agent docs | `charter adf migrate --dry-run` |
-| I want to regenerate CLAUDE.md / AGENTS.md / .cursorrules / GEMINI.md from `.ai/` | `charter adf compile --target all --write` (add `--mode compiled` to convert pointer files) |
+| I want to regenerate CLAUDE.md / AGENTS.md / .cursorrules / GEMINI.md from `.ai/` | `charter adf compile --target all --write` (add `--mode compiled` to convert pointer files; your `## Environment` notes are kept in a retained block at the end) |
 | I want a letter-grade AI-readiness audit | `charter score` |
 | I want to validate governance in CI | `charter validate --ci --format json` |
 | I want to check pattern drift | `charter drift --ci --format json` |
@@ -288,7 +288,7 @@ charter adf init                         # Scaffold .ai/ directory
 charter adf bundle --task "..."          # Merge context for a task
 charter adf evidence --auto-measure      # Validate metric constraints
 charter adf migrate                      # Migrate existing configs
-charter adf compile --target all --write # Re-render compiled-mode CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md
+charter adf compile --target all --write # Re-render compiled-mode CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md (keeps your ## Environment notes)
 charter adf compile --target all --write --mode compiled # Switch pointer files to compiled output
 charter adf vendor status                # Show each vendor file's mode (pointer, compiled, unmanaged)
 charter adf vendor restore CLAUDE.md     # Rewrite one pointer from its template (refuses if content would be lost)