[sroussey]

Fixes #388, #389, #390, #391, #393, #394 and #395, removes the sec-base binary (#386 closed as not planned), and makes check-fixtures pass against live EDGAR again. One commit per area, plus one for review fixes.

Breaking: sec-base removed (212afbf)

sec-base was the generic Workglow CLI with a curated list of sec's tasks. It booted sec's whole runtime before parsing arguments, so --help died unconfigured (#386).

• Removed:
  • the bin entry and its build scripts;
  • src/libs-cli.ts;
  • dist/libs-cli.js from the required tarball contents;
  • the registerSecTasks list, which nothing else read;
  • the plain format: "cik" field widget, which only task run's forms rendered. sec's commands use sec:cik.
• Breaking marker: the commit is refactor!: with a BREAKING CHANGE footer, so the release bump reflects the lost bin entry.
• smoke-bins: now covers the package's only binary, dist/sec.js.

#388: console writes under the live progress frame (64011bf)

The Ink frame erases itself by line count. A console.* write from inside a run shifts that count, so the tail of the message gets overwritten.

• New helper. src/util/diagnostics.ts adds reportDiagnostic / reportDryRun, ported from embarc-data. Messages are held while a run is open and printed once it settles, capped at 200 lines with both ends kept. Outside a run they go straight to console.warn. runWorkflowCli wraps every run in it.
• Conversions. Every console.* call outside src/cli/ is converted (43 under task/storage/sec, plus the ones in src/config):
  • Failures and summaries go through reportDiagnostic.
  • "Would …" lines go through reportDryRun.
  • Bulk-download progress goes through the task's updateProgress.
  • The console.error before a rethrow is removed.
  • sec setup prints its "Wrote …" lines itself, from the task's output, after the run.
• unzip. The bulk-archive extraction no longer sends its per-file listing (up to ~1M lines) to the terminal. Its stderr is read and included in the error message.
• Duplicate CIKs. The Postgres cik_names writer reports one total of collapsed duplicate CIKs instead of a line per 30,000-row slice.
• Guards.
  • consoleDiagnostics.test.ts pins the console-write count per file over every tree except src/cli/, exact in both directions. The only allowed write is the helper's own output.
  • runWorkflow.diagnostics.test.ts sends a failing CIK through fetchAndStoreSubmission inside runWorkflowCli. It checks that nothing reaches the console while the frame is up and that the CIK's message appears after it closes. It fails if runWorkflowCli stops deferring (checked by reverting the wrapper).

#389 and #390: one gate list, pinned Bun (3bb86ef)

• One gate list. scripts/releaseChecks.ts is the single list of checks, and both release-checks and CI run it: format-check → lint → typecheck → build → prepack-check → smoke-bins → test. Under GitHub Actions each gate gets its own log group, and a failure is an annotation naming the gate.
• smoke-bins. Starts the built binary with no SEC_* environment, no .env.local and an empty working directory, and requires --help and --version to exit 0.
• prepack-check. Now refuses any src/ path in the tarball, not just mock_data (checked by adding one source file to files).
• Bun pin. Bun is pinned once, as packageManager: "bun@1.4.3", and setup-bun reads it with bun-version-file: package.json instead of installing latest. toolchainPins.test.ts fails if engines.bun, @types/bun, the workflow or the gate list disagree with the pin, or if a gate is restated as its own workflow step.
• bun-latest.yml. Runs the same gates weekly on the newest Bun and blocks nothing.
• README and .claude/CLAUDE.md name the pinned version and the gate list.

#391: coverage test timeout (ff15b40)

"Loses no content on committed fixtures" is now one it.each case per fixture.

• On an idle machine the four together took ~5 s, the largest 2.7 s. No single case comes near the 30 s limit.
• A slow case or a lost-content failure now names its fixture.

#394: XBRL value_numeric float error (5b0d2e1)

• Reproduced. 0.07 at scale 2 gave 7.000000000000001, 0.57 at scale −2 gave 0.005699999999999999, and empty or whitespace text gave 0. Across 357,150 two-decimal (value, scale) pairs, 36,873 were off before the fix and 0 are off after it.
• Fix. normalizeFactValue folds the scale into the literal's exponent and parses once, so the result is exactly Number(text + "e" + scale).
  • Numeric text must be a decimal literal, optionally with an exponent. "", whitespace, 0x10 and Infinity are now null.
• Tests. parseXbrlCore.test.ts pins the first failing values, the full sweep, the sign, an existing exponent, and the null cases.

#395: unsafe primary_doc spliced into the sec.gov URL (00e90dd)

• Sanitizer. sanitizePrimaryDoc also refuses ?, #, % and control characters (in a URL those mean something other than a filename); safePrimaryDocName is its predicate form.
• Task boundary. SecFetchAccessionDocTask validates fileName in inputToUrl / inputToFileName and percent-encodes the segment, so a refused name throws a TaskError before anything is queued.
• Conversion. conversionCandidates drops an unsafe name, and conversionFetchFileName returns undefined for a primary-document form. The task then fetches nothing and reports the filing.
  • A form read as the whole submission (8-K and the rest) still fetches its .txt, which never uses the name.
  • A filing naming no primary document still falls back to the .txt.
• sec read --fetch. loadFilingHtml refuses an unsafe name with a message saying so.
• Tests. Cover ../../x, a?b and a#b at the sanitizer, the task constructor, inputToUrl, the conversion loader (asserting no context.own, i.e. no fetch task created) and loadFilingHtml.

#393: fixture commands calling fetch() directly (787113f)

• Queued fetches. Both fixture sources fetch through SecFetchTask via a new edgarFixtureFetch.ts. That gives them the queue's caps, the cluster limiter, the 429 retry and SecUserAgent, which trims an empty value and falls back to the compiled-in default. SecFetchTask has no output cache, so the golden audit still sees live bytes.
• Failures surface. The S-1 feeds no longer swallow errors into "". A filer skipped during enrichment is logged with the reason. The hand-rolled sleeps are gone, since the queue paces requests.
• Queue start. startSecJobQueue() registers and starts the fetch queue. bootstrapSecRuntime calls it, and so does scripts/checkFixtures.ts, which has no CLI boot. Without it, FetchUrlTask would build a default queue with none of sec's limits. The script's rate limiter is in-process, since there is no database to share one through.
• Guard. rawFetch.test.ts fails on any bare fetch( under src/ or scripts/, and on any read of SEC_USER_AGENT outside Constants.ts.

check-fixtures: 41 of 43 reported remote-changed (ba97ffc)

• Cause. The filings are byte-identical. sec.gov's CDN rewrites every HTML response to add an empty sensor <script> with a root-relative random path just before </body> (or </BODY>). The response carries x-akamai-transformed and Akamai Bot Manager's _abck / bm_sz cookies.
  • EDGAR refuses active content in a submission, so a script there is never part of the filing.
  • Removing that one tag reproduces the manifest digest for all 41; the other 2 are served untouched.
• Fix. stripCdnInjection removes exactly that tag, matched as served and only before the closing body tag. It works through a latin1 round trip that keeps every other byte. The audit applies it before hashing in both verify and download modes.
• Result. bun run check-fixtures against live EDGAR now reports ok=43, failed=0.
• Pipeline. Unaffected: cached documents also carry the tag, but the HTML parser already drops <script>.
• Tests. Cover the stripping (either case, non-UTF-8 bytes preserved, scripts that aren't the CDN's left alone) and both audit modes over injected responses. The mode test fails without the fix (checked).

Review fixes (759cdbc)

• sec fetch fixtures --list. Its rows used to be written to stdout from inside the run, under the live frame. They are now a listed output that the command prints after the run.
• Write guard. consoleDiagnostics.test.ts also counts process.stdout.write / process.stderr.write. The fixture loggers default to reportDiagnostic, and the output-dir line goes through the injected log.
• S-1 fixture source. The filer-written primaryDocument goes through safePrimaryDocName and is percent-encoded, as on the conversion paths.
• Queue registration. startSecJobQueue throws when a different queue already holds the name (FetchUrlTask's default queue, created by a fetch that ran first), instead of silently leaving it in place.
• Unknown forms. An unknown SEC form is reported once per process rather than once per row.
• Smaller fixes:
  • the SecFetchAccessionDocTask comment now says where a refused name actually fails;
  • two checks in sanitizePrimaryDoc that URL_UNSAFE already covers are gone;
  • two tests follow the T | undefined convention;
  • the smoke-bins report no longer prints nullnull when a spawn fails.

Still open, by design for now:

• Diagnostics are also held when nothing is drawn: piped output, --json, and console child processes.
• Dry-run lines share the 200-line cap.

Both are pinned by tests, so changing either is a behaviour decision, left for a follow-up.

Testing

• bun run release-checks passes locally on the current head, every gate, including the full suite: 1585 passed, 21 skipped.
• The tarball is now 5 files, 648 KB.
• src/verify/coverage.test.ts "loses no content on committed fixtures" times out at 30 s under full-suite load #391's five consecutive loaded full-suite runs were not repeated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FdP6Nwc3vwqfC3vq8urgqT