Skip to content

v1.7.5: read compressed Codex logs - #60

Merged
Anuj7411 merged 9 commits into
mainfrom
codex-zst
Oct 15, 2026
Merged

Anuj7411 merged 9 commits into
mainfrom
codex-zst

Conversation

@Anuj7411

Copy link
Copy Markdown
Owner

v1.7.5: read compressed Codex logs

Codex now compresses its older session logs, and Sipcode reads them.

Fixed

  • Compressed Codex logs are read. Current Codex compresses older session logs to rollout-*.jsonl.zst and keeps only the newest ones as plain .jsonl. Sipcode skipped the compressed ones, so it saw almost none of a Codex user's history: forecast said it needed more days, limits started again at 0%, and stats, why, today, trend, impact and regress missed most sessions. Sipcode now reads them exactly like plain logs, on Node.js 22.15 or newer (zstd ships in Node's own zlib, so there is no new dependency and still no network). A session with both a plain and a compressed log is read once, from the plain log. A damaged or cut-off compressed log is counted as unreadable, never a crash.
  • Older Node.js: a clear warning. On a Node.js without zstd the compressed logs are still counted and reported (see "Logs Sipcode could not read" below). JSON output is unchanged; with --json the note "N compressed Codex log(s) (.jsonl.zst) skipped: reading them needs Node.js 22.15 or newer (you have vX)." goes to stderr.
  • regress sandbox warning with --agent. The "could not be read" WARNING told the user to allow read access to both log folders, and with --agent codex it named an empty one ("allow read access to and ..."). It now names only the folders the run read.
  • regress --bug-report with nothing to report says so. With no finding, the flag used to print only the plain result, which Codex read as "no output". It now ends with "No bug report: there is no finding to report." The answer was and is always on stdout.

Changed

  • Logs Sipcode could not read are impossible to miss. When some log files cannot be read (compressed Codex logs on a Node.js older than 22.15, a damaged or unreadable file, a folder that cannot be listed), stats, today, forecast, trend, impact, why, receipt, drift, limits and regress now start their terminal output with one line per reason, for example "warning: 3 Codex log file(s) could not be read (compressed .jsonl.zst logs need Node.js 22.15 or newer; you have v22.11.0). Numbers below leave them out." It used to be a small note, often the last line and on stderr, and why, receipt and drift did not mention these logs at all. With --json the notes stay on stderr and the JSON is unchanged (stats keeps its warnings array). When every log is read, the output is byte for byte what it was. The four Codex plugin skills say this warning first in their answer.
  • Codex plugin: regress checks the Codex logs by default. A first regress over large Claude Code logs can take a minute or more (49 seconds on 5.4 GB on the maintainer's machine; about 1 second once the usage cache is warm), Codex stopped the command after about 10 seconds with empty output, and a sandbox that cannot write ~/.sipcode never warms the cache. When the user asks about Codex or names no tool, the regress skill now runs sipcode regress --bug-report --agent codex (Codex logs only, about 1 second). It reads the Claude Code logs only when the user asks about Claude Code, says first that the run can take a minute or more, and gives the command a timeout of at least 3 minutes. Empty output is never reported as a result.
  • README: Node.js 22.15 or newer is needed to read compressed Codex logs.

How it works

  • One helper reads every session log (readLogText in src/lib/fs.ts); Codex discovery decodes only line 1 of a compressed log, by streaming up to the first newline. The usage cache keeps working (path, size, mtime).
  • src/lib/zstd.ts walks the zstd frames before decoding, because Node decodes only the first frame of a multi-frame file and returns partial text with no error for a truncated one. Each frame's decoded size is checked against its header.
  • The unread-logs warning has one source (unreadWarnings in src/modules/agents/loadSessions.ts). Period commands collect it per section (SectionOutput.unread) and runSections prints every section's warnings above the banner and the section headers. why / receipt / drift wrap their output in UnreadOutput (src/modules/agents/latest.ts), which prints discovery's counts plus the logs the pick or drift's baseline could not read before the first line the command writes. limits warns on the "no readings" path too; regress puts the warnings above its report.

Notes

  • scripts/verify-counts.mjs (the independent counter) reads .jsonl.zst too, decoding it with node:zlib itself, so it cannot agree with a Sipcode that skips them.
  • Test count: 2,225 to 2,409 (npx vitest run, which includes the 20 e2e tests).

Verification

  • npm run lint, npm run build: pass. npx vitest run: 2,409 passed, twice in a row. npm run test:e2e: 20 passed.
  • New tests: a Codex home laid out plain, mixed, all compressed, and with both forms gives byte-identical output for stats, today, forecast, limits, why (newest, --session, a resumed log), receipt, drift and regress, JSON and terminal. Per reader: discovery line 1, full parse, key scan of logs outside a window, usage cache cold and warm, damaged and truncated files, and a Node.js without zstd (injected, not by removing zlib). Reverting any one reader to a plain read fails these tests.
  • Unread-logs warning: a golden file recorded before the change proves every command's output (stats, today, forecast, trend, impact, why, receipt, drift, limits, regress; terminal and JSON; both tools and each alone) is byte-identical when every log is read. The same 58 cases run with two compressed logs skipped (Node.js without zstd) and with two damaged ones: in the terminal the warning is line 1 with the exact count, followed by the unchanged output, and no stderr note; with --json stdout is the same JSON and the note is on stderr. Also covered: limits with every log skipped (warning before "no readings"), why --agent codex with every log skipped (warning before the error), a skipped and an unreadable log in one run, the regress formatter, runSections and UnreadOutput.
  • Codex plugin: the regress skill's first command is sipcode regress --bug-report --agent codex; the Claude Code commands are listed with the "only when the user explicitly asks" rule; every skill says the warning first. Every command a skill names exists in src and dist.
  • Real data, unread-logs warning (a copy of the maintainer's Codex logs in a scratch folder, scratch HOME; the real folders untouched): with one extra damaged .jsonl.zst in the copy, stats --agent codex starts with "warning: 1 Codex log file(s) could not be read (unreadable or damaged). Numbers below leave them out.", and today, forecast, trend, impact, why, receipt, drift, limits and regress do the same; stats --agent codex --json keeps the JSON (its warnings array has the E003 entry). Without the extra file there is no warning line.
  • Real data, regress --bug-report --agent codex (same copy: 17 Codex logs, 15 compressed; fresh scratch HOME): 1.8 s cold, 1.1 s warm, exit 0; "Codex: 951 requests in 17 sessions, 2026-03-28 to 2026-10-15, versions 0.118.0 to 0.162.0, 4 version/model groups with enough limit data", then "No regressions detected in your logs." and "No bug report: there is no finding to report."
  • Real data (frozen copy of the maintainer's logs, scratch HOME): verify-counts all checks match, exit 0. Codex: 17 logs (15 compressed), 951 requests, 89,451,779 tokens, $56.66; every log equals Codex's own final total; limits again shows the possible cut around Apr 13 (5.1x); forecast has 14 days of history.

🤖 Generated with Claude Code

Anuj7411 and others added 9 commits October 15, 2026 19:16
Codex now compresses older rollouts to rollout-*.jsonl.zst. Add
src/lib/zstd.ts (node:zlib only): it walks the frames first, because
Node decodes only the first frame of a multi-frame file and returns
partial text without an error for a truncated one. A file is decoded
only when every frame is complete, and each frame is checked against
its declared content size. Line 1 can be read by streaming up to the
first newline.

The FileSystem seam gains readBytes (InMemoryFs.writeBytes stores
compressed fixtures), and readLogText / readCompressedLogFirstLine are
the one way to read a session log, plain or compressed. On a Node.js
without zstd they throw ZstdUnavailableError naming the version needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t is read

Current Codex compresses older rollouts to rollout-*.jsonl.zst, and
Sipcode skipped them, so a Codex user saw almost no history: forecast
had 0 days, limits restarted at 0%, why / stats / regress missed most
sessions.

Discovery now lists compressed rollouts (line 1 decoded by streaming up
to the first newline), and every session-log reader goes through
readLogText: the full parse, the request-key scan of files outside a
window, dropCopiedRequests (why / receipt), the limit-reading scan,
latest-session pick, drift. The usage cache stores a compressed log
once it decodes (it is complete by then), keyed by path, size, mtime.

One log per rollout name: when X.jsonl and X.jsonl.zst both exist, in
either folder, the plain one is read (Codex may still append to it);
between two of the same kind sessions/ wins, as before.

On a Node.js without zstd the compressed logs are still counted, with
the note: N compressed Codex log(s) (.jsonl.zst) skipped: reading them
needs Node.js 22.15 or newer (you have vX). A damaged or truncated .zst
is an unreadable file, never a crash.

Tests: a Codex home with plain, mixed, all-compressed and both-forms
layouts gives byte-identical stats / today / forecast / limits / why /
receipt / drift / regress output, JSON and terminal; per-reader tests
for discovery, parse, key scan, usage cache, damaged files and a Node
without zstd (injected, not by removing zlib).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ong first run

Investigated Codex reporting that `sipcode regress --bug-report` returned
no output. Sipcode itself always prints the result to stdout and exits
0 (or 2 when nothing could be read), with nothing on stderr. Two causes
were found from the Codex session's tool-call metadata:

- With no finding, --bug-report printed exactly the plain result and no
  word about the bug report it was asked for. It now ends with: No bug
  report: there is no finding to report. (every state: no regression,
  not enough data, nothing read).
- Codex's script runner stopped the command after about 10 s with empty
  output. A cold run over large Claude Code logs takes much longer (49 s
  measured on 5.4 GB; 1 s warm), and a sandbox that cannot write
  ~/.sipcode never warms the cache. The regress skill now asks for a
  timeout of at least 3 minutes and never to report empty output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ependently

The independent counter skipped .jsonl.zst like Sipcode did, so it would
have agreed with a Sipcode that misses most Codex history. It now walks
the zstd frames itself and decodes each with node:zlib (Node decodes
only the first frame), checks each frame's declared size, and picks one
log per rollout name (the plain .jsonl when both exist). It prints how
many rollouts were compressed.

On a frozen copy of the maintainer's logs: 17 Codex logs (15
compressed), 89,451,779 tokens, $56.66, all checks match, exit 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Version 1.7.5 in package.json, package-lock.json, server.json (both
fields), the plugin manifests, the site Hero and Footer, llms.txt.
CHANGELOG [1.7.5]. README and llms-full.txt: Node.js 22.15 or newer
reads the session logs Codex compresses. Test count 2,225 to 2,275
(npx vitest run, e2e included).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside Codex, the script runner stopped `sipcode regress --bug-report`
after about 10 s. A cold run over every Claude Code log takes about 49 s
(5.4 GB measured), and a sandbox that cannot write ~/.sipcode never warms
the usage cache, so the default run could never finish there.

The sipcode-regress skill now runs `sipcode regress --bug-report --agent
codex` (Codex logs only, about 1 s) when the user asks about Codex or
names no tool. The Claude Code check (`--agent claude-code`, or no
--agent for both tools) runs only when the user explicitly asks about
Claude Code; the skill then says the first run can take a minute or more
and keeps the 3-minute timeout. Every other rule is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Skipped compressed Codex logs (Node.js without zstd), damaged or
unreadable log files and unlistable folders used to be small notes,
often the last line and on stderr; why, receipt and drift did not
mention them at all. A report could look complete while leaving most of
a user's history out.

In the terminal, stats, today, forecast, trend, impact, why, receipt,
drift, limits and regress now start stdout with one line per reason,
with the exact count and the tool's name, then a blank line:

  warning: N Codex log file(s) could not be read (compressed .jsonl.zst
  logs need Node.js 22.15 or newer; you have vX). Numbers below leave
  them out.
  warning: N Codex log file(s) could not be read (unreadable or
  damaged). Numbers below leave them out.
  warning: N Codex log folder(s) could not be read (the folder could
  not be listed). Numbers below leave them out.

- Period commands: SectionOutput.unread collects the lines and
  runSections prints every section's warnings above the banner and the
  section headers (all sections now run before anything is printed).
- why / receipt / drift: UnreadOutput prints discovery's counts, plus
  logs the pick or drift's baseline could not read or parse, before the
  first line the command writes, so a "no sessions found" caused by
  skipped logs carries the warning too.
- limits: also when every log was skipped and there are no readings.
- regress: the warnings lead the report and replace the compressed-log
  note under the coverage lines; its sandbox WARNING now names only the
  folders the run read (with --agent codex it said "allow read access
  to  and ...").
- --json: the notes stay on stderr (named by tool when regress or a
  --session lookup covers both tools); JSON on stdout is unchanged and
  stats keeps its warnings array.

When every log is read, every command's output is byte-identical to
before: a golden file recorded before this change covers stats, today,
forecast, trend, impact, why, receipt, drift, limits and regress,
terminal and JSON, both tools and each alone.

The four Codex plugin skills now say this warning first in the answer
(for --json, the stderr note).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ess default

CHANGELOG [1.7.5]: the warning at the top of the output for logs Sipcode
could not read, the regress skill's Codex default, and the regress
sandbox WARNING that named an empty folder with --agent. README and
llms-full.txt: an older Node.js gets a warning at the top of each
report. Test count 2,275 to 2,409 (npx vitest run, e2e included) in the
README badge, site Hero and Footer, llms.txt and llms-full.txt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receipt rows print the path they wrote: C:/ on this machine, D:/ on
the CI Windows runner, / on Linux. The golden output now holds <cwd>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Anuj7411
Anuj7411 merged commit 586e449 into main Oct 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant