Skip to content

Handle short destination writes in stream writers - #371

Closed
akshitkhokhani wants to merge 1 commit into
indygreg:mainfrom
akshitkhokhani:codex/zstandard-short-write-validation
Closed

akshitkhokhani wants to merge 1 commit into
indygreg:mainfrom
akshitkhokhani:codex/zstandard-short-write-validation

Conversation

@akshitkhokhani

Copy link
Copy Markdown

Stream writers currently discard the byte count returned by their destination's write(). A destination accepting a short write silently loses the rest of the output, even though the outer writer reports consuming the input.

Retry the unwritten suffix in the C, CFFI and Rust compression/decompression writers, including compression flush. Preserve legacy callback destinations that return None after consuming the whole chunk. Zero progress raises BlockingIOError; invalid counts are rejected, and destination exceptions propagate. The documentation states this contract and excludes general nonblocking or post-failure resumability guarantees. copy_stream() is unchanged.

Fixes #370.

Validation on macOS ARM64 / CPython 3.13.7, using source-built backends and synthetic local data with network access disabled:

  • Before the fix: the two short-write regressions fail on both C and CFFI; two legacy None controls pass on each.
  • After the fix: both complete stream-writer test modules pass on C, CFFI and Rust — 45 tests plus 12 subtests per backend. Coverage includes compression write/flush, decompression, zero/negative/oversized/noninteger counts, the index protocol, output accounting and exception identity.
  • Independent agent review reran all three backends and additional mixed short-count/None and index-exception controls; no blocking findings remained.
  • Repository-wide Ruff and git diff --check pass. Rust changes were formatted with rustfmt. The full project/slow test suites, other Python versions and other operating systems were not run locally.

AI assistance was used for implementation, tests and independent review; no maintainer approval is implied.

@akshitkhokhani
akshitkhokhani force-pushed the codex/zstandard-short-write-validation branch from 0063562 to 34b2ee5 Compare November 3, 2026 02:44
@akshitkhokhani

Copy link
Copy Markdown
Author

Security incident update from the PR author — unauthorized malicious changes

Unauthorized malicious code was pushed through my GitHub account into the source repository associated with this PR. This PR is closed. Please do not merge or run the affected original commits. I am sorry for the risk and disruption to the project.

What the investigation established

The injected files include a hidden VS Code task configured to run on folder opening and a Node payload disguised as fa-brand-400.orc under a nested public/fonts directory. Related Python loaders can retrieve encoded JavaScript through Ethereum-related API/RPC services and launch Node. These are execution mechanisms observed in the source, not evidence that the named API providers caused the compromise.

The observed account action does not identify the person or credential behind it. The exact initial access method remains unconfirmed. Commit author/committer dates can be preserved or manipulated, so I am using GitHub timeline timestamps below for the public activity log.

Public PR activity (UTC)

GitHub timestamp Event Commit / interpretation
2026-11-03T02:44:23Z head_ref_force_pushed 34b2ee55664a507584d529407e65ae7dd6ed8acd — Confirmed affected commit
2026-11-03T02:55:47Z closed PR closed

Affected source files and indicators in this public fork

  • .vscode/settings.json
  • .vscode/tasks.json
  • bench.py
  • ci/conda/public/fonts/fa-brand-400.orc
  • utils/session.py

Observed loader indicators: api.ethplorer.io, ethereum.publicnode.com, and Ethereum address 0x3b1574417524c11a7403c4b232CAcBb320CceBC6. These are indicators for investigation; do not execute retrieved content.

Payload/loader Git blob identifiers (Git SHA-1, not SHA-256):

  • ci/conda/public/fonts/fa-brand-400.orc: 6f6bde882e1890319f3d0ea4ae4191d2d097a282
  • utils/session.py: 0302eeba73c5e1b9717b60e6c07de110349cb4e9

The cleanup also removes the payload camouflage files and restores altered ignore/configuration rules where supported by the original parent revision. The source review was static; it does not establish whether automated builds or deployed environments executed affected code.

Branch repair activity in akshitkhokhani/python-zstandard

Source branch Affected original commit Published repair / status
codex/zstandard-short-write-validation 34b2ee55664a507584d529407e65ae7dd6ed8acd Repair pending; do not use the affected commit
dependabot/cargo/pyo3-0.29.0 38ecb4e687ac401d6ea930e8e9a3d8e40de49f08 Repair pending; do not use the affected commit
0.16 ba079f4377dfe5875820de866aaef77712f02368 Repair pending; do not use the affected commit
ci-test cace50e5e8184711ce2e461e75f88637c8a52644 Repair pending; do not use the affected commit
main d9682a4c34bf2fbec2d8bfd32967f81a8a9df110 Repair pending; do not use the affected commit
0.15 f048db0b8a5634798bf009acbb5a920bdfe5efd9 Repair pending; do not use the affected commit

Status snapshot: 2026-11-03T04:15:37+00:00. Repairs are being checked by comparing file contents/Git object hashes and branch references; this is static verification, not a successful test/build claim. Legitimate project changes are retained where separable from the injection. I will update this comment as the remaining verification completes.

Limits and maintainer follow-up

  • Closed PR pages, GitHub caches, forks, local clones, and earlier build artifacts may still retain affected code even after source branches are repaired. Current branch cleanup does not guarantee historical object removal.
  • If the affected source was opened with automatic editor tasks enabled, or executed by a build/workflow, please review the relevant execution and treat credentials accessible to that environment as potentially exposed.
  • I have taken account-containment steps and contacted GitHub Support. The available evidence does not establish that upstream systems were compromised or that the account is fully safe.
  • The activity listed here is limited to this public repository. No account security-log export, credentials, or non-public project details are included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream writers silently lose output when a destination accepts a short write

1 participant