Skip to content

Migrate direct YAML parsing to the maintained fork - #618

Open
Pal Lakatos-Toth (pallakatos) wants to merge 7 commits into
mainfrom
pallakatos-yaml-dependency-migration
Open

Pal Lakatos-Toth (pallakatos) wants to merge 7 commits into
mainfrom
pallakatos-yaml-dependency-migration

Conversation

@pallakatos

@pallakatos Pal Lakatos-Toth (pallakatos) commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Partial progress toward removing unmaintained unsafe-libyaml; the dependency is not fully removed.

  • Migrate controller direct YAML parsing, including signed egress and AGT-profile structural validation, from serde_yaml to the YAML Organization's maintained yaml_serde 0.10.7, using Cargo's documented serde_yaml package alias. Migrate the router's policy-rule counting parser to the same fork. AGT policy loading, enforcement, and the router's AGT context retain legacy serde_yaml::Value: AGT 4.1's public policy API requires that exact type. No mesh/transport changes or compatibility shim.
  • Migrate the unpublished repository-only CNCF conformance tool to the same fork, preserving its YAML value model, stream API and literal tag/non-string-key behavior; its normal graph no longer contains serde_yaml/unsafe-libyaml. The public Rust Value type nevertheless has a new crate identity, so callers of this unpublished library must use the fork's type. Malformed manifests now fail explicitly instead of silently disappearing from the report. The previously evaluated serde-saphyr and its transitive packages are removed.
  • Differentially compare old and fork values, serialized bytes, errors and locations for shipped policy profiles, CRD runtime spec, 01, tags, boolean-like scalars, aliases, literal merge keys, duplicate keys, invalid UTF-8, malformed inputs and multi-document streams. Compare actual egress parsed endpoints, compiled canonical JSON bytes and digest against the legacy parse; verify tools profiles preserve signed raw bytes and digests. Existing canonical golden/negative and router policy-decision tests also pass.
  • Extend the router's yaml_compatibility_policy_counts_and_decisions_preserve_legacy_numeric_versions test using the real Governance::load_policies_from_dir and Governance::evaluate path. For both legacy unquoted version: 1 and version: 1.0, it asserts one rule loads, listed shell:rm is denied, and a different shell:ls action is allowed. This protects current pinned-AGT behavior only; it does not change production semantics or imply compatibility with a future AGT parser.
  • Conformance tests differentially compare all static CRDs and the complete Helm-rendered chart against legacy YAML values, plus 01, tags, aliases, merge keys and non-string keys. Installed Helm must render successfully; only an absent Helm skips the rendered corpus, while static coverage always runs.
  • Independent session-only probe against the YAML Organization's pinned yaml-test-suite v2022-01-17, with its visible whitespace markers decoded: 406 cases, 253 accepted by both and 153 rejected by both, with no value, serialization, error-message or error-location differences. This supplementary probe is not part of CI and is not proof of equivalence for all inputs.

Fork provenance and boundaries:

  • yaml_serde 0.10.7 declares Rust 1.82 and libyaml-rs 0.3.0 declares Rust 1.60. The fork graph uses existing indexmap, itoa, ryu, serde, plus new libyaml-rs (which has no normal dependencies). Compared to main, only yaml_serde and libyaml-rs are new; no existing package versions were upgraded. Scoped tests pass on Rust 1.88.
  • Published Rust source files were compared against the YAML Organization's yaml_serde 0.10.7 tag and libyaml-rs 0.3.0 tag: all 29 and 20 Rust files respectively have matching Git blob hashes. Provenance caveat: the published libyaml-rs archive embeds VCS revision 66d8ae3d348b328ef2159e3dc0bc53f1deb4c3a2, which upstream GitHub does not resolve, rather than release-tag commit e95b9a3e83bfb00728162c82e61e393c393929d1. The reproducible reconstruction below demonstrates identical release content but does not establish how or why the history differs. Crate archives are Cargo-registry checksum verified. The fork is still libyaml translated into unsafe Rust, not a memory-safety rewrite; libyaml-rs is an upstream-maintained fork of archived unsafe-libyaml and its current release history is limited.
  • A broader independent package-content comparison matched all 31 libyaml-rs 0.3.0 and all 40 yaml_serde 0.10.7 packaged source, original manifest, metadata, documentation and script files to their corresponding Git blobs at the public release tags. It excluded Cargo-generated .cargo-ok, .cargo_vcs_info.json, Cargo.lock and normalized Cargo.toml, and mapped Cargo.toml.orig to the tagged Cargo.toml. The normalized manifests explicitly disable build scripts; libyaml-rs has no normal dependencies. This strengthens file-content provenance only.
  • Independently reproduced in a scratch Git repository: git cat-file commit e95b9a3e83bfb00728162c82e61e393c393929d1 | sed 's/1773254475 -0700/1773254376 -0700/g' | git hash-object -t commit --stdin outputs exactly the embedded 66d8ae3d348b328ef2159e3dc0bc53f1deb4c3a2. Only the author/committer timestamps change from 2026-03-11T18:41:15Z to 18:39:36Z; tree ae0644733858560d7def952b0241593059ef1ca0, parent 243f22a105eda9a8612b9b4cbfc81f13a5714bad, identities and message Release 0.3.0 remain identical. The public crates.io index record records publication at 18:39:38Z and checksum 2e126dda6f34391ab7b444f9922055facc83c07a910da3eb16f1e4d9c45dc777, matching Cargo.lock. This explains how the embedded SHA can identify the same release tree; it is not proof of how/why history changed, verified publisher identity, a signed attestation, or current upstream availability of that object.
  • One actual behavior difference is documented by test: on a failing from_reader, the fork exposes the underlying I/O error via Error::source(), whereas the old crate returns no source. Display text is unchanged. The production paths changed here use from_str, not from_reader; tested malformed-input diagnostics and locations match.

Remaining paths and upstream blockers:

  • unsafe-libyaml 0.2.11 still comes through kube-client 3.1.0, AGT agentmesh 4.1.0 pin c3f75d508348a45df9dad7a929d1e137e1df80c0, and the router's direct serde_yaml required for AGT context values. Controller normal graph retains it through kube, not through its own parser. Retain both existing advisory exceptions; this is an informational unmaintained advisory, not a demonstrated exploit.
  • Released kube 4.2.0 requires Rust 1.89 and k8s-openapi 0.28 instead of Kars's 0.27. AGT v4.1.0 still uses serde_yaml; its merged migration targets pending Rust 5.0.0, Rust 1.89 and changed public policy types (migration contract). No unreleased upstream pin or restored mesh fork.
  • The declared workspace Rust 1.88 baseline was already broken before the latest rollup: prior AGT 4.0 also required 1.89 and aes 0.9.4 was already locked. This PR does not change the declared MSRV or upgrade any previously locked package. Scoped controller/conformance tests pass on 1.88, but the workspace check still fails on pre-existing dependencies requiring 1.89.

Related Issues

Relates to #595 (partial; not fixed or closed by this PR). Independent of #594 and #596. Do not merge as part of this work.

Type of Change

  • Bug fix
  • New feature
  • Security hardening
  • Documentation
  • CI/CD
  • Refactor

Checklist

  • Code compiles/builds without errors
  • Tests pass (make test)
  • Linting passes (make lint)
  • Documentation updated (if applicable)
  • No secrets committed
  • Helm chart updated (if K8s changes)

Testing

Local validation at final head 125a53588e3ea256c5a9ab270336b4d0635e5853:

  • cargo test --locked --offline -p kars-inference-router governance::tests::yaml_compatibility_policy_counts_and_decisions_preserve_legacy_numeric_versions — passed; both unquoted numeric versions exercise the real loader/evaluator and assert one loaded rule, deny for shell:rm, and allow for shell:ls.
  • cargo fmt --all -- --check, git diff --check, ./ci/check-loc.sh, and bash ci/security-audit-required.sh — passed; the test remains within the source LOC cap and no gates or production behavior changed.
  • cargo test --locked --offline --workspace --quiet — passed at prior head fa815df4, 2,238 tests, 3 documentation examples ignored.
  • cargo +1.88.0 test --offline -p kars-controller yaml_compatibility — passed, 9 tests. Previous runtime commit 54a93a57 also passed cargo +1.88.0 test --locked --offline -p kars-controller policy_canonical:: (102 tests) and cargo +1.88.0 test --locked --offline -p kars-controller helm_drift:: (18 tests); these production paths did not change subsequently.
  • cargo +1.88.0 test --offline -p kars-cncf-conformance — passed, 4 differential/negative tests and 17 criterion tests; Helm-rendered comparison ran.
  • cargo test --offline -p kars-inference-router governance::tests:: — passed, 19 tests including old/new rule-count comparison and AGT allow/deny decisions at runtime commit 54a93a57; router production code did not change subsequently.
  • cargo clippy --locked --offline -p kars-controller -p kars-inference-router -p kars-cncf-conformance --all-targets --all-features -- -D warnings — passed.
  • cargo deny check advisories bans licenses sources — passed with existing warnings; exceptions retained.
  • bash ci/check-copyright-headers.sh — passed.
  • cargo tree --locked --offline -p kars-cncf-conformance --edges normal and cargo tree --locked --offline -i unsafe-libyaml --edges normal — confirmed the fork-only conformance normal graph and remaining kube/AGT/router paths above.
  • cargo +1.88.0 check --locked --offline --workspace — fails on the pre-existing baseline: agentmesh/agentmesh-mcp, cedar-policy, aes and other already-locked dependencies require 1.89.

GitHub CI at previous head fa815df4ac24e025df47b326619b49559c9694fa: all 40 checks passed (CI run):

  • cargo build --release --workspace — passed.
  • cargo nextest run --workspace --release --no-fail-fast — passed, 2,238 tests, 0 skipped.
  • cargo clippy --all-targets --all-features -- -D warnings — passed.
  • make test-e2e — passed in Kind; chaos fault-injection checks also passed.
  • Same-runner benchmark comparison — passed, all 3 controller and all 3 router groups within the maximum +25% regression threshold versus baseline.
  • security-audit-required, dependency review, RustSec, cargo-deny, CodeQL, secret scanning and other repository gates — passed.

At superseded intermediate head 1e37e156, ci-gates/loc failed because the separate test pushed a 1,229-line source file over its 1,200-line cap. The test was consolidated into the existing compatibility test; the local LOC gate passes at final head 125a53588e3ea256c5a9ab270336b4d0635e5853. Final-head GitHub CI at 125a53588e3ea256c5a9ab270336b4d0635e5853: all 40 checks passed (run), including 2,238 release tests, Kind E2E (98 passed/0 failed), chaos checks, and all six benchmark groups within +25%. No prior-head CI result is presented as current-head validation.

Updated cross-PR integration evidence: git merge-tree for #618 at 125a53588e3ea256c5a9ab270336b4d0635e5853 and #620 at c47251ce14e1a0777caaf94c779a9c94404eab39 reports one Cargo.lock conflict in the controller dependency list; the isolated resolution retains both x509-cert and yaml_serde. Locked/offline metadata contains only crypto-bigint 0.7.11 (no 0.5.5). With CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_BUILD_JOBS=2, 45 targeted tests passed: 9 controller YAML, 3 signed OCI, 21 conformance, 10 OAuth compatibility, 1 preinstalled-provider, and 1 governance numeric-policy regression. Pairwise virtual merges with Headlamp head c9fe3f82 are clean. This is targeted evidence, not exhaustive compatibility; no actual branch was merged.

Full local make test, make lint, release build and E2E were not run; release build and Kind E2E ran in CI at the final head. AKS/live-customer validation was not run. #595 remains open until all unsafe-libyaml paths are removed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

PackageVersionScoreDetails
cargo/libyaml-rs 0.3.0 UnknownUnknown
cargo/yaml_serde 0.10.7 UnknownUnknown
cargo/serde_yaml >= 0.9.0, < 0.10.0 UnknownUnknown
cargo/yaml_serde 0.10.7 UnknownUnknown
cargo/yaml_serde 0.10.7 UnknownUnknown
cargo/serde_yaml >= 0.9.0, < 0.10.0 UnknownUnknown
cargo/yaml_serde 0.10.7 UnknownUnknown

Scanned Files

  • Cargo.lock
  • controller/Cargo.toml
  • inference-router/Cargo.toml
  • tests/cncf-conformance/Cargo.toml

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the repository conformance tool's normal serde_yaml dependency with serde-saphyr, preserving rendered/static Kubernetes manifest values and failing explicitly on invalid manifests. Keep legacy parsing as a test oracle only for this tool.\n\nRetain customer runtime parsers: differential tests reproduce leading-zero and tag incompatibilities that would make a blind substitution breaking. No AGT/kube pins, runtime parser code, MSRV or audit exceptions change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pallakatos Pal Lakatos-Toth (pallakatos) changed the title Test YAML compatibility before parser migration Migrate conformance YAML parser with differential compatibility checks Nov 12, 2026
Migrate controller YAML parsing and router rule counting to the YAML Organization fork, with differential coverage for signed-policy bytes, legacy scalars, tags, aliases, streams, errors and policy counts. Keep AGT's serde_yaml context type and all upstream dependencies unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pallakatos Pal Lakatos-Toth (pallakatos) changed the title Migrate conformance YAML parser with differential compatibility checks Migrate direct YAML parsing with compatibility coverage Nov 12, 2026
Restore the YAML value model and stream API while preserving fail-closed malformed input handling and differential tests. Remove the separate saphyr dependency and compare tags, non-string keys, scalars and rendered manifests against the legacy parser.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pallakatos Pal Lakatos-Toth (pallakatos) changed the title Migrate direct YAML parsing with compatibility coverage Migrate direct YAML parsing to the maintained fork Nov 12, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant