Skip to content

log-shipper: event codes for six node warnings; vendor canonical schema (TC-3006) - #606

Draft
samgbafa wants to merge 1 commit into
mainfrom
feat/tc-3006-node-event-codes
Draft

samgbafa wants to merge 1 commit into
mainfrom
feat/tc-3006-node-event-codes

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

Linear: TC-3006, with TC-2954, TC-2961 and TC-3027. Targets 1.26.7; node main is frozen for the 1.26.6 cut. Canonical schema: tinycloud-dev-sam #336 (stacked on #334).

Behaviour

The log sidecar classifies six node warnings that it used to ship as unclassified. Each ships as a per-minute count (event_code and event_count only), like every other classified diagnostic. None of its fields leaves the node.

node line (target) event code
"closing a pooled connection released inside a transaction" (tinycloud_core::db_pool) db_pool_closed_in_transaction
"Database exceeds the artifact read limit; not loaded" (tinycloud_core::sql::service) sql_database_too_large_to_load
"Database checkpoint save failed; refusing its writes until the back-off ends" (tinycloud_core::sql::service) sql_checkpoint_save_backoff
"kv head size probe failed; size omitted" (#599) kv_head_size_probe_failed
"chain guard wait exceeded the slow-wait threshold" (tinycloud::chain_guard) chain_guard_slow_wait
"request answered with a 5xx" (tinycloud::request) http_5xx
  • Pooled-connection backstop. The warn is emitted inside vendored sqlx-core, so its default target sqlx_core::pool::connection would never classify. It now carries target: "tinycloud_core::db_pool"; this is recorded in vendor/sqlx-core/PATCH.md. The shipper's target allowlist is unchanged.
  • chain_guard_slow_wait. It ships as a count, not as a capped class with its own errors_dropped. A per-minute cap does not fit the TC-1843 reserve: 585.4 MB is used of 600 MB, and even one event per minute adds ≈ 17.4 MB. The arithmetic is in tinycloud-dev-sam feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336. The node already limits the line to one per second per process.
  • http_5xx. It ships as a count. Each 5xx request already ships on its own under the node_request_5xx cap, so a separate event per 5xx would count it twice.

Schema and config

Tests

  • log-shipper/cases.mjs: each of the six lines maps to its code with only the six fixed keys, while extra fields and canaries are dropped. Near-misses stay unclassified: a suffix, a lowercase variant, a truncated message, and the backstop message on the old sqlx_core::pool::connection target.
  • Local runs:
    • node --test log-shipper/budget.test.mjs log-shipper/profile.test.mjs scripts/tc959-docker.test.mjs scripts/tc959-readiness.test.mjs: 41/41 pass.
    • node scripts/tc959-check.mjs passes.
    • Native Vector 0.59.0 (checksum-pinned):
      • vector test: errors_slow 281/281, all_requests 281/281, including the new cases.
      • native-check.mjs: passed, with 0 console canary hits and wire_safe.
      • cap-check.mjs: passed. The five caps are unchanged and there is no chain_guard_slow_wait cap.
      • vector validate on the rendered runtime config: Validated.
  • cargo check -p sqlx-core passes for the one-line vendor change.
  • The hosted compose proof runs in CI (logship workflow).

…point back-off, size probe, chain-guard wait and http_5xx; vendor canonical schema (TC-3006)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant