diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 53131a3b..b3de36e7 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -764,11 +764,12 @@ jobs: # TC-959: main's N0 compose (#349 c33c2302…) with the 95m ingress body # cap (raw d95bfb70…), the TC-1274 + TC-1317 + TC-1482 + c_index + TC-1840 + TC-1843 Vector config and the # optional TC-1482 NODE_STATS_READ_TOKEN env line, the TC-950 pool of 10 and the TC-1777 named-keys gate - # env lines + deletion-ledger volume, the TC-2346 operation-retry warning codes and the TC-2917 - # storage-headroom event in the Vector config - # (raw 5419a6fb…; d703d69b… before TC-2917, 298ea356… before TC-2346, 192dfb7e… before TC-1777), - # its single node digest normalized to 64 zeros. - n0_template_hash = '90841a34e6ea5b581374f50abe70e580496e6814bc83f61f20fd4c9e090e5d51' + # env lines + deletion-ledger volume, the TC-2346 operation-retry warning codes, the TC-2917 + # storage-headroom event and the TC-3006 codes (canonical schema; chain_guard_slow_wait, http_5xx and + # the TC-2954/2961/3027 warning codes) in the Vector config + # (raw 7ee085c6…; 5419a6fb… before TC-3006, d703d69b… before TC-2917, 298ea356… before TC-2346, + # 192dfb7e… before TC-1777), its single node digest normalized to 64 zeros. + n0_template_hash = 'efad41d56df187a7e0ddd1a13f3c4a3c436ad7c0ff141acd32c06e28f9c92a34' node_pin = re.compile(rb'(?m)^ image: ghcr\.io/tinycloudlabs/tinycloud-node@sha256:([0-9a-f]{64})$') def n0_template(raw): pins = list(node_pin.finditer(raw)) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a25f1b9..f131c419 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,16 @@ - Share-email issuer trust bundles can pin one next key for the same issuer DID during an explicit UTC overlap window. The legacy key stops at `issuerNotAfter`; malformed, duplicate, unbounded or expired overlaps fail startup and `--validate-config`. Existing single-key bundles remain unchanged. This is node-side trust support only: the OpenCredentials witness and js-sdk policy contract must also support key rotation before activation. Older node images reject overlap fields; restore the previous single-key bundle before rollback. See `deploy/share-email/production-trust-bundle-contract.md`. - Disk headroom is visible without a shell on the node. Every 5 minutes the node logs a `storage headroom` event per local store, with only `store` (`sql`, and `duckdb` in DuckDB builds) and `free_bytes` (the free bytes on that store's filesystem, rounded down to a whole 64 MiB). The log sidecar ships it to Better Stack as `node_storage_headroom`. The exact value is served as the gauge `tinycloud_storage_free_bytes{store}` on the metrics endpoint, and `tinycloud_disk_headroom_refusals_total{engine="sql"}` counts requests answered `507 sql_disk_headroom`. Details: TC-2917. - **N0 compose.** The embedded Vector configuration is regenerated for the new event, so the reviewed N0 compose template changes to `90841a34…` (raw `5419a6fb…`; `docs/releasing.md`, `.github/workflows/docker.yml`). +- The log sidecar ships six more node warnings to Better Stack as event codes, each a per-minute count of `node_warning` events with no other fields. Until now they shipped as `unclassified`. + - `db_pool_closed_in_transaction`: a pooled connection was released inside a transaction and closed. + - `sql_database_too_large_to_load`: a database over the artifact read limit was not loaded. + - `sql_checkpoint_save_backoff`: a database's writes are refused during its checkpoint back-off. + - `kv_head_size_probe_failed`: the KV head size probe failed. + - `chain_guard_slow_wait`: a chain-guard wait over 500 ms. + - `http_5xx`: the line every 5xx response leaves. Each 5xx request still ships on its own under the `node_request_5xx` cap. + + The vendored log schema now matches the canonical schema, pinned in `log-shipper/canonical.json`. `chain_guard_slow_wait` is no longer a capped class there, since a per-minute cap for it does not fit the TC-1843 reserve. The pooled-connection warning (vendored sqlx-core) now logs under the target `tinycloud_core::db_pool` (`vendor/sqlx-core/PATCH.md`). Details: TC-3006, TC-2954, TC-2961, TC-3027. + - **N0 compose.** The embedded Vector configuration is regenerated for these codes, so the reviewed N0 compose template changes to `efad41d5…` (raw `7ee085c6…`; `docs/releasing.md`, `.github/workflows/docker.yml`). ## [1.26.5] - 2026-11-13 diff --git a/docker-compose.dstack-postgres.yaml b/docker-compose.dstack-postgres.yaml index e2d3f294..dda8ef86 100644 --- a/docker-compose.dstack-postgres.yaml +++ b/docker-compose.dstack-postgres.yaml @@ -174,7 +174,7 @@ services: environment: TINYCLOUD_LOGSHIP_PROFILE: ${TINYCLOUD_LOGSHIP_PROFILE:-errors_slow} BETTERSTACK_SOURCE_TOKEN: ${BETTERSTACK_SOURCE_TOKEN:?seal the node source token before enabling log shipping} - VECTOR_CONFIG_JSON: "{\"data_dir\":\"/var/lib/vector\",\"sources\":{\"gelf_in\":{\"type\":\"socket\",\"mode\":\"udp\",\"address\":\"0.0.0.0:12201\",\"max_length\":262144,\"receive_buffer_bytes\":8388608,\"framing\":{\"method\":\"chunked_gelf\",\"chunked_gelf\":{\"decompression\":\"Auto\"}},\"decoding\":{\"codec\":\"bytes\"}}},\"transforms\":{\"gelf_envelope\":{\"type\":\"remap\",\"inputs\":[\"gelf_in\"],\"source\":\"packet = string(.message) ?? \\\"\\\"\\nif length(packet) > 262144 {\\n . = {\\\"reason\\\": \\\"oversized\\\"}\\n} else {\\n outer, err = parse_json(packet)\\n if err != null || !is_object(outer) || (outer.version != \\\"1.1\\\" && outer.version != \\\"1.0\\\") {\\n . = {\\\"reason\\\": \\\"malformed_envelope\\\"}\\n } else if exists(outer.partial_message) || exists(outer._partial_message) {\\n . = {\\\"reason\\\": \\\"partial\\\"}\\n } else {\\n label = get(outer, [\\\"_tc.logship\\\"]) ?? null\\n . = {\\\"_tc.logship\\\": label, \\\"message\\\": outer.short_message}\\n }\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"logship_gate\":{\"type\":\"remap\",\"inputs\":[\"gelf_envelope\"],\"source\":\"if exists(.reason) {\\n . = {\\\"reason\\\": .reason}\\n} else {\\nlabel = string(get(., [\\\"_tc.logship\\\"]) ?? null) ?? \\\"\\\"\\nraw = .message\\nreason = \\\"\\\"\\nif label == \\\"\\\" { reason = \\\"unlabelled\\\" }\\nelse if !includes([\\\"tinycloud-node\\\", \\\"tinycloud-ingress\\\"], label) { reason = \\\"unexpected_label\\\" }\\nelse if exists(.partial_message) { reason = \\\"partial\\\" }\\nelse if !is_string(raw) { reason = \\\"malformed_envelope\\\" }\\nelse if length(string!(raw)) > 65536 { reason = \\\"oversized\\\" }\\nif reason != \\\"\\\" {\\n . = {\\\"reason\\\": reason}\\n} else {\\n parsed, err = parse_json(string!(raw))\\n if err != null || !is_object(parsed) { . = {\\\"reason\\\": \\\"non_json\\\"} }\\n else if exists(parsed.partial_message) { . = {\\\"reason\\\": \\\"partial\\\"} }\\n else {\\n target = string(parsed.target) ?? \\\"\\\"\\n level = downcase(string(parsed.level) ?? \\\"\\\")\\n if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && target != \\\"tinycloud\\\" && !starts_with(target, \\\"tinycloud::\\\") && target != \\\"tinycloud_core\\\" && !starts_with(target, \\\"tinycloud_core::\\\") {\\n . = {\\\"reason\\\": \\\"unknown_target\\\"}\\n } else if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && !((target == \\\"tinycloud::request\\\" || target == \\\"tinycloud::disk_headroom\\\") && level == \\\"info\\\") {\\n . = {\\\"reason\\\": \\\"filtered_level\\\"}\\n } else { . = {\\\"label\\\": label, \\\"parsed\\\": parsed} }\\n }\\n}\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_route\":{\"type\":\"route\",\"inputs\":[\"logship_gate\"],\"route\":{\"candidate\":\"exists(.parsed)\",\"rejected\":\"exists(.reason)\"}},\"allowlist\":{\"type\":\"remap\",\"inputs\":[\"gate_route.candidate\"],\"source\":\"# Only this fresh object can leave the process boundary. No envelope, span,\\n# message, error, identity, URI or unknown key is copied into it.\\nj = .parsed\\nlabel = .label\\nout = {}\\nts = \\\"\\\"\\nif label == \\\"tinycloud-node\\\" {\\n target = string(j.target) ?? \\\"\\\"\\n level = downcase(string!(j.level))\\n f = object(j.fields) ?? {}\\n msg = string(f.message) ?? \\\"\\\"\\n ts = string!(j.timestamp)\\n # Literal Rocket template aliases only; never split an arbitrary URI.\\n aliases = {\\\"/attestation?\\\":\\\"/attestation\\\",\\\"/.well-known/encryption/network/?\\\":\\\"/.well-known/encryption/network/\\\",\\\"/hooks/events?\\\":\\\"/hooks/events\\\",\\\"/hooks/webhooks?\\\":\\\"/hooks/webhooks\\\",\\\"/public//kv?\\\":\\\"/public//kv\\\"}\\n # TC-1840: the node's one 5xx diagnostic line per database failure ships as an individual event.\\n # The input must carry exactly the declared keys (the message plus these seven, and db_hash only\\n # when it is a valid pseudonym) with exactly the declared types and enum values. Anything else\\n # leaves db_event null and the line falls back to the unclassified count below; no value of it\\n # is ever copied unless it passed, and the shipped object is built from literals and the\\n # validated values only.\\n db_event = null\\n if target == \\\"tinycloud::request\\\" && level == \\\"warn\\\" && msg == \\\"database failure answered with a 5xx\\\" {\\n db_status = f.status\\n db_elapsed = f.elapsed_ms\\n db_code = string(f.code) ?? \\\"\\\"\\n db_service = string(f.service) ?? \\\"\\\"\\n db_kind = string(f.failure_kind) ?? \\\"\\\"\\n db_route = string(f.route) ?? \\\"\\\"\\n db_canonical_route = get(aliases, [db_route]) ?? null\\n if is_string(db_canonical_route) { db_route = db_canonical_route }\\n db_ok = is_integer(db_status) && is_integer(db_elapsed) && is_boolean(f.write_may_have_applied)\\n if db_ok {\\n db_status = int!(db_status)\\n db_elapsed = int!(db_elapsed)\\n db_ok = db_status >= 500 && db_status <= 599 && db_elapsed >= 0 && db_elapsed <= 9007199254740991 && includes([\\\"unavailable_retryable\\\",\\\"outcome_unknown\\\",\\\"internal\\\",\\\"sql_busy\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\"], db_code) && includes([\\\"sql\\\",\\\"duckdb\\\",\\\"kv\\\",\\\"other\\\"], db_service) && includes([\\\"pool_timeout\\\",\\\"pool_closed\\\",\\\"io\\\",\\\"statement_timeout\\\",\\\"serialization\\\",\\\"connection_lost\\\",\\\"artifact_changed\\\",\\\"lock_wait\\\",\\\"hydration\\\",\\\"artifact_transfer\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\",\\\"other\\\"], db_kind) && includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], db_route)\\n }\\n db_hash = string(f.db_hash) ?? \\\"\\\"\\n has_db_hash = exists(f.db_hash)\\n if db_ok {\\n if has_db_hash { db_ok = match(db_hash, r'^[0-9a-f]{64}$') && length(f) == 9 } else { db_ok = length(f) == 8 }\\n }\\n if db_ok {\\n db_event = {\\\"event\\\": \\\"node_warning\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"warn\\\", \\\"event_code\\\": \\\"db_failure_5xx\\\", \\\"status\\\": db_status, \\\"code\\\": db_code, \\\"service\\\": db_service, \\\"route\\\": db_route, \\\"elapsed_ms\\\": db_elapsed, \\\"failure_kind\\\": db_kind, \\\"write_may_have_applied\\\": bool!(f.write_may_have_applied)}\\n if has_db_hash { db_event.db_hash = db_hash }\\n }\\n }\\n # TC-2917: the node's periodic disk headroom line (one per local store) ships as an individual\\n # info event. Exactly the message plus `store` (a declared store) and `free_bytes` (a bounded\\n # integer in whole 64 MiB steps, as the node rounds it); anything else leaves headroom null, and\\n # the info line is then dropped below.\\n headroom = null\\n if target == \\\"tinycloud::disk_headroom\\\" && level == \\\"info\\\" && msg == \\\"storage headroom\\\" && length(f) == 3 {\\n h_store = string(f.store) ?? \\\"\\\"\\n h_free = f.free_bytes\\n if includes([\\\"sql\\\",\\\"duckdb\\\"], h_store) && is_integer(h_free) {\\n h_free = int!(h_free)\\n if h_free >= 0 && h_free <= 9007199254740991 && mod(h_free, 67108864) == 0 {\\n headroom = {\\\"event\\\": \\\"node_storage_headroom\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"store\\\": h_store, \\\"free_bytes\\\": h_free}\\n }\\n }\\n }\\n if target == \\\"tinycloud::request\\\" && level == \\\"info\\\" && msg == \\\"request\\\" {\\n method = string!(f.method)\\n if includes([\\\"TRACE\\\", \\\"CONNECT\\\"], method) { method = \\\"\\\" }\\n route = string!(f.route)\\n canonical_route = get(aliases, [route]) ?? null\\n if is_string(canonical_route) { route = canonical_route }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = f.status\\n latency = f.latency_us\\n if !is_integer(status) || !is_integer(latency) { abort }\\n status = int!(status)\\n latency = int!(latency)\\n if status < 100 || status > 599 || latency < 0 || latency > 9007199254740991 { abort }\\n out = {\\\"event\\\": \\\"node_request\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"latency_us\\\": latency}\\n # N2 correlation fields: copied only when they have exactly the node's emitted shape; anything else is dropped\\n # (the event still ships). The pseudonyms are keyed HMAC-SHA256 digests, never raw identifiers.\\n rid = string(f.request_id) ?? \\\"\\\"\\n if match(rid, r'^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$') { out.request_id = rid }\\n replaced = f.request_id_replaced\\n if is_boolean(replaced) { out.request_id_replaced = replaced }\\n account = string(f.account_hash) ?? \\\"\\\"\\n if match(account, r'^[0-9a-f]{64}$') { out.account_hash = account }\\n space = string(f.space_hash) ?? \\\"\\\"\\n if match(space, r'^[0-9a-f]{64}$') { out.space_hash = space }\\n } else if is_object(db_event) {\\n out = db_event\\n } else if is_object(headroom) {\\n out = headroom\\n } else {\\n if !includes([\\\"warn\\\", \\\"error\\\"], level) { abort }\\n code = \\\"unclassified\\\"\\n if target == \\\"tinycloud\\\" || starts_with(target, \\\"tinycloud::\\\") || target == \\\"tinycloud_core\\\" || starts_with(target, \\\"tinycloud_core::\\\") {\\n codes = {\\\"invocation replay insert failed\\\":\\\"invocation_replay_insert_failed\\\",\\\"invocation replay cleanup failed\\\":\\\"invocation_replay_cleanup_failed\\\",\\\"share-v2 local-tee feature active: deriving a deterministic, non-attested TEE identity from node key material\\\":\\\"share_local_tee_identity\\\",\\\"retention: delivered hook delivery prune failed\\\":\\\"retention_hook_delivery_prune_failed\\\",\\\"retention: dead-letter hook delivery prune failed\\\":\\\"retention_dead_letter_prune_failed\\\",\\\"retention: signed KV ticket prune failed\\\":\\\"retention_signed_ticket_prune_failed\\\",\\\"failed to load tunnel state; tunnel not started\\\":\\\"tunnel_state_load_failed\\\",\\\"invalid loopback API address; tunnel not started\\\":\\\"tunnel_address_invalid\\\",\\\"failed to load link state; LAN listener not started\\\":\\\"link_state_load_failed\\\",\\\"invalid link bind address; LAN listener not started\\\":\\\"link_bind_invalid\\\",\\\"invalid loopback API address; LAN listener not started\\\":\\\"link_address_invalid\\\",\\\"failed to bind link LAN listener; LAN listener not started\\\":\\\"link_bind_failed\\\",\\\"failed to build TLS config; LAN listener not started\\\":\\\"link_tls_config_failed\\\",\\\"link LAN listener exited with error\\\":\\\"link_listener_error\\\",\\\"failed to record link listener bind state\\\":\\\"link_listener_state_write_failed\\\",\\\"failed to record link listener bind failure\\\":\\\"link_bind_state_write_failed\\\",\\\"failed to load config for auto-renew\\\":\\\"link_auto_renew_config_failed\\\",\\\"link auto-renew failed\\\":\\\"link_auto_renew_failed\\\",\\\"failed to hot-reload renewed link cert\\\":\\\"link_cert_reload_failed\\\",\\\"link auto-renew succeeded but no state.json found; skipping cert hot-reload\\\":\\\"link_cert_state_missing\\\",\\\"failed to reload TLS material after link auto-renew\\\":\\\"link_listener_cert_reload_failed\\\",\\\"link disable: service delete failed; local state removed anyway\\\":\\\"link_disable_failed\\\",\\\"link connection error\\\":\\\"link_connection_error\\\",\\\"link listener accept error\\\":\\\"link_listener_accept_failed\\\",\\\"webhook delivery attempt failed\\\":\\\"webhook_delivery_failed\\\",\\\"webhook dispatcher loop failed\\\":\\\"webhook_dispatcher_failed\\\",\\\"addressed delegation authorization failed\\\":\\\"share_authorization_failed\\\",\\\"native share authorization denied\\\":\\\"share_native_authorization_denied\\\",\\\"native share KV put failed\\\":\\\"share_native_kv_put_failed\\\",\\\"share v2 readiness probe failed\\\":\\\"share_v2_readiness_failed\\\",\\\"share v2 is enabled but shareEmail.invitationPublicKey is unset\\\":\\\"share_invitation_key_missing\\\",\\\"configured share invitation key does not match the node's derived signing key\\\":\\\"share_invitation_key_mismatch\\\",\\\"share/v2 policy challenge denied\\\":\\\"share_policy_denied\\\",\\\"failed to read committed hook events\\\":\\\"hook_events_read_failed\\\",\\\"failed to load kv write hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"failed to load kv delete hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"missing committed kv hook row for invocation\\\":\\\"missing_committed_hook_row\\\",\\\"database write committed but webhook enqueue failed\\\":\\\"webhook_enqueue_failed\\\",\\\"tunnel: failed to prepare connection attempt\\\":\\\"tunnel_connection_prepare_failed\\\",\\\"tunnel: no frame within idle timeout; treating connection as dead\\\":\\\"tunnel_idle_timeout\\\",\\\"tunnel: read error\\\":\\\"tunnel_read_failed\\\",\\\"tunnel: dropping malformed frame\\\":\\\"tunnel_malformed_frame\\\",\\\"tunnel: relay sent an error frame\\\":\\\"tunnel_relay_error\\\",\\\"tunnel: failed to write frame\\\":\\\"tunnel_write_failed\\\",\\\"tunnel: failed to record runtime state\\\":\\\"tunnel_state_record_failed\\\",\\\"tunnel: failed to clear runtime state after disable\\\":\\\"tunnel_state_clear_failed\\\",\\\"tunnel authentication rejected\\\":\\\"tunnel_auth_rejected\\\",\\\"network DID resolution enabled: did:web/did:tz fetches use the unhardened ssi client (TC-1080)\\\":\\\"network_did_resolution_enabled\\\",\\\"SQL busy: lock wait\\\":\\\"sql_busy_lock_wait\\\",\\\"SQL busy: hydration\\\":\\\"sql_busy_hydration\\\",\\\"SQL busy: artifact transfer\\\":\\\"sql_busy_artifact_transfer\\\",\\\"Request deadline exceeded\\\":\\\"request_timeout\\\",\\\"Request completed after its deadline\\\":\\\"deadline_overrun\\\",\\\"Database hydration gave up loading its artifact\\\":\\\"sql_hydration_timeout\\\",\\\"Read audit wait expired\\\":\\\"read_audit_busy\\\",\\\"Read audit batch slow\\\":\\\"read_audit_batch_slow\\\",\\\"Read audit batch failed\\\":\\\"read_audit_batch_failed\\\",\\\"Read audit drain stalled\\\":\\\"read_audit_drain_stalled\\\",\\\"retrying operation after a lost database connection\\\":\\\"db_operation_retried\\\",\\\"not retrying operation: its first run took longer than the retry window to fail\\\":\\\"db_operation_retry_skipped_slow\\\",\\\"idx_current_kv_space_deleted_key_c is missing; the node keeps serving and KV list requests are slower until an operator builds it with CREATE INDEX CONCURRENTLY (docs/releasing.md, section idx_current_kv_space_deleted_key_c)\\\":\\\"current_kv_c_index_missing\\\",\\\"idx_current_kv_space_deleted_key_c is invalid or still being built; the node keeps serving and KV list requests are slower. If no CREATE INDEX CONCURRENTLY is running, run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_invalid\\\",\\\"idx_current_kv_space_deleted_key_c exists with the wrong definition and is not used for KV lists; the node keeps serving and KV list requests are slower. Run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_wrong_shape\\\"}\\n known = get(codes, [msg]) ?? null\\n if is_string(known) { code = known }\\n if starts_with(msg, \\\"quota service returned \\\") { code = \\\"quota_fallback\\\" }\\nif starts_with(msg, \\\"Failed to get TDX quote: \\\") { code = \\\"attestation_quote_failed\\\" }\\nif starts_with(msg, \\\"quota service returned an invalid body: \\\") { code = \\\"quota_invalid_body\\\" }\\nif starts_with(msg, \\\"dstack socket available but get_info failed: \\\") { code = \\\"dstack_get_info_failed\\\" }\\n }\\n event = if level == \\\"warn\\\" { \\\"node_warning\\\" } else { \\\"node_error\\\" }\\n out = {\\\"event\\\": event, \\\"svc\\\": \\\"api\\\", \\\"level\\\": level, \\\"event_code\\\": code, \\\"event_count\\\": 1}\\n }\\n} else {\\n ts = string!(j.time)\\n method = string!(j.method)\\n route = string!(j.route)\\n if route == \\\"\\\" { route = \\\"\\\" }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = j.status\\n request_time = j.request_time\\n if !is_integer(status) || !(is_integer(request_time) || is_float(request_time)) { abort }\\n status = int!(status)\\n request_time = to_float!(request_time)\\n if status < 100 || status > 599 || request_time < 0 || request_time > 86400 { abort }\\n out = {\\\"event\\\": \\\"ingress_request\\\", \\\"svc\\\": \\\"ingress\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"request_time_seconds\\\": request_time}\\n}\\nif length(ts) > 64 { abort }\\nparsed_ts = parse_timestamp!(ts, format: \\\"%+\\\")\\nout.ts = format_timestamp!(parsed_ts, format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\")\\n. = out\\n\",\"drop_on_error\":true,\"drop_on_abort\":true,\"reroute_dropped\":true},\"gate_counts\":{\"type\":\"remap\",\"inputs\":[\"gate_route.rejected\",\"gate_route._unmatched\",\"allowlist.dropped\",\"event_route._unmatched\"],\"source\":\"reason = .reason\\nif !includes([\\\"malformed_envelope\\\",\\\"unexpected_label\\\",\\\"unlabelled\\\",\\\"non_json\\\",\\\"oversized\\\",\\\"partial\\\",\\\"invalid_fields\\\",\\\"unknown_target\\\",\\\"filtered_level\\\"], reason) { reason = \\\"invalid_fields\\\" }\\n. = {\\\"reason\\\": reason, \\\"count\\\": 1}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_reduce\":{\"type\":\"reduce\",\"inputs\":[\"gate_counts\"],\"group_by\":[\"reason\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"count\":\"sum\"}},\"gate_output\":{\"type\":\"remap\",\"inputs\":[\"gate_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": \\\"warn\\\", \\\"event\\\": \\\"logship_gate_summary_total\\\", \\\"svc\\\": \\\"logship\\\", \\\"reason\\\": .reason, \\\"count\\\": .count}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"request_profile\":{\"type\":\"filter\",\"inputs\":[\"allowlist\"],\"condition\":\"\\\"$${TINYCLOUD_LOGSHIP_PROFILE}\\\" == \\\"all_requests\\\" || includes([\\\"node_warning\\\", \\\"node_error\\\", \\\"node_storage_headroom\\\"], .event) || (int(.status) ?? 0) >= 400 || (.event == \\\"node_request\\\" && (int(.latency_us) ?? 0) > 1000000) || (.event == \\\"ingress_request\\\" && (to_float(.request_time_seconds) ?? 0.0) > 1.000000)\"},\"event_route\":{\"type\":\"route\",\"inputs\":[\"request_profile\"],\"route\":{\"request\":\"(.event == \\\"node_request\\\" || .event == \\\"ingress_request\\\") && !(((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)) && !((int(.status) ?? 0) >= 500)\",\"node_request_5xx\":\".event == \\\"node_request\\\" && (int(.status) ?? 0) >= 500\",\"ingress_request_5xx\":\".event == \\\"ingress_request\\\" && (int(.status) ?? 0) >= 500\",\"node_request_4xx\":\".event == \\\"node_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"ingress_request_4xx\":\".event == \\\"ingress_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"db_failure_5xx\":\".event == \\\"node_warning\\\" && .event_code == \\\"db_failure_5xx\\\"\",\"storage_headroom\":\".event == \\\"node_storage_headroom\\\"\",\"diagnostic\":\"(.event == \\\"node_warning\\\" && .event_code != \\\"db_failure_5xx\\\") || .event == \\\"node_error\\\"\"}},\"cap_db_failure_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.db_failure_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"db_failure_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"diagnostic_reduce\":{\"type\":\"reduce\",\"inputs\":[\"event_route.diagnostic\"],\"group_by\":[\"event\",\"event_code\",\"level\",\"svc\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"event_count\":\"sum\",\"ts\":\"discard\"}},\"diagnostic_output\":{\"type\":\"remap\",\"inputs\":[\"diagnostic_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": .level, \\\"event\\\": .event, \\\"svc\\\": \\\"api\\\", \\\"event_code\\\": .event_code, \\\"event_count\\\": .event_count}\",\"drop_on_error\":true,\"drop_on_abort\":true}},\"sinks\":{\"betterstack\":{\"type\":\"http\",\"inputs\":[\"event_route.request\",\"event_route.storage_headroom\",\"cap_db_failure_5xx\",\"cap_node_request_5xx\",\"cap_ingress_request_5xx\",\"cap_node_request_4xx\",\"cap_ingress_request_4xx\",\"diagnostic_output\",\"gate_output\"],\"method\":\"post\",\"uri\":\"https://s2795437.us-west-2a.betterstackdata.com/\",\"encoding\":{\"codec\":\"json\"},\"framing\":{\"method\":\"newline_delimited\"},\"compression\":\"gzip\",\"auth\":{\"strategy\":\"bearer\",\"token\":\"SECRET[bs.bs_token]\"},\"batch\":{\"max_bytes\":262144,\"timeout_secs\":1},\"buffer\":{\"type\":\"memory\",\"max_events\":1000,\"when_full\":\"block\"},\"request\":{\"timeout_secs\":10,\"retry_max_duration_secs\":300},\"healthcheck\":{\"enabled\":false}}},\"secret\":{\"bs\":{\"type\":\"directory\",\"path\":\"/var/lib/vector/secrets\",\"remove_trailing_whitespace\":true}}}" + VECTOR_CONFIG_JSON: "{\"data_dir\":\"/var/lib/vector\",\"sources\":{\"gelf_in\":{\"type\":\"socket\",\"mode\":\"udp\",\"address\":\"0.0.0.0:12201\",\"max_length\":262144,\"receive_buffer_bytes\":8388608,\"framing\":{\"method\":\"chunked_gelf\",\"chunked_gelf\":{\"decompression\":\"Auto\"}},\"decoding\":{\"codec\":\"bytes\"}}},\"transforms\":{\"gelf_envelope\":{\"type\":\"remap\",\"inputs\":[\"gelf_in\"],\"source\":\"packet = string(.message) ?? \\\"\\\"\\nif length(packet) > 262144 {\\n . = {\\\"reason\\\": \\\"oversized\\\"}\\n} else {\\n outer, err = parse_json(packet)\\n if err != null || !is_object(outer) || (outer.version != \\\"1.1\\\" && outer.version != \\\"1.0\\\") {\\n . = {\\\"reason\\\": \\\"malformed_envelope\\\"}\\n } else if exists(outer.partial_message) || exists(outer._partial_message) {\\n . = {\\\"reason\\\": \\\"partial\\\"}\\n } else {\\n label = get(outer, [\\\"_tc.logship\\\"]) ?? null\\n . = {\\\"_tc.logship\\\": label, \\\"message\\\": outer.short_message}\\n }\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"logship_gate\":{\"type\":\"remap\",\"inputs\":[\"gelf_envelope\"],\"source\":\"if exists(.reason) {\\n . = {\\\"reason\\\": .reason}\\n} else {\\nlabel = string(get(., [\\\"_tc.logship\\\"]) ?? null) ?? \\\"\\\"\\nraw = .message\\nreason = \\\"\\\"\\nif label == \\\"\\\" { reason = \\\"unlabelled\\\" }\\nelse if !includes([\\\"tinycloud-node\\\", \\\"tinycloud-ingress\\\"], label) { reason = \\\"unexpected_label\\\" }\\nelse if exists(.partial_message) { reason = \\\"partial\\\" }\\nelse if !is_string(raw) { reason = \\\"malformed_envelope\\\" }\\nelse if length(string!(raw)) > 65536 { reason = \\\"oversized\\\" }\\nif reason != \\\"\\\" {\\n . = {\\\"reason\\\": reason}\\n} else {\\n parsed, err = parse_json(string!(raw))\\n if err != null || !is_object(parsed) { . = {\\\"reason\\\": \\\"non_json\\\"} }\\n else if exists(parsed.partial_message) { . = {\\\"reason\\\": \\\"partial\\\"} }\\n else {\\n target = string(parsed.target) ?? \\\"\\\"\\n level = downcase(string(parsed.level) ?? \\\"\\\")\\n if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && target != \\\"tinycloud\\\" && !starts_with(target, \\\"tinycloud::\\\") && target != \\\"tinycloud_core\\\" && !starts_with(target, \\\"tinycloud_core::\\\") {\\n . = {\\\"reason\\\": \\\"unknown_target\\\"}\\n } else if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && !((target == \\\"tinycloud::request\\\" || target == \\\"tinycloud::disk_headroom\\\") && level == \\\"info\\\") {\\n . = {\\\"reason\\\": \\\"filtered_level\\\"}\\n } else { . = {\\\"label\\\": label, \\\"parsed\\\": parsed} }\\n }\\n}\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_route\":{\"type\":\"route\",\"inputs\":[\"logship_gate\"],\"route\":{\"candidate\":\"exists(.parsed)\",\"rejected\":\"exists(.reason)\"}},\"allowlist\":{\"type\":\"remap\",\"inputs\":[\"gate_route.candidate\"],\"source\":\"# Only this fresh object can leave the process boundary. No envelope, span,\\n# message, error, identity, URI or unknown key is copied into it.\\nj = .parsed\\nlabel = .label\\nout = {}\\nts = \\\"\\\"\\nif label == \\\"tinycloud-node\\\" {\\n target = string(j.target) ?? \\\"\\\"\\n level = downcase(string!(j.level))\\n f = object(j.fields) ?? {}\\n msg = string(f.message) ?? \\\"\\\"\\n ts = string!(j.timestamp)\\n # Literal Rocket template aliases only; never split an arbitrary URI.\\n aliases = {\\\"/attestation?\\\":\\\"/attestation\\\",\\\"/.well-known/encryption/network/?\\\":\\\"/.well-known/encryption/network/\\\",\\\"/hooks/events?\\\":\\\"/hooks/events\\\",\\\"/hooks/webhooks?\\\":\\\"/hooks/webhooks\\\",\\\"/public//kv?\\\":\\\"/public//kv\\\"}\\n # TC-1840: the node's one 5xx diagnostic line per database failure ships as an individual event.\\n # The input must carry exactly the declared keys (the message plus these seven, and db_hash only\\n # when it is a valid pseudonym) with exactly the declared types and enum values. Anything else\\n # leaves db_event null and the line falls back to the unclassified count below; no value of it\\n # is ever copied unless it passed, and the shipped object is built from literals and the\\n # validated values only.\\n db_event = null\\n if target == \\\"tinycloud::request\\\" && level == \\\"warn\\\" && msg == \\\"database failure answered with a 5xx\\\" {\\n db_status = f.status\\n db_elapsed = f.elapsed_ms\\n db_code = string(f.code) ?? \\\"\\\"\\n db_service = string(f.service) ?? \\\"\\\"\\n db_kind = string(f.failure_kind) ?? \\\"\\\"\\n db_route = string(f.route) ?? \\\"\\\"\\n db_canonical_route = get(aliases, [db_route]) ?? null\\n if is_string(db_canonical_route) { db_route = db_canonical_route }\\n db_ok = is_integer(db_status) && is_integer(db_elapsed) && is_boolean(f.write_may_have_applied)\\n if db_ok {\\n db_status = int!(db_status)\\n db_elapsed = int!(db_elapsed)\\n db_ok = db_status >= 500 && db_status <= 599 && db_elapsed >= 0 && db_elapsed <= 9007199254740991 && includes([\\\"unavailable_retryable\\\",\\\"outcome_unknown\\\",\\\"internal\\\",\\\"sql_busy\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\"], db_code) && includes([\\\"sql\\\",\\\"duckdb\\\",\\\"kv\\\",\\\"other\\\"], db_service) && includes([\\\"pool_timeout\\\",\\\"pool_closed\\\",\\\"io\\\",\\\"statement_timeout\\\",\\\"serialization\\\",\\\"connection_lost\\\",\\\"artifact_changed\\\",\\\"lock_wait\\\",\\\"hydration\\\",\\\"artifact_transfer\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\",\\\"other\\\"], db_kind) && includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], db_route)\\n }\\n db_hash = string(f.db_hash) ?? \\\"\\\"\\n has_db_hash = exists(f.db_hash)\\n if db_ok {\\n if has_db_hash { db_ok = match(db_hash, r'^[0-9a-f]{64}$') && length(f) == 9 } else { db_ok = length(f) == 8 }\\n }\\n if db_ok {\\n db_event = {\\\"event\\\": \\\"node_warning\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"warn\\\", \\\"event_code\\\": \\\"db_failure_5xx\\\", \\\"status\\\": db_status, \\\"code\\\": db_code, \\\"service\\\": db_service, \\\"route\\\": db_route, \\\"elapsed_ms\\\": db_elapsed, \\\"failure_kind\\\": db_kind, \\\"write_may_have_applied\\\": bool!(f.write_may_have_applied)}\\n if has_db_hash { db_event.db_hash = db_hash }\\n }\\n }\\n # TC-2917: the node's periodic disk headroom line (one per local store) ships as an individual\\n # info event. Exactly the message plus `store` (a declared store) and `free_bytes` (a bounded\\n # integer in whole 64 MiB steps, as the node rounds it); anything else leaves headroom null, and\\n # the info line is then dropped below.\\n headroom = null\\n if target == \\\"tinycloud::disk_headroom\\\" && level == \\\"info\\\" && msg == \\\"storage headroom\\\" && length(f) == 3 {\\n h_store = string(f.store) ?? \\\"\\\"\\n h_free = f.free_bytes\\n if includes([\\\"sql\\\",\\\"duckdb\\\"], h_store) && is_integer(h_free) {\\n h_free = int!(h_free)\\n if h_free >= 0 && h_free <= 9007199254740991 && mod(h_free, 67108864) == 0 {\\n headroom = {\\\"event\\\": \\\"node_storage_headroom\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"store\\\": h_store, \\\"free_bytes\\\": h_free}\\n }\\n }\\n }\\n if target == \\\"tinycloud::request\\\" && level == \\\"info\\\" && msg == \\\"request\\\" {\\n method = string!(f.method)\\n if includes([\\\"TRACE\\\", \\\"CONNECT\\\"], method) { method = \\\"\\\" }\\n route = string!(f.route)\\n canonical_route = get(aliases, [route]) ?? null\\n if is_string(canonical_route) { route = canonical_route }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = f.status\\n latency = f.latency_us\\n if !is_integer(status) || !is_integer(latency) { abort }\\n status = int!(status)\\n latency = int!(latency)\\n if status < 100 || status > 599 || latency < 0 || latency > 9007199254740991 { abort }\\n out = {\\\"event\\\": \\\"node_request\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"latency_us\\\": latency}\\n # N2 correlation fields: copied only when they have exactly the node's emitted shape; anything else is dropped\\n # (the event still ships). The pseudonyms are keyed HMAC-SHA256 digests, never raw identifiers.\\n rid = string(f.request_id) ?? \\\"\\\"\\n if match(rid, r'^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$') { out.request_id = rid }\\n replaced = f.request_id_replaced\\n if is_boolean(replaced) { out.request_id_replaced = replaced }\\n account = string(f.account_hash) ?? \\\"\\\"\\n if match(account, r'^[0-9a-f]{64}$') { out.account_hash = account }\\n space = string(f.space_hash) ?? \\\"\\\"\\n if match(space, r'^[0-9a-f]{64}$') { out.space_hash = space }\\n } else if is_object(db_event) {\\n out = db_event\\n } else if is_object(headroom) {\\n out = headroom\\n } else {\\n if !includes([\\\"warn\\\", \\\"error\\\"], level) { abort }\\n code = \\\"unclassified\\\"\\n if target == \\\"tinycloud\\\" || starts_with(target, \\\"tinycloud::\\\") || target == \\\"tinycloud_core\\\" || starts_with(target, \\\"tinycloud_core::\\\") {\\n codes = {\\\"invocation replay insert failed\\\":\\\"invocation_replay_insert_failed\\\",\\\"invocation replay cleanup failed\\\":\\\"invocation_replay_cleanup_failed\\\",\\\"share-v2 local-tee feature active: deriving a deterministic, non-attested TEE identity from node key material\\\":\\\"share_local_tee_identity\\\",\\\"retention: delivered hook delivery prune failed\\\":\\\"retention_hook_delivery_prune_failed\\\",\\\"retention: dead-letter hook delivery prune failed\\\":\\\"retention_dead_letter_prune_failed\\\",\\\"retention: signed KV ticket prune failed\\\":\\\"retention_signed_ticket_prune_failed\\\",\\\"failed to load tunnel state; tunnel not started\\\":\\\"tunnel_state_load_failed\\\",\\\"invalid loopback API address; tunnel not started\\\":\\\"tunnel_address_invalid\\\",\\\"failed to load link state; LAN listener not started\\\":\\\"link_state_load_failed\\\",\\\"invalid link bind address; LAN listener not started\\\":\\\"link_bind_invalid\\\",\\\"invalid loopback API address; LAN listener not started\\\":\\\"link_address_invalid\\\",\\\"failed to bind link LAN listener; LAN listener not started\\\":\\\"link_bind_failed\\\",\\\"failed to build TLS config; LAN listener not started\\\":\\\"link_tls_config_failed\\\",\\\"link LAN listener exited with error\\\":\\\"link_listener_error\\\",\\\"failed to record link listener bind state\\\":\\\"link_listener_state_write_failed\\\",\\\"failed to record link listener bind failure\\\":\\\"link_bind_state_write_failed\\\",\\\"failed to load config for auto-renew\\\":\\\"link_auto_renew_config_failed\\\",\\\"link auto-renew failed\\\":\\\"link_auto_renew_failed\\\",\\\"failed to hot-reload renewed link cert\\\":\\\"link_cert_reload_failed\\\",\\\"link auto-renew succeeded but no state.json found; skipping cert hot-reload\\\":\\\"link_cert_state_missing\\\",\\\"failed to reload TLS material after link auto-renew\\\":\\\"link_listener_cert_reload_failed\\\",\\\"link disable: service delete failed; local state removed anyway\\\":\\\"link_disable_failed\\\",\\\"link connection error\\\":\\\"link_connection_error\\\",\\\"link listener accept error\\\":\\\"link_listener_accept_failed\\\",\\\"webhook delivery attempt failed\\\":\\\"webhook_delivery_failed\\\",\\\"webhook dispatcher loop failed\\\":\\\"webhook_dispatcher_failed\\\",\\\"addressed delegation authorization failed\\\":\\\"share_authorization_failed\\\",\\\"native share authorization denied\\\":\\\"share_native_authorization_denied\\\",\\\"native share KV put failed\\\":\\\"share_native_kv_put_failed\\\",\\\"share v2 readiness probe failed\\\":\\\"share_v2_readiness_failed\\\",\\\"share v2 is enabled but shareEmail.invitationPublicKey is unset\\\":\\\"share_invitation_key_missing\\\",\\\"configured share invitation key does not match the node's derived signing key\\\":\\\"share_invitation_key_mismatch\\\",\\\"share/v2 policy challenge denied\\\":\\\"share_policy_denied\\\",\\\"failed to read committed hook events\\\":\\\"hook_events_read_failed\\\",\\\"failed to load kv write hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"failed to load kv delete hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"missing committed kv hook row for invocation\\\":\\\"missing_committed_hook_row\\\",\\\"database write committed but webhook enqueue failed\\\":\\\"webhook_enqueue_failed\\\",\\\"tunnel: failed to prepare connection attempt\\\":\\\"tunnel_connection_prepare_failed\\\",\\\"tunnel: no frame within idle timeout; treating connection as dead\\\":\\\"tunnel_idle_timeout\\\",\\\"tunnel: read error\\\":\\\"tunnel_read_failed\\\",\\\"tunnel: dropping malformed frame\\\":\\\"tunnel_malformed_frame\\\",\\\"tunnel: relay sent an error frame\\\":\\\"tunnel_relay_error\\\",\\\"tunnel: failed to write frame\\\":\\\"tunnel_write_failed\\\",\\\"tunnel: failed to record runtime state\\\":\\\"tunnel_state_record_failed\\\",\\\"tunnel: failed to clear runtime state after disable\\\":\\\"tunnel_state_clear_failed\\\",\\\"tunnel authentication rejected\\\":\\\"tunnel_auth_rejected\\\",\\\"network DID resolution enabled: did:web/did:tz fetches use the unhardened ssi client (TC-1080)\\\":\\\"network_did_resolution_enabled\\\",\\\"SQL busy: lock wait\\\":\\\"sql_busy_lock_wait\\\",\\\"SQL busy: hydration\\\":\\\"sql_busy_hydration\\\",\\\"SQL busy: artifact transfer\\\":\\\"sql_busy_artifact_transfer\\\",\\\"Request deadline exceeded\\\":\\\"request_timeout\\\",\\\"Request completed after its deadline\\\":\\\"deadline_overrun\\\",\\\"Database hydration gave up loading its artifact\\\":\\\"sql_hydration_timeout\\\",\\\"Read audit wait expired\\\":\\\"read_audit_busy\\\",\\\"Read audit batch slow\\\":\\\"read_audit_batch_slow\\\",\\\"Read audit batch failed\\\":\\\"read_audit_batch_failed\\\",\\\"Read audit drain stalled\\\":\\\"read_audit_drain_stalled\\\",\\\"retrying operation after a lost database connection\\\":\\\"db_operation_retried\\\",\\\"not retrying operation: its first run took longer than the retry window to fail\\\":\\\"db_operation_retry_skipped_slow\\\",\\\"closing a pooled connection released inside a transaction\\\":\\\"db_pool_closed_in_transaction\\\",\\\"Database exceeds the artifact read limit; not loaded\\\":\\\"sql_database_too_large_to_load\\\",\\\"Database checkpoint save failed; refusing its writes until the back-off ends\\\":\\\"sql_checkpoint_save_backoff\\\",\\\"kv head size probe failed; size omitted\\\":\\\"kv_head_size_probe_failed\\\",\\\"chain guard wait exceeded the slow-wait threshold\\\":\\\"chain_guard_slow_wait\\\",\\\"request answered with a 5xx\\\":\\\"http_5xx\\\",\\\"idx_current_kv_space_deleted_key_c is missing; the node keeps serving and KV list requests are slower until an operator builds it with CREATE INDEX CONCURRENTLY (docs/releasing.md, section idx_current_kv_space_deleted_key_c)\\\":\\\"current_kv_c_index_missing\\\",\\\"idx_current_kv_space_deleted_key_c is invalid or still being built; the node keeps serving and KV list requests are slower. If no CREATE INDEX CONCURRENTLY is running, run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_invalid\\\",\\\"idx_current_kv_space_deleted_key_c exists with the wrong definition and is not used for KV lists; the node keeps serving and KV list requests are slower. Run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_wrong_shape\\\"}\\n known = get(codes, [msg]) ?? null\\n if is_string(known) { code = known }\\n if starts_with(msg, \\\"quota service returned \\\") { code = \\\"quota_fallback\\\" }\\nif starts_with(msg, \\\"Failed to get TDX quote: \\\") { code = \\\"attestation_quote_failed\\\" }\\nif starts_with(msg, \\\"quota service returned an invalid body: \\\") { code = \\\"quota_invalid_body\\\" }\\nif starts_with(msg, \\\"dstack socket available but get_info failed: \\\") { code = \\\"dstack_get_info_failed\\\" }\\n }\\n event = if level == \\\"warn\\\" { \\\"node_warning\\\" } else { \\\"node_error\\\" }\\n out = {\\\"event\\\": event, \\\"svc\\\": \\\"api\\\", \\\"level\\\": level, \\\"event_code\\\": code, \\\"event_count\\\": 1}\\n }\\n} else {\\n ts = string!(j.time)\\n method = string!(j.method)\\n route = string!(j.route)\\n if route == \\\"\\\" { route = \\\"\\\" }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = j.status\\n request_time = j.request_time\\n if !is_integer(status) || !(is_integer(request_time) || is_float(request_time)) { abort }\\n status = int!(status)\\n request_time = to_float!(request_time)\\n if status < 100 || status > 599 || request_time < 0 || request_time > 86400 { abort }\\n out = {\\\"event\\\": \\\"ingress_request\\\", \\\"svc\\\": \\\"ingress\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"request_time_seconds\\\": request_time}\\n}\\nif length(ts) > 64 { abort }\\nparsed_ts = parse_timestamp!(ts, format: \\\"%+\\\")\\nout.ts = format_timestamp!(parsed_ts, format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\")\\n. = out\\n\",\"drop_on_error\":true,\"drop_on_abort\":true,\"reroute_dropped\":true},\"gate_counts\":{\"type\":\"remap\",\"inputs\":[\"gate_route.rejected\",\"gate_route._unmatched\",\"allowlist.dropped\",\"event_route._unmatched\"],\"source\":\"reason = .reason\\nif !includes([\\\"malformed_envelope\\\",\\\"unexpected_label\\\",\\\"unlabelled\\\",\\\"non_json\\\",\\\"oversized\\\",\\\"partial\\\",\\\"invalid_fields\\\",\\\"unknown_target\\\",\\\"filtered_level\\\"], reason) { reason = \\\"invalid_fields\\\" }\\n. = {\\\"reason\\\": reason, \\\"count\\\": 1}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_reduce\":{\"type\":\"reduce\",\"inputs\":[\"gate_counts\"],\"group_by\":[\"reason\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"count\":\"sum\"}},\"gate_output\":{\"type\":\"remap\",\"inputs\":[\"gate_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": \\\"warn\\\", \\\"event\\\": \\\"logship_gate_summary_total\\\", \\\"svc\\\": \\\"logship\\\", \\\"reason\\\": .reason, \\\"count\\\": .count}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"request_profile\":{\"type\":\"filter\",\"inputs\":[\"allowlist\"],\"condition\":\"\\\"$${TINYCLOUD_LOGSHIP_PROFILE}\\\" == \\\"all_requests\\\" || includes([\\\"node_warning\\\", \\\"node_error\\\", \\\"node_storage_headroom\\\"], .event) || (int(.status) ?? 0) >= 400 || (.event == \\\"node_request\\\" && (int(.latency_us) ?? 0) > 1000000) || (.event == \\\"ingress_request\\\" && (to_float(.request_time_seconds) ?? 0.0) > 1.000000)\"},\"event_route\":{\"type\":\"route\",\"inputs\":[\"request_profile\"],\"route\":{\"request\":\"(.event == \\\"node_request\\\" || .event == \\\"ingress_request\\\") && !(((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)) && !((int(.status) ?? 0) >= 500)\",\"node_request_5xx\":\".event == \\\"node_request\\\" && (int(.status) ?? 0) >= 500\",\"ingress_request_5xx\":\".event == \\\"ingress_request\\\" && (int(.status) ?? 0) >= 500\",\"node_request_4xx\":\".event == \\\"node_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"ingress_request_4xx\":\".event == \\\"ingress_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"db_failure_5xx\":\".event == \\\"node_warning\\\" && .event_code == \\\"db_failure_5xx\\\"\",\"storage_headroom\":\".event == \\\"node_storage_headroom\\\"\",\"diagnostic\":\"(.event == \\\"node_warning\\\" && .event_code != \\\"db_failure_5xx\\\") || .event == \\\"node_error\\\"\"}},\"cap_db_failure_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.db_failure_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"db_failure_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"diagnostic_reduce\":{\"type\":\"reduce\",\"inputs\":[\"event_route.diagnostic\"],\"group_by\":[\"event\",\"event_code\",\"level\",\"svc\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"event_count\":\"sum\",\"ts\":\"discard\"}},\"diagnostic_output\":{\"type\":\"remap\",\"inputs\":[\"diagnostic_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": .level, \\\"event\\\": .event, \\\"svc\\\": \\\"api\\\", \\\"event_code\\\": .event_code, \\\"event_count\\\": .event_count}\",\"drop_on_error\":true,\"drop_on_abort\":true}},\"sinks\":{\"betterstack\":{\"type\":\"http\",\"inputs\":[\"event_route.request\",\"event_route.storage_headroom\",\"cap_db_failure_5xx\",\"cap_node_request_5xx\",\"cap_ingress_request_5xx\",\"cap_node_request_4xx\",\"cap_ingress_request_4xx\",\"diagnostic_output\",\"gate_output\"],\"method\":\"post\",\"uri\":\"https://s2795437.us-west-2a.betterstackdata.com/\",\"encoding\":{\"codec\":\"json\"},\"framing\":{\"method\":\"newline_delimited\"},\"compression\":\"gzip\",\"auth\":{\"strategy\":\"bearer\",\"token\":\"SECRET[bs.bs_token]\"},\"batch\":{\"max_bytes\":262144,\"timeout_secs\":1},\"buffer\":{\"type\":\"memory\",\"max_events\":1000,\"when_full\":\"block\"},\"request\":{\"timeout_secs\":10,\"retry_max_duration_secs\":300},\"healthcheck\":{\"enabled\":false}}},\"secret\":{\"bs\":{\"type\":\"directory\",\"path\":\"/var/lib/vector/secrets\",\"remove_trailing_whitespace\":true}}}" entrypoint: - /bin/sh - -ec diff --git a/docs/releasing.md b/docs/releasing.md index 92d141a3..48382f4e 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -657,8 +657,8 @@ Before any CVM mutation the workflow requires: - the main compose, with its one node digest replaced by 64 zeros, to hash to the reviewed N0 template - **`90841a34e6ea5b581374f50abe70e580496e6814bc83f61f20fd4c9e090e5d51`** - (raw `5419a6fbb73efe8b4430e4ab76d5e2970b2c9204525854e23f59dbdc01f4e250` normalized; before the TC-2917 `node_storage_headroom` event (the node's disk headroom line shipped as an individual event) in the Vector config the template was `9d0cd8a2…`, raw `d703d69b…`; before the TC-2346 `db_operation_retried` and `db_operation_retry_skipped_slow` warning codes in the Vector config the template was `c9bcbea9…`, raw `298ea356…`; before the TC-1777 named-keys gate env lines (`TINYCLOUD_ENCRYPTION_NETWORK__NAMED_KEYS_V1`, `…__DELETION_LEDGER_PATH`) and the `deletion-ledger` volume the template was `8df7e0f2…`, raw `192dfb7e407fa591d4f0232b95e43a41a85dd18add1d4976e70ebdbff5ddebca`; before the TC-1843 per-minute caps on 4xx and 5xx request events and on the 5xx diagnostic, and the `errors_dropped` event (the dropped count), in the Vector config the template was `7c3456dc…`, raw `d0e72bde…`; before the TC-1840 `db_failure_5xx` event (the node's 5xx diagnostic line shipped as an individual event) in the Vector config the template was `c91e5b7a…`, raw `99013c40…`; before the TC-950 pool of 10 (`TINYCLOUD_DATABASE__MAX_CONNECTIONS` 15 to 10) the template was `79469f97…`, raw `32a002ef…`; before the three `current_kv_c_index_*` warning codes in the Vector config (TC-1274 `current_kv` C-index migration) the template was `87b5d12a…`, raw `d1fcd665…`; before the optional TC-1482 `NODE_STATS_READ_TOKEN` env line and the TC-1482 route in the Vector config the template was `7fabba70…`, raw `1228d81927fc0a3fbba85745103e77b3112606f5d314d15974d400b222409350`: the 1.21.0 compose + **`efad41d56df187a7e0ddd1a13f3c4a3c436ad7c0ff141acd32c06e28f9c92a34`** + (raw `7ee085c6e5da8a2b7a6e77fa207089b4cef3b8c291b4770f0d0c3336449f90b1` normalized; before the TC-3006 canonical-schema update (`chain_guard_slow_wait` and `http_5xx` as counts, and the TC-2954/2961/3027 warning codes) in the Vector config the template was `90841a34…`, raw `5419a6fb…`; before the TC-2917 `node_storage_headroom` event (the node's disk headroom line shipped as an individual event) in the Vector config the template was `9d0cd8a2…`, raw `d703d69b…`; before the TC-2346 `db_operation_retried` and `db_operation_retry_skipped_slow` warning codes in the Vector config the template was `c9bcbea9…`, raw `298ea356…`; before the TC-1777 named-keys gate env lines (`TINYCLOUD_ENCRYPTION_NETWORK__NAMED_KEYS_V1`, `…__DELETION_LEDGER_PATH`) and the `deletion-ledger` volume the template was `8df7e0f2…`, raw `192dfb7e407fa591d4f0232b95e43a41a85dd18add1d4976e70ebdbff5ddebca`; before the TC-1843 per-minute caps on 4xx and 5xx request events and on the 5xx diagnostic, and the `errors_dropped` event (the dropped count), in the Vector config the template was `7c3456dc…`, raw `d0e72bde…`; before the TC-1840 `db_failure_5xx` event (the node's 5xx diagnostic line shipped as an individual event) in the Vector config the template was `c91e5b7a…`, raw `99013c40…`; before the TC-950 pool of 10 (`TINYCLOUD_DATABASE__MAX_CONNECTIONS` 15 to 10) the template was `79469f97…`, raw `32a002ef…`; before the three `current_kv_c_index_*` warning codes in the Vector config (TC-1274 `current_kv` C-index migration) the template was `87b5d12a…`, raw `d1fcd665…`; before the optional TC-1482 `NODE_STATS_READ_TOKEN` env line and the TC-1482 route in the Vector config the template was `7fabba70…`, raw `1228d81927fc0a3fbba85745103e77b3112606f5d314d15974d400b222409350`: the 1.21.0 compose `d95bfb707871c1d626c4a8f2b8d32d354103edbb5077847c6a21fe5f7f6cfa50`, template `51f8e9e9…`, with only the embedded Vector config regenerated for TC-1274's, TC-1317's and TC-1482's route and warning codes (TC-1274 and TC-1317 were template `7fabba70…`, raw `1228d819…`); TC-1274 alone was template `00938226…`, raw @@ -806,8 +806,8 @@ gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \ The served/OCI identity remains **1.20.6-dstack / e18578ff9162986ddd813259c60f73866e3e60d1**. With the template lock on main this redeploys exactly main's raw compose -`5419a6fbb73efe8b4430e4ab76d5e2970b2c9204525854e23f59dbdc01f4e250` -(the 1.21.0 compose `d95bfb70…` with the TC-1274, TC-1317, TC-1482, `current_kv_c_index_*`, `db_failure_5xx`, the 4xx and 5xx caps, `errors_dropped`, TC-2346 `db_operation_retried` / `db_operation_retry_skipped_slow` and TC-2917 `node_storage_headroom` Vector config, the optional TC-1482 `NODE_STATS_READ_TOKEN` env line, the TC-950 pool of 10, and the TC-1777 `NAMED_KEYS_V1` gate env lines with the `deletion-ledger` volume), whatever digest +`7ee085c6e5da8a2b7a6e77fa207089b4cef3b8c291b4770f0d0c3336449f90b1` +(the 1.21.0 compose `d95bfb70…` with the TC-1274, TC-1317, TC-1482, `current_kv_c_index_*`, `db_failure_5xx`, the 4xx and 5xx caps, `errors_dropped`, TC-2346 `db_operation_retried` / `db_operation_retry_skipped_slow`, TC-2917 `node_storage_headroom` and TC-3006 Vector config, the optional TC-1482 `NODE_STATS_READ_TOKEN` env line, the TC-950 pool of 10, and the TC-1777 `NAMED_KEYS_V1` gate env lines with the `deletion-ledger` volume), whatever digest main's template currently pins. It is not a rollback of a later release across a schema migration. Current pre-N0 tag/pinned equivalents are separately hash-checked; arbitrary same-image config diff --git a/log-shipper/canonical.json b/log-shipper/canonical.json index c74980b5..8c7b63a1 100644 --- a/log-shipper/canonical.json +++ b/log-shipper/canonical.json @@ -1,6 +1,6 @@ { "repo": "samgbafa/tinycloud-dev", - "revision": "4fc32a2f9a24526b8fae6329a04f54230753c8c8", + "revision": "6f0d420211810a533ebfb29b5bb7f34ac2422071", "path": "ops/logging/fields.json", - "sha256": "38dae8e0ce7794d9e626cab30b2b2ab205fce3498588e14b541d6e506e8c0ca4" + "sha256": "c72f50d8fff633f05b9d65d4b81e1dc77364f366679e58191abcac721344847c" } diff --git a/log-shipper/cases.mjs b/log-shipper/cases.mjs index 402fd1fa..f9f3e59a 100644 --- a/log-shipper/cases.mjs +++ b/log-shipper/cases.mjs @@ -130,6 +130,25 @@ export const vectorTests = [ // TC-2346: the node's operation-retry warnings (target tinycloud_core::db_pool). Like every classified diagnostic they ship the code and the count only; operation, error_class and the timings stay in the node's own log. ...[['retrying operation after a lost database connection', 'db_operation_retried'], ['not retrying operation: its first run took longer than the retry window to fail', 'db_operation_retry_skipped_slow']].map(([message, code]) => check(`operation retry warning maps to ${code} with only the fixed diagnostic keys`, envelope({ timestamp, level: 'WARN', target: 'tinycloud_core::db_pool', fields: { message, operation: 'delegate', error_class: 'io', elapsed_ms: 10001, retry_window_ms: 10000, sql: canary } }), `.event == "node_warning" && .event_code == "${code}" && .event_count == 1 && .level == "warn" && keys(.) == ["event", "event_code", "event_count", "level", "svc", "ts"]`)), ...['retrying operation after a lost database connection: x', 'Retrying operation after a lost database connection', 'not retrying operation'].map(message => check(`operation retry warning near-miss "${message}" stays unclassified`, envelope({ timestamp, level: 'WARN', target: 'tinycloud_core::db_pool', fields: { message, operation: 'delegate', error_class: 'io' } }), '.event == "node_warning" && .event_code == "unclassified" && .event_count == 1')), + // TC-2954, TC-2961, TC-3027, TC-3006: the pooled-connection backstop (vendored sqlx-core, target tinycloud_core::db_pool), + // the SQL artifact read limit and checkpoint back-off (tinycloud_core::sql::service), the KV head size probe, the + // chain-guard slow wait (tinycloud::chain_guard) and the every-5xx line (tinycloud::request). Code and count only: + // sizes, delays, kinds, routes, statuses, errors and identifiers stay in the node's own log. + ...[ + ['closing a pooled connection released inside a transaction', 'db_pool_closed_in_transaction', 'tinycloud_core::db_pool', {}], + ['Database exceeds the artifact read limit; not loaded', 'sql_database_too_large_to_load', 'tinycloud_core::sql::service', { service: 'sql', error_class: 'artifact_read_limit', bytes: 300666880, limit_bytes: 67108864 }], + ['Database checkpoint save failed; refusing its writes until the back-off ends', 'sql_checkpoint_save_backoff', 'tinycloud_core::sql::service', { service: 'sql', retry_after_ms: 30000 }], + ['kv head size probe failed; size omitted', 'kv_head_size_probe_failed', 'tinycloud::routes', { error: canary }], + ['chain guard wait exceeded the slow-wait threshold', 'chain_guard_slow_wait', 'tinycloud::chain_guard', { event: 'chain_guard_slow_wait', kind: 'exclusive', elapsed_ms: 612, guards: 3, suppressed: 0 }], + ['request answered with a 5xx', 'http_5xx', 'tinycloud::request', { error_class: 'http_5xx', status: 503, route: '/invoke', classified: true }], + ].map(([message, code, target, fields]) => check(`${code} warning maps to its code with only the fixed diagnostic keys`, envelope({ timestamp, level: 'WARN', target, fields: { message, ...fields, space: canary, db_name: canary } }), `.event == "node_warning" && .event_code == "${code}" && .event_count == 1 && .level == "warn" && keys(.) == ["event", "event_code", "event_count", "level", "svc", "ts"]`)), + ...[ + ['closing a pooled connection released inside a transaction', 'sqlx_core::pool::connection'], + ['closing a pooled connection released inside a transaction: x', 'tinycloud_core::db_pool'], + ['Database exceeds the artifact read limit', 'tinycloud_core::sql::service'], + ['database checkpoint save failed; refusing its writes until the back-off ends', 'tinycloud_core::sql::service'], + ['kv head size probe failed', 'tinycloud::routes'], + ].map(([message, target]) => check(`near-miss "${message}" on ${target} stays unclassified`, envelope({ timestamp, level: 'WARN', target, fields: { message } }), '.event == "node_warning" && .event_code == "unclassified" && .event_count == 1')), check('node 5xx diagnostic ships exactly its eight fields as an individual event', envelope(dbFailure()), `.event == "node_warning" && .svc == "api" && .level == "warn" && .event_code == "db_failure_5xx" && .status == 503 && .code == "sql_busy" && .service == "sql" && .route == "/invoke" && .elapsed_ms == 30001 && .failure_kind == "lock_wait" && .write_may_have_applied == false && .db_hash == "${goodHash}" && !exists(.event_count) && ${keysAre(dbShipped)}`), check('node 5xx diagnostic without a database ships without db_hash', envelope(dbFailure({ db_hash: undefined, status: 500, code: 'internal', service: 'other', failure_kind: 'other' })), `.event_code == "db_failure_5xx" && .status == 500 && .code == "internal" && .service == "other" && .failure_kind == "other" && ${keysAre(dbShipped.filter(key => key !== 'db_hash'))}`), check('node 5xx diagnostic with an unknown outcome ships the write flag', envelope(dbFailure({ status: 500, code: 'outcome_unknown', write_may_have_applied: true })), `.event_code == "db_failure_5xx" && .status == 500 && .code == "outcome_unknown" && .write_may_have_applied == true && ${keysAre(dbShipped)}`), diff --git a/log-shipper/diagnostics.json b/log-shipper/diagnostics.json index 6e51a4a1..93579af0 100644 --- a/log-shipper/diagnostics.json +++ b/log-shipper/diagnostics.json @@ -60,6 +60,12 @@ "Read audit drain stalled": "read_audit_drain_stalled", "retrying operation after a lost database connection": "db_operation_retried", "not retrying operation: its first run took longer than the retry window to fail": "db_operation_retry_skipped_slow", + "closing a pooled connection released inside a transaction": "db_pool_closed_in_transaction", + "Database exceeds the artifact read limit; not loaded": "sql_database_too_large_to_load", + "Database checkpoint save failed; refusing its writes until the back-off ends": "sql_checkpoint_save_backoff", + "kv head size probe failed; size omitted": "kv_head_size_probe_failed", + "chain guard wait exceeded the slow-wait threshold": "chain_guard_slow_wait", + "request answered with a 5xx": "http_5xx", "idx_current_kv_space_deleted_key_c is missing; the node keeps serving and KV list requests are slower until an operator builds it with CREATE INDEX CONCURRENTLY (docs/releasing.md, section idx_current_kv_space_deleted_key_c)": "current_kv_c_index_missing", "idx_current_kv_space_deleted_key_c is invalid or still being built; the node keeps serving and KV list requests are slower. If no CREATE INDEX CONCURRENTLY is running, run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c": "current_kv_c_index_invalid", "idx_current_kv_space_deleted_key_c exists with the wrong definition and is not used for KV lists; the node keeps serving and KV list requests are slower. Run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c": "current_kv_c_index_wrong_shape" diff --git a/log-shipper/fields.json b/log-shipper/fields.json index 76f43bef..53d4478c 100644 --- a/log-shipper/fields.json +++ b/log-shipper/fields.json @@ -34,6 +34,9 @@ "batch_diarization_failed", "batch_provider_rejected_the_service_credential_or_model", "batch_job_failed_at_dispatch", + "batch_provider_blocked", + "batch_provider_recovered", + "batch_region_ungrounded", "batch_worker_heartbeat_failed", "batch_sweep_failed", "batch_worker_iteration_failed", @@ -48,11 +51,19 @@ "stage": { "type": "enum", "values": ["create", "cancel", "delete", "health", "api_unhandled", "admission", "upload", "deletion", "claim_recovery", "reconcile", "vad", "diarize", "pipeline", "fence", "dispatch", "heartbeat", "sweep", "loop"] }, "code": { "type": "enum", "values": ["invalid_request", "diarization_unavailable", "unauthorized", "insufficient_scope", "idempotency_conflict", "transcription_not_found", "active_transcription_exists", "recording_too_large", "quota_exceeded", "service_busy", "service_paused", "service_unavailable", "upload_capability_limit", "upload_capability_invalid", "upload_capability_expired", "upload_length_mismatch", "unsupported_media_type", "upload_in_progress", "upload_interrupted", "upload_rejected", "transcript_expired", "not_found", "internal_error", "upload_expired", "upload_integrity_failed", "invalid_audio", "recording_too_long", "unsupported_recording", "no_speech", "provider_unavailable", "provider_outcome_unknown", "processing_timeout", "processing_failed", "transcription_failed", "cancelled"], "nullable": true }, "alert": { "type": "boolean" }, + "reason": { "type": "string", "regex": "^[A-Za-z0-9_]{1,40}$", "max_len": 40 }, + "http_status": { "type": "integer", "min": 400, "max": 599 }, + "delayed_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "queued_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "processing_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "awaiting_upload_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "oldest_queued_age_s": { "type": "integer", "min": 0, "max": 31536000 }, + "dropped_regions": { "type": "integer", "min": 0, "max": 1000 }, "outcome": { "type": "enum", "values": ["completed", "failed", "no_speech"] }, "duration_seconds": { "type": "number", "min": 0, "max": 86400 }, "tinfoil_calls": { "type": "integer", "min": 0, "max": 100000, "nullable": true }, - "tinfoil_audio_seconds": { "type": "number", "min": 0, "max": 7200, "nullable": true }, - "voiced_seconds": { "type": "number", "min": 0, "max": 7200, "nullable": true }, + "tinfoil_audio_seconds": { "type": "number", "min": 0, "max": 11101, "nullable": true }, + "voiced_seconds": { "type": "number", "min": 0, "max": 11101, "nullable": true }, "dropped_above_floor_pct": { "type": "number", "min": 0, "max": 100, "nullable": true }, "vad_fallback": { "type": "boolean", "nullable": true }, "segments": { "type": "integer", "min": 0, "max": 100000, "nullable": true }, @@ -79,7 +90,7 @@ "status": { "type": "integer", "min": 100, "max": 599 }, "latency_us": { "type": "integer", "min": 0, "max": 9007199254740991 }, "request_time_seconds": { "type": "number", "min": 0, "max": 86400 }, - "event_code": { "type": "enum", "values": ["invocation_replay_insert_failed", "invocation_replay_cleanup_failed", "dstack_get_info_failed", "share_local_tee_identity", "retention_hook_delivery_prune_failed", "retention_dead_letter_prune_failed", "retention_signed_ticket_prune_failed", "quota_invalid_body", "quota_fallback", "tunnel_state_load_failed", "tunnel_address_invalid", "link_state_load_failed", "link_bind_invalid", "link_address_invalid", "link_bind_failed", "link_bind_state_write_failed", "link_tls_config_failed", "link_listener_error", "link_listener_state_write_failed", "link_auto_renew_config_failed", "link_auto_renew_failed", "link_cert_reload_failed", "link_cert_state_missing", "link_listener_cert_reload_failed", "link_disable_failed", "link_connection_error", "link_listener_accept_failed", "webhook_delivery_failed", "webhook_dispatcher_failed", "share_authorization_failed", "share_native_authorization_denied", "share_native_kv_put_failed", "share_v2_readiness_failed", "share_invitation_key_missing", "share_invitation_key_mismatch", "share_policy_denied", "attestation_quote_failed", "hook_events_read_failed", "hook_rows_load_failed", "missing_committed_hook_row", "webhook_enqueue_failed", "tunnel_connection_prepare_failed", "tunnel_idle_timeout", "tunnel_read_failed", "tunnel_malformed_frame", "tunnel_relay_error", "tunnel_write_failed", "tunnel_state_record_failed", "tunnel_state_clear_failed", "tunnel_auth_rejected", "network_did_resolution_enabled", "unclassified", "sql_busy_lock_wait", "sql_busy_hydration", "sql_busy_artifact_transfer", "request_timeout", "deadline_overrun", "sql_hydration_timeout", "read_audit_busy", "read_audit_batch_slow", "read_audit_batch_failed", "read_audit_drain_stalled", "current_kv_c_index_missing", "current_kv_c_index_invalid", "current_kv_c_index_wrong_shape", "db_operation_retried", "db_operation_retry_skipped_slow", "db_failure_5xx", "errors_dropped"] }, + "event_code": { "type": "enum", "values": ["invocation_replay_insert_failed", "invocation_replay_cleanup_failed", "dstack_get_info_failed", "share_local_tee_identity", "retention_hook_delivery_prune_failed", "retention_dead_letter_prune_failed", "retention_signed_ticket_prune_failed", "quota_invalid_body", "quota_fallback", "tunnel_state_load_failed", "tunnel_address_invalid", "link_state_load_failed", "link_bind_invalid", "link_address_invalid", "link_bind_failed", "link_bind_state_write_failed", "link_tls_config_failed", "link_listener_error", "link_listener_state_write_failed", "link_auto_renew_config_failed", "link_auto_renew_failed", "link_cert_reload_failed", "link_cert_state_missing", "link_listener_cert_reload_failed", "link_disable_failed", "link_connection_error", "link_listener_accept_failed", "webhook_delivery_failed", "webhook_dispatcher_failed", "share_authorization_failed", "share_native_authorization_denied", "share_native_kv_put_failed", "share_v2_readiness_failed", "share_invitation_key_missing", "share_invitation_key_mismatch", "share_policy_denied", "attestation_quote_failed", "hook_events_read_failed", "hook_rows_load_failed", "missing_committed_hook_row", "webhook_enqueue_failed", "tunnel_connection_prepare_failed", "tunnel_idle_timeout", "tunnel_read_failed", "tunnel_malformed_frame", "tunnel_relay_error", "tunnel_write_failed", "tunnel_state_record_failed", "tunnel_state_clear_failed", "tunnel_auth_rejected", "network_did_resolution_enabled", "unclassified", "sql_busy_lock_wait", "sql_busy_hydration", "sql_busy_artifact_transfer", "request_timeout", "deadline_overrun", "sql_hydration_timeout", "read_audit_busy", "read_audit_batch_slow", "read_audit_batch_failed", "read_audit_drain_stalled", "current_kv_c_index_missing", "current_kv_c_index_invalid", "current_kv_c_index_wrong_shape", "db_operation_retried", "db_operation_retry_skipped_slow", "db_failure_5xx", "errors_dropped", "http_5xx", "chain_guard_slow_wait", "db_pool_closed_in_transaction", "sql_database_too_large_to_load", "sql_checkpoint_save_backoff", "kv_head_size_probe_failed"] }, "event_count": { "type": "integer", "min": 1, "max": 9007199254740991 }, "reason": { "type": "enum", "values": ["malformed_envelope", "unexpected_label", "unlabelled", "non_json", "oversized", "partial", "invalid_fields", "unknown_target", "filtered_level"] }, "count": { "type": "integer", "min": 1, "max": 9007199254740991 }, @@ -93,14 +104,71 @@ "elapsed_ms": { "type": "integer", "min": 0, "max": 9007199254740991 }, "failure_kind": { "type": "enum", "values": ["pool_timeout", "pool_closed", "io", "statement_timeout", "serialization", "connection_lost", "artifact_changed", "lock_wait", "hydration", "artifact_transfer", "request_timeout", "read_audit_busy", "other"] }, "write_may_have_applied": { "type": "boolean" }, + "classified": { "type": "boolean" }, "class": { "type": "enum", "values": ["db_failure_5xx", "node_request_5xx", "ingress_request_5xx", "node_request_4xx", "ingress_request_4xx"] }, "dropped": { "type": "integer", "min": 1, "max": 9007199254740991 }, "window_s": { "type": "integer", "min": 60, "max": 60 }, + "kind": { "type": "enum", "values": ["shared", "exclusive"] }, + "guards": { "type": "integer", "min": 0, "max": 10000 }, + "suppressed": { "type": "integer", "min": 0, "max": 100000 }, "store": { "type": "enum", "values": ["sql", "duckdb"] }, "free_bytes": { "type": "integer", "min": 0, "max": 9007199254740991 } } }, - "tinychat-backend": { "events": [], "fields": {} }, + "tinychat-backend": { + "events": [ + "backend_startup", + "backend_startup_failed", + "backend_uncaught_exception", + "backend_alert", + "backend_log_error", + "backend_storage_lane_wait", + "backend_diagnostic_summary", + "assemblyai_provider_blocked", + "assemblyai_provider_recovered", + "assemblyai_jobs_failed_in_a_row", + "assemblyai_worker_heartbeat", + "tinychat_request" + ], + "fields": { + "svc": { "type": "enum", "values": ["backend"] }, + "phase": { "type": "enum", "values": ["startup_fatal", "main_rejected", "tick", "tenant_reconcile", "due_timer", "dispatch_lag_status"] }, + "failure": { "type": "enum", "values": ["ledger_config", "secret_reuse", "secret_too_weak", "fireflies_oauth_config", "google_oauth_config", "hosted_spool", "private_cloud_config", "unclassified"] }, + "error_name": { "type": "string", "regex": "^[A-Za-z0-9_.]{1,64}$", "max_len": 64 }, + "component": { "type": "string", "regex": "^[a-z0-9][a-z0-9_.-]{0,39}$", "max_len": 40 }, + "route": { "type": "string", "regex": "^[a-z_]{1,40}$", "max_len": 40 }, + "status": { "type": "integer", "min": 100, "max": 599 }, + "upstream_status": { "type": "integer", "min": 100, "max": 599 }, + "code": { "type": "string", "regex": "^[a-z][a-z0-9_]{0,59}$", "max_len": 60 }, + "reason": { "type": "string", "regex": "^[A-Za-z0-9_]{1,40}$", "max_len": 40 }, + "cid": { "type": "string", "regex": "^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", "max_len": 36 }, + "alert": { "type": "boolean" }, + "http_status": { "type": "integer", "min": 400, "max": 599 }, + "delayed_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "count": { "type": "integer", "min": 0, "max": 1000 }, + "pending_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "consecutive_failed_jobs": { "type": "integer", "min": 0, "max": 100000 }, + "oldest_pending_age_s": { "type": "number", "min": 0, "max": 31536000 }, + "op": { "type": "enum", "values": ["slow-wait", "wait-timeout"] }, + "priority": { "type": "enum", "values": ["foreground", "background"] }, + "waited_ms": { "type": "integer", "min": 0, "max": 3600000 }, + "slow_wait_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "wait_timeout_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "calendar_tick_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "calendar_tenant_reconcile_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "calendar_due_timer_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "calendar_dispatch_lag_status_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "other_count": { "type": "integer", "min": 0, "max": 1000000000 }, + "counts_saturated": { "type": "boolean" }, + "request_id": { "type": "string", "regex": "^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", "max_len": 36 }, + "method": { "type": "enum", "values": ["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "CONNECT", "TRACE", "OTHER"] }, + "duration_ms": { "type": "number", "min": 0, "max": 86400000 }, + "stream": { "type": "enum", "values": ["done", "error", "aborted"] }, + "credits": { "type": "integer", "min": 0, "max": 1000000000 }, + "account_hash": { "type": "string", "regex": "^[0-9a-f]{64}$", "max_len": 64 }, + "route_template": { "type": "string", "regex": "^(?:unmatched|/[A-Za-z0-9_.:*/-]{0,63})$", "max_len": 64 } + } + }, "tinycloud-mcp": { "events": [], "fields": {} }, "ptx-dev": { "events": [], "fields": {} }, "openkey-api": { "events": [], "fields": {} } diff --git a/scripts/ci/fixtures/tc959-n0-compose.yaml b/scripts/ci/fixtures/tc959-n0-compose.yaml index e2d3f294..dda8ef86 100644 --- a/scripts/ci/fixtures/tc959-n0-compose.yaml +++ b/scripts/ci/fixtures/tc959-n0-compose.yaml @@ -174,7 +174,7 @@ services: environment: TINYCLOUD_LOGSHIP_PROFILE: ${TINYCLOUD_LOGSHIP_PROFILE:-errors_slow} BETTERSTACK_SOURCE_TOKEN: ${BETTERSTACK_SOURCE_TOKEN:?seal the node source token before enabling log shipping} - VECTOR_CONFIG_JSON: "{\"data_dir\":\"/var/lib/vector\",\"sources\":{\"gelf_in\":{\"type\":\"socket\",\"mode\":\"udp\",\"address\":\"0.0.0.0:12201\",\"max_length\":262144,\"receive_buffer_bytes\":8388608,\"framing\":{\"method\":\"chunked_gelf\",\"chunked_gelf\":{\"decompression\":\"Auto\"}},\"decoding\":{\"codec\":\"bytes\"}}},\"transforms\":{\"gelf_envelope\":{\"type\":\"remap\",\"inputs\":[\"gelf_in\"],\"source\":\"packet = string(.message) ?? \\\"\\\"\\nif length(packet) > 262144 {\\n . = {\\\"reason\\\": \\\"oversized\\\"}\\n} else {\\n outer, err = parse_json(packet)\\n if err != null || !is_object(outer) || (outer.version != \\\"1.1\\\" && outer.version != \\\"1.0\\\") {\\n . = {\\\"reason\\\": \\\"malformed_envelope\\\"}\\n } else if exists(outer.partial_message) || exists(outer._partial_message) {\\n . = {\\\"reason\\\": \\\"partial\\\"}\\n } else {\\n label = get(outer, [\\\"_tc.logship\\\"]) ?? null\\n . = {\\\"_tc.logship\\\": label, \\\"message\\\": outer.short_message}\\n }\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"logship_gate\":{\"type\":\"remap\",\"inputs\":[\"gelf_envelope\"],\"source\":\"if exists(.reason) {\\n . = {\\\"reason\\\": .reason}\\n} else {\\nlabel = string(get(., [\\\"_tc.logship\\\"]) ?? null) ?? \\\"\\\"\\nraw = .message\\nreason = \\\"\\\"\\nif label == \\\"\\\" { reason = \\\"unlabelled\\\" }\\nelse if !includes([\\\"tinycloud-node\\\", \\\"tinycloud-ingress\\\"], label) { reason = \\\"unexpected_label\\\" }\\nelse if exists(.partial_message) { reason = \\\"partial\\\" }\\nelse if !is_string(raw) { reason = \\\"malformed_envelope\\\" }\\nelse if length(string!(raw)) > 65536 { reason = \\\"oversized\\\" }\\nif reason != \\\"\\\" {\\n . = {\\\"reason\\\": reason}\\n} else {\\n parsed, err = parse_json(string!(raw))\\n if err != null || !is_object(parsed) { . = {\\\"reason\\\": \\\"non_json\\\"} }\\n else if exists(parsed.partial_message) { . = {\\\"reason\\\": \\\"partial\\\"} }\\n else {\\n target = string(parsed.target) ?? \\\"\\\"\\n level = downcase(string(parsed.level) ?? \\\"\\\")\\n if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && target != \\\"tinycloud\\\" && !starts_with(target, \\\"tinycloud::\\\") && target != \\\"tinycloud_core\\\" && !starts_with(target, \\\"tinycloud_core::\\\") {\\n . = {\\\"reason\\\": \\\"unknown_target\\\"}\\n } else if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && !((target == \\\"tinycloud::request\\\" || target == \\\"tinycloud::disk_headroom\\\") && level == \\\"info\\\") {\\n . = {\\\"reason\\\": \\\"filtered_level\\\"}\\n } else { . = {\\\"label\\\": label, \\\"parsed\\\": parsed} }\\n }\\n}\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_route\":{\"type\":\"route\",\"inputs\":[\"logship_gate\"],\"route\":{\"candidate\":\"exists(.parsed)\",\"rejected\":\"exists(.reason)\"}},\"allowlist\":{\"type\":\"remap\",\"inputs\":[\"gate_route.candidate\"],\"source\":\"# Only this fresh object can leave the process boundary. No envelope, span,\\n# message, error, identity, URI or unknown key is copied into it.\\nj = .parsed\\nlabel = .label\\nout = {}\\nts = \\\"\\\"\\nif label == \\\"tinycloud-node\\\" {\\n target = string(j.target) ?? \\\"\\\"\\n level = downcase(string!(j.level))\\n f = object(j.fields) ?? {}\\n msg = string(f.message) ?? \\\"\\\"\\n ts = string!(j.timestamp)\\n # Literal Rocket template aliases only; never split an arbitrary URI.\\n aliases = {\\\"/attestation?\\\":\\\"/attestation\\\",\\\"/.well-known/encryption/network/?\\\":\\\"/.well-known/encryption/network/\\\",\\\"/hooks/events?\\\":\\\"/hooks/events\\\",\\\"/hooks/webhooks?\\\":\\\"/hooks/webhooks\\\",\\\"/public//kv?\\\":\\\"/public//kv\\\"}\\n # TC-1840: the node's one 5xx diagnostic line per database failure ships as an individual event.\\n # The input must carry exactly the declared keys (the message plus these seven, and db_hash only\\n # when it is a valid pseudonym) with exactly the declared types and enum values. Anything else\\n # leaves db_event null and the line falls back to the unclassified count below; no value of it\\n # is ever copied unless it passed, and the shipped object is built from literals and the\\n # validated values only.\\n db_event = null\\n if target == \\\"tinycloud::request\\\" && level == \\\"warn\\\" && msg == \\\"database failure answered with a 5xx\\\" {\\n db_status = f.status\\n db_elapsed = f.elapsed_ms\\n db_code = string(f.code) ?? \\\"\\\"\\n db_service = string(f.service) ?? \\\"\\\"\\n db_kind = string(f.failure_kind) ?? \\\"\\\"\\n db_route = string(f.route) ?? \\\"\\\"\\n db_canonical_route = get(aliases, [db_route]) ?? null\\n if is_string(db_canonical_route) { db_route = db_canonical_route }\\n db_ok = is_integer(db_status) && is_integer(db_elapsed) && is_boolean(f.write_may_have_applied)\\n if db_ok {\\n db_status = int!(db_status)\\n db_elapsed = int!(db_elapsed)\\n db_ok = db_status >= 500 && db_status <= 599 && db_elapsed >= 0 && db_elapsed <= 9007199254740991 && includes([\\\"unavailable_retryable\\\",\\\"outcome_unknown\\\",\\\"internal\\\",\\\"sql_busy\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\"], db_code) && includes([\\\"sql\\\",\\\"duckdb\\\",\\\"kv\\\",\\\"other\\\"], db_service) && includes([\\\"pool_timeout\\\",\\\"pool_closed\\\",\\\"io\\\",\\\"statement_timeout\\\",\\\"serialization\\\",\\\"connection_lost\\\",\\\"artifact_changed\\\",\\\"lock_wait\\\",\\\"hydration\\\",\\\"artifact_transfer\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\",\\\"other\\\"], db_kind) && includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], db_route)\\n }\\n db_hash = string(f.db_hash) ?? \\\"\\\"\\n has_db_hash = exists(f.db_hash)\\n if db_ok {\\n if has_db_hash { db_ok = match(db_hash, r'^[0-9a-f]{64}$') && length(f) == 9 } else { db_ok = length(f) == 8 }\\n }\\n if db_ok {\\n db_event = {\\\"event\\\": \\\"node_warning\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"warn\\\", \\\"event_code\\\": \\\"db_failure_5xx\\\", \\\"status\\\": db_status, \\\"code\\\": db_code, \\\"service\\\": db_service, \\\"route\\\": db_route, \\\"elapsed_ms\\\": db_elapsed, \\\"failure_kind\\\": db_kind, \\\"write_may_have_applied\\\": bool!(f.write_may_have_applied)}\\n if has_db_hash { db_event.db_hash = db_hash }\\n }\\n }\\n # TC-2917: the node's periodic disk headroom line (one per local store) ships as an individual\\n # info event. Exactly the message plus `store` (a declared store) and `free_bytes` (a bounded\\n # integer in whole 64 MiB steps, as the node rounds it); anything else leaves headroom null, and\\n # the info line is then dropped below.\\n headroom = null\\n if target == \\\"tinycloud::disk_headroom\\\" && level == \\\"info\\\" && msg == \\\"storage headroom\\\" && length(f) == 3 {\\n h_store = string(f.store) ?? \\\"\\\"\\n h_free = f.free_bytes\\n if includes([\\\"sql\\\",\\\"duckdb\\\"], h_store) && is_integer(h_free) {\\n h_free = int!(h_free)\\n if h_free >= 0 && h_free <= 9007199254740991 && mod(h_free, 67108864) == 0 {\\n headroom = {\\\"event\\\": \\\"node_storage_headroom\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"store\\\": h_store, \\\"free_bytes\\\": h_free}\\n }\\n }\\n }\\n if target == \\\"tinycloud::request\\\" && level == \\\"info\\\" && msg == \\\"request\\\" {\\n method = string!(f.method)\\n if includes([\\\"TRACE\\\", \\\"CONNECT\\\"], method) { method = \\\"\\\" }\\n route = string!(f.route)\\n canonical_route = get(aliases, [route]) ?? null\\n if is_string(canonical_route) { route = canonical_route }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = f.status\\n latency = f.latency_us\\n if !is_integer(status) || !is_integer(latency) { abort }\\n status = int!(status)\\n latency = int!(latency)\\n if status < 100 || status > 599 || latency < 0 || latency > 9007199254740991 { abort }\\n out = {\\\"event\\\": \\\"node_request\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"latency_us\\\": latency}\\n # N2 correlation fields: copied only when they have exactly the node's emitted shape; anything else is dropped\\n # (the event still ships). The pseudonyms are keyed HMAC-SHA256 digests, never raw identifiers.\\n rid = string(f.request_id) ?? \\\"\\\"\\n if match(rid, r'^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$') { out.request_id = rid }\\n replaced = f.request_id_replaced\\n if is_boolean(replaced) { out.request_id_replaced = replaced }\\n account = string(f.account_hash) ?? \\\"\\\"\\n if match(account, r'^[0-9a-f]{64}$') { out.account_hash = account }\\n space = string(f.space_hash) ?? \\\"\\\"\\n if match(space, r'^[0-9a-f]{64}$') { out.space_hash = space }\\n } else if is_object(db_event) {\\n out = db_event\\n } else if is_object(headroom) {\\n out = headroom\\n } else {\\n if !includes([\\\"warn\\\", \\\"error\\\"], level) { abort }\\n code = \\\"unclassified\\\"\\n if target == \\\"tinycloud\\\" || starts_with(target, \\\"tinycloud::\\\") || target == \\\"tinycloud_core\\\" || starts_with(target, \\\"tinycloud_core::\\\") {\\n codes = {\\\"invocation replay insert failed\\\":\\\"invocation_replay_insert_failed\\\",\\\"invocation replay cleanup failed\\\":\\\"invocation_replay_cleanup_failed\\\",\\\"share-v2 local-tee feature active: deriving a deterministic, non-attested TEE identity from node key material\\\":\\\"share_local_tee_identity\\\",\\\"retention: delivered hook delivery prune failed\\\":\\\"retention_hook_delivery_prune_failed\\\",\\\"retention: dead-letter hook delivery prune failed\\\":\\\"retention_dead_letter_prune_failed\\\",\\\"retention: signed KV ticket prune failed\\\":\\\"retention_signed_ticket_prune_failed\\\",\\\"failed to load tunnel state; tunnel not started\\\":\\\"tunnel_state_load_failed\\\",\\\"invalid loopback API address; tunnel not started\\\":\\\"tunnel_address_invalid\\\",\\\"failed to load link state; LAN listener not started\\\":\\\"link_state_load_failed\\\",\\\"invalid link bind address; LAN listener not started\\\":\\\"link_bind_invalid\\\",\\\"invalid loopback API address; LAN listener not started\\\":\\\"link_address_invalid\\\",\\\"failed to bind link LAN listener; LAN listener not started\\\":\\\"link_bind_failed\\\",\\\"failed to build TLS config; LAN listener not started\\\":\\\"link_tls_config_failed\\\",\\\"link LAN listener exited with error\\\":\\\"link_listener_error\\\",\\\"failed to record link listener bind state\\\":\\\"link_listener_state_write_failed\\\",\\\"failed to record link listener bind failure\\\":\\\"link_bind_state_write_failed\\\",\\\"failed to load config for auto-renew\\\":\\\"link_auto_renew_config_failed\\\",\\\"link auto-renew failed\\\":\\\"link_auto_renew_failed\\\",\\\"failed to hot-reload renewed link cert\\\":\\\"link_cert_reload_failed\\\",\\\"link auto-renew succeeded but no state.json found; skipping cert hot-reload\\\":\\\"link_cert_state_missing\\\",\\\"failed to reload TLS material after link auto-renew\\\":\\\"link_listener_cert_reload_failed\\\",\\\"link disable: service delete failed; local state removed anyway\\\":\\\"link_disable_failed\\\",\\\"link connection error\\\":\\\"link_connection_error\\\",\\\"link listener accept error\\\":\\\"link_listener_accept_failed\\\",\\\"webhook delivery attempt failed\\\":\\\"webhook_delivery_failed\\\",\\\"webhook dispatcher loop failed\\\":\\\"webhook_dispatcher_failed\\\",\\\"addressed delegation authorization failed\\\":\\\"share_authorization_failed\\\",\\\"native share authorization denied\\\":\\\"share_native_authorization_denied\\\",\\\"native share KV put failed\\\":\\\"share_native_kv_put_failed\\\",\\\"share v2 readiness probe failed\\\":\\\"share_v2_readiness_failed\\\",\\\"share v2 is enabled but shareEmail.invitationPublicKey is unset\\\":\\\"share_invitation_key_missing\\\",\\\"configured share invitation key does not match the node's derived signing key\\\":\\\"share_invitation_key_mismatch\\\",\\\"share/v2 policy challenge denied\\\":\\\"share_policy_denied\\\",\\\"failed to read committed hook events\\\":\\\"hook_events_read_failed\\\",\\\"failed to load kv write hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"failed to load kv delete hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"missing committed kv hook row for invocation\\\":\\\"missing_committed_hook_row\\\",\\\"database write committed but webhook enqueue failed\\\":\\\"webhook_enqueue_failed\\\",\\\"tunnel: failed to prepare connection attempt\\\":\\\"tunnel_connection_prepare_failed\\\",\\\"tunnel: no frame within idle timeout; treating connection as dead\\\":\\\"tunnel_idle_timeout\\\",\\\"tunnel: read error\\\":\\\"tunnel_read_failed\\\",\\\"tunnel: dropping malformed frame\\\":\\\"tunnel_malformed_frame\\\",\\\"tunnel: relay sent an error frame\\\":\\\"tunnel_relay_error\\\",\\\"tunnel: failed to write frame\\\":\\\"tunnel_write_failed\\\",\\\"tunnel: failed to record runtime state\\\":\\\"tunnel_state_record_failed\\\",\\\"tunnel: failed to clear runtime state after disable\\\":\\\"tunnel_state_clear_failed\\\",\\\"tunnel authentication rejected\\\":\\\"tunnel_auth_rejected\\\",\\\"network DID resolution enabled: did:web/did:tz fetches use the unhardened ssi client (TC-1080)\\\":\\\"network_did_resolution_enabled\\\",\\\"SQL busy: lock wait\\\":\\\"sql_busy_lock_wait\\\",\\\"SQL busy: hydration\\\":\\\"sql_busy_hydration\\\",\\\"SQL busy: artifact transfer\\\":\\\"sql_busy_artifact_transfer\\\",\\\"Request deadline exceeded\\\":\\\"request_timeout\\\",\\\"Request completed after its deadline\\\":\\\"deadline_overrun\\\",\\\"Database hydration gave up loading its artifact\\\":\\\"sql_hydration_timeout\\\",\\\"Read audit wait expired\\\":\\\"read_audit_busy\\\",\\\"Read audit batch slow\\\":\\\"read_audit_batch_slow\\\",\\\"Read audit batch failed\\\":\\\"read_audit_batch_failed\\\",\\\"Read audit drain stalled\\\":\\\"read_audit_drain_stalled\\\",\\\"retrying operation after a lost database connection\\\":\\\"db_operation_retried\\\",\\\"not retrying operation: its first run took longer than the retry window to fail\\\":\\\"db_operation_retry_skipped_slow\\\",\\\"idx_current_kv_space_deleted_key_c is missing; the node keeps serving and KV list requests are slower until an operator builds it with CREATE INDEX CONCURRENTLY (docs/releasing.md, section idx_current_kv_space_deleted_key_c)\\\":\\\"current_kv_c_index_missing\\\",\\\"idx_current_kv_space_deleted_key_c is invalid or still being built; the node keeps serving and KV list requests are slower. If no CREATE INDEX CONCURRENTLY is running, run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_invalid\\\",\\\"idx_current_kv_space_deleted_key_c exists with the wrong definition and is not used for KV lists; the node keeps serving and KV list requests are slower. Run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_wrong_shape\\\"}\\n known = get(codes, [msg]) ?? null\\n if is_string(known) { code = known }\\n if starts_with(msg, \\\"quota service returned \\\") { code = \\\"quota_fallback\\\" }\\nif starts_with(msg, \\\"Failed to get TDX quote: \\\") { code = \\\"attestation_quote_failed\\\" }\\nif starts_with(msg, \\\"quota service returned an invalid body: \\\") { code = \\\"quota_invalid_body\\\" }\\nif starts_with(msg, \\\"dstack socket available but get_info failed: \\\") { code = \\\"dstack_get_info_failed\\\" }\\n }\\n event = if level == \\\"warn\\\" { \\\"node_warning\\\" } else { \\\"node_error\\\" }\\n out = {\\\"event\\\": event, \\\"svc\\\": \\\"api\\\", \\\"level\\\": level, \\\"event_code\\\": code, \\\"event_count\\\": 1}\\n }\\n} else {\\n ts = string!(j.time)\\n method = string!(j.method)\\n route = string!(j.route)\\n if route == \\\"\\\" { route = \\\"\\\" }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = j.status\\n request_time = j.request_time\\n if !is_integer(status) || !(is_integer(request_time) || is_float(request_time)) { abort }\\n status = int!(status)\\n request_time = to_float!(request_time)\\n if status < 100 || status > 599 || request_time < 0 || request_time > 86400 { abort }\\n out = {\\\"event\\\": \\\"ingress_request\\\", \\\"svc\\\": \\\"ingress\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"request_time_seconds\\\": request_time}\\n}\\nif length(ts) > 64 { abort }\\nparsed_ts = parse_timestamp!(ts, format: \\\"%+\\\")\\nout.ts = format_timestamp!(parsed_ts, format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\")\\n. = out\\n\",\"drop_on_error\":true,\"drop_on_abort\":true,\"reroute_dropped\":true},\"gate_counts\":{\"type\":\"remap\",\"inputs\":[\"gate_route.rejected\",\"gate_route._unmatched\",\"allowlist.dropped\",\"event_route._unmatched\"],\"source\":\"reason = .reason\\nif !includes([\\\"malformed_envelope\\\",\\\"unexpected_label\\\",\\\"unlabelled\\\",\\\"non_json\\\",\\\"oversized\\\",\\\"partial\\\",\\\"invalid_fields\\\",\\\"unknown_target\\\",\\\"filtered_level\\\"], reason) { reason = \\\"invalid_fields\\\" }\\n. = {\\\"reason\\\": reason, \\\"count\\\": 1}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_reduce\":{\"type\":\"reduce\",\"inputs\":[\"gate_counts\"],\"group_by\":[\"reason\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"count\":\"sum\"}},\"gate_output\":{\"type\":\"remap\",\"inputs\":[\"gate_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": \\\"warn\\\", \\\"event\\\": \\\"logship_gate_summary_total\\\", \\\"svc\\\": \\\"logship\\\", \\\"reason\\\": .reason, \\\"count\\\": .count}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"request_profile\":{\"type\":\"filter\",\"inputs\":[\"allowlist\"],\"condition\":\"\\\"$${TINYCLOUD_LOGSHIP_PROFILE}\\\" == \\\"all_requests\\\" || includes([\\\"node_warning\\\", \\\"node_error\\\", \\\"node_storage_headroom\\\"], .event) || (int(.status) ?? 0) >= 400 || (.event == \\\"node_request\\\" && (int(.latency_us) ?? 0) > 1000000) || (.event == \\\"ingress_request\\\" && (to_float(.request_time_seconds) ?? 0.0) > 1.000000)\"},\"event_route\":{\"type\":\"route\",\"inputs\":[\"request_profile\"],\"route\":{\"request\":\"(.event == \\\"node_request\\\" || .event == \\\"ingress_request\\\") && !(((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)) && !((int(.status) ?? 0) >= 500)\",\"node_request_5xx\":\".event == \\\"node_request\\\" && (int(.status) ?? 0) >= 500\",\"ingress_request_5xx\":\".event == \\\"ingress_request\\\" && (int(.status) ?? 0) >= 500\",\"node_request_4xx\":\".event == \\\"node_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"ingress_request_4xx\":\".event == \\\"ingress_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"db_failure_5xx\":\".event == \\\"node_warning\\\" && .event_code == \\\"db_failure_5xx\\\"\",\"storage_headroom\":\".event == \\\"node_storage_headroom\\\"\",\"diagnostic\":\"(.event == \\\"node_warning\\\" && .event_code != \\\"db_failure_5xx\\\") || .event == \\\"node_error\\\"\"}},\"cap_db_failure_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.db_failure_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"db_failure_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"diagnostic_reduce\":{\"type\":\"reduce\",\"inputs\":[\"event_route.diagnostic\"],\"group_by\":[\"event\",\"event_code\",\"level\",\"svc\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"event_count\":\"sum\",\"ts\":\"discard\"}},\"diagnostic_output\":{\"type\":\"remap\",\"inputs\":[\"diagnostic_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": .level, \\\"event\\\": .event, \\\"svc\\\": \\\"api\\\", \\\"event_code\\\": .event_code, \\\"event_count\\\": .event_count}\",\"drop_on_error\":true,\"drop_on_abort\":true}},\"sinks\":{\"betterstack\":{\"type\":\"http\",\"inputs\":[\"event_route.request\",\"event_route.storage_headroom\",\"cap_db_failure_5xx\",\"cap_node_request_5xx\",\"cap_ingress_request_5xx\",\"cap_node_request_4xx\",\"cap_ingress_request_4xx\",\"diagnostic_output\",\"gate_output\"],\"method\":\"post\",\"uri\":\"https://s2795437.us-west-2a.betterstackdata.com/\",\"encoding\":{\"codec\":\"json\"},\"framing\":{\"method\":\"newline_delimited\"},\"compression\":\"gzip\",\"auth\":{\"strategy\":\"bearer\",\"token\":\"SECRET[bs.bs_token]\"},\"batch\":{\"max_bytes\":262144,\"timeout_secs\":1},\"buffer\":{\"type\":\"memory\",\"max_events\":1000,\"when_full\":\"block\"},\"request\":{\"timeout_secs\":10,\"retry_max_duration_secs\":300},\"healthcheck\":{\"enabled\":false}}},\"secret\":{\"bs\":{\"type\":\"directory\",\"path\":\"/var/lib/vector/secrets\",\"remove_trailing_whitespace\":true}}}" + VECTOR_CONFIG_JSON: "{\"data_dir\":\"/var/lib/vector\",\"sources\":{\"gelf_in\":{\"type\":\"socket\",\"mode\":\"udp\",\"address\":\"0.0.0.0:12201\",\"max_length\":262144,\"receive_buffer_bytes\":8388608,\"framing\":{\"method\":\"chunked_gelf\",\"chunked_gelf\":{\"decompression\":\"Auto\"}},\"decoding\":{\"codec\":\"bytes\"}}},\"transforms\":{\"gelf_envelope\":{\"type\":\"remap\",\"inputs\":[\"gelf_in\"],\"source\":\"packet = string(.message) ?? \\\"\\\"\\nif length(packet) > 262144 {\\n . = {\\\"reason\\\": \\\"oversized\\\"}\\n} else {\\n outer, err = parse_json(packet)\\n if err != null || !is_object(outer) || (outer.version != \\\"1.1\\\" && outer.version != \\\"1.0\\\") {\\n . = {\\\"reason\\\": \\\"malformed_envelope\\\"}\\n } else if exists(outer.partial_message) || exists(outer._partial_message) {\\n . = {\\\"reason\\\": \\\"partial\\\"}\\n } else {\\n label = get(outer, [\\\"_tc.logship\\\"]) ?? null\\n . = {\\\"_tc.logship\\\": label, \\\"message\\\": outer.short_message}\\n }\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"logship_gate\":{\"type\":\"remap\",\"inputs\":[\"gelf_envelope\"],\"source\":\"if exists(.reason) {\\n . = {\\\"reason\\\": .reason}\\n} else {\\nlabel = string(get(., [\\\"_tc.logship\\\"]) ?? null) ?? \\\"\\\"\\nraw = .message\\nreason = \\\"\\\"\\nif label == \\\"\\\" { reason = \\\"unlabelled\\\" }\\nelse if !includes([\\\"tinycloud-node\\\", \\\"tinycloud-ingress\\\"], label) { reason = \\\"unexpected_label\\\" }\\nelse if exists(.partial_message) { reason = \\\"partial\\\" }\\nelse if !is_string(raw) { reason = \\\"malformed_envelope\\\" }\\nelse if length(string!(raw)) > 65536 { reason = \\\"oversized\\\" }\\nif reason != \\\"\\\" {\\n . = {\\\"reason\\\": reason}\\n} else {\\n parsed, err = parse_json(string!(raw))\\n if err != null || !is_object(parsed) { . = {\\\"reason\\\": \\\"non_json\\\"} }\\n else if exists(parsed.partial_message) { . = {\\\"reason\\\": \\\"partial\\\"} }\\n else {\\n target = string(parsed.target) ?? \\\"\\\"\\n level = downcase(string(parsed.level) ?? \\\"\\\")\\n if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && target != \\\"tinycloud\\\" && !starts_with(target, \\\"tinycloud::\\\") && target != \\\"tinycloud_core\\\" && !starts_with(target, \\\"tinycloud_core::\\\") {\\n . = {\\\"reason\\\": \\\"unknown_target\\\"}\\n } else if label == \\\"tinycloud-node\\\" && !includes([\\\"warn\\\", \\\"error\\\"], level) && !((target == \\\"tinycloud::request\\\" || target == \\\"tinycloud::disk_headroom\\\") && level == \\\"info\\\") {\\n . = {\\\"reason\\\": \\\"filtered_level\\\"}\\n } else { . = {\\\"label\\\": label, \\\"parsed\\\": parsed} }\\n }\\n}\\n}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_route\":{\"type\":\"route\",\"inputs\":[\"logship_gate\"],\"route\":{\"candidate\":\"exists(.parsed)\",\"rejected\":\"exists(.reason)\"}},\"allowlist\":{\"type\":\"remap\",\"inputs\":[\"gate_route.candidate\"],\"source\":\"# Only this fresh object can leave the process boundary. No envelope, span,\\n# message, error, identity, URI or unknown key is copied into it.\\nj = .parsed\\nlabel = .label\\nout = {}\\nts = \\\"\\\"\\nif label == \\\"tinycloud-node\\\" {\\n target = string(j.target) ?? \\\"\\\"\\n level = downcase(string!(j.level))\\n f = object(j.fields) ?? {}\\n msg = string(f.message) ?? \\\"\\\"\\n ts = string!(j.timestamp)\\n # Literal Rocket template aliases only; never split an arbitrary URI.\\n aliases = {\\\"/attestation?\\\":\\\"/attestation\\\",\\\"/.well-known/encryption/network/?\\\":\\\"/.well-known/encryption/network/\\\",\\\"/hooks/events?\\\":\\\"/hooks/events\\\",\\\"/hooks/webhooks?\\\":\\\"/hooks/webhooks\\\",\\\"/public//kv?\\\":\\\"/public//kv\\\"}\\n # TC-1840: the node's one 5xx diagnostic line per database failure ships as an individual event.\\n # The input must carry exactly the declared keys (the message plus these seven, and db_hash only\\n # when it is a valid pseudonym) with exactly the declared types and enum values. Anything else\\n # leaves db_event null and the line falls back to the unclassified count below; no value of it\\n # is ever copied unless it passed, and the shipped object is built from literals and the\\n # validated values only.\\n db_event = null\\n if target == \\\"tinycloud::request\\\" && level == \\\"warn\\\" && msg == \\\"database failure answered with a 5xx\\\" {\\n db_status = f.status\\n db_elapsed = f.elapsed_ms\\n db_code = string(f.code) ?? \\\"\\\"\\n db_service = string(f.service) ?? \\\"\\\"\\n db_kind = string(f.failure_kind) ?? \\\"\\\"\\n db_route = string(f.route) ?? \\\"\\\"\\n db_canonical_route = get(aliases, [db_route]) ?? null\\n if is_string(db_canonical_route) { db_route = db_canonical_route }\\n db_ok = is_integer(db_status) && is_integer(db_elapsed) && is_boolean(f.write_may_have_applied)\\n if db_ok {\\n db_status = int!(db_status)\\n db_elapsed = int!(db_elapsed)\\n db_ok = db_status >= 500 && db_status <= 599 && db_elapsed >= 0 && db_elapsed <= 9007199254740991 && includes([\\\"unavailable_retryable\\\",\\\"outcome_unknown\\\",\\\"internal\\\",\\\"sql_busy\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\"], db_code) && includes([\\\"sql\\\",\\\"duckdb\\\",\\\"kv\\\",\\\"other\\\"], db_service) && includes([\\\"pool_timeout\\\",\\\"pool_closed\\\",\\\"io\\\",\\\"statement_timeout\\\",\\\"serialization\\\",\\\"connection_lost\\\",\\\"artifact_changed\\\",\\\"lock_wait\\\",\\\"hydration\\\",\\\"artifact_transfer\\\",\\\"request_timeout\\\",\\\"read_audit_busy\\\",\\\"other\\\"], db_kind) && includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], db_route)\\n }\\n db_hash = string(f.db_hash) ?? \\\"\\\"\\n has_db_hash = exists(f.db_hash)\\n if db_ok {\\n if has_db_hash { db_ok = match(db_hash, r'^[0-9a-f]{64}$') && length(f) == 9 } else { db_ok = length(f) == 8 }\\n }\\n if db_ok {\\n db_event = {\\\"event\\\": \\\"node_warning\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"warn\\\", \\\"event_code\\\": \\\"db_failure_5xx\\\", \\\"status\\\": db_status, \\\"code\\\": db_code, \\\"service\\\": db_service, \\\"route\\\": db_route, \\\"elapsed_ms\\\": db_elapsed, \\\"failure_kind\\\": db_kind, \\\"write_may_have_applied\\\": bool!(f.write_may_have_applied)}\\n if has_db_hash { db_event.db_hash = db_hash }\\n }\\n }\\n # TC-2917: the node's periodic disk headroom line (one per local store) ships as an individual\\n # info event. Exactly the message plus `store` (a declared store) and `free_bytes` (a bounded\\n # integer in whole 64 MiB steps, as the node rounds it); anything else leaves headroom null, and\\n # the info line is then dropped below.\\n headroom = null\\n if target == \\\"tinycloud::disk_headroom\\\" && level == \\\"info\\\" && msg == \\\"storage headroom\\\" && length(f) == 3 {\\n h_store = string(f.store) ?? \\\"\\\"\\n h_free = f.free_bytes\\n if includes([\\\"sql\\\",\\\"duckdb\\\"], h_store) && is_integer(h_free) {\\n h_free = int!(h_free)\\n if h_free >= 0 && h_free <= 9007199254740991 && mod(h_free, 67108864) == 0 {\\n headroom = {\\\"event\\\": \\\"node_storage_headroom\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"store\\\": h_store, \\\"free_bytes\\\": h_free}\\n }\\n }\\n }\\n if target == \\\"tinycloud::request\\\" && level == \\\"info\\\" && msg == \\\"request\\\" {\\n method = string!(f.method)\\n if includes([\\\"TRACE\\\", \\\"CONNECT\\\"], method) { method = \\\"\\\" }\\n route = string!(f.route)\\n canonical_route = get(aliases, [route]) ?? null\\n if is_string(canonical_route) { route = canonical_route }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = f.status\\n latency = f.latency_us\\n if !is_integer(status) || !is_integer(latency) { abort }\\n status = int!(status)\\n latency = int!(latency)\\n if status < 100 || status > 599 || latency < 0 || latency > 9007199254740991 { abort }\\n out = {\\\"event\\\": \\\"node_request\\\", \\\"svc\\\": \\\"api\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"latency_us\\\": latency}\\n # N2 correlation fields: copied only when they have exactly the node's emitted shape; anything else is dropped\\n # (the event still ships). The pseudonyms are keyed HMAC-SHA256 digests, never raw identifiers.\\n rid = string(f.request_id) ?? \\\"\\\"\\n if match(rid, r'^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$') { out.request_id = rid }\\n replaced = f.request_id_replaced\\n if is_boolean(replaced) { out.request_id_replaced = replaced }\\n account = string(f.account_hash) ?? \\\"\\\"\\n if match(account, r'^[0-9a-f]{64}$') { out.account_hash = account }\\n space = string(f.space_hash) ?? \\\"\\\"\\n if match(space, r'^[0-9a-f]{64}$') { out.space_hash = space }\\n } else if is_object(db_event) {\\n out = db_event\\n } else if is_object(headroom) {\\n out = headroom\\n } else {\\n if !includes([\\\"warn\\\", \\\"error\\\"], level) { abort }\\n code = \\\"unclassified\\\"\\n if target == \\\"tinycloud\\\" || starts_with(target, \\\"tinycloud::\\\") || target == \\\"tinycloud_core\\\" || starts_with(target, \\\"tinycloud_core::\\\") {\\n codes = {\\\"invocation replay insert failed\\\":\\\"invocation_replay_insert_failed\\\",\\\"invocation replay cleanup failed\\\":\\\"invocation_replay_cleanup_failed\\\",\\\"share-v2 local-tee feature active: deriving a deterministic, non-attested TEE identity from node key material\\\":\\\"share_local_tee_identity\\\",\\\"retention: delivered hook delivery prune failed\\\":\\\"retention_hook_delivery_prune_failed\\\",\\\"retention: dead-letter hook delivery prune failed\\\":\\\"retention_dead_letter_prune_failed\\\",\\\"retention: signed KV ticket prune failed\\\":\\\"retention_signed_ticket_prune_failed\\\",\\\"failed to load tunnel state; tunnel not started\\\":\\\"tunnel_state_load_failed\\\",\\\"invalid loopback API address; tunnel not started\\\":\\\"tunnel_address_invalid\\\",\\\"failed to load link state; LAN listener not started\\\":\\\"link_state_load_failed\\\",\\\"invalid link bind address; LAN listener not started\\\":\\\"link_bind_invalid\\\",\\\"invalid loopback API address; LAN listener not started\\\":\\\"link_address_invalid\\\",\\\"failed to bind link LAN listener; LAN listener not started\\\":\\\"link_bind_failed\\\",\\\"failed to build TLS config; LAN listener not started\\\":\\\"link_tls_config_failed\\\",\\\"link LAN listener exited with error\\\":\\\"link_listener_error\\\",\\\"failed to record link listener bind state\\\":\\\"link_listener_state_write_failed\\\",\\\"failed to record link listener bind failure\\\":\\\"link_bind_state_write_failed\\\",\\\"failed to load config for auto-renew\\\":\\\"link_auto_renew_config_failed\\\",\\\"link auto-renew failed\\\":\\\"link_auto_renew_failed\\\",\\\"failed to hot-reload renewed link cert\\\":\\\"link_cert_reload_failed\\\",\\\"link auto-renew succeeded but no state.json found; skipping cert hot-reload\\\":\\\"link_cert_state_missing\\\",\\\"failed to reload TLS material after link auto-renew\\\":\\\"link_listener_cert_reload_failed\\\",\\\"link disable: service delete failed; local state removed anyway\\\":\\\"link_disable_failed\\\",\\\"link connection error\\\":\\\"link_connection_error\\\",\\\"link listener accept error\\\":\\\"link_listener_accept_failed\\\",\\\"webhook delivery attempt failed\\\":\\\"webhook_delivery_failed\\\",\\\"webhook dispatcher loop failed\\\":\\\"webhook_dispatcher_failed\\\",\\\"addressed delegation authorization failed\\\":\\\"share_authorization_failed\\\",\\\"native share authorization denied\\\":\\\"share_native_authorization_denied\\\",\\\"native share KV put failed\\\":\\\"share_native_kv_put_failed\\\",\\\"share v2 readiness probe failed\\\":\\\"share_v2_readiness_failed\\\",\\\"share v2 is enabled but shareEmail.invitationPublicKey is unset\\\":\\\"share_invitation_key_missing\\\",\\\"configured share invitation key does not match the node's derived signing key\\\":\\\"share_invitation_key_mismatch\\\",\\\"share/v2 policy challenge denied\\\":\\\"share_policy_denied\\\",\\\"failed to read committed hook events\\\":\\\"hook_events_read_failed\\\",\\\"failed to load kv write hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"failed to load kv delete hook rows\\\":\\\"hook_rows_load_failed\\\",\\\"missing committed kv hook row for invocation\\\":\\\"missing_committed_hook_row\\\",\\\"database write committed but webhook enqueue failed\\\":\\\"webhook_enqueue_failed\\\",\\\"tunnel: failed to prepare connection attempt\\\":\\\"tunnel_connection_prepare_failed\\\",\\\"tunnel: no frame within idle timeout; treating connection as dead\\\":\\\"tunnel_idle_timeout\\\",\\\"tunnel: read error\\\":\\\"tunnel_read_failed\\\",\\\"tunnel: dropping malformed frame\\\":\\\"tunnel_malformed_frame\\\",\\\"tunnel: relay sent an error frame\\\":\\\"tunnel_relay_error\\\",\\\"tunnel: failed to write frame\\\":\\\"tunnel_write_failed\\\",\\\"tunnel: failed to record runtime state\\\":\\\"tunnel_state_record_failed\\\",\\\"tunnel: failed to clear runtime state after disable\\\":\\\"tunnel_state_clear_failed\\\",\\\"tunnel authentication rejected\\\":\\\"tunnel_auth_rejected\\\",\\\"network DID resolution enabled: did:web/did:tz fetches use the unhardened ssi client (TC-1080)\\\":\\\"network_did_resolution_enabled\\\",\\\"SQL busy: lock wait\\\":\\\"sql_busy_lock_wait\\\",\\\"SQL busy: hydration\\\":\\\"sql_busy_hydration\\\",\\\"SQL busy: artifact transfer\\\":\\\"sql_busy_artifact_transfer\\\",\\\"Request deadline exceeded\\\":\\\"request_timeout\\\",\\\"Request completed after its deadline\\\":\\\"deadline_overrun\\\",\\\"Database hydration gave up loading its artifact\\\":\\\"sql_hydration_timeout\\\",\\\"Read audit wait expired\\\":\\\"read_audit_busy\\\",\\\"Read audit batch slow\\\":\\\"read_audit_batch_slow\\\",\\\"Read audit batch failed\\\":\\\"read_audit_batch_failed\\\",\\\"Read audit drain stalled\\\":\\\"read_audit_drain_stalled\\\",\\\"retrying operation after a lost database connection\\\":\\\"db_operation_retried\\\",\\\"not retrying operation: its first run took longer than the retry window to fail\\\":\\\"db_operation_retry_skipped_slow\\\",\\\"closing a pooled connection released inside a transaction\\\":\\\"db_pool_closed_in_transaction\\\",\\\"Database exceeds the artifact read limit; not loaded\\\":\\\"sql_database_too_large_to_load\\\",\\\"Database checkpoint save failed; refusing its writes until the back-off ends\\\":\\\"sql_checkpoint_save_backoff\\\",\\\"kv head size probe failed; size omitted\\\":\\\"kv_head_size_probe_failed\\\",\\\"chain guard wait exceeded the slow-wait threshold\\\":\\\"chain_guard_slow_wait\\\",\\\"request answered with a 5xx\\\":\\\"http_5xx\\\",\\\"idx_current_kv_space_deleted_key_c is missing; the node keeps serving and KV list requests are slower until an operator builds it with CREATE INDEX CONCURRENTLY (docs/releasing.md, section idx_current_kv_space_deleted_key_c)\\\":\\\"current_kv_c_index_missing\\\",\\\"idx_current_kv_space_deleted_key_c is invalid or still being built; the node keeps serving and KV list requests are slower. If no CREATE INDEX CONCURRENTLY is running, run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_invalid\\\",\\\"idx_current_kv_space_deleted_key_c exists with the wrong definition and is not used for KV lists; the node keeps serving and KV list requests are slower. Run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c\\\":\\\"current_kv_c_index_wrong_shape\\\"}\\n known = get(codes, [msg]) ?? null\\n if is_string(known) { code = known }\\n if starts_with(msg, \\\"quota service returned \\\") { code = \\\"quota_fallback\\\" }\\nif starts_with(msg, \\\"Failed to get TDX quote: \\\") { code = \\\"attestation_quote_failed\\\" }\\nif starts_with(msg, \\\"quota service returned an invalid body: \\\") { code = \\\"quota_invalid_body\\\" }\\nif starts_with(msg, \\\"dstack socket available but get_info failed: \\\") { code = \\\"dstack_get_info_failed\\\" }\\n }\\n event = if level == \\\"warn\\\" { \\\"node_warning\\\" } else { \\\"node_error\\\" }\\n out = {\\\"event\\\": event, \\\"svc\\\": \\\"api\\\", \\\"level\\\": level, \\\"event_code\\\": code, \\\"event_count\\\": 1}\\n }\\n} else {\\n ts = string!(j.time)\\n method = string!(j.method)\\n route = string!(j.route)\\n if route == \\\"\\\" { route = \\\"\\\" }\\n if !includes([\\\"GET\\\",\\\"HEAD\\\",\\\"POST\\\",\\\"PUT\\\",\\\"PATCH\\\",\\\"DELETE\\\",\\\"OPTIONS\\\",\\\"\\\"], method) || !includes([\\\"/healthz\\\",\\\"/info\\\",\\\"/version\\\",\\\"/invoke\\\",\\\"/delegate\\\",\\\"/revoke\\\",\\\"/attestation\\\",\\\"/signed/kv\\\",\\\"/signed/kv/\\\",\\\"/admin/quota\\\",\\\"/admin/quota/\\\",\\\"/admin/usage\\\",\\\"/admin/stats\\\",\\\"/hooks/events\\\",\\\"/hooks/tickets\\\",\\\"/hooks/webhooks\\\",\\\"/hooks/webhooks/\\\",\\\"/delegation/query\\\",\\\"/delegation/status\\\",\\\"/encryption/networks\\\",\\\"/encryption/networks/\\\",\\\"/encryption/networks//revoke\\\",\\\"/encryption/networks//decrypt\\\",\\\"/policy/v3/status\\\",\\\"/policy/v3/status/\\\",\\\"/policy/v3/policies\\\",\\\"/policy/v3/challenges\\\",\\\"/policy/v3/delegations\\\",\\\"/policy/v3/enforcer-bindings\\\",\\\"/policy/v3/deliveries/authorize\\\",\\\"/.well-known/tinycloud/node-keys\\\",\\\"/.well-known/encryption/network/\\\",\\\"/public//kv\\\",\\\"/public//kv/\\\",\\\"/peer/generate/\\\",\\\"/evidences/\\\",\\\"/share/v2/readiness\\\",\\\"/share/v2/policies\\\",\\\"/share/v2/policy/challenges\\\",\\\"/share/v2/policy/session\\\",\\\"/share/v2/invoke\\\",\\\"/share/v2/deliveries/authorize\\\",\\\"/<_s..>\\\",\\\"/public/<_space_id>/kv/<_key..>\\\",\\\"\\\"], route) { abort }\\n status = j.status\\n request_time = j.request_time\\n if !is_integer(status) || !(is_integer(request_time) || is_float(request_time)) { abort }\\n status = int!(status)\\n request_time = to_float!(request_time)\\n if status < 100 || status > 599 || request_time < 0 || request_time > 86400 { abort }\\n out = {\\\"event\\\": \\\"ingress_request\\\", \\\"svc\\\": \\\"ingress\\\", \\\"level\\\": \\\"info\\\", \\\"method\\\": method, \\\"route\\\": route, \\\"status\\\": status, \\\"request_time_seconds\\\": request_time}\\n}\\nif length(ts) > 64 { abort }\\nparsed_ts = parse_timestamp!(ts, format: \\\"%+\\\")\\nout.ts = format_timestamp!(parsed_ts, format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\")\\n. = out\\n\",\"drop_on_error\":true,\"drop_on_abort\":true,\"reroute_dropped\":true},\"gate_counts\":{\"type\":\"remap\",\"inputs\":[\"gate_route.rejected\",\"gate_route._unmatched\",\"allowlist.dropped\",\"event_route._unmatched\"],\"source\":\"reason = .reason\\nif !includes([\\\"malformed_envelope\\\",\\\"unexpected_label\\\",\\\"unlabelled\\\",\\\"non_json\\\",\\\"oversized\\\",\\\"partial\\\",\\\"invalid_fields\\\",\\\"unknown_target\\\",\\\"filtered_level\\\"], reason) { reason = \\\"invalid_fields\\\" }\\n. = {\\\"reason\\\": reason, \\\"count\\\": 1}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"gate_reduce\":{\"type\":\"reduce\",\"inputs\":[\"gate_counts\"],\"group_by\":[\"reason\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"count\":\"sum\"}},\"gate_output\":{\"type\":\"remap\",\"inputs\":[\"gate_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": \\\"warn\\\", \\\"event\\\": \\\"logship_gate_summary_total\\\", \\\"svc\\\": \\\"logship\\\", \\\"reason\\\": .reason, \\\"count\\\": .count}\",\"drop_on_error\":true,\"drop_on_abort\":true},\"request_profile\":{\"type\":\"filter\",\"inputs\":[\"allowlist\"],\"condition\":\"\\\"$${TINYCLOUD_LOGSHIP_PROFILE}\\\" == \\\"all_requests\\\" || includes([\\\"node_warning\\\", \\\"node_error\\\", \\\"node_storage_headroom\\\"], .event) || (int(.status) ?? 0) >= 400 || (.event == \\\"node_request\\\" && (int(.latency_us) ?? 0) > 1000000) || (.event == \\\"ingress_request\\\" && (to_float(.request_time_seconds) ?? 0.0) > 1.000000)\"},\"event_route\":{\"type\":\"route\",\"inputs\":[\"request_profile\"],\"route\":{\"request\":\"(.event == \\\"node_request\\\" || .event == \\\"ingress_request\\\") && !(((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)) && !((int(.status) ?? 0) >= 500)\",\"node_request_5xx\":\".event == \\\"node_request\\\" && (int(.status) ?? 0) >= 500\",\"ingress_request_5xx\":\".event == \\\"ingress_request\\\" && (int(.status) ?? 0) >= 500\",\"node_request_4xx\":\".event == \\\"node_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"ingress_request_4xx\":\".event == \\\"ingress_request\\\" && ((int(.status) ?? 0) >= 400 && (int(.status) ?? 0) < 500)\",\"db_failure_5xx\":\".event == \\\"node_warning\\\" && .event_code == \\\"db_failure_5xx\\\"\",\"storage_headroom\":\".event == \\\"node_storage_headroom\\\"\",\"diagnostic\":\"(.event == \\\"node_warning\\\" && .event_code != \\\"db_failure_5xx\\\") || .event == \\\"node_error\\\"\"}},\"cap_db_failure_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.db_failure_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"db_failure_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_5xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_5xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_5xx\\\"\\nlocal cap = 10\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_node_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.node_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"node_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"cap_ingress_request_4xx\":{\"type\":\"lua\",\"version\":\"2\",\"inputs\":[\"event_route.ingress_request_4xx\"],\"hooks\":{\"process\":\"process\"},\"timers\":[{\"interval_seconds\":60,\"handler\":\"flush\"}],\"source\":\"local class = \\\"ingress_request_4xx\\\"\\nlocal cap = 3\\nlocal passed = 0\\nlocal dropped = 0\\nlocal ticks = 0\\nfunction process(event, emit)\\n if passed < cap then\\n passed = passed + 1\\n emit(event)\\n else\\n dropped = dropped + 1\\n end\\nend\\nfunction flush(emit)\\n ticks = ticks + 1\\n if ticks == 1 then return end\\n if dropped > 0 then\\n emit({log = {ts = os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"), level = \\\"warn\\\", event = \\\"node_warning\\\", svc = \\\"api\\\", event_code = \\\"errors_dropped\\\", class = class, dropped = dropped, window_s = 60}})\\n end\\n passed = 0\\n dropped = 0\\nend\"},\"diagnostic_reduce\":{\"type\":\"reduce\",\"inputs\":[\"event_route.diagnostic\"],\"group_by\":[\"event\",\"event_code\",\"level\",\"svc\"],\"end_every_period_ms\":60000,\"expire_after_ms\":75000,\"flush_period_ms\":1000,\"merge_strategies\":{\"event_count\":\"sum\",\"ts\":\"discard\"}},\"diagnostic_output\":{\"type\":\"remap\",\"inputs\":[\"diagnostic_reduce\"],\"source\":\". = {\\\"ts\\\": format_timestamp!(now(), format: \\\"%Y-%m-%dT%H:%M:%S%.3fZ\\\"), \\\"level\\\": .level, \\\"event\\\": .event, \\\"svc\\\": \\\"api\\\", \\\"event_code\\\": .event_code, \\\"event_count\\\": .event_count}\",\"drop_on_error\":true,\"drop_on_abort\":true}},\"sinks\":{\"betterstack\":{\"type\":\"http\",\"inputs\":[\"event_route.request\",\"event_route.storage_headroom\",\"cap_db_failure_5xx\",\"cap_node_request_5xx\",\"cap_ingress_request_5xx\",\"cap_node_request_4xx\",\"cap_ingress_request_4xx\",\"diagnostic_output\",\"gate_output\"],\"method\":\"post\",\"uri\":\"https://s2795437.us-west-2a.betterstackdata.com/\",\"encoding\":{\"codec\":\"json\"},\"framing\":{\"method\":\"newline_delimited\"},\"compression\":\"gzip\",\"auth\":{\"strategy\":\"bearer\",\"token\":\"SECRET[bs.bs_token]\"},\"batch\":{\"max_bytes\":262144,\"timeout_secs\":1},\"buffer\":{\"type\":\"memory\",\"max_events\":1000,\"when_full\":\"block\"},\"request\":{\"timeout_secs\":10,\"retry_max_duration_secs\":300},\"healthcheck\":{\"enabled\":false}}},\"secret\":{\"bs\":{\"type\":\"directory\",\"path\":\"/var/lib/vector/secrets\",\"remove_trailing_whitespace\":true}}}" entrypoint: - /bin/sh - -ec diff --git a/vendor/sqlx-core/PATCH.md b/vendor/sqlx-core/PATCH.md index 9a077e3b..aa5dd9fc 100644 --- a/vendor/sqlx-core/PATCH.md +++ b/vendor/sqlx-core/PATCH.md @@ -145,6 +145,9 @@ released with a transaction open. Closing the session makes the server roll the transaction back, and a `warn!` ("closing a pooled connection released inside a transaction") records each such leak. The pool opens a replacement on the next acquire. + The `warn!` carries `target: "tinycloud_core::db_pool"`, so the node's log + shipper classifies it as `db_pool_closed_in_transaction`; the default + `sqlx_core::…` target would ship as unclassified (TC-3006). ### Re-applying on a vendor bump diff --git a/vendor/sqlx-core/src/pool/connection.rs b/vendor/sqlx-core/src/pool/connection.rs index 8daa4217..353ecba0 100644 --- a/vendor/sqlx-core/src/pool/connection.rs +++ b/vendor/sqlx-core/src/pool/connection.rs @@ -346,8 +346,12 @@ impl Floating> { // including any drop-time rollback. A session still inside a transaction would run // the next borrower's statements in it, so close it; ending the session ends the // transaction. Bounded like the close above. - // A leak the drop guards missed: logged so it is seen, not only absorbed. - tracing::warn!("closing a pooled connection released inside a transaction"); + // A leak the drop guards missed: logged so it is seen, not only absorbed. The target + // is the node's pool module so the log shipper classifies it (TC-3006). + tracing::warn!( + target: "tinycloud_core::db_pool", + "closing a pooled connection released inside a transaction" + ); let _ = crate::rt::timeout(CLOSE_ON_DROP_TIMEOUT, self.close()).await; false } else {