Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -764,11 +764,12 @@ jobs:
# TC-959: main's N0 compose (#349 c33c2302…) with the 95m ingress body
# cap (raw d95bfb70…), the TC-1274 + TC-1317 + TC-1482 + c_index + TC-1840 + TC-1843 Vector config and the
# optional TC-1482 NODE_STATS_READ_TOKEN env line, the TC-950 pool of 10 and the TC-1777 named-keys gate
# env lines + deletion-ledger volume, the TC-2346 operation-retry warning codes and the TC-2917
# storage-headroom event in the Vector config
# (raw 5419a6fb…; d703d69b… before TC-2917, 298ea356… before TC-2346, 192dfb7e… before TC-1777),
# its single node digest normalized to 64 zeros.
n0_template_hash = '90841a34e6ea5b581374f50abe70e580496e6814bc83f61f20fd4c9e090e5d51'
# env lines + deletion-ledger volume, the TC-2346 operation-retry warning codes, the TC-2917
# storage-headroom event and the TC-3006 codes (canonical schema; chain_guard_slow_wait, http_5xx and
# the TC-2954/2961/3027 warning codes) in the Vector config
# (raw 7ee085c6…; 5419a6fb… before TC-3006, d703d69b… before TC-2917, 298ea356… before TC-2346,
# 192dfb7e… before TC-1777), its single node digest normalized to 64 zeros.
n0_template_hash = 'efad41d56df187a7e0ddd1a13f3c4a3c436ad7c0ff141acd32c06e28f9c92a34'
node_pin = re.compile(rb'(?m)^ image: ghcr\.io/tinycloudlabs/tinycloud-node@sha256:([0-9a-f]{64})$')
def n0_template(raw):
pins = list(node_pin.finditer(raw))
Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@
- Share-email issuer trust bundles can pin one next key for the same issuer DID during an explicit UTC overlap window. The legacy key stops at `issuerNotAfter`; malformed, duplicate, unbounded or expired overlaps fail startup and `--validate-config`. Existing single-key bundles remain unchanged. This is node-side trust support only: the OpenCredentials witness and js-sdk policy contract must also support key rotation before activation. Older node images reject overlap fields; restore the previous single-key bundle before rollback. See `deploy/share-email/production-trust-bundle-contract.md`.
- Disk headroom is visible without a shell on the node. Every 5 minutes the node logs a `storage headroom` event per local store, with only `store` (`sql`, and `duckdb` in DuckDB builds) and `free_bytes` (the free bytes on that store's filesystem, rounded down to a whole 64 MiB). The log sidecar ships it to Better Stack as `node_storage_headroom`. The exact value is served as the gauge `tinycloud_storage_free_bytes{store}` on the metrics endpoint, and `tinycloud_disk_headroom_refusals_total{engine="sql"}` counts requests answered `507 sql_disk_headroom`. Details: TC-2917.
- **N0 compose.** The embedded Vector configuration is regenerated for the new event, so the reviewed N0 compose template changes to `90841a34…` (raw `5419a6fb…`; `docs/releasing.md`, `.github/workflows/docker.yml`).
- The log sidecar ships six more node warnings to Better Stack as event codes, each a per-minute count of `node_warning` events with no other fields. Until now they shipped as `unclassified`.
- `db_pool_closed_in_transaction`: a pooled connection was released inside a transaction and closed.
- `sql_database_too_large_to_load`: a database over the artifact read limit was not loaded.
- `sql_checkpoint_save_backoff`: a database's writes are refused during its checkpoint back-off.
- `kv_head_size_probe_failed`: the KV head size probe failed.
- `chain_guard_slow_wait`: a chain-guard wait over 500 ms.
- `http_5xx`: the line every 5xx response leaves. Each 5xx request still ships on its own under the `node_request_5xx` cap.

The vendored log schema now matches the canonical schema, pinned in `log-shipper/canonical.json`. `chain_guard_slow_wait` is no longer a capped class there, since a per-minute cap for it does not fit the TC-1843 reserve. The pooled-connection warning (vendored sqlx-core) now logs under the target `tinycloud_core::db_pool` (`vendor/sqlx-core/PATCH.md`). Details: TC-3006, TC-2954, TC-2961, TC-3027.
- **N0 compose.** The embedded Vector configuration is regenerated for these codes, so the reviewed N0 compose template changes to `efad41d5…` (raw `7ee085c6…`; `docs/releasing.md`, `.github/workflows/docker.yml`).

## [1.26.5] - 2026-11-13

Expand Down
2 changes: 1 addition & 1 deletion docker-compose.dstack-postgres.yaml

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions docs/releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -657,8 +657,8 @@ Before any CVM mutation the workflow requires:

- the main compose, with its one node digest replaced by 64 zeros, to hash
to the reviewed N0 template
**`90841a34e6ea5b581374f50abe70e580496e6814bc83f61f20fd4c9e090e5d51`**
(raw `5419a6fbb73efe8b4430e4ab76d5e2970b2c9204525854e23f59dbdc01f4e250` normalized; before the TC-2917 `node_storage_headroom` event (the node's disk headroom line shipped as an individual event) in the Vector config the template was `9d0cd8a2…`, raw `d703d69b…`; before the TC-2346 `db_operation_retried` and `db_operation_retry_skipped_slow` warning codes in the Vector config the template was `c9bcbea9…`, raw `298ea356…`; before the TC-1777 named-keys gate env lines (`TINYCLOUD_ENCRYPTION_NETWORK__NAMED_KEYS_V1`, `…__DELETION_LEDGER_PATH`) and the `deletion-ledger` volume the template was `8df7e0f2…`, raw `192dfb7e407fa591d4f0232b95e43a41a85dd18add1d4976e70ebdbff5ddebca`; before the TC-1843 per-minute caps on 4xx and 5xx request events and on the 5xx diagnostic, and the `errors_dropped` event (the dropped count), in the Vector config the template was `7c3456dc…`, raw `d0e72bde…`; before the TC-1840 `db_failure_5xx` event (the node's 5xx diagnostic line shipped as an individual event) in the Vector config the template was `c91e5b7a…`, raw `99013c40…`; before the TC-950 pool of 10 (`TINYCLOUD_DATABASE__MAX_CONNECTIONS` 15 to 10) the template was `79469f97…`, raw `32a002ef…`; before the three `current_kv_c_index_*` warning codes in the Vector config (TC-1274 `current_kv` C-index migration) the template was `87b5d12a…`, raw `d1fcd665…`; before the optional TC-1482 `NODE_STATS_READ_TOKEN` env line and the TC-1482 route in the Vector config the template was `7fabba70…`, raw `1228d81927fc0a3fbba85745103e77b3112606f5d314d15974d400b222409350`: the 1.21.0 compose
**`efad41d56df187a7e0ddd1a13f3c4a3c436ad7c0ff141acd32c06e28f9c92a34`**
(raw `7ee085c6e5da8a2b7a6e77fa207089b4cef3b8c291b4770f0d0c3336449f90b1` normalized; before the TC-3006 canonical-schema update (`chain_guard_slow_wait` and `http_5xx` as counts, and the TC-2954/2961/3027 warning codes) in the Vector config the template was `90841a34…`, raw `5419a6fb…`; before the TC-2917 `node_storage_headroom` event (the node's disk headroom line shipped as an individual event) in the Vector config the template was `9d0cd8a2…`, raw `d703d69b…`; before the TC-2346 `db_operation_retried` and `db_operation_retry_skipped_slow` warning codes in the Vector config the template was `c9bcbea9…`, raw `298ea356…`; before the TC-1777 named-keys gate env lines (`TINYCLOUD_ENCRYPTION_NETWORK__NAMED_KEYS_V1`, `…__DELETION_LEDGER_PATH`) and the `deletion-ledger` volume the template was `8df7e0f2…`, raw `192dfb7e407fa591d4f0232b95e43a41a85dd18add1d4976e70ebdbff5ddebca`; before the TC-1843 per-minute caps on 4xx and 5xx request events and on the 5xx diagnostic, and the `errors_dropped` event (the dropped count), in the Vector config the template was `7c3456dc…`, raw `d0e72bde…`; before the TC-1840 `db_failure_5xx` event (the node's 5xx diagnostic line shipped as an individual event) in the Vector config the template was `c91e5b7a…`, raw `99013c40…`; before the TC-950 pool of 10 (`TINYCLOUD_DATABASE__MAX_CONNECTIONS` 15 to 10) the template was `79469f97…`, raw `32a002ef…`; before the three `current_kv_c_index_*` warning codes in the Vector config (TC-1274 `current_kv` C-index migration) the template was `87b5d12a…`, raw `d1fcd665…`; before the optional TC-1482 `NODE_STATS_READ_TOKEN` env line and the TC-1482 route in the Vector config the template was `7fabba70…`, raw `1228d81927fc0a3fbba85745103e77b3112606f5d314d15974d400b222409350`: the 1.21.0 compose
`d95bfb707871c1d626c4a8f2b8d32d354103edbb5077847c6a21fe5f7f6cfa50`, template
`51f8e9e9…`, with only the embedded Vector config regenerated for TC-1274's,
TC-1317's and TC-1482's route and warning codes (TC-1274 and TC-1317 were template `7fabba70…`, raw `1228d819…`); TC-1274 alone was template `00938226…`, raw
Expand Down Expand Up @@ -806,8 +806,8 @@ gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \

The served/OCI identity remains **1.20.6-dstack / e18578ff9162986ddd813259c60f73866e3e60d1**.
With the template lock on main this redeploys exactly main's raw compose
`5419a6fbb73efe8b4430e4ab76d5e2970b2c9204525854e23f59dbdc01f4e250`
(the 1.21.0 compose `d95bfb70…` with the TC-1274, TC-1317, TC-1482, `current_kv_c_index_*`, `db_failure_5xx`, the 4xx and 5xx caps, `errors_dropped`, TC-2346 `db_operation_retried` / `db_operation_retry_skipped_slow` and TC-2917 `node_storage_headroom` Vector config, the optional TC-1482 `NODE_STATS_READ_TOKEN` env line, the TC-950 pool of 10, and the TC-1777 `NAMED_KEYS_V1` gate env lines with the `deletion-ledger` volume), whatever digest
`7ee085c6e5da8a2b7a6e77fa207089b4cef3b8c291b4770f0d0c3336449f90b1`
(the 1.21.0 compose `d95bfb70…` with the TC-1274, TC-1317, TC-1482, `current_kv_c_index_*`, `db_failure_5xx`, the 4xx and 5xx caps, `errors_dropped`, TC-2346 `db_operation_retried` / `db_operation_retry_skipped_slow`, TC-2917 `node_storage_headroom` and TC-3006 Vector config, the optional TC-1482 `NODE_STATS_READ_TOKEN` env line, the TC-950 pool of 10, and the TC-1777 `NAMED_KEYS_V1` gate env lines with the `deletion-ledger` volume), whatever digest
main's template currently pins. It is not a rollback of a
later release across a schema migration. Current pre-N0 tag/pinned
equivalents are separately hash-checked; arbitrary same-image config
Expand Down
4 changes: 2 additions & 2 deletions log-shipper/canonical.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"repo": "samgbafa/tinycloud-dev",
"revision": "4fc32a2f9a24526b8fae6329a04f54230753c8c8",
"revision": "6f0d420211810a533ebfb29b5bb7f34ac2422071",
"path": "ops/logging/fields.json",
"sha256": "38dae8e0ce7794d9e626cab30b2b2ab205fce3498588e14b541d6e506e8c0ca4"
"sha256": "c72f50d8fff633f05b9d65d4b81e1dc77364f366679e58191abcac721344847c"
}
19 changes: 19 additions & 0 deletions log-shipper/cases.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,25 @@ export const vectorTests = [
// TC-2346: the node's operation-retry warnings (target tinycloud_core::db_pool). Like every classified diagnostic they ship the code and the count only; operation, error_class and the timings stay in the node's own log.
...[['retrying operation after a lost database connection', 'db_operation_retried'], ['not retrying operation: its first run took longer than the retry window to fail', 'db_operation_retry_skipped_slow']].map(([message, code]) => check(`operation retry warning maps to ${code} with only the fixed diagnostic keys`, envelope({ timestamp, level: 'WARN', target: 'tinycloud_core::db_pool', fields: { message, operation: 'delegate', error_class: 'io', elapsed_ms: 10001, retry_window_ms: 10000, sql: canary } }), `.event == "node_warning" && .event_code == "${code}" && .event_count == 1 && .level == "warn" && keys(.) == ["event", "event_code", "event_count", "level", "svc", "ts"]`)),
...['retrying operation after a lost database connection: x', 'Retrying operation after a lost database connection', 'not retrying operation'].map(message => check(`operation retry warning near-miss "${message}" stays unclassified`, envelope({ timestamp, level: 'WARN', target: 'tinycloud_core::db_pool', fields: { message, operation: 'delegate', error_class: 'io' } }), '.event == "node_warning" && .event_code == "unclassified" && .event_count == 1')),
// TC-2954, TC-2961, TC-3027, TC-3006: the pooled-connection backstop (vendored sqlx-core, target tinycloud_core::db_pool),
// the SQL artifact read limit and checkpoint back-off (tinycloud_core::sql::service), the KV head size probe, the
// chain-guard slow wait (tinycloud::chain_guard) and the every-5xx line (tinycloud::request). Code and count only:
// sizes, delays, kinds, routes, statuses, errors and identifiers stay in the node's own log.
...[
['closing a pooled connection released inside a transaction', 'db_pool_closed_in_transaction', 'tinycloud_core::db_pool', {}],
['Database exceeds the artifact read limit; not loaded', 'sql_database_too_large_to_load', 'tinycloud_core::sql::service', { service: 'sql', error_class: 'artifact_read_limit', bytes: 300666880, limit_bytes: 67108864 }],
['Database checkpoint save failed; refusing its writes until the back-off ends', 'sql_checkpoint_save_backoff', 'tinycloud_core::sql::service', { service: 'sql', retry_after_ms: 30000 }],
['kv head size probe failed; size omitted', 'kv_head_size_probe_failed', 'tinycloud::routes', { error: canary }],
['chain guard wait exceeded the slow-wait threshold', 'chain_guard_slow_wait', 'tinycloud::chain_guard', { event: 'chain_guard_slow_wait', kind: 'exclusive', elapsed_ms: 612, guards: 3, suppressed: 0 }],
['request answered with a 5xx', 'http_5xx', 'tinycloud::request', { error_class: 'http_5xx', status: 503, route: '/invoke', classified: true }],
].map(([message, code, target, fields]) => check(`${code} warning maps to its code with only the fixed diagnostic keys`, envelope({ timestamp, level: 'WARN', target, fields: { message, ...fields, space: canary, db_name: canary } }), `.event == "node_warning" && .event_code == "${code}" && .event_count == 1 && .level == "warn" && keys(.) == ["event", "event_code", "event_count", "level", "svc", "ts"]`)),
...[
['closing a pooled connection released inside a transaction', 'sqlx_core::pool::connection'],
['closing a pooled connection released inside a transaction: x', 'tinycloud_core::db_pool'],
['Database exceeds the artifact read limit', 'tinycloud_core::sql::service'],
['database checkpoint save failed; refusing its writes until the back-off ends', 'tinycloud_core::sql::service'],
['kv head size probe failed', 'tinycloud::routes'],
].map(([message, target]) => check(`near-miss "${message}" on ${target} stays unclassified`, envelope({ timestamp, level: 'WARN', target, fields: { message } }), '.event == "node_warning" && .event_code == "unclassified" && .event_count == 1')),
check('node 5xx diagnostic ships exactly its eight fields as an individual event', envelope(dbFailure()), `.event == "node_warning" && .svc == "api" && .level == "warn" && .event_code == "db_failure_5xx" && .status == 503 && .code == "sql_busy" && .service == "sql" && .route == "/invoke" && .elapsed_ms == 30001 && .failure_kind == "lock_wait" && .write_may_have_applied == false && .db_hash == "${goodHash}" && !exists(.event_count) && ${keysAre(dbShipped)}`),
check('node 5xx diagnostic without a database ships without db_hash', envelope(dbFailure({ db_hash: undefined, status: 500, code: 'internal', service: 'other', failure_kind: 'other' })), `.event_code == "db_failure_5xx" && .status == 500 && .code == "internal" && .service == "other" && .failure_kind == "other" && ${keysAre(dbShipped.filter(key => key !== 'db_hash'))}`),
check('node 5xx diagnostic with an unknown outcome ships the write flag', envelope(dbFailure({ status: 500, code: 'outcome_unknown', write_may_have_applied: true })), `.event_code == "db_failure_5xx" && .status == 500 && .code == "outcome_unknown" && .write_may_have_applied == true && ${keysAre(dbShipped)}`),
Expand Down
6 changes: 6 additions & 0 deletions log-shipper/diagnostics.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@
"Read audit drain stalled": "read_audit_drain_stalled",
"retrying operation after a lost database connection": "db_operation_retried",
"not retrying operation: its first run took longer than the retry window to fail": "db_operation_retry_skipped_slow",
"closing a pooled connection released inside a transaction": "db_pool_closed_in_transaction",
"Database exceeds the artifact read limit; not loaded": "sql_database_too_large_to_load",
"Database checkpoint save failed; refusing its writes until the back-off ends": "sql_checkpoint_save_backoff",
"kv head size probe failed; size omitted": "kv_head_size_probe_failed",
"chain guard wait exceeded the slow-wait threshold": "chain_guard_slow_wait",
"request answered with a 5xx": "http_5xx",
"idx_current_kv_space_deleted_key_c is missing; the node keeps serving and KV list requests are slower until an operator builds it with CREATE INDEX CONCURRENTLY (docs/releasing.md, section idx_current_kv_space_deleted_key_c)": "current_kv_c_index_missing",
"idx_current_kv_space_deleted_key_c is invalid or still being built; the node keeps serving and KV list requests are slower. If no CREATE INDEX CONCURRENTLY is running, run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c": "current_kv_c_index_invalid",
"idx_current_kv_space_deleted_key_c exists with the wrong definition and is not used for KV lists; the node keeps serving and KV list requests are slower. Run DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c; and then the CREATE INDEX CONCURRENTLY in docs/releasing.md, section idx_current_kv_space_deleted_key_c": "current_kv_c_index_wrong_shape"
Expand Down
Loading
Loading