[pblazej]

Summary

livekit-telemetry — the shared client-telemetry core — and its UniFFI surface for the Swift, Kotlin and Dart SDKs. Wire spec (events, attributes, cadence, upload policy): livekit-telemetry/SPEC.md.

#1566 stacks on this PR: it takes every telemetry name from the protocol registry.

Changes

• livekit-ffi/src/core_modules/telemetry.rs: configure and configure-pulled, scopes, spans, stats, the app API, the opt-out, device state
• log_forward.rs: the core's own warnings and errors copied into telemetry
• Dart tests
• The telemetry_ping example and the README's local testing guide
• One lifecycle task carries out configure, shutdown and the opt-out's purge one at a time, in the order they were sent; telemetry_configure waits until its pipeline is installed
• A replacing exporter starts only once the one before it stopped (drained within its bound, or cancelled and joined), so two exporters never replay a shared cache at once
• telemetry_shutdown returns once every exporter queued before it stopped, and a shutdown that resumes late never ends a newer lifecycle
• The pull queue never hands out or keeps a request once finish returned, and a pulled configure refused after the opt-out finishes its queue
• Platform log lines passed to telemetry_log or TelemetryScope.log get the same credential masking as the core's own copied lines

What a platform calls, and when

When	Call
SDK init	telemetry_configure(config, transport, instruments) — storage and tuning only; Dart: telemetry_configure_pulled(config, []) + queue.try_next()
Room created	telemetry_scope()
Connect, and every token refresh	scope.set_server(url, token) — cheap, idempotent
Room lifecycle	scope.start(SpanName, parent) → step / set_track / end; subscribe_started / subscribed (tracks already in the room at join: subscribe_started at connect, so lk.subscribe measures from join; a lone subscribed still opens the span as a fallback) / subscribe_failed; track_ended(sid); set_room; disconnected(reason)
Stats	every scope.stats_poll_interval_ms(): scope.record_peer_stats(report, {trackId → sid}, ts) per peer connection
OS signals	telemetry_set_device_state (thermal Unknown / low power None without a source), telemetry_device_event, telemetry_log
App API (the only public surface)	scope.emit_custom(name, {k: v}), scope.set_attribute(key, value?), telemetry_disable() — synchronous, in effect when it returns (TODO on platforms: token discussion); telemetry_is_disabled() reads that opt-out process-wide (every thread and Dart isolate), for platforms to check right before each getStats or submit

No endpoint, header or sink exists in any client-language API. Local end-to-end tests set LK_TELEMETRY_ENDPOINT (read by the core; see the README).

Local testing against a local OpenTelemetry backend (LGTM)

Everything can be tried against a local OpenTelemetry backend (LGTM) in a couple of minutes; no LiveKit server is needed for the Rust example.

# 1. Local OTel backend (OTLP/HTTP on :4318, UI on :3000)
docker run -d --name lk-lgtm -p 3000:3000 -p 4318:4318 grafana/otel-lgtm

# 2. Record a small session (lk.connect with its checkpoints, lk.publish, lk.subscribe ended by
#    first media, a custom event) and export it to the local backend; it prints the trace id
LK_TELEMETRY_ENDPOINT=http://localhost:4318 cargo run -p telemetry_ping

# 3. Optional: the same through the Dart bindings (pull queue)
(cd livekit-ffi && cargo make dart-package) && cd livekit-ffi/packages/dart && dart pub get \
  && LK_TELEMETRY_ENDPOINT=http://localhost:4318 dart test test/live_collector_test.dart

# 4. Optional: the file cache — stop the backend, run step 2 with LK_TELEMETRY_DIR=/tmp/lk-telemetry,
#    start the backend again and run once more to watch the cached batches replay

Then open http://localhost:3000 → Explore: Loki {service_name="telemetry_ping"} for the log records (otel.event.name: custom.ping, lk.rtc.stats.sample, lk.telemetry.report), Tempo with the printed trace id for the spans (lk.connect, lk.publish, lk.subscribe). Platform e2e tests set LK_TELEMETRY_ENDPOINT the same way (with livekit-server --dev for a real Room).

Verification

At a2558a40, from a clean checkout:

• cargo fmt --check
• cargo clippy -p livekit-telemetry --all-targets -- -D warnings, also with --all-features and as a cargo hack --feature-powerset --depth 2 run
• cargo test -p livekit-telemetry: 275 unit, 2 doc
• RUSTDOCFLAGS="-D warnings" cargo doc -p livekit-telemetry --no-deps
• cargo test -p livekit-ffi --no-default-features --features core-modules: 34
• cargo clippy -p livekit-ffi --no-default-features --features core-modules --all-targets: only the 7 warnings that main (6473ec17) has
• cargo clippy -p telemetry_ping --all-targets -- -D warnings
• Kotlin: cargo make --profile release android-package in livekit-ffi generates the bindings and assembleRelease succeeds, and a fresh ./gradlew compileReleaseKotlin --rerun-tasks passes. android-check-size fails: the arm64 library is 2,298,256 bytes against a limit of 1,769,472 bytes (main: 1,593,408 bytes)
• Dart: cargo make dart-package, dart pub get, dart test: 7 pass, 1 skip. The skipped live collector test passes against a local collector

Stages 2–7 pass the same livekit-telemetry gates on their own tips (see each PR).

[xianshijing-lk]

there can be a race condition here, like serve() hands a request to the host after finish() has returned:

finish() (751) sets finished inside the inflight lock. A serve already holding that lock has read finished == false, so it inserts and returns Some(export) while finish blocks behind it; finish then clears the map. next() has no re-check on the Some path. The doc on inflight ("nothing is handed out or kept once finish returned") is only half true — the existing test nothing_is_served_after_finish_returns asserts the map is empty but never checks what serve returned.

Impact: the host performs one more upload — carrying Authorization: Bearer — after the queue was finished, and since its entry is gone, complete() is ignored and the exporter retries → duplicate batch. Not an opt-out leak: serve's separate process && global::is_disabled() check covers that path. This is the shutdown/queue-replacement path.

I reproduced it deterministically at both levels, and this 7-line fix makes both go green with all 18 of the PR's telemetry tests still passing:

pub fn finish(&self) {

• // Before the lock: a serve already holding it then sees the flag on its way out.
• self.finished.store(true, Ordering::SeqCst);
  {
  let mut inflight = self.inflight.lock()...;

•    self.finished.store(true, Ordering::SeqCst);
     inflight.clear();
  

  }
  // in serve(), just before the insert:
  +if self.finished.load(Ordering::SeqCst) {

• return None;
  +}
  inflight.retain(|_, done| !done.is_closed());

[xianshijing-lk]

what will happen if telemetry is off ? does telemetry_copy cost anything or trigger some warning logs ?

[xianshijing-lk]

it looks like this telemetry_configure can block the caller thread ?

This can be very problematic.

@1egoman and @lukasIO have some experience with AIC and Krisp plugins, and it took us a few iterations to fix the register problems. maybe we can get some suggestions here ?

[xianshijing-lk]

I am concerned on this approach, how long will telemetry_shutdown() will delay the shutdown ?
I wonder if we should just stop networking and save the data on device, and the data will be uploaded after the app get opened again