[jimdowling]

This PR (logicalclocks.github.io)

New guide user_guides/mlops/serving/transformation-tracing.md (nav: Model Serving > Transformation Tracing): enabling tracing at deploy time, switching it from the SDK, REST and UI, the metrics, the trace layout and the measured overhead. markdownlint and snakeoil pass; the strict build resolves the new page's API links once the hopsworks-api PR is merged.

FSTORE-2153: metrics and traces for feature view transformations in model deployments

JIRA: https://hopsworks.atlassian.net/browse/FSTORE-2153

A model or feature view deployment runs its feature view's on-demand and model-dependent transformations, a DAG of transformation functions, on every request. This work times them:

• Metrics on every request (while tracing is on), on the predictor's Prometheus endpoint: per feature view and per transformation function, cumulative calls, seconds and errors, plus the fastest and slowest call of the last 60 s window, and the DAG's edges. The Grafana deployment dashboard charts them.
• Sampled traces: a share of requests (10% by default) is recorded as a trace, predict > feature_lookup > transformations > each phase > one span per function in DAG order, then model_predict and feature_log. The feature log write is a background span in the same trace.
• Deployment page: a Transformations section lists the traced requests and draws each one's spans; the background feature log write is drawn as a separate process on its own time axis.
• Switching it live: a deployment created with tracing (model.deploy(tracing={"enabled": True})) can be switched off and on, and resampled, while it runs: PUT /project/{id}/serving/{servingId}/tracing, deployment.set_tracing(...), or the switch on the section. The backend writes the state to a config map mounted in the predictor; each serving worker applies it within about a minute, without a restart. A deployment created without tracing is refused (240058), since its pods have no collector sidecar.
• No cost while off: an untraced deployment runs the upstream code after one flag check per call site, with no telemetry objects, clock reads or allocations. Measured on 5 transformation functions: 39 to 41 µs per feature vector off, +17 µs on and not sampled, +0.4 ms sampled.

It also fixes the default predictor OOM found while testing: the predictor read model_schema.json through the Go HopsFS client (libhdfs-golang via pyarrow), whose idle heap bursts took it past 1 GiB. ModelEngine._read_json now reads it over REST; peak memory fell from the 1 GiB limit to 438 to 483 MiB. The client defects are HOPSFS-425 (logicalclocks/hopsfs-go-client#46).

Testing

• Loadtest workflows/model_serving/python_driver/external/test_transformation_tracing.py, 3/3 passed on jim-dag-stats (job loadtest-1792732688, SDK from this branch): the tracing config survives an update; a request records the nested transformation spans with one row span; tracing switched off records nothing and switched back on records again, with the deployment running throughout. Run before the merge of upstream on 2026-10-23; not re-run after it.
• Cluster checks on jim-dag-stats: refusal on an untraced deployment; off left hopsworks_transformations_calls_total at 40 and traces at 281, on at 50% added 80 calls and 24 traces for 40 requests, same pod, 0 restarts; the config map reached the pod in 8 s and 48 s; the UI switch, the disabled-tracing notice and the background log process drawn on real traces.
• Not covered by the loadtest: the Prometheus metrics themselves (the predictor's metrics port is not reachable from the test pod; covered by unit tests and checked by hand on the cluster) and agent deployments, whose tracing is unchanged.
• Codex pre-review plus two adversarial rounds on the untraced-overhead question, converged; findings applied, accepted risks recorded.

Merge order

hopsworks-api, then hopsworks-ee, then docker-images (its wrapper calls the new hopsworks_common.tracing functions and its tests install the client from main), then hopsworks-front, loadtest and logicalclocks.github.io (its API links resolve once hopsworks-api is merged).

Related PRs

• [FSTORE-2153] Metrics and traces for feature view transformations in model deployments hopsworks-api#1354
• logicalclocks/hopsworks-ee#3578
• logicalclocks/docker-images#1140
• logicalclocks/hopsworks-front#2203
• logicalclocks/loadtest#1189
• [FSTORE-2153] Metrics and traces for feature view transformations in model deployments #718
• [HOPSFS-425] hopsfs-go-client allocates 336 MiB per RPC and redials every 5 s when the namenode expects TLS hopsfs-go-client#46 (HOPSFS-425, the Go client defects behind the OOM)

🤖 Generated with Claude Code