Repository navigation
feat!: trace and measure authsome on forge v1.12.9 and grove v1.8.0 - #137
Merged
Merged
Conversation
This moves forge and forge/extensions/auth to v1.12.9, grove and its three drivers to v1.8.0, chronicle to v1.9.0, dispatch to v1.9.1 and herald, keysmith, ledger, relay, vault and warden to v1.8.0. go-utils lands on v1.4.1 because forge needs it. You get forge's server spans and semconv HTTP metrics on every route, grove's db.client spans and query durations, and the telemetry each library added in its own release. Nothing authsome calls in those libraries changed shape, so this commit builds and vets as it is.
forge v1.12.9 describes a pointer element, such as the items of []*Connection, as oneOf a $ref and a null-only object. The generators read neither oneOf nor nullable, so eight list fields across the Go, TypeScript and Dart SDKs and both embedded clients would have turned into []any, unknown[] and List<dynamic> on the next regeneration. The spec loader now folds that exact pair back into the $ref. Any other oneOf is left as it was, so a real union is never narrowed to one of its members. The SDKs regenerate byte for byte as before.
Two things move in spec.json. forge now marks the items of pointer lists as nullable (see the previous commit for why the SDKs don't change), and the accept-invitation description picks up the sentence 8126ef8 added to the route. That second one is why the SDK Generation job has been red on main since that commit; this brings it back to green. You can reproduce it with make -C sdkgen generate SDK_OUT=../sdk, generate-ui-core and generate-flutter-core under GOWORK=off. Only spec.json differs.
internal/telemetry is what every instrumented path in the engine and its plugins goes through. It holds the tracer scope (github.com/xraph/authsome), the bounded label values, the instruments recorded into a go-utils metrics sink, and a helper that marks a span failed with the error's type and never its text. It also has ProviderTransport, the client every third-party call will use. A call is measured by provider kind and result, and traced as a client span under the caller's span with the method, the status and (for vendor hosts) server.address on it. The URL, its path and its query never reach the span, and the request goes out with no traceparent, tracestate or baggage. A call made outside any request is measured but starts no trace of its own. telemetrytest gives the tests that follow a real Forge registry with its Prometheus scrape and a span recorder installed as the global provider.
… by route or tenant
The extension now passes the app's registry to a new option,
WithMetricsSink, and stops wrapping authsome's routes in
middleware.Metrics. That middleware is gone. forge already records
semconv http.server metrics for every route, so it double-measured each
request, and it labelled by method plus raw path. Some paths carry
secrets and IDs: the shared-signals push secret, session IDs, passkey
credential IDs.
With a sink configured the engine counts hook-bus events as
authsome_events_total{action,resource,outcome}, and the operational
signals become typed families: authsome_controls_degraded_total{control},
authsome_requests_throttled_total{limiter},
authsome_ratelimiter_errors_total{limiter} and
authsome_audit_records_failed_total. None of them is labelled by tenant,
which is an app ID and unbounded.
WithMetrics(bridge.MetricsCollector) keeps working and still receives
every event and counter it did, so a collector you wrote yourself sees no
change. The metricsadapter that implements it is fixed too: the tenant is
ignored, every series in a family carries the same label keys (labels()
used to drop empty values), counters get the _total suffix, and free-text
values fold into "other".
BREAKING CHANGE: dashboards reading forge_authsome_events_total by tenant,
action="http.request" or a route label, or the old counter names
(forge_authsome_audit_record_failed, forge_authsome_control_degraded,
forge_authsome_ratelimit_rejected and friends), need the new names. The
docs have the mapping. middleware.Metrics and middleware.RouteLabel are
removed.
…nd plugin hooks
Every sign-in path reaches IssueSession, so that is where an attempt is
counted: authsome_signin_attempts_total{method,result}, with the method
folded into a bounded set (social:github is social, sso:<connection> is
sso, password+mfa is password, a custom plugin's method is other). A call
that completes an MFA challenge is not a new attempt and is not counted
again. Password sign-in also counts the refusals that happen before
IssueSession: invalid, locked, rate_limited, unverified, password_expired,
banned, and denied when a plugin's before hook refuses.
A wrong password and an identifier nobody holds are both "invalid", and
they look the same in a trace too. The unknown identifier's stand-in hash
runs under the same verify password span a real check gets, and that span
carries no algorithm and no outcome, so nothing says which one it was.
Spans you get, all internal: issue session (method, result, MFA required
or just verified, opaque or jwt), refresh session (which credential, and
a result that tells replay_detected and dpop_rejected apart while the
caller still gets the same generic error), hash password, verify password,
and run plugin hook for each plugin's auth hook, so a slow risk check or
geo lookup shows up as itself. authsome_sessions_refreshed_total{result}
and authsome_password_hash_duration_seconds{operation,algorithm} come with
them.
forge runs Engine.Start inside a "start authsome" span, and grove opens a
db.client span under any valid span context, even one that ended at boot.
The legacy token sweep, webhook adoption and the retention sweeper all
inherited that span, so every query they made, idle ticks included, landed
in the boot trace forever. Start now strips the span where those contexts
are made and keeps their values and cancellation as before.
A retention sweep that wins the lease is now a sweep retention root span,
with the rows it removed and an event (kind and error type only) for any
kind that failed. Losing the lease starts no span. It is counted in
authsome_retention_sweeps_attempted_total{result} and
authsome_retention_rows_deleted_total{kind}; plugin sweepers share one
"plugins" kind because a custom plugin's name is not a bounded value.
The JWT session check ran on context.Background(), so its store lookup was
in no trace at all. The middleware now calls a context-taking variant
(SessionBindingConfig.JWTSessionContextChecker, added at the end of the
struct, and Engine.JWTSessionContextChecker) that runs on
context.WithoutCancel of the request: same cancellation and deadlines as
before, which is to say none, but under the request's span.
…s or baggage
Every send through Engine.Mailer() and Engine.SMSSender() is now a send
email or send sms span and an observation in
authsome_provider_request_duration_seconds{kind,result}, whichever bridge
you configured. The no-op mailer is handed back untouched. A provider's
error goes on the span as its type only, since those messages quote
addresses and codes.
The HTTP clients authsome builds for Twilio, Resend, Turnstile, the HIBP
breach check and the webhook URL test now go through the shared provider
transport. Each call is a client span with the method, the status and an
error class, and server.address only for vendor hosts (the webhook test
hides the customer's host). No URL, path or query is recorded, which
matters here: HIBP's path carries the password's SHA-1 prefix and
Twilio's carries the account SID. No traceparent, tracestate or baggage is
sent to any of them.
The breach check used to run on context.Background(). It now runs on
context.WithoutCancel of the request through a new
BreachChecker.IsBreachedContext, so the call sits in the request's trace
with exactly the cancellation and five-second timeout it had.
NewBreachCheckerWithClient and captcha.DefaultTurnstileTimeout are new and
exported for this.
…th2 token request
Magic link, phone, passkey, social and SSO now count their own refusals in
authsome_signin_attempts_total (an expired link, a wrong code, a failed
assertion, a provider that turned the user away) and hand the count to
IssueSession once they reach it, so each attempt is counted once. The
method is the bounded one; a social provider or an SSO connection name is
never a label.
MFA counts every code check in authsome_mfa_verifications_total{flow,
method,result}: the sign-in challenge (ok, invalid, replayed, expired
ticket, exhausted) and the verify endpoints for TOTP, SMS and recovery
codes.
An API key authentication is an authenticate api key span and a count in
authsome_apikey_authentications_total{result}. authsome reads keysmith's
store directly, so keysmith's own validate key span never fires for these
requests; this one stands in for it. An unknown prefix and a wrong secret
are both "invalid".
The OAuth2 token endpoint counts requests in
authsome_oauth2_tokens_requested_total{grant_type,result}, and each
client-secret bcrypt check is a verify client secret span. The changes to
plugins/oauth2provider stay small on purpose: an unmerged branch touches
issueTokens.
Social token exchange and profile fetches, and SSO OIDC discovery, token
exchange and userinfo, go through the provider transport. SSO hides the
host (the IdP is the customer's). OIDC discovery used to run on
context.Background(); in the callback it now runs on context.WithoutCancel
of the request with the same ten-second timeout. The login-URL paths have
no context to pass and keep Background. The SAML metadata fetch is
measured, and checking the assertion's signature is a verify saml
assertion span.
… when there is work
The CRM outbox worker ticks every 30 seconds and usually finds nothing. An
idle tick now makes its claim query outside any trace and starts no span.
A tick that claims jobs is one deliver retention jobs root span with a
deliver retention job child per job, and the hourly purge is a purge
retention outbox root span when it runs. Each job's outcome (done, retry,
dead, deferred, suppressed) goes on its span and into
authsome_retention_jobs_processed_total{provider,outcome}, where provider
is the CRM's type (hubspot, generic, other) and never the name an operator
gave it. A CRM's error reaches the span as its type only; those messages
quote response bodies.
The built-in HubSpot and generic providers' HTTP clients go through the
provider transport (kind=crm). HubSpot's host is a vendor's and is
recorded; a generic provider's URL is the customer's, so its host is not.
The shared-signals JWKS refresher starts a refresh stream keys root span
only for a round with a key set due for a fetch, using a new
jwksclient.Client.Due. Key-set fetches are client spans under it with no
host on them, since the JWKS URI belongs to the transmitter.
The privacy test drives the real HTTP routes with values that are easy to spot: a wrong password, an unknown identifier, the MFA gate and a wrong then right TOTP code, a refresh, a good and a guessed API key, and the OAuth2 client-credentials grant with a right and a wrong secret, all from a recognisable address and user agent. It then checks that none of the passwords, codes, ticket, tokens, keys, client secret, email, IPs, user agent, user ID or app ID appears on any span authsome started (names, attributes, events, statuses) or anywhere in the metric scrape. The values are built at runtime, so nothing here looks like a credential to a secret scanner. forge's own server span is left out of it on purpose. It records url.path and user_agent.original, which is forge's to redact. The extension test registers authsome in a real forge app and checks that its families land on /_/metrics, that no per-request authsome event or route label survives, and that no authsome series carries a tenant or an app ID.
…stopped being true A new Observability page lists every span (name, kind, attributes) and every metric as you'll scrape it from a Forge app (forge_authsome_..., with each label's values), says what never reaches telemetry, explains why a wrong password and an unknown identifier look the same, notes what forge records on its own server span, and maps the old metric names to the new ones. The Metrics bridge page described an interface with two methods, gauges nothing emits, an event table of webhook names, a duration the engine never passes and an OpenTelemetry example that exports nothing in a Forge app. It now matches the code: three methods, the events and counters the engine actually sends, why the tenant shouldn't be a label, and what the bundled adapter records. The audit trail, rate limiting, architecture, bridges and configuration pages point at the new families, and the hook-bus example stops labelling by app.
Since 1c0f0bc the workflow defaults to contents: read and each job widens what it needs, but prepare never got its own block. On a dispatch it runs git push origin "$TAG" and then creates the GitHub release, and both need contents: write, so the next release would stop at the tag push with a 403. No release has run since that commit, which is why nobody has hit it yet.
The social, SSO, magic link, phone, passkey and password sign-ins check the app's DPoP mandate before they reach IssueSession. A refusal there comes back as a 400, which the attempt counter read as "invalid", while the docs (and IssueSession's own DPoP gate) call it "denied". It is a policy refusal, not a bad credential, so each of those checks now records "denied". /v1/signin refuses before SignIn runs, so its handler counts the refusal itself. A malformed proof or a nonce challenge is still "invalid": those are the client getting the proof wrong, not the app saying no.
controls_degraded_total{control="breach_check"} could never fire. The
HIBP checker failed open by returning (false, nil) for a failed request,
a non-200 answer or an unreadable body, so the sign-up branch that counted
a degraded check was dead, and a password reset or change ignored the
error altogether.
IsBreachedContext, new in this branch, now returns
ErrBreachCheckUnavailable in those cases. It still reports false, and the
transport's error isn't wrapped because it quotes the URL and the URL
carries the hash prefix. IsBreached keeps its old contract exactly: it
fails open with a nil error. Sign-up, password reset and password change
go through one helper that counts a check that did not run, so nothing
about who gets through changes, only that you can now see the skip.
The HubSpot provider's client spans recorded server.address because api.hubapi.com is a vendor's host. ProviderConfig.BaseURL can point that provider anywhere, a proxy or something of the customer's, so the host is now only recorded when the base URL is HubSpot's own.
…d metrics A sign-up with an address that is already taken answers like a fresh one and spends the same hash, so the response time gives nothing away. The stand-in hash ran outside any span and outside the hash histogram, though, while a fresh sign-up recorded both. Engine.ConsumeDummyHashContext runs it under the same hash password span, with the same algorithm attribute, and records it in the same histogram, and the duplicate path uses it. The DPoP and sign-up telemetry tests in the api package now share telemetry_signin_test.go.
The dead-letter warning logged cause.Error(), and a CRM's error carries up to 512 bytes of its response body, which can echo the contact's email back. That line has been live since the plugin got the engine's logger. It now logs the job ID, the provider, the error's type and, when the CRM answered, the status class (4xx, 5xx). ProviderError gains a Status field so the class is there to log. The full text still goes into the job row through MarkDead and MarkRetry, which is where you look when you investigate a job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Forge app running authsome now exposes fifteen
forge_authsome_*families on/_/metrics, every label a bounded enum. Traces get spans for work forge can't see, and background work gets a root span only when it does something.Re-pins: forge and forge/extensions/auth v1.12.9, grove and drivers v1.8.0, chronicle v1.9.0, dispatch v1.9.1, and herald, keysmith, ledger, relay, vault and warden v1.8.0.
What breaks
relay_deliveries_totaltorelay_deliveries_attempted_totalandrelay_delivery_latency_secondstorelay_delivery_duration_seconds.middleware.Metricsandmiddleware.RouteLabelare gone: forge measures routes, and that middleware's raw-path label leaked the push secret and session and credential IDs.Engine.Mailer()andSMSSender()return unexported tracing wrappers: asserting to a concrete type fails, though no-op bridges pass through.WithMetrics(bridge.MetricsCollector)still works; the extension now usesWithMetricsSink.Spans
Client spans never record a URL, send traceparent or baggage, or name a customer's host (overridden HubSpot base URLs included).
Metrics
Counters
signin_attempts,mfa_verifications,sessions_refreshed,apikey_authentications,oauth2_tokens_requested,retention_sweeps_attempted,retention_rows_deleted,retention_jobs_processed,controls_degraded,requests_throttled,ratelimiter_errors,audit_records_failedandevents; histogramspassword_hash_duration_secondsandprovider_request_duration_seconds.Wrong password and unknown identifier look identical, as do fresh and duplicate sign-ups; a DPoP mandate refusal is
deniedeverywhere.Worth your eyes
forge's server span records raw
url.pathanduser_agent.original, so path secrets and IDs reach traces; that wants forge-side redaction, orexclude_pathsmeanwhile.fix/oauth2-jwt-scope-parkingmerges cleanly in Go (oauth2provider changes stay minimal for it); two docs files conflict only because it carries pre-rebase copies of main's docs commits, so take this side.The JWT session check, HIBP and OIDC discovery now use
context.WithoutCancelof the request (same timeouts, now traced).breach_checknow fires when HIBP can't answer;IsBreachedand who gets through don't change.No log line became live. A dead-lettered CRM job logs its ID, error type and status class, not the CRM's response, which can echo emails.
Side fixes: sdkgen keeps element types for forge's nullable list items, turning main's SDK Generation job green, and release
prepareregainscontents: write(missing since 1c0f0bc).Release
After merge:
gh workflow run release.yml --ref main -f tag=v1.8.0(modules defaults to all; npm publishes 1.8.0, Flutter rehearses).Testing
Race tests, uncapped golangci-lint, gosec, govulncheck, goimports, tidy and store conformance (Postgres, Mongo) pass under GOWORK=off. A privacy test pushes recognisable values through sign-in, MFA, refresh, API key and OAuth2 and fails if any reaches a span or the scrape.