Skip to content
RushObservabilityPublic

About

AI-powered SRE investigation agent for the rush observability platform

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

sre-agent

It reads the dashboards so you don't have to.

ci license

Give sre-agent an alert or a plain-English question and it forms a hypothesis and goes looking — across traces, logs, metrics, Kubernetes, ArgoCD, and deploy history — until it can name a likely cause. It streams its reasoning as it works, so you watch the investigation rather than wait for a verdict.

Under the hood it's a ReAct loop over an OpenAI-compatible model with built-in investigation tools. The interesting problems here aren't calling the LLM; they're knowing when to stop, what to keep in a small context window, and how to keep the model from chasing its own tail.

Not a standalone product. sre-agent is one service in a Rush deployment and expects the rest to be running.

How it works

Investigations follow a five-phase playbook — orient, hypothesize, gather evidence, verify, conclude — and the agent keeps a small working memory (suspect services, confirmed facts, things ruled out) that survives transcript compaction. Duplicate tool calls come back as errors written to teach the model to self-correct. Parse retries are counted apart from real work, so a malformed response doesn't eat the investigation budget. A run of empty results forces a summary instead of more thrashing.

It reads telemetry, configuration, custom skills, and investigation history through query-api. The agent receives no database credentials. Kubernetes and ArgoCD/Flux access uses its in-cluster ServiceAccount.

Read-only GitHub source access

The agent can inspect code linked to an observed service without webhooks and without a general-purpose shell. Create a GitHub App with only the repository permission Contents: Read-only, install it on selected repositories, and mount its PEM private key into the agent. Each repository link can carry its operator-approved GitHub installation and stable repository IDs. The policy is keyed by Rush tenant, so API callers cannot claim another tenant's installation.

Required environment variables:

GITHUB_APP_ID=<numeric app id>
GITHUB_APP_PRIVATE_KEY_PATH=/var/run/rush-github/private-key.pem
SRE_AGENT_GITHUB_REPOSITORY_POLICY={"acme":[{"repository":"acme/api","installationId":654321,"repositoryId":123456789}]}
REPOSITORY_CACHE_DIR=/var/run/rush-repositories

GITHUB_API_URL is optional for GitHub Enterprise Server. There is deliberately no global installation fallback. The query API and agent both require an exact tenant/repository/installation/repository-ID policy match before access, including cached access. The agent mints a short-lived installation token scoped by stable repository ID and contents: read, downloads a bounded tar snapshot, rejects links/special files/path traversal, and exposes only list_repository_files, search_repository, and read_repository_file. list_changes uses the same token to call GitHub's compare endpoint between the commit SHAs of two deploy markers, so deploy events need commit_sha. PR numbers and titles come from merge and squash commit messages, which needs no extra GitHub permission; rebase merges show as plain commits. Repository code is never executed. Successful source reads are sent to query-api's tamper-evident audit log without tokens or source contents.

Tools

Tool Purpose
query_traces / get_trace search spans; pull a full trace by ID
search_logs logs by severity and text
get_insights quick overview: anomalies in the window, repeated WARN/ERROR log patterns, health score
query_metrics request rate, error rate, p50/p99
list_services / service_dependencies health snapshot; call graph
compare_service_windows / rank_slow_dependencies incident-vs-baseline service comparison; slow downstream ranking
analyze_trace_critical_path identify spans dominating a trace's duration
inspect_profiles check CPU profiles for a suspect app; compare function hotspots and call paths across incident and baseline windows
get_resource_saturation / list_metric_catalog resource pressure; available metric names and labels
detect_service_silence distinguish missing traffic from a healthy low-volume service
inspect_postgresql correlate an app's PostgreSQL spans with slow-query, lock, advisor, replication, recovery, and planning evidence from the existing read-only PostgreSQL collector
list_deploys / get_anomaly_context / search_past_incidents deploys; anomaly context; prior investigation leads
get_argocd_app / get_flux_resource ArgoCD and Flux resource health
kube_describe / kube_events describe resources in the caller's mapped namespaces; namespace events
search_kubernetes_access correlate a suspected operator action with tenant-scoped Kubernetes access metadata when the paid add-on is enabled
list_repository_files / search_repository / read_repository_file bounded, read-only access to operator-approved GitHub repositories
list_changes PRs, commits, and changed files (with bounded diffs) between the deploy running at a given time and the previous deploy
load_skill load an investigation playbook

The built-in skills are error_rate_spike, latency_degradation, deploy_regression, dependency_failure, argocd_unhealthy, flux_unhealthy, throughput_anomaly, and postgresql_diagnostics.

Custom skills are managed from Settings → AI Agent → Custom skills in the frontend. They are stored by query-api, loaded fresh for the next investigation, and merged with the built-ins. The agent only receives enabled skills. Custom skill bodies are treated as untrusted advisory content and never override the agent's system rules.

Profiling evidence

For latency, CPU pressure, and deploy investigations, the agent is instructed to check profiling for the suspect apps when the investigation includes profiles or all scope. It uses the investigation's tenant and exact time windows, and returns up to 15 functions and five call paths per window. It tries cpu first, then sampled_cpu if there are no CPU samples; it never combines the two types.

The app must already be sending profiles to Rush. Update and restart both query-api and sre-agent for this tool: query-api must allow the agent's internal credential to read GET /api/v1/profiles. No separate integration or profiling license is required. For example, ask: "Investigate articles latency over the last hour and check its CPU profiles."

Self CPU belongs to the leaf function; total CPU includes its callees, counting recursive functions once per stack. The agent compares CPU seconds and function shares, not request durations. Captured CPU divided by the whole window is an average across the selected processes and can be affected by collection gaps. Missing profiles or a missing baseline are reported as evidence gaps. The agent continues with traces, logs, and metrics rather than calling the app healthy.

Running it

Needs a running query-api v0.1.11 or later, since every telemetry query goes through its versioned endpoint, /api/v1/query/execute. Provider connections and API keys are configured in Settings → AI Agent → Providers. The agent never connects to ClickHouse or receives database or LLM credentials.

export QUERY_API_URL=http://localhost:8080
export SRE_AGENT_INTERNAL_TOKEN=dev-local-agent-token
make run

make docker        # build image
make docker-push

The container uses Chainguard's glibc-dynamic runtime, pinned to a multi-architecture digest for amd64 and arm64. It runs as UID/GID 65532, includes CA certificates, and has no shell or package manager. Any mounted repository-cache directory must be writable by that UID/GID. Dependabot tracks runtime digest updates. Trivy findings are advisory: fixable high/critical vulnerabilities leave a failed scan step and a warning in the release job, but do not block publication. The job summary distinguishes findings from scanner errors or missing reports. JSON findings and a Markdown summary are retained for 90 days as workflow artifacts and attached to the GitHub release alongside the SPDX SBOM. SBOM generation, attestations, tests, and the non-root check remain required; pull-request dependency checks are unchanged.

Variable Default
SRE_AGENT_PORT 8081 listen port
SRE_AGENT_MAX_CONCURRENT_INVESTIGATIONS 4 maximum investigations executing at once
SRE_AGENT_MAX_QUEUED_INVESTIGATIONS 16 maximum investigations waiting for a slot
SRE_AGENT_MAX_TOOL_STEPS 40 fallback maximum tool-bearing rounds per investigation; Settings takes precedence
SRE_AGENT_MAX_LLM_CALLS 55 fallback maximum total provider calls, including retries and final review; Settings takes precedence
SRE_AGENT_RUNTIME_METRICS_INTERVAL_SECS 15 process/runtime metric sampling interval
SRE_AGENT_INTERNAL_TOKEN required shared query-api-to-agent credential; never expose it to browsers
QUERY_API_URL required query-api base URL used for telemetry, configuration, and investigation state
ARGOCD_NAMESPACE argocd where ArgoCD Application CRDs live

The same agent settings can be managed at runtime in the frontend under Settings → AI Agent:

  • Tenant access enables the agent for all enabled tenants or an explicit list.
  • Providers stores encrypted provider credentials and routing policy in query-api.
  • Models defines the named models, default model, and reasoning levels available to users.
  • Investigation limits changes the tool-step and LLM-call budgets for new investigations.
  • Custom skills creates, edits, enables/disables, and deletes user-authored playbooks.

Runtime settings are stored by query-api and take effect for new investigations (budget changes may remain cached for up to 30 seconds).

The standalone sre_evals harness still accepts OPENAI_API_KEY and optional OPENAI_BASE_URL so it can use an independent judge. Those variables are not read by the production server.

Kubernetes access boundaries

Kubernetes inspection is deny-by-default. Set SRE_AGENT_KUBE_TENANT_NAMESPACES to a JSON object that maps Rush tenant IDs to the namespaces they may inspect:

SRE_AGENT_KUBE_TENANT_NAMESPACES={"acme":["acme-prod"],"*":["shared-observability"]}

The optional * entry is a shared-namespace allowlist; it does not grant access to arbitrary namespaces. Cluster-scoped resources such as nodes and namespace enumeration are denied unless both SRE_AGENT_KUBE_ALLOW_CLUSTER_SCOPED=true and the authenticated caller has the explicit kube_cluster scope (the Rush admin role is the only role that receives it). The Helm chart creates a dedicated service account and namespace RoleBindings from sreAgent.kube.tenantNamespaces; it does not grant the agent Secrets, pod-log, node, or namespace permissions.

API

POST /api/v1/investigate starts an investigation and returns a Server-Sent Events stream:

{ "event_id": "", "question": "why is checkout slow?", "additional_context": "" }

Events: session_created (for a new interactive session), thinking_delta (incremental reasoning), tool_call, tool_result, summary (final or preliminary report), error, and done (token usage + round count). Sessions can be continued with follow-up questions from the frontend.

GET /healthz is a cheap liveness check. GET /readyz verifies query-api and LLM configuration and returns 503 until both are available. GET /metrics exports low-cardinality Prometheus metrics for investigation admission, investigation outcomes, report-kind outcomes, investigation work and report sizes, process/runtime health, SSE streams, LLM calls and token usage, query-api requests and tool calls; it uses the same x-rush-internal-token header as the API.

Part of Rush

  • query-api — query, ingest, config backend
  • frontend — where investigations are launched and streamed
  • helm-charts — deploys all of it together

License

Business Source License 1.1.

About

AI-powered SRE investigation agent for the rush observability platform

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages