Repository navigation
fix(aio): keep content out of ai events in privacy mode - #1105
Open
carlos-marchal-ph wants to merge 1 commit into
Open
carlos-marchal-ph wants to merge 1 commit into
carlos-marchal-ph wants to merge 1 commit into
Conversation
![posthog[bot]](https://avatars.githubusercontent.com/in/1203049?s=60&v=4){kind=small}
Comment on lines
+48
to
+54
| _CONTENT_ATTRIBUTE_ROOTS = ( | ||
| "gen_ai.prompt", | ||
| "gen_ai.completion", | ||
| "llm.prompts", | ||
| "llm.input_messages", | ||
| "llm.output_messages", | ||
| ) |
Contributor
There was a problem hiding this comment.
P1 · Redact OpenInference function calls and prompt-template content
With OpenInference LangChain instrumentation, spans carry model-generated arguments in llm.function_call and user inputs in llm.prompt_template.variables. Neither matches _is_content_attribute, so both OTel integrations export this content with privacy_mode=True, even after removing the message attributes. Extend the filter to cover these content fields.
Comment on lines
+112
to
+114
| attributes=attributes, | ||
| events=events, | ||
| links=span.links, |
Contributor
There was a problem hiding this comment.
P2 · Preserve collection types supported by older OTel encoders
With supported OpenTelemetry SDK 1.20.0, privacy-mode spans retaining any attribute fail OTLP serialization: ReadableSpan.dropped_attributes reads .dropped, which this plain dict lacks. Nonempty event lists and link tuples fail similarly. Both integrations lose the export batch; use compatible bounded collections or raise the minimum OTel version.
Contributor
posthog-python Compliance ReportDate: 2026-10-22T16:32:00.814446+00:00 ✅ All Tests Passed!121/121 tests passed Capture_V1 Tests✅ 95/95 tests passed View Details
Capture_Ai Tests✅ 5/5 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
Feature_Flags_Local_Evaluation Tests✅ 4/4 tests passed View Details
|
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| "gen_ai.completion", | ||
| "llm.prompts", | ||
| "llm.input_messages", | ||
| "llm.output_messages", |
There was a problem hiding this comment.
High: OpenInference payloads bypass privacy redaction
OpenInference emits completion text under llm.choices.*, prompt values under llm.prompt_template.variables, and function arguments under llm.function_call, none of which this filter removes. (github.com) The telemetry recipient can therefore read this private content even with privacy_mode=True in either OTel integration, because the surviving attributes are forwarded to OTLP and serialized unchanged; include these payload attributes in the redaction rules. (raw.githubusercontent.com)
Suggested change
| "llm.output_messages", | |
| "llm.output_messages", | |
| "llm.choices", | |
| "llm.function_call", | |
| "llm.prompt_template.template", | |
| "llm.prompt_template.variables", |
PR overviewThe PR updates privacy-mode filtering for AI telemetry events in the OpenTelemetry integrations to keep content out of emitted events. One issue remains open: the filter misses OpenInference attributes containing completion text, prompt variables, and function arguments. These attributes are forwarded unchanged to the telemetry recipient even when privacy_mode=True in either integration, allowing private content to leave the application despite the privacy setting. Open issues (1)
Fixed/addressed: 0 · PR risk: 7/10 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
privacy mode doesn't keep content out of AI events:
$ai_errorisstr(exc), never redacted..parse()validation errors quote model output, gemini client-side validation quotes the prompt, langchainOutputParserExceptionincludes the full output, and openai-compatible servers echo the request in 4xx bodies.$exceptionautocapture ignores privacy.headers/authorizationare sent verbatim in$ai_tools, privacy or not.posthog_privacy_mode=Falseundid a client default ofTrue.model_configprompt/instructions and$ai_trace_metadatasent under privacy.context: https://posthog.slack.com/archives/C0ACMS2BNTU/p1792232645298019
Changes
$ai_errorunder privacy (client or per-call/handler flag): built only from structured fields that pass a strict grammar, never message/traceback/attributes. format, shared with posthog-js:{Name}[: {status}][ {code}…][ param={param}][ request_id={id}][ (caused by A, B)]e.g.
BadRequestError: 400 invalid_request_error param=messages[3].content request_id=req_123. applied to openai / anthropic / gemini (incl. embeddings), langchain generation/span/trace, openai agents. the middleware's fixed "Tool returned an error" text is kept.$exceptionskipped under privacy.headers/authorizationvalues ontype == "mcp"tools become[REDACTED], always, without mutating the caller's objects. tool definitions otherwise unchanged and still sent under privacy.model_configprompt/instructionsand$ai_trace_metadataset toNone.privacy_modeandclientkwargs onPostHogSpanProcessor/PostHogTraceExporter; also follows the module-level client. content attributes and exception messages stripped, list derived from the server's otel →$ai_*mapping. tool definitions andexception.typekept. if redaction fails the span is dropped, not sent.Notes for reviewers
capture-aionly defines privacy as stripping content properties; this widens it to errors and OTel. spec update to follow.references/public_api_snapshot.txt).💚 How did you test it?
posthog/test/ai/test_privacy_mode.py: summarizer against real openai / anthropic / google-genai / pydantic errors and grammar rejections; MCP masking across providers and tool shapes; canary tests with a realClient(privacy_mode=True, enable_exception_autocapture=True)asserting a canary string appears nowhere in any captured event, incl.$exception, across every integration and OTel.pytest posthog/test/ai1288 passed; ruff, mypy (baseline),make public_api_checkclean.📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
$ai_errorover text scrubbing: regex/PII detection can't guarantee anything. rejected a full schema allowlist for$ai_tools(would silently drop new tool types); only the documented MCP credential slots are masked.fix(aio)/ai-privacy-mode-leaks.🤖 Generated with Claude Code