Skip to content

fix: track new users and rejected agent runs in telemetry - #493

Open
angeloashmore wants to merge 1 commit into
mainfrom
fix/telemetry-identify-new-users
Open

angeloashmore wants to merge 1 commit into
mainfrom
fix/telemetry-identify-new-users

Conversation

@angeloashmore

@angeloashmore angeloashmore commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Resolves:

Description

We are measuring the new developer onboarding funnel: copy the init command, run prismic init, then create a page type.

Before this PR, the CLI undercounted the init step. A new user who logged in during init was not linked to their account for that run. An agent run without --task-id or --user-intent was rejected before telemetry started, so it was not tracked at all.

Amplitude, Nov 10–16, staff excluded:

  • 33 users copied the init command.
  • 17 have a successful init tracked.
  • 16 have no init event under their user.

After this PR, a run that includes a login is attributed to the user who logged in. A rejected agent run sends a failed Start and End event with the error. The rejection message and exit code are unchanged.

Checklist

  • If my changes require tests, I added them.
  • If my changes affect backward compatibility, it has been discussed.
  • If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

How to QA 1

  1. Log out, then run prismic init and log in when asked. The Start and End events of that run carry your user ID.
  2. Run AI_AGENT=test prismic docs list with no flags. It prints the same error as before, and sends a Start and an End event with success: false.

Generated by Claude Code


Note

Low Risk
Telemetry-only behavior changes; login and agent validation UX and exit codes are unchanged.

Overview
Fixes onboarding funnel telemetry so first-time logins (e.g. during prismic init) are attributed to the user, and agent runs rejected for missing --task-id / --user-intent still emit command Start/End events.

After tokens are saved, login completion now calls trackUser(profile) right after fetching the profile (alongside the existing forgetTrackedUser), so that command run is linked to the user without waiting for the main entry path’s async profile lookup.

Agent validation no longer short-circuits before telemetry init: Sentry/tracking still initialize, then on validation failure the CLI keeps the same error message and exit code but records trackCommandStart and trackCommandEnd({ error }) so failed agent attempts show up as unsuccessful runs.

Reviewed by Cursor Bugbot for commit 9826613. Bugbot is set up for automated code reviews on this repo. Configure here.

Footnotes

  1. Please use these labels when submitting a review:
    ❓ #ask: Ask a question.
    💡 #idea: Suggest an idea.
    ⚠️ #issue: Strongly suggest a change.
    🎉 #nice: Share a compliment. ↩

A user who logs in during a command, such as `prismic init`, was not
linked to their account for that command. Their events carried only
the machine ID.

An agent run without `--task-id` or `--user-intent` was rejected before
telemetry started, so the run was not tracked.

The CLI now identifies the user right after a login. A rejected agent
run now sends a failed Start and End event. The error message and exit
code do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1dGD8RJNc2KTciWMJh6cq

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9826613. Configure here.

Comment thread src/index.ts
trackCommandStart(command);
trackCommandEnd({ error: agentOptionsError });
return;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent rejection waits on profile fetch

Medium Severity

The agent-options rejection now runs after telemetry init, so a logged-in run with no stored user id (including every PRISMIC_TOKEN run) starts a fire-and-forget getUserProfile. That in-flight fetch keeps the process alive after the error is printed, so exit waits on the profile request. If it hangs, the new Start and End events never flush.

Additional Locations (1)
Fix in Cursor Fix in Web

Triggered by team rule: Prismic Pull Request Review Method

Reviewed by Cursor Bugbot for commit 9826613. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants