Skip to content

Open an evergreen docs PR after each SDK publish - #225

Merged
rgarcia merged 3 commits into
mainfrom
hypeship/update-docs-on-sdk-publish
Oct 30, 2026
Merged

rgarcia merged 3 commits into
mainfrom
hypeship/update-docs-on-sdk-publish

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an Update Docs workflow that runs after Publish NPM and keeps one evergreen PR open on kernel/docs (branch sdk-docs-update) with doc updates for the new SDK release. It follows the same structure as update-mcp-coverage.yml: a Pi agent edits the docs checkout, the workflow keeps only allowlisted paths, validates, and a separate job pushes the branch and opens or updates the PR.

Differences from the MCP updater:

  • Baseline version. The docs repo has no SDK pin, so the workflow reads the last covered version from Docs: Update for Node SDK X.Y.Z commit subjects in the docs history. The bot commit and the PR title use the same text, so the marker survives a squash merge. If no marker exists, it uses the previous stable SDK tag. Runs for a version that's already covered are skipped.
  • Multi-language examples. The Python and Go SDKs are checked out at the matching v$SDK_VERSION tag, or the default branch if that tag isn't published yet, so the agent can check Python and Go examples against real methods. The docs style guide requires TypeScript and Python examples.
  • Agent inputs. The agent gets the Node SDK diff (CHANGELOG.md, api.md, src/resources) and the OpenAPI spec diff for the same window.
  • Allowlist. Only docs.json and .mdx files are published. Hidden directories, changelog.mdx, and the CLI and MCP server reference pages are excluded because they're maintained separately.
  • Validation. jq empty docs.json, mint broken-links (same mint@4.2.930 pin as the docs repo CI), and git diff --check.

Testing

  • actionlint with shellcheck passes. The only finding is the info-level SC2016 for backticks in the PR body printf, which update-mcp-coverage.yml also gets.
  • Ran the "Prepare update branch and SDK diff" script locally against real kernel/docs and kernel-node-sdk clones (without the bot git config lines):
    • no marker: diff from v0.135.0 to v0.136.0
    • a 0.130.0 marker: diff covers 0.131.0 through 0.136.0
    • a 0.136.0 marker: skipped as stale
  • Checked the path allowlist script against sample paths.
  • The workflow hasn't run end to end in Actions yet. After merge, trigger it with workflow_dispatch to get a first PR. It uses the same ADMIN_APP_* secrets, which already open PRs on kernel/docs.

@rgarcia
rgarcia requested a review from hiroTamada October 29, 2026 21:15

@hiroTamada hiroTamada left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed. the overall structure mirrors the MCP updater nicely, but there are two bugs in the version baseline and one bigger design question.

Bugs

  • .github/workflows/update-docs.yml:30,42 — on workflow_dispatch, SDK_SHA is main HEAD rather than the v$version tag. With no docs marker, line 149 then picks the current release as the baseline and the run skips as stale (simulated on a post-merge main: SDK_VERSION=0.138.0 old_version=0.138.0), so the suggested first manual run does nothing. With a marker, the diff includes unreleased commits. Consider resolving sha from refs/tags/v$version.
  • .github/workflows/update-docs.yml:145,367,374 — the marker is read from the update branch's own history, so each run diffs only since the last unmerged bot commit, and gh pr edit --body-file replaces the body with that increment's report. Reviewers lose the description of earlier changes still in the PR. Consider reading the marker from origin/main so each run's report covers the whole PR. The prompt already has the agent read the full diff against origin/main.

Suggestions

  • .github/workflows/update-docs.yml:159,162 — the SDK diff skips src/lib/ (browser-routing, browser-fetch, audit-log-download, config-registry-wait) and src/client.ts, which are hand-written and user-facing. The agent only sees the CHANGELOG line for those. Consider adding them to the pathspec.
  • .github/prompts/update-docs.md:45-46 — the spec marks /credentials*, /org/credential_providers*, and /mpp/browsers as x-hidden, but the SDK still exposes client.credentials.*. Consider telling the agent not to document x-hidden / x-stainless-skip operations.

Questions

  • .github/prompts/update-docs.md:25-30 — most SDK releases need no guide changes, and the docs team already ships feature docs within about a day (e.g. #729, #730, #724, #602). The prompt's field-level triggers ("new response fields, new enum values") overlap with the generated API reference, and the agent can't see existing or in-flight human docs. Consider defaulting to no change, editing only guides that are now wrong, passing in recent docs commits and open PRs, and listing missing coverage in the report instead of writing pages. A dry run over v0.130.0 to v0.138.0 against what was actually merged would show the noise level.
  • .github/workflows/update-docs.yml:89-97 — the Go SDK tags trail Node (currently v0.137.0 vs v0.138.0), so the default-branch fallback will be routine. Is it OK that Go examples get dropped for that release?
  • .github/workflows/update-docs.yml:145-151 — a no-op run records no marker, so once a marker exists, a run of no-docs releases keeps widening the audit window. Is that acceptable given the 55-minute timeout?
  • .github/workflows/update-docs.yml:160,164,186 — the spec is taken from kernel/kernel main at the release commit time, so it can be ahead of what the SDK codegen picked up. Since the prompt treats the spec as the source of truth, the agent could document behavior the released SDK doesn't have yet.

Nits

  • .github/scripts/docs-update-disallowed-paths.sh:8 — .* only matches top-level hidden paths (foo/.hidden/x.mdx passes), but the prompt says hidden directories are excluded. Consider adding */.*.
  • .github/workflows/update-docs.yml:368 — the Validation list leaves out git diff --check.

@rgarcia

rgarcia commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. All of it is addressed in a394d50, except the Go question below.

Bugs

  • Dispatch SHA. sha now resolves from refs/tags/v$version, so manual runs use the published release instead of main HEAD.
  • Report only covering the latest increment. I changed the baseline instead of reading the marker from origin/main. With a default of no change, most runs commit nothing, so a main-based window would keep growing (your 55-minute question). Each run now audits exactly one release (previous tag..v$SDK_VERSION). The report is still cumulative: the agent gets the current PR body as previous-report.md and has to describe the whole diff against origin/main. If the branch gets regenerated from main, the baseline drops back to main's last docs update so dropped bot changes are re-audited. workflow_dispatch also takes a since_version input for widening the window by hand, for example after a failed run.

Suggestions

  • src/client.ts and src/lib are now in the SDK diff.
  • The prompt says not to document x-hidden or x-stainless-skip operations, even if an SDK exposes them.

Questions

  • Default to no change. Agreed. The prompt now opens with "most SDK releases need no docs change, finishing with no changes is the expected outcome". It only edits guide text or examples that are now wrong or misleading, or adds a short addition where a list presented as complete would mislead. It never creates pages and lists missing coverage in the report instead. The agent gets docs commits since the baseline (recent-docs-commits.txt) and open docs PRs with their files (open-docs-prs.txt) and skips anything they cover. No-op runs write the report, including missing coverage, to the job summary.
  • Go tags trailing. v0.139.0 has no Go tag yet, so the fallback is routine. The agent now leaves an existing Go example unchanged when the Go SDK lacks the method and says so in the report. It doesn't drop or invent Go code. The next release's run catches up once Go is tagged.
  • Spec ahead of the SDK. The prompt now treats the checked-out SDKs as the source of truth for what exists. The spec is used only for the meaning of operations and fields the SDK already has.

Nits

  • The allowlist now matches */.*.
  • git diff --check is in the Validation list.

Dry run: 0.130.0 → 0.139.0 against current docs main

I ran the updated prompt locally with since_version=0.130.0 (about 10 minutes).

  • 3 lines changed in 2 pages:
    • human_intervention and AWAITING_HUMAN_INTERVENTION added to the choice-type list and flow-step table in auth/programmatic.mdx.
    • shop_pay_oauth added to the action.name list in integrations/wallets/overview.mdx.
    • All of these values exist in src/resources/auth/connections.ts at v0.139.0, and mint broken-links passed.
  • Left alone because recent docs commits or open docs PRs already cover them: saving running browsers to profiles, passkeys, 1Password changes, projects_disabled, and wallet payment methods (#728).
  • 6 gaps listed for a person (hitl setup, Shop Pay guide, fingerprint override, payment_window_expired, leased_at, last_used_at), none written.

@rgarcia
rgarcia requested a review from hiroTamada October 30, 2026 14:39

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a394d50. Configure here.

old_version=$SINCE_VERSION
if [[ -z "$old_version" && "${regenerated:-}" == true ]]; then
old_version=$covered
fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Audit window skips uncovered SDK versions

Medium Severity

The audit baseline now defaults to the previous SDK tag and only uses the docs coverage marker after a conflict regeneration. A failed or skipped generate therefore never gets re-audited: the next publish diffs only the newest tag. Regeneration has the same hole when main has no marker yet, which is the first evergreen PR — the branch is reset and earlier unmerged audits are dropped instead of replayed.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a394d50. Configure here.

@rgarcia
rgarcia merged commit 7a68698 into main Oct 30, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants