Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/promotion.yml
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,13 @@ jobs:
paths=(':(glob).vapi-state.*.json')
if [[ "$PROMOTION_OUTCOME" == "success" ]]; then
paths+=(resources)
elif [[ -s tmp/promotion-applied.txt ]]; then
# A later transition failed: keep the files of the transitions
# that finished applying, so git matches what reached the
# platform. The failed transition's rewrites are discarded below.
while IFS= read -r applied; do
[[ -n "$applied" ]] && paths+=(":(literal)$applied")
done < tmp/promotion-applied.txt
fi

if [[ -z "$(git status --porcelain -- "${paths[@]}")" ]]; then
Expand All @@ -110,6 +117,10 @@ jobs:
git config user.email "vapi-gitops[bot]@users.noreply.github.com"
git add -A -- "${paths[@]}"
git commit -m "chore: record promoted Vapi state [skip promotion]"
# Unstaged rewrites from a failed transition would make the rebase
# refuse to run, and then nothing would be pushed, state included.
git reset --hard HEAD
git clean -fd -- resources
for attempt in 1 2 3; do
git pull --rebase origin main
if git push; then
Expand Down
56 changes: 56 additions & 0 deletions improvements.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ you which stack PR closes the row.**
| 30 | Tool-linking pass could PATCH a raw assistant slug | Mid-push 400 naming the wrong resource | None | RESOLVED 2026-08-03 (#51) |
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | Open |
| 32 | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56) |
| 35 | A failed promotion pushed nothing, not even state | git lost track of resources already on the platform | None | RESOLVED 2026-10-01 |

**Active backlog after cleanup:** `#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.

Expand Down Expand Up @@ -1711,6 +1712,61 @@ errors today; widening `include` is a separate change.

---

## 35. A failed promotion pushed nothing, not even state

**[RESOLVED 2026-10-01]**

**Discovered:** 2026-10-01, while planning the promotion check gate (TEST-141).

### Problem

When a multi-transition promotion failed partway, the workflow meant to
commit the UUID state and leave resource files alone. But the failing
transition had already rewritten tracked files in its target org, so the
commit step's `git pull --rebase` refused ("You have unstaged changes") and
the job pushed nothing: not the state, and not the files of the transitions
that had already reached the platform.

### Current behavior (Verified, before the fix)

- `.github/workflows/promotion.yml` "Commit reconciled files and UUID state"
staged only `.vapi-state.*.json` on a non-success outcome, then ran
`git pull --rebase origin main` with the failed transition's rewrites
still in the working tree.
- `promotionPlanApply` writes target files before `apply.ts` runs, so any
update to an existing target file left a tracked modification behind.
- Reproduced in a scratch repo: transitions a→b (applies) then b→c (fails
with an existing file in c) → `error: cannot pull with rebase`, exit 128,
origin unchanged.

### Risk

Git and the platform disagree after any partial failure: b's resources are
live but not in git, and the next promotion plans from stale files.

### Current mitigation

None needed once the fix below lands.

### Possible fix (landed)

- `src/promote-cmd.ts` truncates `tmp/promotion-applied.txt` at the start of
each `--apply` run and, after each successful `apply.ts`, appends the paths
`git status` reports under `resources/<target>/` — read from git rather
than the plan, because apply's own pull and push can rewrite other files.
- On a non-success outcome the commit step adds the state files plus exactly
those paths, commits, then `git reset --hard HEAD && git clean -fd --
resources` before rebasing, so the failed transition's rewrites can't block
the push.
- `promotionCommandRun(args, deps)` takes an injectable child runner for
tests (`tests/promote-cmd.test.ts`).

### Status

**RESOLVED 2026-10-01.**

---

## Out of scope (intentionally not improvements)

- **State file is identity-only and not git-ignored.** It's intentionally
Expand Down
71 changes: 68 additions & 3 deletions src/promote-cmd.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { resolveApiKey } from "./api-key.ts";
import { spawnSync } from "node:child_process";
import { existsSync, readFileSync } from "node:fs";
import { resolve } from "node:path";
import { execFileSync, spawnSync } from "node:child_process";
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import type { PromotionConfig, PromotionPipeline } from "./promotion.ts";
import {
Expand Down Expand Up @@ -36,6 +36,16 @@ const ROOT_DIR = resolve(
process.env.VAPI_GITOPS_ROOT ?? fileURLToPath(new URL("..", import.meta.url)),
);

// Files that transitions which finished applying left changed, one path per
// line. When a later transition fails, the promotion workflow commits exactly
// these (plus state) and discards the failed transition's rewrites, so what
// reached the platform is still recorded in git.
export const APPLIED_PATHS_FILE = "tmp/promotion-applied.txt";

export interface PromotionDeps {
childRun: typeof childRun;
}

function argumentsParse(args: string[]): PromotionArguments {
const parsed: PromotionArguments = { all: false, apply: false };
for (let index = 0; index < args.length; index++) {
Expand Down Expand Up @@ -189,6 +199,51 @@ function childRun(
if (result.status !== 0) throw new Error(`${script} failed for ${org}`);
}

// The paths git reports changed under resources/<org>/, including new and
// deleted files. Read from git rather than the plan because apply's pull and
// push can rewrite files the plan didn't name.
function changedPathsRead(org: string): string[] {
const output = execFileSync(
"git",
[
"status",
"--porcelain",
"-z",
"--untracked-files=all",
"--",
`resources/${org}`,
],
{ cwd: ROOT_DIR, encoding: "utf8" },
);
const entries = output.split("\0").filter(Boolean);
const paths: string[] = [];
for (let index = 0; index < entries.length; index++) {
const entry = entries[index]!;
paths.push(entry.slice(3));
// A rename or copy is followed by its original path.
if (entry[0] === "R" || entry[0] === "C") index++;
}
return paths;
}

function appliedPathsRecord(org: string): void {
const file = resolve(ROOT_DIR, APPLIED_PATHS_FILE);
const recorded = existsSync(file)
? readFileSync(file, "utf8").split("\n").filter(Boolean)
: [];
let changed: string[];
try {
changed = changedPathsRead(org);
} catch (error) {
console.warn(
`⚠️ Could not record applied paths for ${org}: ${error instanceof Error ? error.message : String(error)}`,
);
return;
}
const paths = [...new Set([...recorded, ...changed])];
writeFileSync(file, paths.length > 0 ? `${paths.join("\n")}\n` : "");
}

function stateLoad(org: string) {
const path = resolve(ROOT_DIR, `.vapi-state.${org}.json`);
if (!existsSync(path))
Expand All @@ -204,7 +259,9 @@ async function transitionRun(
apply: boolean,
tokens: Map<string, string>,
allowEmptySourceDeletion: boolean,
deps: PromotionDeps,
): Promise<boolean> {
const { childRun } = deps;
if (apply) {
childRun(
"src/pull.ts",
Expand Down Expand Up @@ -246,11 +303,13 @@ async function transitionRun(
connectionLoad(config, transition.target, tokens),
["--force", "--allow-new-files", "--resolve=ours", ...changedPaths],
);
appliedPathsRecord(transition.target);
return plan.changes.some((change) => change.kind === "delete");
}

export async function promotionCommandRun(
args = process.argv.slice(2),
deps: PromotionDeps = { childRun },
): Promise<void> {
const parsed = argumentsParse(args);
const configPath = resolve(ROOT_DIR, "promotion.yml");
Expand All @@ -259,6 +318,11 @@ export async function promotionCommandRun(
const config = promotionConfigParse(readFileSync(configPath, "utf8"));
const tokens = parsed.apply ? tokensParse() : new Map<string, string>();
delete process.env.VAPI_PROMOTION_TOKENS;
if (parsed.apply) {
const file = resolve(ROOT_DIR, APPLIED_PATHS_FILE);
mkdirSync(dirname(file), { recursive: true });
writeFileSync(file, "");
}
// Applying a deletion removes the intermediate org's state entry. Carry the
// reviewed authorization forward so the same deletion can reach later orgs.
const deletionAuthorizedSources = new Set<string>();
Expand All @@ -270,6 +334,7 @@ export async function promotionCommandRun(
parsed.apply,
tokens,
deletionAuthorizedSources.has(sourceKey),
deps,
);
if (deleted)
deletionAuthorizedSources.add(
Expand Down
Loading
Loading