test(autoland): pin step execution behavior with golden scenarios

tests/golden/autoland/README.md

@@ -0,0 +1,28 @@
+# Autoland golden files
+
+Expected output of the end-to-end autoland scenarios in
+`tests/test_autoland_golden.py`. Each scenario runs `stack-pr autoland` against
+a scripted GitHub and local clone (`tests/autoland_scenarios.py`) and writes:
+
+| File                         | What it holds                                                         |
+| ---------------------------- | --------------------------------------------------------------------- |
+| `<scenario>.out.txt`         | Everything the run printed (plain console, no rich), including input |
+| `<scenario>.checkpoint.json` | The last checkpoint the run saved (a successful run deletes it after) |
+| `<scenario>.status.txt`      | `stack-pr autoland --status` over that checkpoint                     |
+| `<scenario>.status.json`     | `stack-pr autoland --status -o json` over that checkpoint             |
+| `<scenario>.plan.txt`        | That checkpoint's plan, as the `-i` editor would show it              |
+
+Temporary paths are replaced with `<TMP>` and the checkpoint's save time with
+`<SAVED>`; everything else (SHAs included) is deterministic.
+
+## Regenerating
+
+When a change to autoland's behavior is intended, regenerate the files and
+review the diff like any other change:
+
+```bash
+UPDATE_GOLDEN=1 uv run --no-sync pytest tests/test_autoland_golden.py
+git diff tests/golden/autoland
+```
+
+A refactor that is meant to change nothing should pass without regenerating.

tests/golden/autoland/checks_fail.checkpoint.json

@@ -0,0 +1,45 @@
+{
+  "version": 1,
+  "branch": "feat",
+  "base": "main",
+  "plan_file": null,
+  "current_step": 0,
+  "last_landed_sha": "",
+  "abort_reason": "PR #101 checks failed after retries",
+  "stack": [
+    {
+      "pr_url": "https://github.com/o/r/pull/101",
+      "pr_number": 101,
+      "branch": "TestBot/stack/101",
+      "title": "Add the widget model",
+      "review_decision": "APPROVED",
+      "state": "failed",
+      "check_retries": 1,
+      "queue_retries": 0,
+      "error_message": "Checks failed after 1 retries: ci/test"
+    },
+    {
+      "pr_url": "https://github.com/o/r/pull/102",
+      "pr_number": 102,
+      "branch": "TestBot/stack/102",
+      "title": "Expose widgets over the API",
+      "review_decision": "APPROVED",
+      "state": "pending",
+      "check_retries": 0,
+      "queue_retries": 0,
+      "error_message": ""
+    }
+  ],
+  "plan": [
+    {
+      "type": "land",
+      "entry_index": 0,
+      "pr_number": 101
+    },
+    {
+      "type": "land",
+      "entry_index": 1,
+      "pr_number": 102
+    }
+  ]
+}

tests/golden/autoland/checks_fail.out.txt

@@ -0,0 +1,33 @@
+
+Discovering stack...
+
+
+Autoland plan · 2 steps · 0 done · 2 remaining
+
+→ ⬜ 1. Pending | Land | #101 - Add the widget model
+  ⬜ 2. Pending | Land | #102 - Expose widgets over the API
+State file: <TMP>/home/.stack-pr/autoland/feat.json
+
+
+============================================================
+Step 1/2: Landing PR #101 — Add the widget model
+============================================================
+Rerunning failed checks (attempt 1/1): ci/test
+$ gh run rerun 9101 --failed
+PR #101: Failed: ci/test — polling in 120s
+PR #101: Checks in progress... — polling in 120s
+
+
+
+Autoland plan · 2 steps · 0 done · 2 remaining
+
+→ ❌ 1. Failed  | Land | #101 - Add the widget model
+        ↳ Checks failed after 1 retries: ci/test · retries CI 1 / MQ 0
+  ⬜ 2. Pending | Land | #102 - Expose widgets over the API
+
+ABORTED: PR #101 checks failed after retries
+
+Landing failed: PR #101 checks failed after retries
+
+State saved to <TMP>/home/.stack-pr/autoland/feat.json — resume with --resume
+

tests/golden/autoland/checks_fail.plan.txt

@@ -0,0 +1,13 @@
+# Autoland plan — edit steps below.
+# l [<pr>]        = land that PR (a number, or its URL); a bare 'l'
+#                   lands the next PR in the stack instead
+# w <workflow>    = wait for a workflow to complete
+# c [<condition>] = pause for manual confirmation; the optional
+#                   condition names what to verify before proceeding
+#                   (e.g. 'c QA sign-off complete')
+#
+# Lines starting with # are comments and are ignored.
+# Blank lines are ignored.
+#
+l 101                         # Add the widget model
+l 102                         # Expose widgets over the API

tests/golden/autoland/checks_fail.status.json

@@ -0,0 +1,42 @@
+{
+  "status": "stopped",
+  "branch": "feat",
+  "base": "main",
+  "state_file": "<TMP>/home/.stack-pr/autoland/feat.json",
+  "state_file_exists": true,
+  "lock_file": "<TMP>/home/.stack-pr/autoland/feat.json.lock",
+  "pid": null,
+  "last_saved": "<SAVED>",
+  "abort_reason": "PR #101 checks failed after retries",
+  "resume_command": "stack-pr autoland --resume",
+  "plan": {
+    "total": 2,
+    "done": 0,
+    "remaining": 2,
+    "steps": [
+      {
+        "number": 1,
+        "type": "land",
+        "outcome": "failed",
+        "status": "Failed",
+        "is_next": true,
+        "pr_number": 101,
+        "pr_url": "https://github.com/o/r/pull/101",
+        "title": "Add the widget model",
+        "detail": "Checks failed after 1 retries: ci/test \u00b7 retries CI 1 / MQ 0"
+      },
+      {
+        "number": 2,
+        "type": "land",
+        "outcome": "pending",
+        "status": "Pending",
+        "is_next": false,
+        "pr_number": 102,
+        "pr_url": "https://github.com/o/r/pull/102",
+        "title": "Expose widgets over the API",
+        "detail": null
+      }
+    ]
+  },
+  "other_runs": []
+}

tests/golden/autoland/checks_fail.status.txt

@@ -0,0 +1,18 @@
+Autoland status
+
+  Status:     Stopped — not running, can be resumed
+  Branch:     feat
+  Base:       main
+  State file: <TMP>/home/.stack-pr/autoland/feat.json
+  Last saved: <SAVED>
+
+
+Autoland plan · 2 steps · 0 done · 2 remaining
+
+→ ❌ 1. Failed  | Land | #101 - Add the widget model
+        ↳ Checks failed after 1 retries: ci/test · retries CI 1 / MQ 0
+  ⬜ 2. Pending | Land | #102 - Expose widgets over the API
+
+ABORTED: PR #101 checks failed after retries
+
+Resume with: stack-pr autoland --resume

tests/golden/autoland/confirm_eof.checkpoint.json

@@ -0,0 +1,50 @@
+{
+  "version": 1,
+  "branch": "feat",
+  "base": "main",
+  "plan_file": "<TMP>/plan.autoland-plan",
+  "current_step": 1,
+  "last_landed_sha": "78e4ec1063c72debbb00f2ff966f72377789cdae",
+  "abort_reason": "Confirm step received EOF \u2014 cannot confirm in non-interactive mode",
+  "stack": [
+    {
+      "pr_url": "https://github.com/o/r/pull/101",
+      "pr_number": 101,
+      "branch": "TestBot/stack/101",
+      "title": "Add the widget model",
+      "review_decision": "APPROVED",
+      "state": "merged",
+      "check_retries": 0,
+      "queue_retries": 0,
+      "error_message": ""
+    },
+    {
+      "pr_url": "https://github.com/o/r/pull/102",
+      "pr_number": 102,
+      "branch": "TestBot/stack/102",
+      "title": "Expose widgets over the API",
+      "review_decision": "APPROVED",
+      "state": "pending",
+      "check_retries": 0,
+      "queue_retries": 0,
+      "error_message": ""
+    }
+  ],
+  "plan": [
+    {
+      "type": "land",
+      "entry_index": 0,
+      "pr_number": 101
+    },
+    {
+      "type": "confirm",
+      "condition": "QA sign-off complete",
+      "confirmed": false
+    },
+    {
+      "type": "land",
+      "entry_index": 1,
+      "pr_number": 102
+    }
+  ]
+}

tests/golden/autoland/confirm_eof.out.txt

@@ -0,0 +1,52 @@
+
+Discovering stack...
+
+
+Autoland plan · 3 steps · 0 done · 3 remaining
+
+→ ⬜ 1. Pending | Land    | #101 - Add the widget model
+  ⬜ 2. Pending | Confirm | QA sign-off complete
+  ⬜ 3. Pending | Land    | #102 - Expose widgets over the API
+State file: <TMP>/home/.stack-pr/autoland/feat.json
+
+
+============================================================
+Step 1/3: Landing PR #101 — Add the widget model
+============================================================
+PR #101 is mergeable (mergeStateStatus=CLEAN)
+
+Adding PR #101 to merge queue
+$ gh pr merge 101 --squash
+PR #101: in merge queue (0m) — polling in 120s
+
+PR #101 merged!
+
+Rebasing stack on origin/main...
+$ git fetch origin main
+$ git rebase origin/main
+Re-submitting stack...
+
+============================================================
+Step 2/3: Manual confirmation required
+============================================================
+
+Confirm "QA sign-off complete" is complete — ready to proceed?
+
+Next steps:
+  1. Land PR #102: Expose widgets over the API
+Type y/Y then Enter to continue (Ctrl+C to abort): 
+
+
+
+Autoland plan · 3 steps · 1 done · 2 remaining
+
+  ✅ 1. Merged                | Land    | #101 - Add the widget model
+→ ⏳ 2. Awaiting confirmation | Confirm | QA sign-off complete
+  ⬜ 3. Pending               | Land    | #102 - Expose widgets over the API
+
+ABORTED: Confirm step received EOF — cannot confirm in non-interactive mode
+
+Landing failed: Confirm step received EOF — cannot confirm in non-interactive mode
+
+State saved to <TMP>/home/.stack-pr/autoland/feat.json — resume with --resume
+

tests/golden/autoland/confirm_eof.plan.txt

@@ -0,0 +1,14 @@
+# Autoland plan — edit steps below.
+# l [<pr>]        = land that PR (a number, or its URL); a bare 'l'
+#                   lands the next PR in the stack instead
+# w <workflow>    = wait for a workflow to complete
+# c [<condition>] = pause for manual confirmation; the optional
+#                   condition names what to verify before proceeding
+#                   (e.g. 'c QA sign-off complete')
+#
+# Lines starting with # are comments and are ignored.
+# Blank lines are ignored.
+#
+l 101                         # Add the widget model
+c QA sign-off complete
+l 102                         # Expose widgets over the API

tests/golden/autoland/confirm_eof.status.json

@@ -0,0 +1,51 @@
+{
+  "status": "stopped",
+  "branch": "feat",
+  "base": "main",
+  "state_file": "<TMP>/home/.stack-pr/autoland/feat.json",
+  "state_file_exists": true,
+  "lock_file": "<TMP>/home/.stack-pr/autoland/feat.json.lock",
+  "pid": null,
+  "last_saved": "<SAVED>",
+  "abort_reason": "Confirm step received EOF \u2014 cannot confirm in non-interactive mode",
+  "resume_command": "stack-pr autoland --resume",
+  "plan": {
+    "total": 3,
+    "done": 1,
+    "remaining": 2,
+    "steps": [
+      {
+        "number": 1,
+        "type": "land",
+        "outcome": "done",
+        "status": "Merged",
+        "is_next": false,
+        "pr_number": 101,
+        "pr_url": "https://github.com/o/r/pull/101",
+        "title": "Add the widget model",
+        "detail": null
+      },
+      {
+        "number": 2,
+        "type": "confirm",
+        "outcome": "active",
+        "status": "Awaiting confirmation",
+        "is_next": true,
+        "condition": "QA sign-off complete",
+        "detail": null
+      },
+      {
+        "number": 3,
+        "type": "land",
+        "outcome": "pending",
+        "status": "Pending",
+        "is_next": false,
+        "pr_number": 102,
+        "pr_url": "https://github.com/o/r/pull/102",
+        "title": "Expose widgets over the API",
+        "detail": null
+      }
+    ]
+  },
+  "other_runs": []
+}

tests/golden/autoland/confirm_eof.status.txt

@@ -0,0 +1,18 @@
+Autoland status
+
+  Status:     Stopped — not running, can be resumed
+  Branch:     feat
+  Base:       main
+  State file: <TMP>/home/.stack-pr/autoland/feat.json
+  Last saved: <SAVED>
+
+
+Autoland plan · 3 steps · 1 done · 2 remaining
+
+  ✅ 1. Merged                | Land    | #101 - Add the widget model
+→ ⏳ 2. Awaiting confirmation | Confirm | QA sign-off complete
+  ⬜ 3. Pending               | Land    | #102 - Expose widgets over the API
+
+ABORTED: Confirm step received EOF — cannot confirm in non-interactive mode
+
+Resume with: stack-pr autoland --resume