Skip to content

fix(stovepipe): separate record dlq handler - #726

Merged
mnoah1 merged 8 commits into
mainfrom
mnoah1/stovepipe-promotion-dlq-reconciliation
Sep 22, 2026
Merged

mnoah1 merged 8 commits into
mainfrom
mnoah1/stovepipe-promotion-dlq-reconciliation

Conversation

@mnoah1

@mnoah1 mnoah1 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

During the record step, we were previously reusing the existing record handler as the dlq handler. Since record has outbound side effects (e.g. Promote), this can lead to 1000x attempts to external APIs even if the request will never succeed.

This PR adds a separate DLQ handler for record, focused on repair/recording an event for the abandoned record attempt, rather than repeated attempts that will likely continue to fail if the original retry budget is exhausted.

Intent:

  • Ensure record-stage work that exhausts the primary retry budget reaches an observable stopping point instead of retrying indefinitely on the DLQ.
  • Preserve whatever durable effects already succeeded without trying to reconstruct or complete the original workflow.
  • Keep fail-closed behavior when no validation fact was recorded. Persistent promotion permission failures are one example of this broader case.

Changes:

  • Add a standalone record DLQ controller that reloads the durable Request and appends an idempotent record_abandoned history event.
  • Keep the original failure in structured logs without adding an event-reason or arbitrary metadata contract.
  • Acknowledge malformed or unresolvable poison after logging and counting it; retry only Request reads and history persistence.
  • Do not create facts, advance bookmarks, call source control, or publish hooks from the record DLQ.
  • Rely on the queue-scoped storage contract instead of rechecking the loaded Request queue.

Summary:
Intent:
- Prevent permanent promotion failures from occupying the record DLQ indefinitely.
- Preserve successful validation state while making abandoned promotion visible in request history.

Changes:
- Add record-specific DLQ handling that does not repeat known promotion failures.
- Retain an idempotent promotion_failed history event before acknowledging abandoned promotion work.
- Keep non-promotion reconciliation under the existing DLQ retry policy.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@mnoah1 mnoah1 changed the title fix(stovepipe): bound record promotion retries fix(stovepipe): terminate failed record work Sep 18, 2026
@mnoah1 mnoah1 changed the title fix(stovepipe): terminate failed record work fix(stovepipe): terminate abandoned record work Sep 21, 2026
@mnoah1 mnoah1 changed the title fix(stovepipe): terminate abandoned record work fix(stovepipe): record dlq handler Sep 21, 2026
@mnoah1 mnoah1 changed the title fix(stovepipe): record dlq handler fix(stovepipe): separate record dlq handler Sep 21, 2026
@mnoah1
mnoah1 marked this pull request as ready for review September 21, 2026 14:43
@mnoah1
mnoah1 requested review from a team, behinddwalls and sbalabanov as code owners September 21, 2026 14:43
@mnoah1
mnoah1 enabled auto-merge September 21, 2026 14:43
Comment thread stovepipe/controller/record/dlq.go
@mnoah1
mnoah1 added this pull request to the merge queue Sep 22, 2026
Merged via the queue into main with commit 01a4dd2 Sep 22, 2026
16 checks passed
@mnoah1
mnoah1 deployed to stack-rebase September 22, 2026 19:22 — with GitHub Actions Active
@behinddwalls
behinddwalls deleted the mnoah1/stovepipe-promotion-dlq-reconciliation branch September 22, 2026 19:22

This branch was successfully deployed

1 active deployment
stack-rebase a73ce0df Deployed Sep 22, 2026 by mnoah1 via Rebase Stack #529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants