Apply the modeling provenance cascade migration to TACC's Hasura

Question

Run the six-FK change from graphql_engine PR #14 against TACC's Postgres, so delete works at mint.tacc.utexas.edu.

Why this is a ticket and not part of the flip

#99 proved the user role cannot delete a problem statement, a task or a thread by any order of operations. The fix is a schema migration. It does not ride along with the deploy: charts/mint/templates/post-install-hasura.yaml carries "helm.sh/hook": post-install only, so a helm upgrade — which is what the flip is — never runs hasura migrate apply.

It does not block #77. The Lit UI's delete mutations are byte-identical, so delete is already broken at TACC and the flip is not a regression (decided 2026-08-09).

What to do

Prefer a surgical run_sql over hasura migrate apply. TACC's migration state is unverified, so migrate apply would drag in whatever else is pending, and hasura seeds apply sits in that same job.

Read TACC's current FK state first — confdeltype for the six constraints. The dev cluster was at 1771200016000, exactly the repo's tip; TACC's version is unknown.
Apply migrations/1771200017000_modeling_provenance_cascade_on_delete/up.sql.
Record the version in hdb_catalog.schema_migrations so a later migrate apply does not re-run it.
Confirm with one delete through the signed-in app. That also closes the one thing Deleting a problem statement, task or thread leaves the row behind and reports success #99 could not prove: the round trip through Hasura under a real Tapis token, rather than at the SQL layer.

Safe for the Lit UI either way: its deletes were already no-ops, so nothing about it changes.

Facts to reuse

The dev-cluster rehearsal, the exact SQL and the FK state before and after are on #99.

Bumped the TACC hasura image to the commit that carries the migration: In-For-Disaster-Analytics/dynamo#9.

components.hasura.image.tag goes 0d74aa1 -> c274ce0. That is one commit, mintproject/graphql_engine#14, which adds 1771200017000_modeling_provenance_cascade_on_delete.

This does not do the work in this ticket. The image only carries the migration files. The run_sql step and the hdb_catalog.schema_migrations row are still open.

It does make the image and the database agree after the manual step. Merge it first.

The schema change is applied on TACC production. One step is still open.

Verified

The six FKs now read confdeltype = c. All of them, in one reading.
No session is blocked.
TACC had no drift. hasura migrate status showed every version through 1771200016000 applied, and 1771200017000 as the only pending one. The worry in this ticket — that migrate apply would drag in what else is pending — did not apply.

Still open

Step 4, the delete through the signed-in app. Nothing here proves the round trip through Hasura under a Tapis token. Do not close this ticket until someone deletes a problem statement at mint.tacc.utexas.edu and sees it gone after a reload.

What happened

The first hasura migrate apply stalled at 0 / 1 for five minutes.

One session was idle in transaction for 100 days, holding locks from an abandoned COPY public.dataslice. ALTER TABLE ... DROP CONSTRAINT needs an ACCESS EXCLUSIVE lock on the parent tables, so it waited. A waiting request of that kind also blocks every later reader, so app traffic queued behind it.

pg_cancel_backend on a duplicate apply, then pg_terminate_backend on the 100-day session. The first apply took the lock and committed at once.

The idle session is now #119. It also stalled vacuum for 100 days.

Artifacts

Runbook: docs/runbook-116-cascade-migration.md — docs: runbook for the TACC cascade migration #118. Step 1c checks for long-lived transactions before the apply, because of this run.
Image bump: In-For-Disaster-Analytics/dynamo#9 moves components.hasura.image.tag to c274ce0. Merge it so the image and the database agree.

Step 4 is done. The delete round trip works through Hasura under a real Tapis token.

Result

Created a throwaway problem statement in the signed-in app, deleted it, and reloaded. It is gone. The mutation went out as the bottom-up document and the server removed real rows:

delete_thread                    affected_rows 1
delete_task                      affected_rows 1
delete_problem_statement_by_pk   { "id": "mpxnbljmmsnxmgk0" }

_by_pk returned a row, not null. That is the exact value #99 could never get.

The success also proves the migration. Those FKs were ON DELETE RESTRICT. Under RESTRICT this delete raises a foreign-key violation. It committed, so the six FKs are ON DELETE CASCADE in the running database, verified from the app rather than from confdeltype.

Verified as anonymous after the delete — the role that still sees an orphan:

problem_statement_by_pk("mpxnbljmmsnxmgk0")  null
task(problem_statement_id = ...)             []
thread(task.problem_statement_id = ...)      []

Token: sub mosorio@portals, tenant portals, tapis/client_id mint-localhost-3000. Role user.

Where it was run, and why that is enough

Run at localhost:3000 against TACC's production Hasura (https://graphql.mint.tacc.utexas.edu/v1/graphql) and production Postgres, not at mint.tacc.utexas.edu.

The literal test is not runnable today. mint.tacc.utexas.edu still serves Lit — #77 has not happened. Lit's delete_problem_statement deletes problem_statement_provenance and problem_statement_permission as root fields 5 and 6, before delete_problem_statement_by_pk. Hasura runs one mutation's root fields in a single transaction in document order, so the CREATE provenance that authorises the delete is gone before the row delete runs. A delete there fails for a reason that has nothing to do with the migration. It is a false negative, not a proof.

Origin does not enter this round trip: same endpoint, same database, same user role, same Tapis tenant. What localhost cannot prove is the deployment — TLS on the right host, nginx SPA fallback, the entrypoint writing AUTH_CLIENT_ID: mint. That is #77's job, and a constraint is landed there to re-confirm the delete at the real origin after the flip.

Negative control, unplanned but useful

The first two runs went to a stale dev server on port 3000 from another worktree (orca/workspaces/monorepo/08-09-18-44, started Aug 9), which predates #115. So they exercised the old client against the migrated database, and reproduced #99 exactly:

delete_thread_provenance              affected_rows 1
delete_task_provenance                affected_rows 1
delete_problem_statement_provenance   affected_rows 1
delete_thread                         affected_rows 0
delete_task                           affected_rows 0
delete_problem_statement_by_pk        null

HTTP 200, no errors key, and the row survived. The migration alone does not fix delete — the client fix is required, and Lit stays broken at TACC after the flip until it is unrouted. Worth stating plainly, because #116 could otherwise read as "migration applied, delete fixed".

delete_*_permission affected 0 rows in every run: no permission rows exist for these entities. The gate is provenance alone.

Still to run: two orphan rows at TACC production

The negative control left two orphans. Their CREATE provenance is deleted, and that provenance is the user delete permission — so the fixed client cannot remove them either. Same state as the #92 orphan. They need the admin secret.

Filed as #120 with the SQL.

problem_statement	task	thread
`dqu2ft1qmsnwzf7l`	`ck7ck3zwmsnwzfsi`	`l7gs1mozmsnwzg27`
`dkgbvxpzmsnx9u6o`	`1ier73rsmsnx9uy8`	`5wjaat9bmsnx9v7y`

One trap worth keeping

--strictPort when you start the dev server. Vite silently fell back to 3001 because another worktree held 3000, and the browser kept talking to day-old code that looked right. Two runs and a wrong diagnosis went by before the request body gave it away. Read the request body before believing a live verification.

Resolved. Step 4 verified: the delete round trip works through TACC's production Hasura under a real Tapis token, and the successful delete_problem_statement_by_pk proves the six FKs are CASCADE in the running database. Run at localhost:3000 because the host still serves Lit, where the literal test cannot pass for an unrelated reason. Re-confirm at the real origin after the flip — constraint landed on #77. Orphan cleanup in #120.

Navigation Menu

Repository navigation

Apply the modeling provenance cascade migration to TACC's Hasura #116

Question

Why this is a ticket and not part of the flip

What to do

Facts to reuse

Verified

Still open

What happened

Artifacts

Result

Where it was run, and why that is enough

Negative control, unplanned but useful

Still to run: two orphan rows at TACC production

One trap worth keeping

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Apply the modeling provenance cascade migration to TACC's Hasura #116

Description

Question

Why this is a ticket and not part of the flip

What to do

Facts to reuse

Activity

mosoriob commented on Aug 10, 2026

mosoriob commented on Aug 11, 2026

Verified

Still open

What happened

Artifacts

mosoriob commented on Aug 11, 2026

Result

Where it was run, and why that is enough

Negative control, unplanned but useful

Still to run: two orphan rows at TACC production

One trap worth keeping

mosoriob commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions