Apply the indicator standard-variable FK migration to TACC's Hasura

Question

Apply the #106 migration to TACC's Hasura, and confirm the Variables step saves an
indicator there.

The code is merged in two halves:

Schema: Point thread and task indicator columns at modelcatalog_standard_variable graphql_engine#15 — migration
1771200018000_response_variable_standard_variable_fk plus the metadata move of
four array relationships from variable to modelcatalog_standard_variable.
Client: Read the thread indicator as a standard variable (#106) #135.

Why a ticket of its own

helm upgrade does not run migrations — the migration job is a post-install hook
only (see #117). #116 established the same pattern for the cascade migration, and
docs/runbook-116-cascade-migration.md is the runbook to copy.

Order matters

The migration must land before the client. The client selects
response_variable { label }, which does not resolve while the relationship still
points at public.variable.

What to verify at TACC

Four constraints report confrelid = modelcatalog_standard_variable:
thread_response_variable_id_fkey, thread_driving_variable_id_fkey,
task_response_variable_id_fkey, task_driving_variable_id_fkey.
get_inconsistent_metadata returns is_consistent: true. Without the metadata
half it returns four inconsistent array relationships — measured on the dev
cluster.
Thread nzGQkhtRsudfSEHkDGct carries
https://w3id.org/okn/i/mint/TOTAL_WATER_STORAGE. It is the only row the
migration touches at TACC.
Signed in, pick an indicator in the Variables step and press Continue. It must
save. This is the defect Variables step cannot save any indicator: it writes a standard-variable URI into a column that references the legacy variable table #106 opened on, and it has only ever been proved fixed
under the admin role on the dev cluster.

Note on the dev cluster

The shared MicroK8s instance is already migrated — DB and metadata — from the
#106 session. It is ahead of graphql_engine main until mintproject/graphql_engine#15
merges. A metadata apply from main would revert the relationship move and leave it
inconsistent against the migrated database.

`scripts/deploy-hasura.sh` cannot land this migration

Tested read-only against the dev cluster. Two facts change how this ticket must be done.

1. Migrations are baked into the hasura image. The pod's /hasura/migrations stops at
1771200017000. hasura migrate apply only applies what the running image carries, so it
can never apply 1771200018000 until the image is rebuilt from a merged
graphql_engine main. Bump the hasura image tag first, or run the SQL by hand as #116 did.

2. hasura metadata apply reverts the relationship move. Measured with
hasura metadata diff in the pod, dev cluster, migration already applied:

server (live)                      image (what apply installs)
  modelcatalog_standard_variable     variable
    - tasksByDrivingVariable           - tasksByDrivingVariable
    - tasksByResponseVariable          - tasksByResponseVariable
    - threadsByDrivingVariable         - threadsByDrivingVariable
    - threadsByResponseVariable        - threadsByResponseVariable

An image built before mintproject/graphql_engine#15 puts the four back on variable, where
the constraints no longer are. That is the inconsistent state again. So the image must carry
both halves, or neither.

hasura migrate status in the pod reports 25 migrations, all Present / Present, so the
migration state itself is clean.

Two unrelated bugs in that script

Lines 85-99 give false assurance. An unchecked-out submodule is an empty directory,
pushd into it still succeeds, and git rev-parse then resolves against the parent repo.
It reported model-catalog-api (fix/106-indicator-standard-variable): all commits pushed,
which is the monorepo's branch. Guard on .git existing, not on the directory.
Line 75 prints kubectl: found (). kubectl version --client --short no longer exists.

The `user` role is now proved, on the dev cluster

The open question this ticket carried — the write was measured only under the admin
role — is answered for the dev cluster. It is still open for TACC.

Max signed in at http://localhost:3000 with the mint-localhost-3000 Tapis client, against
the dev cluster's Hasura through a port-forward. The dev webhook validates the same tenant as
TACC (portals.tapis.io), and it resolved the token:

[INFO] accountType user

He then set an indicator and an adjustable variable in the Variables step and pressed
Continue. The write landed:

thread nzGQkhtRsudfSEHkDGct
  response_variable_id  https://w3id.org/okn/i/mint/7a3b4c37-...  label 10hr_dead_moisture
  driving_variable_id   https://w3id.org/okn/i/mint/06100430-...

Both columns took a standard variable URI, the FK held, and the relationship resolved to a
label. This is the exact operation that failed with thread_response_variable_id_fkey when
#106 was opened.

The row is restored to .../TOTAL_WATER_STORAGE with no driving variable, which is where the
migration left it. The dev cluster is back to its prior state.

What is still this ticket's job: the same walk at TACC, after the migration is applied
there.

Done end to end on the dev cluster, and it found a third bug in the script

The dev cluster now runs the migration the supported way, not by hand.

helm list -n mint
mint  mint  66  deployed  MINT-9.0.0-beta.5

helm upgrade to 9.0.0-beta.5 (#107), then
./scripts/deploy-hasura.sh --skip-restart.

check	result
`1771200018000`	`Present / Present`
`get_inconsistent_metadata`	`is_consistent: true`
four FKs	all `-> modelcatalog_standard_variable`
thread `nzGQkhtRsudfSEHkDGct`	`.../TOTAL_WATER_STORAGE`, label `total_water_storage`

Two things worth carrying to TACC.

The migration is idempotent. The dev database already had the SQL applied by hand, so
hasura migrate apply re-ran up.sql over it. It succeeded: the DROP finds the constraint,
both data steps match no rows, and the ADD re-creates it. Metadata came from the image and
matched what had been applied by hand.

Rendering is minimal. helm template of 9.0.0-beta.2 vs 9.0.0-beta.5 against the dev
cluster's own values differs only in the hasura image tag, in both places that consume it,
plus helm.sh/chart labels. ui_react stays unrendered (enabled: false).

Third bug: `scripts/deploy-hasura.sh:150` picks a dying pod

POD=$(kubectl get pod -n "$NAMESPACE" -l app=mint-hasura -o jsonpath='{.items[0].metadata.name}')

items[0] is not filtered by phase. Run right after a rollout, it selected the Terminating
pod, which still carried the pre-upgrade image. migrate status then reported the old
migration list, and metadata apply failed against the already-migrated database:

error applying metadata: cannot continue due to inconsistent metadata
in table "variable": in relationship "threadsByDrivingVariable":
no foreign constraint exists on the given column(s)

Nothing was damaged — the apply is atomic and was refused. Re-running once the old pod had
gone worked first time. But --skip-restart exists precisely for the case where a rollout
just happened, which is exactly when this misfires. Add
--field-selector=status.phase==Running, or select on the ready condition.

This joins the two already noted above: the submodule check's false positive (lines 85-99)
and the dead kubectl version --short (line 75).

This is now a hard blocker, and it breaks more than the Variables step

Measured at TACC on 2026-08-29 while resolving #110, on develop (e25656f).

The whole thread page fails to load. /modeling/thread/:id renders only:

Failed to load thread: field 'label' not found in type: 'variable'

Not the Variables step — the page. GetThread selects driving_variable { id label } and
response_variable { id label } since #106 (04c5b67), and TACC's Hasura still resolves
both relationships to the legacy variable table:

thread.driving_variable   -> variable
thread.response_variable  -> variable

variable has name, not label, so Hasura rejects the document at validation and no
field on the page survives. It fails for a thread that carries no indicator at all —
jx8ibyilmsmg1pvt has response_variable_id: null — because validation is schema-level.

So #136 now blocks #77. Before this it was "the indicator cannot be saved at TACC";
now it is "no user can open a thread at TACC on the current client". Wired as a dependency.

To finish #110's verification I ran against a local client with those two selections
reverted to name — that isolates #110 from this, and is not a state anything ships in.

One consequence worth keeping: this is the first defect on the map where the client and
TACC's schema disagree in a way that takes the whole page down rather than one control.
The #106 client change and the graphql_engine migration are a matched pair, and develop
now carries only half of it against TACC.

The migration breaks the deployed Lit UI

Do not apply it before the flip. It stops the Lit problem statement list.

thread.response_variable is a foreign_key_constraint_on relationship. It follows the
constraint. After the migration it points to modelcatalog_standard_variable. That table has
id, label and description. It has no name.

Lit selects response_variable { name } in
ui/src/queries/fragments/thread_previews.graphql. That fragment is part of
problem_statement_info. The problem statement list, the get query and both subscriptions use
it. Hasura refuses the document at validation. The pages fail. They do not lose one field.

The commit TACC runs today is 888ec50. It has that selection.

Measured live, as anonymous, in both directions:

endpoint	state	`{ name }`	`{ label }`
`graphql.mint.tacc.utexas.edu`	not migrated	works	`validation-failed`
`graphql.mint.local` (dev)	migrated	`validation-failed`	works

The two apps are a matched pair with the schema, in opposite directions. TACC cannot serve both.

This corrects the Order matters section. The migration must not simply come before the
client. It must come with the flip, or Lit must stop reading that field first.

Cost to Lit

One cosmetic item. screens/modeling/actions.ts:361 is the only consumer. It fills
problem["preview"], the variable chips on a problem statement card. At TACC 1 of 165 threads
carries a response variable. The chip is empty on every other card.

Three sequences

A. Migrate, then flip at once. No Lit change. The Lit list is dead between the steps. It
stays dead if the flip is rolled back. Confirm SUBSIDE can live without the un-ported Lit screens #69 made unrouted Lit the only undo for the flip. This
breaks the undo when it is needed.
B. Patch Lit to read label, deploy, then migrate. The same coupling, reversed. The
patched Lit is broken until the migration lands.
C. Make Lit agnostic first. Delete the response_variable { name } selection. Lit then
works against both schemas. The migration can land at any time. The flip keeps a working undo.
Cost: one line, one Lit image, one ui tag bump in dynamo, one helm upgrade.

C is the recommendation. It makes two safe steps out of one timed step. Max decides.

Prepared

dynamo#12 bumps the hasura
image to 06492bd. The image carries the migration and the metadata. A helm upgrade runs
neither. ghcr.io/mintproject/graphql-engine:06492bd… is published and pullable.
#155 adds
docs/runbook-136-indicator-fk-migration.md. It copies the Apply the modeling provenance cascade migration to TACC's Hasura #116 runbook and corrects four
things: the metadata apply is required, the image must carry both halves, the lock footprint
is wider, and the rollback is lossy. It also records the three scripts/deploy-hasura.sh
bugs.

Still open

The apply at TACC needs MFA. The step 6 walk needs a signed-in browser. Both need Max.

Sequence C, decided by Max on 2026-09-12. Step 1 is built.

Make Lit migration-agnostic first, then the migration and the flip become two independent
steps and #77 keeps a working undo.

mint-ui-lit#662 — thread_previews
selects the thread id instead of response_variable { name }.

Verified live and anonymously, both directions, which is what makes this safe to merge
before the migration:

Fragment	`graphql.mint.tacc.utexas.edu` (not migrated)	`graphql.mint.local` (migrated)
the PR	data	data
`main` today	data	`validation-failed`

It really is one line. A sweep of the repository confirms thread_previews.graphql is the
only place Lit traverses the relationship — thread-info, task-info and both update
mutations use the raw response_variable_id column, which the migration does not change.
screens/modeling/actions.ts:361 already guards on th.response_variable && th.response_variable.name, so it needs no change; problem["preview"] is simply empty. There
is no codegen in that repo.

The hasura image: #12 is superseded

dynamo #16 pins
global.imageTag to the monorepo develop head
9d58ec77a08151ffa6eacc6aa50648a9128718a0. That image carries both halves of this change:
the migration 1771200018000 and the metadata with the four array relationships already on
modelcatalog_standard_variable. graphql_engine now lives in the monorepo, so the separate
06492bd build that dynamo#12
bumped to is no longer the right source. Close #12 in favour of #16.

New constraint on the runbook: five later migrations must not ride along

The runbook was written when 1771200018000 was the newest migration. It is not any more.
develop carries five more that TACC has never applied:

1771200020000_modelcatalog_etl_process
1771200021000_standard_variable_embeddings
1771200022000_modflow_2005_hydraulic_head_output
1771200023000_modflow_2005_barton_springs_workflow_repair
1771200024000_modflow_2005_tapis_component_urls

So two steps in docs/runbook-136-indicator-fk-migration.md need reading with care.

Step 1a says to stop if anything before 1771200018000 is Not Present. Expect the
five after it to read Present / Not Present. That is correct and is not a reason to stop.
Step 2 already uses hasura migrate apply --version 1771200018000 --type up. Keep that
form. A bare hasura migrate apply would now apply all six.

Order of operations

Merge and deploy mint-ui-lit#662 —
build, then bump ui.image in dynamo, then helm upgrade. Lit is then safe against both
schemas.
Merge dynamo#16 and
helm upgrade. This rolls the hasura pod onto the image that carries the migration.
Do not run hasura metadata apply yet — the new metadata puts the four array
relationships on modelcatalog_standard_variable while the constraints still point at
variable, which is the inconsistent state.
Run the runbook: migration 1771200018000 by version, then metadata apply, then verify.
The flip, #77.

Steps 1 to 3 all need ssh and MFA, so they need Max.

Applied at TACC on 2026-09-12. All four checkpoints pass.

Check	Result
Four constraints point at `modelcatalog_standard_variable`	pass
`metadata is consistent`	pass
Thread `nzGQkhtRsudfSEHkDGct` carries `TOTAL_WATER_STORAGE`	pass
Variables step saves an indicator, signed in	pass

The runbook alone did not work

The database had to move first. Migration 1771200021000_standard_variable_embeddings
runs CREATE EXTENSION vector. TACC ran PostgreSQL 10 on postgis/postgis:10-3.2-alpine,
which carries no pgvector, so every hasura migrate apply failed:

could not open extension control file ".../vector.control"

TACC now runs ghcr.io/mintproject/postgres-pgvector:sha-9d58ec7: PostgreSQL 16.9,
PostGIS 3.5, pgvector 0.8.6. PostgreSQL 16 cannot read PostgreSQL 10 data files, so the
window was dump, empty the data directory, upgrade, restore. All 31 migrations are now
applied and the metadata applies. 668 standard variables.

Chart: #117, version 9.0.0-beta.13. Values: the
private deployment repo.

Facts later tickets depend on

Two rollback copies exist. Do not delete them yet.
~/tacc-values/tacc-hasura-pg10.dump on the login node, md5
0238080b3e7ed462eabaf1f9e4eee163, 97 tables. /mnt/hasura_db.pg10 inside the PVC
data-mint-hasura-db-0, 156 MB. The old data directory was renamed, not deleted, so a
rollback is a rename back. Delete both after a few days of normal use.
kubectl scale fails at TACC. The account holds patch on deployments.apps but
not on the deployments/scale subresource, and the bound role mint-role does not
exist. Use kubectl patch deploy <name> -p '{"spec":{"replicas":0}}'.
helm upgrade resets every replica to 1. Services restart mid-window. Stop the
writers again after the upgrade and before the restore.
Hasura recreates hdb_catalog in an empty database within seconds, and pg_restore
then collides with it. Drop that schema before the restore.
The Hasura CLI needs no install on the login node. Run it as a one-off pod from the
graphql-engine image: cd /hasura && hasura migrate apply.
mint-semantic-search runs at TACC for the first time. It writes 384-dimension
vectors. It has never run at production scale.

Evidence for #117

helm upgrade did not apply the migrations. A one-off pod did.

Navigation Menu

Repository navigation

Apply the indicator standard-variable FK migration to TACC's Hasura #136

Question

Why a ticket of its own

Order matters

What to verify at TACC

Note on the dev cluster

`scripts/deploy-hasura.sh` cannot land this migration

Two unrelated bugs in that script

The `user` role is now proved, on the dev cluster

Done end to end on the dev cluster, and it found a third bug in the script

Third bug: `scripts/deploy-hasura.sh:150` picks a dying pod

This is now a hard blocker, and it breaks more than the Variables step

The migration breaks the deployed Lit UI

Cost to Lit

Three sequences

Prepared

Still open

Sequence C, decided by Max on 2026-09-12. Step 1 is built.

The hasura image: #12 is superseded

New constraint on the runbook: five later migrations must not ride along

Order of operations

The runbook alone did not work

Facts later tickets depend on

Evidence for #117

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Apply the indicator standard-variable FK migration to TACC's Hasura #136

Description

Question

Why a ticket of its own

Order matters

What to verify at TACC

Note on the dev cluster

Activity

mosoriob commented on Aug 30, 2026

scripts/deploy-hasura.sh cannot land this migration

Two unrelated bugs in that script

mosoriob commented on Aug 30, 2026

The user role is now proved, on the dev cluster

mosoriob commented on Aug 30, 2026

Done end to end on the dev cluster, and it found a third bug in the script

Third bug: scripts/deploy-hasura.sh:150 picks a dying pod

mosoriob commented on Aug 30, 2026

This is now a hard blocker, and it breaks more than the Variables step

mosoriob commented on Aug 30, 2026

The migration breaks the deployed Lit UI

Cost to Lit

Three sequences

Prepared

Still open

mosoriob commented on Sep 12, 2026

Sequence C, decided by Max on 2026-09-12. Step 1 is built.

The hasura image: #12 is superseded

New constraint on the runbook: five later migrations must not ride along

Order of operations

mosoriob commented on Sep 12, 2026

The runbook alone did not work

Facts later tickets depend on

Evidence for #117

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`scripts/deploy-hasura.sh` cannot land this migration

The `user` role is now proved, on the dev cluster

Third bug: `scripts/deploy-hasura.sh:150` picks a dying pod