Skip to content

fix: Reachable and Filter read every custom-role grant a key held on the listed type, and sought every role giving the permission however many there were - #151

Draft
Michael McQuade (giraffesyo) wants to merge 12 commits into
canaryfrom
fix/custom-arm-giving-roles
Draft

Michael McQuade (giraffesyo) wants to merge 12 commits into
canaryfrom
fix/custom-arm-giving-roles

Conversation

@giraffesyo

@giraffesyo Michael McQuade (giraffesyo) commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Reachable and Filter read every custom-role grant a key held on the listed type, whatever the role gave
    • The problem: past 64 grants of a key on the listed type, searches read the grants of built-in roles a role at a time but every grant of a custom role, by rbac_grants_custom_principal. Deciding which custom roles the members' and the public's grants name (broadArgs) read them all as well. Beside a tenant's custom viewer (cluster:view) granted to the members on 20,000 clusters, a member's Reachable for cluster:login read 8,927 pages of rbac_grants, and a host's query of Filter's condition read 17,922. Granted to the subject itself, they read 4,474 and 13,377. Authorize read 14.
    • The fix: the role-at-a-time read now reads a key's grants of custom roles on the type apart. It reads them by the scope, by rbac_grants_custom_principal as before, up to 64. Past that, where those 64 name 16 roles or fewer, it finds the custom roles the key holds: it steps from one to the next in rbac_grants_principal_role, one seek each, looks each up in rbac_roles by (tenant, id), and keeps those giving the permission on the type. It keeps them with the predicate givingSQL decides with, in the same statement, so a concurrent SetRole is seen by both alike. It then seeks the grants of each of those roles alone, or reads by the scope while the key holds fewer than 4 for each. It stops after 16 custom roles, and reads a key holding more by the scope, as before: each step costs about what reading a few grants does, whatever the role gives or wherever it is granted. Past 64, broadOn tests each giving custom role for a grant on the type with EXISTS, under the same bounds, instead of reading every custom grant. No migration is needed.
  • The role-at-a-time read sought every role giving the permission, however many there were
    • The problem: a key holding 64 grants on the type cost one index descent for each role giving the permission, for every key and type. Beside a catalog of 1,000 roles giving cluster:login, a subject in 20 groups each holding 100 grants read 60,775 pages, against about 700 reading by the scope.
    • The fix: a key's grants of built-in roles are read a role at a time only once it holds max(64, 4 × the built-in roles giving the permission) grants on the type (heldPerRole). A key is still read one way, once.

Pages of rbac_grants read, before → after (plan for the arguments / generic plan):

Case Before After
Custom viewer to the members, 20,000 clusters: Reachable 8,927 / 8,915 183 / 147
… a query of Filter's condition 17,922 / 17,883 446 / 347
Custom viewer to the subject, 20,000 clusters: Reachable 4,474 / 4,462 160 / 124
… a query of Filter's condition 13,377 / 13,338 423 / 324
1,000 more roles giving the permission: Reachable 60,775 / 60,755 733 / 673
… Filter 60,790 / 60,755 768 / 673
Stock catalog, same subject: Reachable 775 / 755 815 / 755
The members' grants of 300 custom roles giving nothing, one on each of 300 clusters: Reachable 219 / 209 239 / 239
… a query of Filter's condition 465 / 465 535 / 525

The first six rows and the stock catalog's are counted by the round 20 audits' tests, which leave the grants' partial indexes empty; the last two by this branch's tests' harness, which fills them (filled). Under a plan for the arguments, which is made for every call, part of the stock catalog's 40 more pages is the metapages of those indexes, read in planning (see Testing); under a generic plan it reads what canary does. Where a key holds 64 grants on the type of more than 16 custom roles, it reads its first 64 by the scope twice, about 10 to 15% more than canary, a cost that doesn't grow with the grants or the roles. An earlier version of this branch stepped through every custom role a key held, and read 1,912 / 1,902 pages there.

As of a time, searches read as before: by the scope, every grant of a key on the type, of any role. The docs now say so.

Testing

  • TestSearchesReadOnlyTheCustomGrantsOfRolesGivingThePermission counts the pages Reachable, Filter and a query of Filter's condition read beside 500 and 5,000 grants of a custom role giving nothing. It covers grants to the members, to the subject and to a group, under force_custom_plan and force_generic_plan; under the generic plan it counts the run after the one that builds the plan. Without the fix it fails, for example Reachable read 322 pages ... the members', and 2310 beside 5000. With only broadOn reverted, the members' case still fails (Reachable read 212 ... and 1208).
  • TestSearchesReadByTheScopeBesideManyRolesGivingThePermission covers 20 groups each holding 100 grants, with and without 200 more roles giving the permission. Without the fix it fails, for example Reachable read 780 pages ... and 12780 beside 200 more.
  • The round 19 randomized list harnesses (TestRvListsAgreeInRandomWorlds and TestAuditListsAgreeInRandomWorlds, 6 seeds each) were run on this branch and on canary. The branch ran at heldByScope 0, 1, 64 and the default, with heldPerRole at 0 and at the default, so both the role-at-a-time read and the two-step bound are exercised. canary ran at 0, 1, 64 and the default. On fffaeb5, all six branch runs (heldByScope/heldPerRole 0/0, 1/0, 0/default, 1/default, 64 and the default) and canary's 0, 1 and 64 were IDENTICAL to canary's default once grant UUIDs were masked. The audit harness passed at each of those settings on both heads. On 988f8d1, the default and heldByScope 1 with heldPerRole 0 were IDENTICAL to canary's default and heldByScope 1. On bec9812 and on 21c9082, which merges canary at a1f1583, both settings were IDENTICAL as well. They weren't run again once the custom grants were read apart; the randomized differential test below covers that head.
  • TestSearchesStepThroughFewOfTheCustomRolesAKeyHolds covers the members' and the subject's grants: beside 100 grants of a custom viewer on clusters, grants of 100 and of 1,000 roles on another type; and grants of 300 custom roles giving nothing, one on each of 300 clusters, against reading by the scope. Against this branch's earlier head 21c9082, which stepped through every custom role a key held, it fails: Reachable read 721 pages beside grants of 100 custom roles elsewhere that are the members', and 8212 beside 1000, and Reachable read 1912 pages beside grants of 300 custom roles that are the members', and 188 by the scope. Without the check of how many roles the first 64 grants name, it fails: Reachable read 341 pages ... and 198 by the scope. On canary only the members' query of Filter's condition fails (306 → 493). TestCallsReadByKeyUnderAPlanMadeWhileTheTablesWereSmall failed on a version that read the tenant's custom roles giving the permission by the tenant; this one reads them by key.
  • A randomized differential test built 80 worlds: 0 to 40 custom roles per tenant, with ids at the 128-character limit and with dots, hyphens and underscores, giving the permission or not, on the type or not, public or privileged or neither, and a grant of a deleted role. Each of the subject, its two groups, the members and the public held 0 to 130 grants, many at 63 to 81, of up to four roles, some expired, some expiring and some conditional. Each world's Reachable and query of Filter's condition for cluster:login and cluster:view were compared under heldByScope/heldPerRole 64/4, 0/0, 1/0, 1/4, 16/4, 64/0 and 3/1, which step through every custom role, through 16, 4, 3 or none, against reading by the scope alone, under both plan modes. All agreed on b6b2c57, and the results matched canary's and 21c9082's.
  • TestFilterCountsAFewOfItsSubjectsGrantsOfBuiltInRoles caught a first version of that scan, which read and sorted every grant the subject held through rbac_grants_key. It passes.
  • TestSearchesReadOnlyTheGrantsOfRolesGivingThePermission and the three new tests count each search on the run after the one that plans it, under the generic plan. They also keep autovacuum off rbac_grants and put one grant of another tenant in each of its partial indexes (filled).
    • Planning reads the metapage of every empty btree index for every reference to the table, because PostgreSQL keeps no height for an empty index. The test database leaves the custom-role, conditional, expiring and activation indexes empty, and this branch adds references to rbac_grants. So on bec9812, planning the broadArgs statement and Filter's count read 25 and 50 pages, against 5 and 30 on canary.
    • With those indexes holding a row, bec9812 and canary read 0 under a generic plan, and 3 and 68 under a plan for the arguments against canary's 2 and 66.
    • The test had counted whichever run planned. A plan rebuilt in the counted run, after an autovacuum or another session's schema change, read about 80 extra pages here against about 40 on canary.
    • The changed test still fails on fix: Reachable and Filter read every grant their subject, its groups, the members or the public held on the listed type, of any role, so grants of roles giving nothing there slowed them #146's parent: Filter read 169 pages beside 500 grants that are the members', and 1186 beside 5000.
    • A database with none of those kinds of grants pays those metapage reads, a buffer hit per empty index per reference to rbac_grants, on each plan, and this branch has more references than canary. Under a plan for the arguments, which is made for every call, the empty indexes cost each call, beside 300 custom roles: Reachable 33 to 43 hits (canary 2 to 7), Filter 55 to 57 (19 to 28), and a query of Filter's condition 85 to 99 (39 to 46). They don't grow with the grants or roles, and a generic plan, once made, pays none.
  • The package's list and search tests pass, and golangci-lint reports no issues.
  • make tidy-check lint vulncheck and GODEBUG=fips140=only go test -race -shuffle=on -cover -timeout 60m ./... against PostgreSQL 18 on 55432, on a026738 (merged with canary 9872532): RESULT.

Checklist

  • make check passes
  • No exported name, stored value or table was renamed (they are contract once released)

🤖 Generated with pw code

…s held on the listed type, and sought each role giving the permission however many did

Past heldByScope a key's grants on a type are read a role at a time for the
built-in and the tenant's custom roles giving the permission, and only where
the key holds heldPerRole grants for each of those roles, so a catalog of many
giving roles keeps the read by the scope. broadArgs tests the broad keys'
custom roles so too.
…m ones too, and by the scope beside many giving roles
… grown with the roles giving the permission, custom ones included
… a key holds, by key, rather than reading the tenant's custom roles

A key's custom roles are found by stepping through rbac_grants_principal_role
a role at a time and looked up in rbac_roles by key; the roles giving the
permission and the bound they set are read once for each key and type.
…ype to find which custom roles it held

Each step of the scan through a key's custom roles is a row comparison in
rbac_grants_principal_role's order, which no index leading to the scope after
the principal reads in order.
…he run after the one planning its statements
…ted, so planning a statement again reads none of them
…ose giving the permission, whatever they gave and wherever they were granted

A key's grants of custom roles on a type are read apart from its grants of
built-in roles: by the scope up to 64, and past that a role at a time only
where those 64 name 16 roles or fewer and the key holds 16 custom roles or
fewer, found by stepping through them by key; otherwise by the scope, as
before. The members' grants of 300 custom roles, one on each of 300 clusters,
cost a member's Reachable 1,900 pages, where by the scope it read 210.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant