[dwolfson]

Totals count the whole repository by default (push-down). Owner decision O-cap-1, 2026-11-08: "we should pushdown and get all totals not just visible ones (at least for now - we have some ideas on pushdown that may allow us to more efficiently do pushdown with zone sensitivity)". Every total that a report spec or analytic function shows now counts with population="repository" (push_down=True). On the live platform that takes an Asset count from a 30 s timeout to 0.9 s.

Branch feat/totals-repository-population off upstream/main 5ac4bb5 (#555). Two commits: 03b854d (repository default) and 4c64296 (read-back P1-1: relationship counts get a client that can answer them). No Egeria writes. No version bump.

Release note: totals now count the whole repository, including elements the viewer may not be able to see. This is a deliberate interim choice until zone-aware push-down exists; pass population="viewer" for a viewer-scoped count.

Rules followed

• Viewer stays available, but only on request. Every count helper still accepts population="viewer". scope=None still reaches the server default. Only the default changed.
• One population per payload; a ratio never mixes populations (the Portal's rule, overview_handler.py "Count scope"). A numerator from a find only sees what the viewer may see, so it is never divided by a repository total. Each routine took one of two paths:

  • Made wholly native, in the repository population:

    • term_definition_completeness (Glossary Definition Completeness)
    • context_readiness_funnel
    • business_value_signals
    • semantic_grounding

    "Has a description" is now count_elements_by_property(..., "description", "", operator="NEQ").

  • Kept wholly viewer-scoped, explicitly: these payloads come from scans that a native count cannot express (relationship-end distinct GUIDs, update times, certification end dates). Their native total now passes population="viewer".

    • certifications_summary
    • orphan_glossary_terms
    • stale_assets
    • contextualised_coverage
    • solution_component_realisation
• A failed repository count is None plus an unmeasured note, strict or not. The find fallback only sees what the viewer may see, so it is not used, and a 0 would read as an empty catalog. The non-strict 0 now survives only for a failed viewer count, which no dashboard or analytic default reaches.
• Each routine declares a client that can answer its own counts (read-back P1-1). A repository relationship count needs MetadataExpert's native count_relationships_between_elements. A ClassificationExplorer lacks it (checked live), and the Portal's supply builds exactly the declared class.
  • feedback_summary and count_relationships therefore declare egeria_client:EgeriaTech, as asset_composition already does. EgeriaTech has both get_relationships and the native count.
  • metric_trend can't declare EgeriaTech twice, because an input may be declared only once. So it binds a target's ce slot to its own EgeriaTech when the target declares EgeriaTech there.
  • The sweep found nothing else: semantic_grounding and the funnel count through mgr (a MetadataExpert), and certifications_summary is viewer-scoped.
• R4: semantic_grounding passes expert=mgr. count_relationships in the repository population also uses ce itself when it has the native method; the EgeriaTech that PlainSupply hands out does. A bare ClassificationExplorer gets None with the words "no native relationship count is available (it needs a MetadataExpert client)".

Changes

• overview_metrics:
  • These helpers now default to "repository": count_elements, count_elements_by_property (which gains operator="EQ"|"NEQ"), count_relationships, counts_by_type, people_counts, usage_context_counts, feedback_summary, business_value_signals, and growth_series (new population param).
  • Docstrings say what the repository population includes.
  • Notes for a property count name the condition, for example GlossaryTerm with description != ''.
  • The native-method cache is keyed by class name. It no longer hands a cached method name to a client that lacks it.
• The server default is forwarded explicitly. The keyword wrapper reads population=None as "not given". So a composite that forwarded scope=None used to get the repository default instead. It now forwards scope=None, through _pop(scope), in counts_by_type, people_counts, usage_context_counts, feedback_summary, business_value_signals and growth_series.
• Declarations (analytic_registry):
  • The population param defaults to repository.
  • Every measure of the routines above declares population="repository".
  • A new REPOSITORY_WORDS sentence is appended to each one's description: the totals include elements you may not be able to see.
  • Cost lines and not_measured sentences are updated.
  • Business Value and Term Completeness lose their "read"/"capped find" wording.
  • The MeasureDeclaration.population docstring now says it is the default population when the routine takes a population param.
  • The scan-backed routines keep population="viewer".
• Demo specs: six descriptions now say the counts are native over the whole repository. analytic_spec_params is unchanged; the defaults carry the population.

Live before/after (read-only)

Method:

• 2026-11-08 19:53–19:56Z.
• User resourceexplorernpa, default 30 s client timeout (what the tiles use).
• Before = fix(overview-metrics): a capped or timed-out count is not measured, never the cap #555 (5ac4bb5, the same tree as 43f6e38), loaded as a separate module. After = this branch.
• The catalog was growing during the run: Asset read 27,026 at 18:05Z and 27,402 at 19:55Z.

figure	before (#555)	after (this PR)
Asset count	33.8 s, not measured (native timed out; fallback "at least 500")	0.9 s, 27,395
Process count	31.4 s, not measured (same)	0.9 s, 24,840
GlossaryTerm count	7.2 s, 1,061	0.9 s, 1,061
Glossary Definition Completeness	2.7 s, total 500 (page cap, not the total), defined 177, undefined 65 %	2.1 s, total 1,061, defined 536, undefined 49 %
Semantic Grounding	39.5 s, links/pct not measured (Asset timed out; get_relationships capped at 500)	2.0 s, 767 links, 3 %
Business Value Signals (after only)	n/a	5.2 s: assets 27,400, confidential 11, described 12,194, duplicates 0 (was one capped 500-asset find)
AI Context Readiness funnel (after only)	n/a	2.9 s: cataloged 27,402, documented 12,194, classified 884, lineage 2,120
Growth series 7d, 3 points (after only)	n/a	4.1 s: assets 287 → 287 → 27,402. Push-down honours asOfTime (probe: GlossaryTerm 30 days ago = 200 in both populations)

Before this PR, the completeness tile's "total" was the 500-term page, so its 65 % described only that page. The ratio is now two native counts in one population.

One definitional shift: "defined" now means the description is set and not the empty string. A whitespace-only description counts as defined; live that is 9 of 1,061 terms. A full viewer scan gives 527 non-blank against 536 NEQ "". "Not blank" cannot be counted natively: Egeria's LIKE (regex) condition returned 0 for every pattern tried (.+, .*\S.*, (?s).*[a-zA-Z0-9].*), in both populations. That looks like an Egeria bug, worth its own issue.

Feedback Summary through a Portal-shaped supply (live, 20:13Z, read-only)

The supply builds exactly the declared client class, as the Portal's does.

version	client the routine receives	result
#555 (ClassificationExplorer, server-default get_relationships path)	ClassificationExplorer	0.8 s, total 61 (noteLogs 61)
03b854d (ClassificationExplorer, repository default)	ClassificationExplorer	total None for every type, "no native relationship count is available (it needs a MetadataExpert client)": the P1
this PR (4c64296)	EgeriaTech (the only class asked for)	0.7 s, total 61 (noteLogs 61), native, push-down
this PR, population=viewer	EgeriaTech	1.3 s, total 61

Tests

• pytest tests/micro-tests on Python 3.13.2: 2,295 collected, 2,292 passed, 3 skipped, 0 failed (junit count, after 4c64296; 03b854d alone was 2,284 / 3 / 0).
• New in 4c64296, test_portal_supply_relationship_counts.py:
  • feedback_summary and count_relationships run through a RoleSupply that hands out a client shaped like the class asked for. The test checks that the declared class (EgeriaTech) is the one used, with push-down.
  • A bare ClassificationExplorer cannot count the repository.
  • metric_trend of feedback_summary gives its EgeriaTech to the ce slot.
  • A sweep runs every demo spec through that supply, and none may come back "no native relationship count is available". I checked that the sweep fails if the old declaration is restored.
• Also new: test_scope_none_reaches_the_server_default_through_every_composite.
• New or rewritten tests:
  • Repository default. Push-down is True by default, scope=None omits it, and population="viewer" sets it False, for business_value_signals and growth_series. An unknown population raises.
  • Failed repository count. It is None with a "has no fallback" note, strict or not, and the find is never asked (test_a_timed_out_repository_count_is_not_measured_and_never_asks_the_find, test_a_failed_repository_count_is_none_never_zero).
  • Dashboard path. A non-strict tile is never a confident zero; the result is absent, with the reason. A viewer tile still says "at least 500".
  • Ratios.
    • Term completeness divides two repository counts, with no find call, and gives no percentage when the numerator is not measured.
    • The funnel makes four native push-down counts, and its documented is NEQ "" on Asset.
    • Business value is native and never capped (26,994 stays 26,994).
    • Semantic grounding counts relationships through mgr, with push-down.
  • Relationship counts. With no native counter, a count is None with its words. Feedback counts the repository through an EgeriaTech-like client. The count_elements_by_property NEQ body is checked, and a bad operator raises.
• Updated tests:
  • Tests that exercised the find fallback now pass population="viewer". That fallback only exists there now.
  • Composite tests now use native-count fakes instead of find pages.
  • test_rs_w2_3's fake EgeriaTech gains count_relationships_between_elements, as the real one has.
• ruff: clean on the changed code. The remaining findings in the touched files pre-exist.
• I did not re-run with egeria-trellis-analytics installed, and did not re-count the spec census separately. The suite's own census tests pass.

Consumers (not changed here)

• Portal overview_handler.py already binds scope="repository" explicitly through _scoped, and scope="viewer" for its background viewer Asset count, so its own counts don't change. These calls do change:
  • business_value_signals(mgr, as_of) now returns repository counts instead of a 500-capped find, so _reconcile_business_value sees a consistent assetTotal.
  • context_readiness_funnel: _strict_funnel_counts calls count_elements without a scope, and those calls now default to repository too. This matches the payload's countScope.
  • Still mixed in the Portal payload: orphan_glossary_terms and stale_assets stay viewer-scoped inside a payload labelled countScope: "repository". Their native totals are viewer Asset/GlossaryTerm counts, and the Asset one times out at 30 s. That mix predates this PR, but it is worth labelling in egeria-workspaces.
• Portal local-dashboards.html: the fix(overview-metrics): a capped or timed-out count is not measured, never the cap #555 finding stands. It needs an absent branch and unmeasured labels.

Follow-ups (not changed here)

• P2, out of scope here per the read-back: a population param for the routines that hard-code repository (semantic_grounding, context_readiness_funnel, term_definition_completeness).
• Egeria: the LIKE condition counts 0 for every regex tried. This blocks a native "not blank" count.
• Egeria: the viewer-population COUNT on a supertype takes about 50 s, and needs the zone-aware push-down the owner mentioned.
• The five scan-backed routines are still viewer-only and capped at a page: certifications_summary.active, stale_assets.staleCount, contextualised_coverage.contextualisedCount, orphan_glossary_terms.referencedCount, and solution_component_realisation. Moving them to repository needs native counts Egeria does not offer yet (distinct relationship ends, update-time filters).

🤖 Generated with Claude Code