Skip to content

feat(lapis): api to compute the count for every lineage including sub lineages - #1931

Open
fengelniederhammer wants to merge 17 commits into
mainfrom
1876-api-to-compute-the-count-for-every-lineage-including-sub-lineages
Open

fengelniederhammer wants to merge 17 commits into
mainfrom
1876-api-to-compute-the-count-for-every-lineage-including-sub-lineages

Conversation

@fengelniederhammer

@fengelniederhammer fengelniederhammer commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

resolves #1876

Summary

Adds sublineage-inclusive aggregation on /sample/aggregated: append * to a lineage field in fields (e.g. fields=country,pangoLineage*) to get a count for every lineage including all of its sublineages. Counts do not partition — a sequence is counted into each of its ancestor lineages.

Example: GET /sample/aggregated?fields=pangoLineage* → rows like { "pangoLineage*": "B.1.1*", "count": 1234 } (values carry the trailing *).

  • <field>* syntax — only on the aggregated endpoint, only for fields with a lineage index; matched case-insensitively. Each matched sequence rolls up into its full ancestor chain, resolving aliases and alias-of-alias (e.g. BQ.1.1 → BE.1 → BA.5.3.1 → B.1.1.529 → B) and stopping the climb at recombinant edges.
  • Reject plain + star on the same field — fields=pangoLineage,pangoLineage* returns 400. Combinations with other fields (country,pangoLineage*) are allowed.

Requires a RhyDB build with concat and transitiveClosure

PR Checklist

  • All necessary documentation has been adapted.
  • All necessary changes are explained in the llms.txt.
  • The implemented feature is covered by an appropriate test.

@vercel

vercel Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
lapis Ready Ready Preview Oct 30, 2026 9:24am UTC

Request Review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved aggregation issues misclassify leaf recombinant samples into the null bucket and cause ambiguous counts or rejected valid requests when temporary column names collide with metadata fields.

3 open issues
What changed in this PR

Adds sublineage-inclusive counts to LAPIS’s existing /sample/aggregated endpoint, addressing #1876 through a trailing * on lineage field names.

Changes:

  • Adds lineage-field parsing, ordering support, and validation against requesting both plain and starred forms.
  • Introduces composable SaneQL query builders and ancestor aggregation with alias handling.
  • Adds unit and end-to-end coverage, API documentation, and RhyDB compatibility information.
File Description
README.md Adds a RhyDB compatibility entry.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​rhydb/​SaneQlAstTest.kt Updates receiver-based pipeline tests.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​rhydb/​RhyDbQueryToSaneQlTest.kt Tests lineage aggregation query rendering.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​rhydb/​RhyDbQueryDslTest.kt Tests the new query builders.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​request/​converter/​LineageFieldConverterTest.kt Tests starred lineage parsing and validation.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​request/​converter/​AggregatedFieldConverterTest.kt Tests lineage-field conversion integration.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​request/​AggregatedFiltersRequestTest.kt Tests rejection of conflicting field forms.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​model/​RhyDbQueryModelTest.kt Tests forwarding lineage fields to query actions.
lapis/​src/​test/​kotlin/​org/​genspectrum/​lapis/​controller/​LapisControllerTest.kt Tests GET and POST handling and validation.
lapis/​src/​main/​resources/​templates/​llms.txt Documents sublineage-inclusive aggregation.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​rhydb/​SaneQlAst.kt Supports composable pipeline receivers and keywords.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​rhydb/​RhyDbQueryDsl.kt Adds fluent SaneQL expression and pipeline builders.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​rhydb/​RhyDbQuery.kt Integrates query builders and lineage action fields.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​rhydb/​AggregatedActionSteps.kt Builds ancestor closure joins and rolled-up counts.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​request/​Field.kt Adds the starred lineage field representation.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​request/​converter/​OrderByFieldsCleaner.kt Canonicalizes starred lineage ordering fields.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​request/​converter/​LineageFieldConverter.kt Parses starred fields with lineage indexes.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​request/​converter/​AggregatedFieldConverter.kt Enables starred fields in aggregation requests.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​request/​AggregatedFiltersRequest.kt Rejects plain and starred forms of the same field.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​openApi/​OpenApiDocs.kt Documents starred fields, responses, and ordering.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​model/​RhyDbQueryModel.kt Passes lineage fields into aggregation actions.
lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​controller/​ControllerDescriptions.kt Explains inclusive counts and overlapping totals.
lapis-e2e/​test/​aggregatedQueries/​groupByPangoLineageIncludingSublineagesWithDateFilter.json Adds date-filtered lineage aggregation expectations.
lapis-e2e/​test/​aggregatedQueries/​groupByPangoLineageIncludingSublineages.json Adds lineage-filtered aggregation expectations.
lapis-e2e/​test/​aggregatedQueries/​groupByCountryAndPangoLineageIncludingSublineages.json Adds combined division and lineage grouping expectations.
lapis-e2e/​test/​aggregated.spec.ts Checks count consistency with lineage filters and aliases.
lapis-docs/​src/​content/​docs/​references/​introduction.mdx Introduces starred lineage aggregation syntax.
lapis-docs/​src/​content/​docs/​concepts/​lineage-queries.mdx Adds examples and explains overlapping counts.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Filtering incoming edges drops recombinant leaf lineages from the closure, incorrectly placing samples assigned to them or their aliases in the null bucket rather than their own lineage bucket.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Preserve recombinant leaf lineages in closure buckets

lapis/​src/​main/​kotlin/​org/​genspectrum/​lapis/​rhydb/​AggregatedActionSteps.kt:120

When a recombinant lineage has no non-recombinant children, filtering its incoming edges removes it entirely from the closure. Samples assigned to it or its aliases then contribute to the null bucket instead of its own <lineage>* bucket. Union self-edges from the unfiltered lineage table before taking the closure, and update the rendering expectations and regression coverage for a recombinant leaf.

3 resolved since last review

Review effort: Balanced

This branch was successfully deployed

1 active deployment
Preview — 0ca71beb Deployed Oct 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API to compute the count for every lineage including sub-lineages

2 participants