feat: release history via get_release_changes

[fedorov]

Closes #40.

Summary

The served release already carries the full history of earlier releases, but nothing on the tool surface pointed at it. This PR turns that into a first-class capability and fixes the discoverability gaps the issue diagnosed.

• New: GET /v3/releases/changes?version=N / MCP get_release_changes(version) — diff of release N vs N-1: series added / revised / removed (TB, patients affected), collections new / updated / removed, analysis results that gained series. Defaults to the served release; accepts 24 or "v24".
• How: every series version IDC ever served is either the current row in index (valid from series_revised_idc_version) or a row in prior_versions_index (valid min_idc_version–max_idc_version). Comparing which versions span N vs N-1 gives an exact diff, including removed series, which the issue's query can't see.
• Discoverability (recs 2–5): get_idc_version description points at release history; one-line INSTRUCTIONS touch + "Release history" section in idc://guide; whats_new MCP prompt; prior_versions_index table/column descriptions (fill-ins only where upstream is empty); notable_columns on list_tables entries.
• Not done (rec 6): per-release announcement URL/DOI — no such data in the index; belongs upstream in idc-index-data. The user guide links the IDC release notes page instead.

Validation

Checked against idc-index-data 25.0.4 and again on 25.0.5:

Release	Result
v24	Reproduces the issue's numbers (39,872 added, 5.663 TB, 15 new collections, 7 revised), plus 1,034 series removed
v20	590 revised; bucket-only moves are no longer counted as revisions
v5	apollo removed and apollo_5_lscc new (the collection move)

A cold call takes about 1–1.5 s, and results are memoized per version.

tests/test_releases.py covers:

• self-consistency of the served release;
• past releases: v20 revisions checked against an independent count, and the v5 collection move;
• 24 / "v24" accepted on core, REST and MCP;
• bad versions as clean errors (400 on REST);
• core == REST == MCP parity;
• the v1/v2 warnings and the memo;
• discoverability.

Full suite: 151 passed.

Also bumps idc-index 0.13.3 → 0.13.4 (idc-index-data 25.0.5), which ships the prior_versions_index descriptions from ImagingDataCommons/idc-index-data#189. This server therefore adds no descriptions of its own for that table.

Known limitations

• analysis_results cannot attribute series that are no longer served today, because prior_versions_index has no analysis_result_id.
• v1 and v2 are incomplete, and those responses carry a warnings entry. prior_versions_index has no rows starting at v1, because its build drops the original idc-open-idc bucket.

Docs: docs/user-guide.md (new Release history section, endpoint/tool tables), CHANGELOG.md under [Unreleased].

🤖 Generated with Claude Code

[fedorov]

Review: correct for the served release, wrong for past ones

For the served release (v24) the diff matches the issue's numbers. For earlier releases, two problems in the diff SQL cause wrong results. The tests only check the served release, which is the one case where neither problem can happen.

Fix before merge

1. Bucket moves reported as revisions (releases.py:38). prior_versions_index splits one content version into two rows when only the storage bucket changes. Of the 2,843 series reported as revised in v20, 2,253 have the same crdc_series_uuid in v19: they were moved from public-datasets-idc to idc-open-data, not changed.
2. Series that move between collections are misclassified (releases.py:32). At v5, apollo goes from 43 series to 0 and apollo_5_lscc from 0 to 43. The diff reports apollo_5_lscc as updated (43 revised) and doesn't list apollo at all.
3. MCP version: int rejects "v23" (server.py:264). That is the format the tool itself returns in previous_version. The PR description says "v24" is accepted, but the core's handling of the v prefix can't be reached from either adapter.
4. The new prior_versions_index docs describe one row per version (schema.py:155/172, _GUIDE, user guide). The data has 2,253 uuids with two rows each, so SQL written from these docs will repeat problem 1.

Also worth fixing: patients_affected isn't scoped to collection. The analysis_results section and the collections section can disagree. "Current version" is read from a different source than get_idc_version. The whats_new prompt normalizes the version in the adapter. isdigit() accepts characters like '²'. The query is recomputed on every call. There are also a few nits. Details are in the inline comments.

All numbers above were checked with DuckDB against idc-index-data 24.2.2.

---

_{🤖 Drafted with Claude Code (automated review, posted at my request).}

[fedorov]

Bucket moves are reported as content revisions. c.lo = N marks a series as revised whenever its current version row starts at N, but prior_versions_index also starts a new row when only the storage bucket changes. It contains 2,253 crdc_series_uuids with two contiguous rows each (e.g. 19–19 and 20–21).

At v20 this reports series_revised = 2,843. 2,253 of those have the same crdc_series_uuid in v19: they only moved from public-datasets-idc to idc-open-data (mostly cptac_ccrcc, cptac_ucec, cptac_pda and ccdi_mci). The real count is about 590, and those collections are wrongly shown as updated.

Fix: carry crdc_series_uuid in v and use WHEN c.crdc_series_uuid <> p.crdc_series_uuid THEN 'revised', or merge adjacent prior rows with the same uuid first.

[fedorov]

Series that move between collections get attributed to the new collection. COALESCE(c.collection_id, p.collection_id) assigns each joined row to its current collection, so the previous-release membership is credited to the new collection.

v5 example: apollo goes from 43 series to 0 and apollo_5_lscc from 0 to 43. This returns apollo_5_lscc as updated with 43 revised, leaves it out of new_collections, returns removed_collections=[], and has no row for apollo.

Fix: join on (SeriesInstanceUID, collection_id) (a move then counts as removed from one collection and added to the other), or compute collection presence for each release separately.

[fedorov]

count(DISTINCT PatientID) isn't scoped to collection. PatientIDs are only unique within a collection (80 PatientIDs in the current index appear in more than one), so the top-level total merges different patients and can be less than the sum of the per-collection counts. Suggest count(DISTINCT (collection_id, PatientID)).

[fedorov]

This section uses series_init_idc_version on current index rows, while the collections section uses version rows. So:

• is_new = min(init) = N ignores analysis-result series that were later removed, so a result can be reported as new in a release after the one where it actually first appeared.
• An analysis result whose series were only revised in N shows up as revisions in collections but is missing from analysis_results.

Deriving both sections from the same v/cur/prev sets would keep them consistent.

[fedorov]

"Current version" comes from max(version_metadata_index.idc_version) here, but DiscoveryService.version() (what get_idc_version returns) derives it from the idc-index-data version. If an idc-index-data release ever ships a version_metadata_index that is out of step with its own version, current_version here will differ from get_idc_version. It is also bound as hi for every index row, so the default call would describe the wrong release. Consider reusing the same helper.

[fedorov]

version: int | None makes pydantic reject "v23", which is the format this tool returns in idc_version / previous_version and the format get_idc_version returns. An agent that passes previous_version back in gets ToolError: Input should be a valid integer, and REST ?version=v23 returns 422. The core's v prefix handling (and its test) can't be reached from either adapter, even though the PR description says "v24" is accepted. Suggest int | str | None here and on the REST query param, and let _parse_version validate.

[fedorov]

This repeats the version normalization in the adapter with no validation (against the "adapters are thin" rule in CLAUDE.md), and pastes the raw text into the generated call. whats_new(version="latest") produces get_release_changes(version=latest). Suggest reusing the core _parse_version (made public) and falling back to the no-argument form on invalid input.

[fedorov]

The data contradicts these descriptions. prior_versions_index is not one row per version: 2,253 crdc_series_uuids have two contiguous rows that differ only by bucket. So "the next release revised or removed it" is wrong for those rows (see the comment on releases.py:38). The same claim is in the table description (line 155), _GUIDE, and docs/user-guide.md. An LLM that follows these docs in run_sql (e.g. max_idc_version = 19 AND SeriesInstanceUID IN index meaning "revised in v20") will count those 2,253 bucket moves as revisions.

Also consider adding crdc_series_uuid to NOTABLE_COLUMNS for this table, since it's the column that tells the two cases apart.

[fedorov]

Nit: status only ever holds 'new' | 'removed' | 'updated'. Literal["new", "removed", "updated"] would validate it and put the enum in /v3/openapi.json and the MCP output schema.

[fedorov]

These invariants only run against the served release, which is the one case where neither the split-row problem nor the collection-move problem can happen, so both pass the suite. Suggest adding past-release checks, e.g.:

• every series counted as revised in N has a different crdc_series_uuid in N-1 (catches the v20 bucket moves);
• release_changes(5) lists apollo_5_lscc in new_collections and apollo in removed_collections.

[fedorov]

Rebased onto main (idc-index-data 25.0.4) and addressed the review in 77fad01. All numbers below are from 25.0.4.

Fix before merge

1. Bucket moves: "revised" now means a different crdc_series_uuid. v20 goes from 11,928 revised (on 25.0.4 the bucket split affects 11,338 versions, more than the 2,253 seen on 24.2.2) to 590.
2. Collection moves: per collection, a moved series is removed from one collection and added to the other. The totals join on the series alone, so a move is not counted as an addition to IDC. v5 now lists apollo_5_lscc in new_collections and apollo in removed_collections.
3. "v23": REST (?version=v23) and MCP both accept it, and the core parser validates it.
4. prior_versions_index docs: rewritten. The description says the table is not a record of what is new, and that a row also ends when a version only moved bucket. The text matches docs: describe prior_versions_index in its schema sidecar idc-index-data#189 and applies here only while upstream is empty.

Also fixed

• 48 v23 versions have two overlapping rows. Each release is now collapsed to one row per series before joining, so they're no longer double-counted.
• patients_affected counts (collection_id, PatientID) pairs.
• analysis_results comes from the same per-release sets as collections, and now also reports revised and removed series.
• The current version comes from the idc-index-data version, the same source as get_idc_version.
• whats_new reuses the core parser and falls back to the served release on bad input.
• isascii() and isdigit().
• Results are memoized per version. A cold call takes about 1–1.5 s.
• The SQL binds 2 positional parameters, once, in a params CTE.
• Shared mb_to_tb() replaces the four copies.
• status is a Literal.
• Dropped the redundant dates.get guard.

New finding: prior_versions_index has no rows starting at v1, because its build drops the original idc-open-idc bucket. So the v1 and v2 diffs are incomplete. Those responses carry a warnings entry.

Tests now cover past releases: v20 revisions checked against an independent count, the v5 collection move, both adapters accepting "v23", the prompt fallback, the warnings and the memo. 151 passed.

---

_{🤖 Drafted with Claude Code, posted at my request.}

[Copilot]

🟡 Changes recommended

Version validation and historical-detail guidance need correction before approval.

2 open issues

• Handle oversized version strings as InvalidQueryError · New
• Clarify index limitations for historical release comparisons · New

What changed in this PR

Adds release-history summaries to the shared core, REST API, and MCP server, addressing #40’s discoverability gaps.

Changes:

• Adds release comparisons with series, collection, and analysis-result summaries.
• Improves discovery through tool descriptions, guides, notable columns, and a prompt.
• Adds regression and adapter-parity tests; updates pinned index data.

File	Description
uv.lock	Locks updated index dependencies.
tests/​test_releases.py	Tests release comparisons and discoverability.
src/​idc_api/​rest/​app.py	Exposes the release-changes endpoint.
src/​idc_api/​mcp/​server.py	Adds release tool, prompt, and guidance.
src/​idc_api/​core/​services/​releases.py	Computes and caches release comparisons.
src/​idc_api/​core/​services/​query.py	Returns notable columns.
src/​idc_api/​core/​services/​licenses.py	Reuses size conversion.
src/​idc_api/​core/​services/​discovery.py	Reuses size and release helpers.
src/​idc_api/​core/​services/​cohort.py	Reuses size conversion.
src/​idc_api/​core/​services/​_common.py	Defines shared helpers.
src/​idc_api/​core/​services/​__init__.py	Exports the release service.
src/​idc_api/​core/​schema.py	Curates notable columns.
src/​idc_api/​core/​models.py	Defines release-summary response models.
src/​idc_api/​core/​context.py	Initializes the release service.
pyproject.toml	Updates the exact index pin.
docs/​user-guide.md	Documents release-history access.
dev/​deployment.md	Updates the dependency-pin example.
CHANGELOG.md	Records user-visible changes.

_{Review effort: Balanced}

[fedorov]

Fixed in b6fc0ef: parse_idc_version rejects versions longer than 4 digits before calling int(), so an overlong string is a clean InvalidQueryError (400 on REST), and whats_new falls back to the served release. Regression cases cover "9" * 5000 in core, REST and the prompt. No real release comes close to that length, so the only impact was a 500 instead of a 400.

---

_{🤖 Drafted with Claude Code, posted at my request.}

[fedorov]

Agreed, fixed in b6fc0ef. The tool description, _NOTE, idc://guide and the user guide now limit the index shortcut to series still served. They say that removed series and earlier versions of revised ones are only in prior_versions_index, and explain how to compare two releases: build each from both tables (an index row is in release R when series_revised_idc_version <= R, a prior_versions_index row when min_idc_version <= R <= max_idc_version), then compare membership and crdc_series_uuid, never row ranges.

---

_{🤖 Drafted with Claude Code, posted at my request.}