Skip to content

feat(roboinvestor): ingest a report from a company that is not on the platform - #2085

Merged
jfrench9 merged 7 commits into
mainfrom
feature/report-import
Nov 19, 2026
Merged

jfrench9 merged 7 commits into
mainfrom
feature/report-import

Conversation

@jfrench9

@jfrench9 jfrench9 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

A fund can now load a report from a company that keeps its books elsewhere. create-issuer adds the company as an entity, and ingest-report loads its report from a holon or from rows read out of a PDF or a workbook. Rows are assembled into a holon first, so every report in the graph was loaded from a serialization that is stored and can be downloaded. The report is checked before anything is written, and one that does not check is refused with its reasons.

Changes

Importer (shared kernel)

  • operations/roboledger/commands/report_import.py (new): import_report reads a holon with xbrlkit, runs xbrlkit's report check, stores the file whole under the report's bundle key, and writes an uploaded report with its fact sets and facts on a named entity. Each fact keeps the source the file gave it, on asserted provenance.
  • Concepts and sections go through the existing reporting-extension command rather than being written here: a library concept is the library's own (pulled in when the tenant's curated copy left it out), and a concept the library does not have joins one "Received Reports" extension per graph, marked as received rather than as a chart account.
  • A second import for the same entity, period end and kind replaces the first. A dry_run checks and reports what would load without writing.
  • The stored files follow the transaction: a replaced report's file is deleted when the replacement commits, and a report whose transaction is undone takes its own file with it. The entity is locked for the import.
  • A concept the graph already holds in another vocabulary (one a share brought, or a firm's own extension) is linked from the report's section. A line presented twice under one parent is stored as one association.
  • commands/reports.py: deleting an uploaded report removes the sections it brought.

RoboInvestor operations (each also an MCP tool)

  • operations/roboinvestor/commands/reports.py (new), routers/extensions/roboinvestor/operations.py, models/api/extensions/investor.py: create-issuer and ingest-report, with their request and response models. A duplicate identifier is 409; a refused report is 422 naming each reason; an unknown entity is 404.
  • middleware/mcp/tools/annotations.py: hints for the two new write tools.

Reads

  • operations/roboledger/reads/reports.py: the reports list shows the fund what it received about a counterparty, accepts a counterparty's id to list that company's reports, and gives an uploaded report only the sections it brought.

Demo

  • examples/roboinvestor_demo: two new steps. A holding whose issuer keeps its books elsewhere gets an issuer and its monthly package from rows (a mistyped copy is refused, a dry run writes nothing). Then the holon of the report the share delivered is downloaded and ingested as a file, and the run asserts it lands the same facts and values as the shared copy.

Docs

  • docs/technical/RoboInvestor-Operations.md: a section on receiving a report from a company that is not on the platform, and the operation count.

Breaking Changes

None.

  • Additive API: two operations (create-issuer, ingest-report) and their models, generated tier, so a client minor on the next regen.
  • Behavior change in a read: the fund's default reports list now includes reports received about a counterparty entity, and reports(entityId:) accepts a counterparty's id (it returned an entity-not-found error before). Lists that held only authored and shared reports are unchanged.
  • MCP: an investor graph gains two write tools.

Testing

  • just test-all, exit 0, at the first four commits: 19,376 passed, 44 skipped, 2 xfailed; the code-quality stages and the dbt tests passed. The later commits were covered by the affected suites (2,678 passed after the review fixes) and by CI.
  • just demo-roboinvestor on the local stack: 45/45 checks, including the route check (62 facts by the share, 62 by the file, the same total).
  • 20 new database-backed tests on a provisioned tenant (test_report_import_db.py, test_report_ingest_db.py): a package lands with its values, sources, sections and calculation arcs; a curated-out library concept is pulled rather than duplicated; replace; delete; dry run; refusals for a total that does not foot, a missing fact, an unreadable file, an unplaceable row, an unknown entity and a duplicate issuer; listing for the fund and for the company.
  • On the local stack, against an investor graph carrying roboinvestor alone: created an issuer, dry-ran and ingested two months from rows, confirmed a package that does not foot returns 422 with the numbers, added a security and a position, rebuilt the graph, and ran Portfolio → Position → Security → Entity → Report → Fact to the ingested facts. Listed the reports over GraphQL both ways, and downloaded the stored holon through reportDownloadUrl, read it back and re-ran the check on it.
  • The JSON example in the doc was run as written against the local API as a dry run.
  • Not covered: a holon produced by a third party (xbrlkit-built and ledger-published holons are both exercised); a report with dimensional facts (they are counted and left in the stored file, by design, with no test on a real one); reportPackage renders only the statement sections of an uploaded report, not a custom table.

Certification

  • I have the right to submit this work under the Apache 2.0 license, and do so. Where any part of it is owned by my employer, I have their permission.

🤖 Generated with Claude Code

A holon is read and checked with xbrlkit, stored whole under the report's
bundle key, and projected into the tenant as an uploaded report on a named
entity. Its concepts and sections go through the reporting-extension command
(the library's own concept where it has one, pulled in when the tenant's copy
left it out); this writes the report, its fact sets and its facts, each fact
keeping its source. A second import for the same entity, period end and kind
replaces the first, and deleting an uploaded report takes its sections with
it. A file that cannot be read or does not check is refused with its reasons.
Two operations on the RoboInvestor surface, each also an MCP tool.
create-issuer adds a company that keeps its books elsewhere as an entity
securities and reports attach to. ingest-report loads a report onto an entity
from a holon, or from rows (concepts, sections and values with the page or
cell each came from) that the server assembles into a holon first, so every
report in the graph was loaded from a stored serialization. dry_run checks
and says what would load without writing.
…ompany

A concept a report brings is a received concept, not one of the graph's
accounts, and keeps its qname in the graph. The reports list shows the fund
what it received about a counterparty, takes a counterparty to list that
company's reports, and shows an uploaded report only the sections it brought.
@github-actions github-actions Bot added the change:normal Normal change: automated review documented on the pull request before merge label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Change classification: normal — platform or application code changed.

A normal change carries a documented automated review on this pull request before merge, in place of a second human approver. The change-classification job requests that review and passes once it has been posted. Recorded by the change-classification job.

@jfrench9

Copy link
Copy Markdown
Member Author

@claude please review this PR

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @jfrench9's task in 39s —— View job


PR review: ingest a report from a company that is not on the platform

I read the importer, the RoboInvestor commands, the router registration, the read changes and the API models. I did not run the tests. Your PR description reports just test-all green with 14 new DB-backed tests.

The design holds up. Rows are assembled into a holon, which is checked before anything is written and then stored. Vocabulary goes through the existing reporting_extension command. The error mapping (409 / 422 / 404) is sensible. I found one real defect and a few smaller issues.

Should fix

1. Replacing a report orphans its stored holon in S3.

  • ImportedReport.replaced_report_ids is documented as "Reports this one replaced, whose stored artifacts the caller deletes after commit" (report_import.py:94-96).
  • The ingest-report operation never does that. ingest_report in operations/roboinvestor/commands/reports.py:156 only copies the ids into the response, and nothing calls delete_report_artifacts for them.
  • The other delete paths do it. routers/extensions/roboledger/operations/reports.py:250, middleware/mcp/tools/report_tools.py:195 and distribution.py:670 call it after commit.
  • Each re-ingest of a month therefore leaves a full holon (up to 25 MB) under the old report's bundle prefix. The rows are gone, so nothing references the file.
  • Deleting an uploaded report through delete_report has the same gap. I saw no S3 cleanup in commands/reports.py:738-748. The existing routers do it after commit.
  • Suggestion: have the ingest-report handler call delete_report_artifacts(graph_id, result.replaced_report_ids) after commit. Check whether OperationSpec has a post-commit hook for this. If not, a small wrapper in the router works.

Worth addressing

2. The holon is uploaded to S3 before the DB commit.

  • _store runs at the end of import_report (report_import.py:252), and the caller commits afterwards.
  • If the commit fails, the S3 object is orphaned. Nothing deletes it.
  • The inverse case is acceptable: a crash before the upload leaves no row.
  • This is small, but the same cleanup helper could cover it.

2a. Dry run is not entirely free of side effects, and the check comes after the parse.

  • holon has no max_length on IngestReportRequest. The 25 MB cap in _read runs only after FastAPI has parsed the whole body.
  • Consider a max_length on the field, or a body-size limit, so the cap applies before parsing.

3. _element_ids looks up by qname alone, across all taxonomies in the tenant schema (report_import.py:626-634).

  • If two taxonomies hold the same qname, the dict keeps an arbitrary row and the fact binds to that element.
  • Candidates are a tenant's curated copy versus the library, and a firm's own extension prefix.
  • Scoping the lookup to the library, the received-reports extension and the tenant's own chart, or at least checking for ambiguity, would make the binding deterministic.

4. A numeric concept with a null numeric_value is silently stored as an empty string.

  • _fact_row falls through to the Nonnumeric branch with string_value="" (report_import.py:665-678).
  • It's probably unreachable after check_report, but a refusal would be safer than a blank non-numeric fact on a numeric concept.

5. Concurrent ingests for the same entity, period end and kind can both succeed.

  • _delete_previous is select-then-delete with no lock and no unique constraint.
  • Two racing requests could leave two reports for one period.
  • The same applies to create_issuer: its duplicate check is also select-then-insert.
  • A SELECT ... FOR UPDATE on the entity row, or an advisory lock keyed on the entity, would fix both.
  • create_issuer also accepts an issuer with no identifier at all, so duplicates by name are unprevented. That may be intended.

Nits

  • own_structures evaluates set(brought) inside the comprehension, once per structure (reads/reports.py, own_structures). Hoist it out of the comprehension.
  • _holon_from_rows falls back to a placeholder entity identity when the entity is missing, so that the importer can raise the 404. It works, but it builds a report with fake identity first. Raising ReportImportEntityNotFoundError directly would be clearer.
  • _refusal joins every problem into one string. For rows with many problems this makes a large 422 detail. Returning the list in a structured field would be easier for MCP clients to act on.
  • The PR notes that dimensional facts are counted but not loaded, and that no test uses a real one. That's documented and surfaced in the response, which is fine. A synthetic dimensional fact in the DB tests would still pin the count.

Checks

  • The counterparty read clause (Report.origin != "authored" on linked entities) is consistent, and origin is NOT NULL with a check constraint, so there are no null-origin surprises.
  • CLAUDE.md rules look followed: no os.getenv, config via env, logic in the operations layer, no hand-written migrations (there are none).
  • I couldn't tell from the diff whether the ingest-report operation commits through execute_operation. I assumed it does, based on the other registrars.

Verdict: the data model and checks are solid. I'd merge after fixing #1. The rest can follow.
· branch feature/report-import

A replaced report's file is deleted once the replacement commits, and a
report whose transaction is undone takes its own file with it, on every
surface. The entity is locked for the import so two for one company cannot
both find nothing to replace. A nil value is not loaded, the holon field has a
size limit, and rows for an entity outside the graph are refused before
anything is built.
@jfrench9

Copy link
Copy Markdown
Member Author

Review follow-up, pushed:

  • Item 1 (fixed). A replaced report's stored file is deleted once the replacement commits. It is tied to the session's commit rather than to a router hook, so it holds on REST and MCP alike. The delete path was already covered: the existing delete-report handlers remove artifacts after commit.
  • Item 2 (fixed). The same mechanism removes the new file when the transaction is undone, so a failed commit leaves no orphan. A database-backed test covers both outcomes.
  • Item 2a. holon has a max_length.
  • Item 3 (no change). elements.qname carries a unique index per tenant (idx_elements_qname), so a qname resolves to one row.
  • Item 4 (fixed). A nil value is not loaded; a numeric fact with no number is already refused by the check.
  • Item 5 (fixed for ingest). The entity row is locked for the import. create-issuer is unchanged: an issuer may have no identifier, and two with the same name are allowed.
  • Nits. The set is hoisted, and rows for an entity outside the graph are refused before a report is built. The 422 detail stays one string for now, as every operation's is.

Affected suites rerun: 2,678 passed.

Two things a real ledger-authored report carries that a built one does not.
A concept the graph already holds in another vocabulary (one a share brought,
or a firm's own extension) is linked from the report's section rather than
refused by the extension command, which resolves only its own vocabulary and
the library. And a line presented twice under one parent is one association.
…latform

Two steps. A holding whose issuer keeps its books elsewhere gets an issuer
entity and its monthly package ingested from rows, after a mistyped copy is
refused with the numbers and a dry run writes nothing. Then the holon of the
report the share delivered is downloaded and ingested as a file, and the run
asserts it lands the same facts and values as the shared copy. A refusal now
writes its amounts out in full.
@jfrench9
jfrench9 merged commit 3dbc053 into main Nov 19, 2026
10 checks passed
@jfrench9
jfrench9 deleted the feature/report-import branch November 19, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

change:normal Normal change: automated review documented on the pull request before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant