[dimalvovs]

This pull request restructures and generalizes the preprocessing pipeline, renaming and reorganizing modules to improve clarity, modularity, and support for both MultiQC and generic section-keyed JSON input. The main entrypoint is renamed, and the extraction, annotation, and I/O logic are now split into clear, single-responsibility modules under the new ingest package. Documentation and CI instructions are updated to reflect these changes, and the pipeline is now more robust to missing optional files and non-MultiQC data.

Major pipeline and API restructuring:

• The main entrypoint script is renamed from pipeline.py to llmize.py, and all references in documentation, Docker, and CI are updated accordingly. [1] [2] [3] [4]
• The preprocessing layer is refactored and renamed from json_reduction to ingest, splitting responsibilities into io_utils.py (I/O and path resolution), extract.py (input flattening/extraction), and annotate.py (section annotation and MultiQC-specific merging). The public API is re-exported from ingest/__init__.py. [1] [2] [3] [4]
• The annotation step (annotate()) now accepts any section-keyed JSON, with descriptor schema being optional; if missing, sections are passed through unannotated instead of failing. [1] [2]

Documentation and CI updates:

• All documentation, including the README and CI instructions, is updated to reference the new llmize.py entrypoint and ingest/ package. The documentation now clearly states that the pipeline supports both MultiQC and generic JSON, and describes the new module structure and conventions. [1] [2] [3] [4] [5]
• CI now runs unit tests in addition to syntax checks, and the instructions for running and validating the pipeline are updated. [1] [2]

Improved flexibility and error handling:

• The descriptor schema is now treated as optional: if it is missing, annotation falls back gracefully and emits a warning instead of failing.
• Extraction and annotation steps are opportunistic: MultiQC-specific logic only triggers when relevant section names are present, making the pipeline robust for generic JSON input as well. [1] [2] [3]

These changes make the pipeline easier to maintain, extend, and use with a wider range of input data.

[Copilot]

🟡 Changes recommended

Unresolved moderate issues remain in generic-section handling, glossary preservation, and Nextflow descriptor handling.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Refactors preprocessing into the modular ingest package, renames the entrypoint to llmize.py, and adds generic section-keyed JSON support.

Changes:

• Splits I/O, extraction, and annotation responsibilities.
• Updates Docker, Nextflow, documentation, and CI.
• Adds ingest tests and optional descriptor handling; removes glossary integration.

File summaries

File	Summary
tests/test_ingest.py	Adds extraction and annotation tests.
tests/__init__.py	Adds a test package marker.
README.md	Documents the new entrypoint, package layout, and generic JSON support.
nextflow.config	Updates the descriptor schema path. Moderate (1 vote): Missing descriptors still fail during staging.
main.nf	Uses llmize.py and updates input messaging. Moderate (1 vote): Descriptor handling remains mandatory before fallback can run.
llmize.py	Uses the new ingest API and optional descriptor handling. Nit (1 vote): Missing descriptors do not emit the documented warning.
json_reduction/cell_type_glossary.json	Removes the legacy glossary asset.
interpret.py	Removes glossary prompt context. Moderate (3 votes): Existing glossary definitions are lost.
ingest/io_utils.py	Adds path resolution and JSON I/O utilities.
ingest/extract.py	Adds MultiQC extraction and generic passthrough.
ingest/descriptor_schema.json	Adds the relocated descriptor schema. Nit (3 votes): Contains “once” instead of “ones.”
ingest/annotate.py	Adds annotation and merging logic. Moderate (1 vote): Broad substring matching can transform unrelated generic sections.
ingest/__init__.py	Re-exports the ingest public API.
Dockerfile	Updates the container entrypoint.
docker/entrypoint.sh	Runs llmize.py.
check_env.py	Treats the descriptor schema as optional.
.github/workflows/test.yml	Adds unit-test execution and updates compile checks.
.github/copilot-instructions.md	Updates development guidance. Nit (3 votes): References nonexistent ingest/main.py.

Review details

Suppressed comments (4)

ingest/annotate.py:100

• Because annotate() now accepts arbitrary section-keyed JSON, the existing substring classification ("spatial_neighbors" in section / "co_occurrence" in section) can capture unrelated generic sections such as my_spatial_neighbors_summary. Those sections are then renamed/reshaped (and co-occurrence payloads can be dropped), contradicting the promised generic passthrough; restrict the special cases to the documented MultiQC section names and numeric suffixes.
  llmize.py:173
• When the descriptor path is missing, this expression silently substitutes {}. The PR description says the optional-schema fallback emits a warning, but a normal llmize.py run gives no indication that all sections are being left unannotated; please log a warning on this fallback (the environment check does not cover ordinary runs).
  main.nf:40
• The CLI now treats a missing descriptor as valid, but this Nextflow process still requires and stages descriptor via channel.fromPath(..., checkIfExists: true) before llmize.py runs. A missing custom descriptor therefore fails the workflow instead of reaching the {} fallback described by this refactor; make the descriptor input/channel optional and add --descriptor only when it exists.

    python3 ${home}/llmize.py \\

nextflow.config:5

• The Python entry point now treats a missing descriptor as optional, but the Nextflow workflow still stages params.descriptor with checkIfExists: true, so a missing descriptor fails before llmize.py can apply this fallback. Make the Nextflow descriptor input conditional too, or scope the optional-file claim to direct invocation.

    descriptor   = "${projectDir}/ingest/descriptor_schema.json"

• Files reviewed: 16/18 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.