MetaPathways 3.5.2: Nextflow local/Slurm workflows, PGDB builds and explorer

[RyloByte]

Problem and resulting behavior

MetaPathways 3.5.2 moves workflow orchestration and reference construction to Nextflow while retaining the existing commands and sample outputs. The complete analysis_wf command runs multiple assemblies, optional reads and genome maps through annotation, abundance, community/MAG PGDB inference and combined reporting on a local server or Slurm.

This PR targets dev and should remain draft while the local/HPC benchmarks and independent review are pending. Production promotion will be a separate dev → master PR. No release tag or registry publication is part of this PR.

Changes to review

• Nextflow task dependencies, threaded-tool budgets, concurrent samples/PGDBs, bounded Slurm submission, modular workflow generation for large task graphs, durable checkpoints and retained logs/resource traces.
• Automatic input discovery and explicit sample-ID manifests; optional per-sample compact results retain report inputs while removing large intermediates and workflow caches.
• Correct paired-read handling and exact CoverM BAM selection; consistent feature coordinates/counts and database-specific taxonomy/LCA without mixing annotation sources.
• build_pt creates and registers a private Apptainer image from a licensed installer, applies official vendor patches and validates BLAST support. Optional matching MetaCyc reference preparation includes indexes and mapping tables. PGDB staging supplies sequences and coordinates to community and MAG builds; optional MAG failures retain explicit outcomes.
• SQLite-backed reports and the sample-first explorer connect annotation, taxonomy, genome membership, abundance and pathways, with filtering and CSV export.
• Bundled three-sample CAMI II reviewer inputs, pinned MAGSplitter/Camelot installation, package/container release validation, and rebuilt Read the Docs guides with workflow diagrams and tool/data citations.

Validation

• Runtime changes reviewed/tested at a320bb4f6bf81f62a1a560d609b49ceaab3f7c7d; subsequent PR-preparation changes are documentation only. Record the final tested PR commit before sign-off.
• 129 runtime/regression tests passed: 128 passed in the initial run; the report HTTP test was blocked by sandbox socket permissions and passed when rerun with loopback access. No analysis benchmark was launched by these checks.
• Fresh release-control suite: 17 tests passed. These are local tests, not a registry publication or final package-build certification.
• Warning-as-error Sphinx build passed after the PR-preparation changes; links and anchors checked in 37 HTML pages, plus source/CLI documentation checks.
• All three detailed workflow Mermaid diagrams were rendered and visually reviewed in a browser.
• The committed three-sample reviewer audit records 51/51 successful tasks, valid SQLite integrity/foreign keys, nine genome bins and reconciliation of annotation/abundance relationships. That run used --skip_ptools; it is historical evidence, not a fresh licensed-PGDB or final-commit test.

Remaining merge gates

• Review the completed full local benchmark: required tasks, optional MAG outcomes, PGDB exports, report joins and measured resources.
• Review the completed HPC benchmark: Slurm resource requests/submission limits, compact cleanup, offline compute-node behavior and complete results.
• Independent tester sign-off on the final candidate, including single/two-sample workflows, unchanged rerun checkpoints, explorer filters and CSV export.
• Review licensed Pathway Tools community/MAG outcomes separately from public reviewer tests.
• Green PR smoke/documentation CI for the final candidate.
• Run the nonpublishing Release workflow on the candidate with blank release_tag and source_run_id; retain package artifacts, optionally Docker/SIF validation and security reports, and add run links here.

Expected no-pathway or optional MAG outcomes must remain distinguishable from task failures. The final installed Conda/container artifacts require their own checks; older candidate builds do not validate this exact revision. Benchmark results are pending and are not claimed as passed.

Follow the tester checklist and release process. Keep licensed installers, private images, MetaCyc data, credentials and full benchmark outputs out of the PR.

[read-the-docs-community]

Documentation build overview

📚 HallamLab-MetaPathways | 🛠️ Build #34977879 | 📁 Comparing 05aaa6b against latest (9d6990a)

  🔍 Preview build  

23 files changed · + 2 added · ± 21 modified

+ Added

• test-bundle.html
• test.html

± Modified

• benchmarking.html
• cami-references.html
• cli-reference.html
• commands.html
• containers.html
• detailed-workflow.html
• documentation.html
• execution.html
• getting-started.html
• inputs.html
• and 11 more...