Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
57e8839
Add filter-paralogs subcommand and MADC count overflow warning
alex-sandercock Nov 2, 2026
84e1f95
Move filter-paralogs to its own module; read HapApp fixed-ID MADCs
alex-sandercock Nov 2, 2026
cd2d7cb
Add filter-paralogs --clone-positions for other assemblies
alex-sandercock Nov 2, 2026
17a44e4
updated tile documentation
alex-sandercock Nov 2, 2026
d31c594
Initial plan
Copilot Nov 2, 2026
d2fa3ef
Run Python 3.9 tests on Ubuntu 24.04
Copilot Nov 2, 2026
112df67
Fix filter-paralogs placement and scoring defects found in review
alex-sandercock Nov 2, 2026
4f2298e
Add genotype-calling engine adapted from polyRAD
alex-sandercock Nov 12, 2026
5f0242c
Add 'madcats call': genotype VCFs from a MADC
alex-sandercock Nov 12, 2026
75f911d
Add 'call --overdispersion auto': maximum-likelihood estimate
alex-sandercock Nov 12, 2026
e1ad5eb
call: skip readCountSum column; record real-data comparison with polyRAD
alex-sandercock Nov 12, 2026
7d991ce
call: estimate the overdispersion by default
alex-sandercock Nov 12, 2026
6fa798a
call: --selfing-rate for the Hardy-Weinberg prior
alex-sandercock Nov 12, 2026
265b098
call: wrap two over-long lines
alex-sandercock Nov 12, 2026
c6b3a4b
Record the comparison with updog's norm model
alex-sandercock Nov 12, 2026
a9e8f94
call: --prior norm, updog's normal-shaped genotype distribution
alex-sandercock Nov 12, 2026
22cd16d
call: allelic bias, background-read rate, faster two-allele records
alex-sandercock Nov 12, 2026
1bd6887
call: refine every promising starting bias; add a free-weights prior
alex-sandercock Nov 12, 2026
745ca48
call: estimate the read model with the free-weights prior
alex-sandercock Nov 12, 2026
2aea31b
Record the finished call model: bias, background rate, benchmark
alex-sandercock Nov 13, 2026
d6b4c9d
Development record: overdispersion recovery with the finished estimate
alex-sandercock Nov 13, 2026
5cad8bb
Development record: joint estimate re-run with the finished estimator
alex-sandercock Nov 13, 2026
4fde575
Development record: separate the estimate from the calling in the tim…
alex-sandercock Nov 13, 2026
cac710f
Development record: potato DArTag against SNP-array genotypes
alex-sandercock Nov 13, 2026
8974ec1
call: --prior structure, a prior for subpopulations, clines and families
alex-sandercock Nov 13, 2026
565ba72
Development record: what the structure prior was not tested on
alex-sandercock Nov 13, 2026
8654ed2
Development record: family priors were measured and not built
alex-sandercock Nov 13, 2026
5d9d2fc
Development record: priors on haplotype records; a simulated F1 with …
alex-sandercock Nov 13, 2026
277cdc5
Development record: three figures corrected against the run output
alex-sandercock Nov 13, 2026
0f1e6ec
call: stop repeating work that cannot change between passes
alex-sandercock Nov 13, 2026
5d7c419
call: one iteration loop for every prior
alex-sandercock Nov 13, 2026
0dd27ab
call: remove unused parameters and repeated helpers
alex-sandercock Nov 13, 2026
2d41169
call: names that say what the thing is
alex-sandercock Nov 13, 2026
36a2b5d
call: name the source of every adopted or adapted part
alex-sandercock Nov 13, 2026
feb0a90
call: the norm prior's fit no longer stalls at a start with no curvature
alex-sandercock Nov 13, 2026
a832601
call tests: one helper each for the model line and for an expected error
alex-sandercock Nov 13, 2026
6e706e4
Development record: source audit, off-target accuracy, timings after …
alex-sandercock Nov 13, 2026
a5dcf8a
Development record: three ways to shorten the estimate with a fitted …
alex-sandercock Nov 13, 2026
e93972b
Development record: call on two public data sets with truth
alex-sandercock Nov 13, 2026
8c66b81
filter-paralogs: three defects found in review
alex-sandercock Nov 13, 2026
a80b188
filter-paralogs: the table filter in parts, and less repeated work
alex-sandercock Nov 13, 2026
cf237e1
filter-paralogs: names, comments and sources
alex-sandercock Nov 13, 2026
593eaba
filter-paralogs tests: planted copies, thread counts, and the unteste…
alex-sandercock Nov 13, 2026
136db5e
Development record: filter-paralogs reviewed, and checked with known …
alex-sandercock Nov 13, 2026
96593d0
Development record: the smelt comparison on the same genotypes for ev…
alex-sandercock Nov 13, 2026
8b84640
call docs: allopolyploids are outside the model
alex-sandercock Nov 13, 2026
6b2ef78
Development record: the whole potato deposit, and off-target substitu…
alex-sandercock Nov 13, 2026
30597ed
Development record: a vendor MADC with diploid and tetraploid samples…
alex-sandercock Nov 13, 2026
3d1b4f6
Development record: filter-paralogs on alfalfa, hexaploid sweetpotato…
alex-sandercock Nov 13, 2026
672818f
filter-paralogs: a haplotype-resolved reference, measured on alfalfa
alex-sandercock Nov 13, 2026
f71c930
Development record: polyploids pooled from real diploid reads; the va…
alex-sandercock Nov 13, 2026
0357f60
Development record: hexaploid microhaplotype tables
alex-sandercock Nov 13, 2026
76d7ac1
Development record: the pooled table with every grouping that finished
alex-sandercock Nov 13, 2026
cd275ae
Development record: other seeds and mixed ploidy for the simulated ta…
alex-sandercock Nov 13, 2026
d6800e8
filter-paralogs on the alfalfa panel's design genome
alex-sandercock Nov 13, 2026
b5d8964
filter-paralogs page: one figure corrected
alex-sandercock Nov 13, 2026
4512600
call: --allele-bias auto keeps a fitted bias only where it gains a ma…
alex-sandercock Nov 13, 2026
84c64c4
call: rewrap one paragraph of the module's opening comment
alex-sandercock Nov 13, 2026
fcfa80c
call: INFO/BIAS is always the fitted bias; a record called without it…
alex-sandercock Nov 13, 2026
53b59ad
Merge branch 'generate-locus-group-ties' into madcats-genotype-calling
alex-sandercock Nov 13, 2026
66d8ca7
Merge branch 'copilot/fix-github-actions-job' into madcats-genotype-c…
alex-sandercock Nov 13, 2026
4d1b261
Tests: tiled markers through generate, filter-paralogs and call
alex-sandercock Nov 13, 2026
e016cdb
Version 0.14.0
alex-sandercock Nov 13, 2026
443c574
Development record: which thresholds on a call VCF remove the worst m…
alex-sandercock Nov 13, 2026
02ccd0c
Help text of call and filter-paralogs: option groups, shorter descrip…
alex-sandercock Nov 13, 2026
a421388
README and docs entry pages: list call and filter-paralogs
alex-sandercock Nov 14, 2026
91a3181
call page: which settings to use, and no bias for inbred lines
alex-sandercock Nov 14, 2026
f490f64
Development record: --allele-bias auto across population types and on…
alex-sandercock Nov 14, 2026
9c5f538
call --allele-bias auto: reject a bias at a search limit, and call re…
alex-sandercock Nov 14, 2026
bf86167
Docs: a genotype-calling tutorial and a shorter README introduction
alex-sandercock Nov 14, 2026
6527d12
Docs: call page says what to use and how; its measurements move to th…
alex-sandercock Nov 14, 2026
ea1eab8
Tutorial: a starting recommendation for call, and the same model for …
alex-sandercock Nov 14, 2026
0b22b25
Docs: accuracy across replicate draws and calibration for the bias se…
alex-sandercock Nov 14, 2026
b0534f1
Validation page: the norm prior against hwe across replicate draws
alex-sandercock Nov 14, 2026
a2944fa
Development record: replicates and calibration of off, fit and auto
alex-sandercock Nov 14, 2026
e089ee1
README: restore the introduction and Highlights
alex-sandercock Nov 14, 2026
ce37766
call: write MPP and PMC, updog's maxpostprob and prop_mis
alex-sandercock Nov 15, 2026
a39ca4e
Docs: MPP and PMC in call's VCFs
alex-sandercock Nov 15, 2026
90bd0da
Development record: MPP and PMC against updog and known genotypes
alex-sandercock Nov 15, 2026
78388c0
Docs: style edits in progress
alex-sandercock Nov 15, 2026
75652e7
Docs: plainer wording on the documentation site and README
alex-sandercock Nov 15, 2026
a403213
Development record: plainer wording
alex-sandercock Nov 15, 2026
e474442
Development record: the suite has 257 tests; ignore minimap2 indexes
alex-sandercock Nov 15, 2026
908e372
CI and installation: build mappy for one architecture on macOS
alex-sandercock Nov 15, 2026
db886c7
filter-paralogs on lettuce: the kept rows, and blastn beside bwa mem
alex-sandercock Nov 15, 2026
ce886d2
Review: selfing rate of 1 at odd ploidy, and missing genotypes in the…
alex-sandercock Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,12 @@ jobs:
matrix:
os: [ubuntu-latest]
python: ["3.9", "3.10", "3.11", "3.12", "3.13"]
exclude:
- os: ubuntu-latest
python: "3.9"
include:
- os: ubuntu-24.04
python: "3.9"
- os: macos-latest
python: "3.13"
runs-on: ${{ matrix.os }}
Expand All @@ -25,6 +30,13 @@ jobs:
with:
python-version: ${{ matrix.python }}

- name: Install the optional genome aligner (filter-paralogs tests)
run: |
# mappy has no wheel for this macOS Python, and its source does not compile as the
# universal (arm64 + x86_64) binary that setup-python's interpreter asks for.
if [ "$RUNNER_OS" = "macOS" ]; then export ARCHFLAGS="-arch $(uname -m)"; fi
python -m pip install mappy

- name: Run tests
run: python -m unittest discover -s tests -v

Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -185,3 +185,6 @@ docs/book/
# macOS
.DS_Store
examples/madcats-demo/results/

# minimap2 indexes built by filter-paralogs
*.mmi
66 changes: 40 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,19 @@

`madcats` is a command-line tool that turns targeted-sequencing FASTQs into MADC microhaplotype
count tables. It counts the alleles a panel was designed for and finds additional ones in the reads.
The output is a raw MADC table, which goes through HapApp before dosage calling and VCF conversion.
The output is a raw MADC table, which goes through HapApp before dosage calling and VCF conversion,
or straight to `madcats call` for genotypes and VCFs.

The same code handles DArTag, AgriSeq (Ion Torrent), GT-seq, rhAmpSeq and Flex-Seq capture data.
It needs only the Python 3 standard library. Every analysis threshold is a command-line option, and
each run writes a JSON file recording its parameters and read counts.

> **A prior MADC is not an input.** Generation uses only the FASTQs plus the assay design: the
> **A prior MADC is not an input.** Generation uses only the FASTQs plus the assay design, either the
> DArTag primer/probe files, a probe design alone, or a platform-neutral Ref/Alt tag manifest.

## Highlights

- Closely matches vendor output: read counts correlate at r = 0.9997 with the vendor MADC on a
- Closely matches vendor output. Read counts correlate at r = 0.9997 with the vendor MADC on a
sweetpotato DArTag panel, and dosage calls agree 95–99% across five DArTag panels.
- Designed to be platform independent, with testing performed on DArTag, GT-Seq, rhAmpSeq, Flex-Seq, and AgriSeq.
- Builds the panel from vendor design files (`madcats design`), from the probe design and the reads
Expand All @@ -34,8 +35,13 @@ each run writes a JSON file recording its parameters and read counts.
paired-end reads (`madcats merge-pairs`).
- Writes raw MADC tables for HapApp, which prepares them for BIGr and other tools. Each row also
lists its variants and marker type.
- Pure Python with no dependencies. Runs samples in parallel and caches per-sample results, so
reruns are fast.
- Calls diploid and polyploid genotypes from the MADC (`madcats call`) and writes VCFs of the
designed targets, of all tag variants, or of whole microhaplotypes.
- Optional genome-based paralog filter (`madcats filter-paralogs`, needs
`pip install 'madcats[genome]'`) for panels where co-amplified paralogous copies inflate
heterozygosity. It works on vendor and HapApp fixed-allele-ID MADCs too.
- Pure Python with no dependencies for everything else. Runs samples in parallel and caches
per-sample results, so reruns are fast.

## Installation

Expand All @@ -53,11 +59,11 @@ To keep it isolated from other Python packages, use [pipx](https://pipx.pypa.io/

### Pre-built Binaries

Standalone executables that need no Python installation are attached to each
[release](https://github.com/Breeding-Insight/MADCATS/releases):
- `madcats-linux-x86_64.tar.gz`: Linux
- `madcats-macos-x86_64.tar.gz`: macOS (Intel)
- `madcats-macos-aarch64.tar.gz`: macOS (Apple Silicon)
Each [release](https://github.com/Breeding-Insight/MADCATS/releases) includes standalone
executables that need no Python installation:
- `madcats-linux-x86_64.tar.gz` for Linux
- `madcats-macos-x86_64.tar.gz` for macOS (Intel)
- `madcats-macos-aarch64.tar.gz` for macOS (Apple Silicon)

```bash
# Download and extract (example: Linux)
Expand Down Expand Up @@ -103,6 +109,12 @@ madcats validate \
--generated results/project_MADC.csv \
--expected vendor_MADC.csv \
--report results/validation.json

# 4) Optional: call genotypes and write a VCF
madcats call \
--madc results/project_MADC.csv \
--ploidy 4 \
--output-prefix results/project
```

No primer file? Infer the panel from the probe design and the reads:
Expand All @@ -114,7 +126,7 @@ madcats infer-design \
--output reference_tags.csv
```

Reads that are not primer-started (for example AgriSeq on Ion Torrent) are normalized first:
Normalize reads that are not primer-started (for example AgriSeq on Ion Torrent) first:

```bash
madcats normalize-reads --probe-design markers.tsv --fastq-dir raw_fastqs/ --out-dir normalized/ --jobs 8
Expand All @@ -127,13 +139,13 @@ madcats generate --fastq-dir normalized/ --fastq-glob "*.fastq" \

Full documentation is available at [madcats.readthedocs.io](https://madcats.readthedocs.io), including:

- [Getting Started](https://madcats.readthedocs.io/en/latest/getting-started/installation.html): installation and first steps
- [Tutorials](https://madcats.readthedocs.io/en/latest/tutorials/dartag.html): walkthroughs for DArTag, primer-free, AgriSeq and paired-end data
- [Command Reference](https://madcats.readthedocs.io/en/latest/commands/overview.html): every subcommand and option
- [Evidence Model](https://madcats.readthedocs.io/en/latest/reference/evidence-model.html): how alleles are counted, discovered and denoised
- [Getting Started](https://madcats.readthedocs.io/en/latest/getting-started/installation.html) for installation and first steps
- [Tutorials](https://madcats.readthedocs.io/en/latest/tutorials/dartag.html) with step-by-step examples for DArTag, primer-free, AgriSeq and paired-end data, the paralog filter and genotype calling
- [Command Reference](https://madcats.readthedocs.io/en/latest/commands/overview.html) for every subcommand and option
- [Evidence Model](https://madcats.readthedocs.io/en/latest/reference/evidence-model.html) on how MADCATS counts, discovers and denoises alleles

The design rationale and version-by-version measurements behind every default are recorded in the
[development record](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md).
The [development record](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md)
gives the design rationale and version-by-version measurements behind every default.

## Commands

Expand All @@ -146,6 +158,8 @@ The design rationale and version-by-version measurements behind every default ar
| `madcats window-scan` | Sweep candidate tag lengths and recommend one from the project's own reads |
| `madcats generate` | Generate the MADC count table |
| `madcats validate` | Compare a generated MADC against a reference MADC |
| `madcats filter-paralogs` | Drop or flag MADC rows that a reference genome places at another locus (needs `mappy`) |
| `madcats call` | Call genotypes from a MADC and write target, all-variant or haplotype VCFs |

Run `madcats <command> --help` for full options.

Expand All @@ -162,18 +176,18 @@ Chr01_000123,ACGTTGCAGGTCAAGT,ACGTTGCAGGTCAAGTTCA...,ACGTTGCAGGTCAAGTTCG...
`LocusGroup` are optional.

### Probe design (TSV/CSV)
`MarkerName` and `TargetSequence` with one bracketed variant: a SNP such as `[C/T]` or an
`MarkerName` and `TargetSequence` with one bracketed variant, either a SNP such as `[C/T]` or an
anchored indel such as `[A/ACTTG]`.

### FASTQs
One FASTQ (optionally gzipped) per sample in a folder. Sample columns are named by the FASTQ stem,
or by a `--sample-lookup` table.
One FASTQ (optionally gzipped) per sample in a folder. MADCATS names sample columns by the FASTQ
stem, or by a `--sample-lookup` table.

## Output

| File | Contents |
|------|----------|
| `project_MADC.csv` | MADC table: `AlleleID`, `CloneID`, `AlleleSequence`, summary columns, one read-count column per sample |
| `project_MADC.csv` | MADC table with `AlleleID`, `CloneID`, `AlleleSequence`, summary columns and one read-count column per sample |
| `project_MADC.csv.qc.json` | Parameters, provenance, and per-sample read accounting |
| `project_MADC.csv.variants.csv` | One line per variant event per row (with `--annotations on`, the default) |

Expand All @@ -185,13 +199,13 @@ compared with the reference genotypes.

| Platform | Accuracy | Validated against |
|----------|----------|-------------------|
| DArTag (sweetpotato, red clover, blackberry, alfalfa, pecan) | count r = 0.9997; dosage agreement 95.4–98.8% | vendor MADC, vendor/updog dosage, families, skim-seq VCF |
| AgriSeq, Ion Torrent (honeybee, oyster) | 94.9% vs Torrent VariantCaller; 98.0% replicate concordance | vendor caller + replicates |
| DArTag (sweetpotato, red clover, blackberry, alfalfa, pecan) | count r = 0.9997, dosage agreement 95.4–98.8% | vendor MADC, vendor/updog dosage, families, skim-seq VCF |
| AgriSeq, Ion Torrent (honeybee, oyster) | 94.9% vs Torrent VariantCaller, 98.0% replicate concordance | vendor caller + replicates |
| GT-seq (delta smelt) | 99.49% | independent BWA/bcftools truth VCF |
| Flex-Seq capture (cranberry) | 95.13% genotype agreement | published population VCF + 20 trios |
| rhAmpSeq (8 grape families) | 99.92–99.99% family transmission | Mendelian constraints + same-reads head-to-head |

Details: [Validation & Accuracy](https://madcats.readthedocs.io/en/latest/reference/validation.html).
See [Validation & Accuracy](https://madcats.readthedocs.io/en/latest/reference/validation.html) for details.

## Development

Expand All @@ -209,12 +223,12 @@ If MADCATS is useful in your work, please cite:
> Sandercock, A.M. (2026). MADCATS: Marker-Anchored Discovery and Counting of Alleles from Targeted
> Sequencing. GitHub repository: https://github.com/Breeding-Insight/MADCATS

*(A formal article/DOI is planned; this section will be updated when available.)*
*(A formal article/DOI is planned, and this section will be updated when available.)*

For the MADC format, see Breeding Insight's
[MADC file format overview](https://breeding-insight.github.io/BIGr_documentation/posts/madc-file-format/).

**AI assistance:** MADCATS was developed in collaboration with Claude Opus 5.5 and Fable 5.1
**AI assistance.** MADCATS was developed in collaboration with Claude Opus 5.5 and Fable 5.1
(Anthropic), which helped write its code and documentation. The authors directed the work, reviewed
and tested the results, and are ultimately responsible for the application.

Expand Down
Loading
Loading