Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 85 additions & 8 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,17 @@ which inputs they require:
library's common sequence `GATGGATCCAGTGC` in front of every amplicon.
`--leading-anchor GATGGATCCAGTGC` cuts each read where that sequence
ends (it must end within 40 bases beyond its own length) and counts
reads without it as `reads_no_leading_anchor`. Preprocessed deliveries
reads without it as `reads_no_leading_anchor`. With no exact copy, the
one placement within a single substitution is used instead
(`--leading-anchor-mismatches 1`, the default since 0.14.0; counted as
`reads_leading_anchor_mismatch`). Preprocessed deliveries
are primer-started and need no anchor. The anchor is part of the cache
key and the provenance line. `infer-design` takes the same option, so
the primer-free route reads raw deliveries too. Both commands check the
first FASTQ before any read pass. When almost no read starts with a
panel tag but most carry one after a constant prefix, the run stops and
names that prefix (the suffix shared by the bases in front of the tag),
key and the provenance line. `infer-design` takes `--leading-anchor`
as well, so the primer-free route reads raw deliveries too. Both commands
check a sample of the FASTQs (up to 24, spread across the run) before
any read pass. When almost no read starts with a panel tag but most
carry one after a constant prefix, the run stops and names that prefix
(the suffix that the sampled files' prefixes share),
and `--leading-anchor auto` applies the detected sequence instead,
recording it in the QC as detected.
2. **Vendor-MADC bootstrap** (`scripts/madc_to_reference_tags.py`). It
Expand Down Expand Up @@ -1293,6 +1297,33 @@ DArTag panels do. The primer-free route on the raw alfalfa reads, with
no primer file, resolves 2,999 of 3,000 loci in the exact tier, all
identical to the design tags.

**One substitution in the leading anchor (0.14.0).** A read whose common
sequence carries a sequencing error has no exact anchor, and 0.13.0
counted it as `reads_no_leading_anchor`. With
`generate --leading-anchor-mismatches 1` (the default), such a read is cut
after the one placement within a single substitution in the search window
and counted as `reads_leading_anchor_mismatch`. An exact copy is tried
first, so a read that has one is cut as before. A read with no such
placement, or more than one, still counts as having no anchor. Indels are
not matched, because they would move the cut. An anchor shorter than 12
bases is always matched exactly. The setting joins the cache key only when
an anchor is matched with a substitution, so other caches stay valid.
`infer-design` still matches the anchor exactly, since its reads only vote.
Measured on the salmon 3K DArTag plate (DSalm21-6350, 1,104 raw FASTQs,
54-bp tags, `--gapped-rescue off`): with `0`, the table is byte for byte
the 0.13.0 table after the provenance cell. With `1`, reads without an
anchor fall from 13,505,332 to 2,048,910 (2.4% to 0.4% of reads),
2,789,193 more reads are assigned (+1.06%) and 2,304,149 more reach emitted
rows (+0.90%, and 0.65% to 3.20% per sample). 16 new rows pass the evidence
rules, none is lost and no row changes category. The Alt fraction per
marker and sample (depth of 20 or more) changes by 0.0000 at the median and
0.0185 at the 99th percentile, and 379 of 2,501,106 cells move by more than
0.05. Against the vendor MADC, recall is unchanged (99.50%), precision goes
from 77.40% to 77.27% (the 16 rows), and the share of cells with exactly
the vendor's count rises from 64.65% to 66.95%. In one sample, 2,441 of
5,468 recovered on-target reads had the substitution at the anchor's last
base (C read as T in 2,223), spread over 1,106 markers.

**In the design route, a tag carries its variant.** Tag reconstruction from the
primer table and probe design now keeps only the primer-started windows
that contain the bracketed variant. The pecan design has one marker
Expand Down Expand Up @@ -1343,6 +1374,52 @@ byte-identically from cache). Protection, the final bookkeeping and the
table write stay in the main process. The table is unchanged with
`--jobs 1`.

**Raw deliveries with barcodes of several lengths (0.14.0).** The 0.12.0
detection of the leading anchor pooled the
sampled reads and built the anchor from the prefixes at the modal tag
offset. DArT inline barcodes have 10 to 14 bases, about a fifth of the
samples each, so of 8 sampled files only two or three sit at the modal
offset, and the anchor grew into the barcode bases those few shared. On
the salmon validation plate (DSalm21-6350: 1,104 FASTQs, 2,979 loci,
common sequence `CGATGGATCCAGTGC` in the delivery's targets file; three
truncated files left out of the draws), detection from random subsets of
1, 2, 3 and 4 files (60 draws each) and of 8 files (150 draws) returned
the exact 15 bases in 0, 9, 19, 39 and 133 draws. Of the 17 misses with 8
files, 13 were one base too long and 4 two. A barcode on this plate ends
in A, G or T, 30–36% each, so an anchor one base too long fits about a
third of the samples and the others lose every read.

Detection now works per file. A barcode is constant within a file, so
each file gives one prefix, the most common among its shifted reads
(seen at least twice). That also absorbs a tag found a base early, which
the modal offset used to. The anchor is the longest suffix that every
file's prefix shares, and the check samples 24 files instead of 8. On the
same draws the exact sequence came back in 0, 39, 57, 58 and 150, and in
150 of 150 draws of 16 and of 24 files. In 3,000 further draws of 8 files
two anchors were one base too long, about what the barcode endings
predict (the chance that 8 barcodes end alike is 1 in 1,900). In 1,000
further draws of 24 files none was. Sampling 24 files of the plate takes
about 0.02 s.

Two rules cover files that are not one clean sample. A file that mixes
barcodes has its prefix shortened until 90% of its shifted reads would be
cut by it. The plate's three truncated files are such files (variants of
one barcode, the commonest in 18–27% of the shifted reads), and alone each gives
the common sequence plus the 2 or 3 bases its variants share, which cuts
95–96% of its tag-carrying reads. Files holding 90% of the shifted reads
must end in the same 8 bases, and the rest are set aside, so a failed
sample does not hide the anchor.

With up to 24 files the sample is the whole run, and the anchor is in
every file's prefix by construction. It then cuts every file where the
amplicon starts, even when it is longer than the common sequence: from
one file it is the whole prefix, barcode included, and from two files it
kept one to five barcode bases in 21 of 60 draws. In 40 draws each of 1, 2
and 3 files, the detected anchor was found in at least 97% of the
tag-carrying reads of every file in the run. So `auto` uses an anchor from
fewer than three files and warns not to reuse it for other samples, and
the stop message for an undeclared prefix says the same.

## Truncated reads: partial-read salvage with a locus guard (0.11.0)

A DArTag order whose delivered files were trimmed to 109–113 bp per file,
Expand Down Expand Up @@ -4191,8 +4268,8 @@ By default sample columns use the unique FASTQ filename stem.
python3 -m unittest discover -s tests -v
```

The suite contains 257 unit/end-to-end tests in six files.
`test_madcats.py` (86) covers `generate` and the commands around it,
The suite contains 264 unit/end-to-end tests in six files.
`test_madcats.py` (93) covers `generate` and the commands around it,
`test_call.py` (112) the genotype caller, `test_filter_paralogs.py` (49) the
paralog filter, `test_tiled_markers.py` (5) tiled markers through all three,
`test_example.py` (4) the bundled example and `test_docs.py` (1) the match
Expand Down
3 changes: 2 additions & 1 deletion docs/src/commands/generate.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de
| Option | Default | Description |
|--------|---------|-------------|
| `--min-identity <FLOAT>` | 0.80 | Minimum substitution-only identity to a Ref or Alt tag. A gapped alignment uses the same edit limit |
| `--leading-anchor <SEQ\|auto>` | none | Constant sequence before the amplicon in raw deliveries (for DArT, `GATGGATCCAGTGC`). Each read is cut where the anchor ends (it must end within 40 bases beyond its own length). `auto` detects it from the first FASTQ |
| `--leading-anchor <SEQ\|auto>` | none | Constant sequence before the amplicon in raw deliveries (for DArT, `GATGGATCCAGTGC`). MADCATS cuts each read where the anchor ends, and the anchor must end within 40 bases beyond its own length. `auto` detects it from a sample of up to 24 FASTQs. With one or two FASTQs the detected anchor may include the sample barcode, and the run warns about it |
| `--leading-anchor-mismatches {0,1}` | 1 | Substitutions allowed in the leading anchor. With `1`, MADCATS cuts a read that has no exact copy after its only one-substitution match, and counts it as `reads_leading_anchor_mismatch`. A read with no such match, or more than one, counts as `reads_no_leading_anchor`. Anchors under 12 bases must match exactly. `0` requires an exact match for every anchor, as in 0.13.0 |
| `--primer-mismatches {0,1}` | 1 | Mismatches allowed while assigning a primer/anchor. With `0`, reads from Match haplotypes whose distinguishing SNP falls inside the anchor are lost |
| `--gapped-rescue {on,off}` | `on` | Give reads that fail substitution-only identity one gapped alignment, so the counts include undesigned indels |
| `--max-indel-length <INT>` | 10 | Widest gap (alignment band) a gapped alignment may use |
Expand Down
19 changes: 19 additions & 0 deletions docs/src/development/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,15 @@ Notable changes by version. Measurements behind each change are in the
overflow and record it in their QC reports. Neither command corrects the counts, since the
wrapped ones cannot be recovered.
The lettuce 3K vendor MADC has 90,913 such cells across 1,108 markers.
- `--leading-anchor auto`, and the raw-delivery check that names the anchor, find the common
sequence when inline barcodes differ in length. Each sampled FASTQ gives its own prefix, and the
anchor is the suffix that all of them share. 24 files are sampled instead of 8. Before, the
anchor came only from reads at the most common tag offset. DArT barcodes have 10 to 14 bases, so
few files share that offset, and the anchor could keep one or two barcode bases and drop every
read of the samples whose barcode ended otherwise. On a salmon DArTag plate (1,101 samples),
detection from 8 random files returned the exact common sequence in 133 of 150 draws before and
in 150 of 150 now. With one or two FASTQs the anchor may still include barcode bases. It cuts
those files correctly, and `auto` warns not to reuse it for other samples.
- For tiled markers, `generate` credits a read that fits several tiles of one amplicon to each of
them, instead of dropping it as a tie. Tiles are markers that read the same tag window, with a
shared or nested left primer and Ref tags that agree except at the markers' variant positions.
Expand All @@ -111,6 +120,16 @@ Notable changes by version. Measurements behind each change are in the
(now 6,562–103,852 each), and one pair with identical tags had split its reads 12,299 / 80,306
(now about 112,400 each). All other markers' rows are unchanged.
- The tile families are part of the per-sample cache key. Panels without tiles keep their caches.
- `infer-design --leading-anchor`: the fuzzy pass cuts raw reads at the anchor, as the exact pass
does. It had dropped the anchor before scanning, so on a raw delivery its reads did not vote.
- `generate --leading-anchor-mismatches {0,1}` (default 1). A read with no exact copy of the
leading anchor is cut after the one placement within a single substitution, and the QC counts it
as `reads_leading_anchor_mismatch`. Anchors shorter than 12 bases are always matched exactly,
and `0` matches every anchor exactly, as 0.13.0 did. On the salmon 3K DArTag plate (1,104 raw
FASTQs), `0` reproduces the 0.13.0 table byte for byte. With `1`, reads without an anchor fall
from 13.5 million to 2.0 million, 1.06% more reads are assigned, and no row changes category.
The setting is part of the cache key only when an anchor is matched with a substitution, so
other caches stay valid. `infer-design` still matches the anchor exactly.

## 0.13.0

Expand Down
14 changes: 10 additions & 4 deletions docs/src/tutorials/dartag.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,20 @@ The defaults are the recommendation for diploid and polyploid DArTag projects al
## Raw DArT Deliveries

Older DArT orders deliver reads with an inline barcode and the library's common sequence
`GATGGATCCAGTGC` in front of the amplicon, so nothing is primer-started. MADCATS checks the first
FASTQ before any read pass. When almost no read starts with a panel tag but most carry one after a
constant prefix, the run stops and names that prefix. Re-run with:
`GATGGATCCAGTGC` in front of the amplicon, so nothing is primer-started. MADCATS checks a sample
of the FASTQs (up to 24) before any read pass. When almost no read starts with a panel tag but most
carry one after a constant prefix, the run stops and names that prefix. Re-run with:

```bash
madcats generate ... --leading-anchor auto
# or explicitly
madcats generate ... --leading-anchor GATGGATCCAGTGC
```

The QC counts reads without the anchor as `reads_no_leading_anchor`.
With one or two FASTQs, the anchor that `auto` finds may include the sample barcode. It fits those
files, and the run warns not to reuse it for other samples.

A read whose anchor carries one sequencing error is still cut where the anchor ends, as long as
there is only one such placement (`--leading-anchor-mismatches 1`, the default). The QC counts
those reads as `reads_leading_anchor_mismatch`, and reads without the anchor as
`reads_no_leading_anchor`.
Loading
Loading