From 6fc3a89bc103ccfb6cdb1200f304ae6fd55da84c Mon Sep 17 00:00:00 2001 From: Alexander Sandercock <39815775+alex-sandercock@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:14:07 -0500 Subject: [PATCH 1/2] Docs: how call relates to polyRAD and updog, and a shorter 0.14.0 changelog The tutorial's table headed "The same model in call" claimed more than was tested, and gave call's defaults as polyRAD's IterateHWE. The section now says what call adapts from polyRAD (the likelihood, the Hardy-Weinberg prior with its selfing rate, the iteration) and from updog (the allelic bias and the norm prior), what is its own, and which settings come closest to each package, with how close they came. Each package ran at its defaults on four simulated tables of 18,000 genotypes. With polyRAD's settings call gave polyRAD's genotype in every cell. Its defaults do not. Beside updog norm and hw, call with the bias fitted gave the same genotype for 99% or more in the diploid and the tetraploids and 95 to 96% in the hexaploid. The development record has the table. The 0.14.0 changelog entry was 1,456 words, against 11 to 110 for every earlier version. It is now 294, and says what the release adds. The tile measurements it held move to the development record, with a check on 12 lettuce samples against 0.13.0 and against published genotypes. --- docs/DEVELOPMENT.md | 68 +++++++++++++ docs/src/commands/call.md | 10 +- docs/src/development/changelog.md | 130 +++++-------------------- docs/src/tutorials/genotype-calling.md | 39 +++++--- 4 files changed, 126 insertions(+), 121 deletions(-) diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 6addbc4..2f0337f 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -1479,6 +1479,44 @@ Their target sits at position 110, one base past the 109-bp window (the vendor's SNP report lists them without a position, and the 110th base segregates in the plates whose reads are longer than 109 bp). +## Tiled markers share their reads (`generate --tile-groups`, 0.14.0) + +Tiles are markers that read the same tag window. They have a shared or +nested left primer, and Ref tags that agree except at the markers' variant +positions. A read that is reference at every one of those positions fits +each tile equally well, and up to 0.13.0 `generate` dropped it as a tie. +It now credits such a read to every tile of the amplicon that it fits. +`--tile-groups auto|declared|off` (default `auto`) links `LocusGroup` +members and finds undeclared tiles, and the design route also reads +`LocusGroup` from the probe design. Ties across tile families are still +dropped. The QC counts the extra credits as `tile_extra_credits_*`, so +`reads_assigned_full_length` still counts each read once. The tile families +are part of the per-sample cache key, and panels without tiles keep their +caches. + +On the lettuce 3K DArTag panel (376 samples), 13 markers in 6 amplicons now +get, in one full-panel run, the same table cells as running each tile in a +panel of its own. 11 had zero reads before (now 6,562–103,852 each), and one +pair with identical Ref tags had split its reads 12,299 / 80,306 (now about +112,400 each). All other markers' rows are unchanged. + +Against 0.13.0, and against published genotypes. On 12 of those samples +0.14.0 with `--tile-groups off` writes the MADC and the variants table of +0.13.0, byte for byte. With the default only the 13 tiled markers differ, +and the 8,898 rows of the other 2,987 markers are the same. `call --ploidy +2` then called both tables, and the calls at the tiled markers were set +beside the genotypes of the published realignment (96 sample-by-marker +cells have one): + +| Table | Right | Wrong | Not called | +|---|---|---|---| +| 0.13.0 | 11 | 0 | 85 | +| 0.14.0 | 92 | 0 | 4 | + +In 0.13.0 a tile kept only the reads that carry its own alternate allele, +so a sample that is reference at the marker had no reads there and no call. +The samples are inbred lines, so this did not test a heterozygote. + ## Genome paralog filter (`filter-paralogs`, 0.14.0) On the public lettuce 3K DArTag data about two thirds of the reads come from @@ -2051,6 +2089,36 @@ and 6) and stores the values in `tests/fixtures/polyrad_parity.json`: | Allele frequencies and posteriors after `IterateHWE(tol = 1e-10)` | 7 decimal places | | Genotypes after the correction to the ploidy | identical (144 sample-by-allele values) | +**Closest settings to each package, end to end.** The table above compares +single functions on shared inputs. Here each package ran at its defaults on +whole tables, and `call` ran with the settings nearest to it. updog's +simulator (`rgeno`, `rflexdog`) drew four tables of 120 markers and 150 +samples in Hardy-Weinberg proportions, with a sequencing error of 0.005 and +an overdispersion of 0.01. Two have an allelic bias drawn from a log-normal +with a standard deviation of 0.5. Depth is 30 reads in the diploid, 60 in +the tetraploids and 100 in the hexaploid. Each cell is the share of the +table's 18,000 genotypes that `call` and the package gave alike, with +`--min-dp 0 --min-gq 0`: + +| `call` settings, beside | Tetraploid, biased | Tetraploid, unbiased | Hexaploid, biased | Diploid, unbiased | +|---|---|---|---|---| +| `--prior hwe --overdispersion 9 --contamination-rate 0.001 --background-rate 0`, beside polyRAD `IterateHWE` | 100% | 100% | 100% | 100% | +| The defaults, beside polyRAD `IterateHWE` | 87.1% | 87.5% | 81.0% | 97.4% | +| `--prior norm --allele-bias fit`, beside updog `norm` | 99.0% | 99.4% | 94.8% | 99.99% | +| `--prior norm --allele-bias auto`, beside updog `norm` | 97.3% | 98.7% | 86.6% | 100% | +| `--prior hwe --allele-bias fit`, beside updog `hw` | 99.2% | 99.5% | 95.6% | 100% | + +With polyRAD's settings `call` gave polyRAD's genotype in all 72,000 cells. +Its defaults do not, because they estimate the overdispersion and the +background-read rate where polyRAD takes 9 and a contamination rate. Against +updog the genotypes are near but not the same. updog estimates a sequencing +error and an overdispersion at every marker, and `call` one background-read +rate and one overdispersion for the table. The shares of genotypes equal to +the simulated ones were alike, 94.0% for `call` with `norm` and `fit` and +94.2% for updog `norm` in the biased tetraploid, 94.7% and 94.6% in the +unbiased one, 84.7% and 83.6% in the hexaploid, and 100% in the diploid. +This is one draw per table, from updog's own simulator. + **Correction to the ploidy.** With more than two alleles the model finds the most probable dosage of each allele separately, and the dosages need not add up to the ploidy. polyRAD adds or removes one copy at a time. An exact search diff --git a/docs/src/commands/call.md b/docs/src/commands/call.md index a75dd3d..563448e 100644 --- a/docs/src/commands/call.md +++ b/docs/src/commands/call.md @@ -73,8 +73,10 @@ The measurements behind this section are in ### The genotype model `call` adapts the model of [polyRAD](https://github.com/lvclark/polyRAD) (Clark, Lipka and Sacks -2019), with the `norm` prior of [updog](https://github.com/dcgerard/updog) (Gerard and Ferrão -2020) and a prior for structured populations as options. Within a record the model considers each +2019), with the allelic bias and the `norm` prior of [updog](https://github.com/dcgerard/updog) +(Gerard et al. 2018, Gerard and Ferrão 2020) and a prior for structured populations as options. +[How `call` relates to polyRAD and updog](../tutorials/genotype-calling.md#how-call-relates-to-polyrad-and-updog) +gives the settings closest to each package. Within a record the model considers each allele separately, against all the other alleles together: 1. **Likelihood.** In a sample of ploidy *P* an allele can be present in 0 to *P* copies (its @@ -113,8 +115,8 @@ homozygous dosages. **`norm`** makes no assumption about mating. Its dosage probabilities are proportional to a normal density at the dosages 0 to *P*, so it can be narrower than Hardy-Weinberg proportions (one family) or wider (subpopulations), and when the dosages have two peaks (inbred lines) it flattens -out. It is updog's `norm` genotype distribution, fitted the way updog fits it. With the same -likelihood, `call` and updog reach the same prior and the same genotypes. `call` fits it separately +out. It is updog's `norm` genotype distribution. With the same likelihood, `call` and updog +reached the same prior and the same genotypes on stored reference fits. `call` fits it separately for each ploidy among the samples, since a shape over the dosages of one ploidy says nothing about another. A ploidy with fewer than 10 samples uses Hardy-Weinberg proportions instead, and `call` prints a note and lists those ploidies in the QC JSON. In a haplotype record every allele gets its diff --git a/docs/src/development/changelog.md b/docs/src/development/changelog.md index 390742e..3817090 100644 --- a/docs/src/development/changelog.md +++ b/docs/src/development/changelog.md @@ -5,112 +5,30 @@ Notable changes by version. Measurements behind each change are in the ## 0.14.0 -- New command `call`, in its own module, `madcats_call.py`. It calls genotypes from a MADC (from - `generate` or `filter-paralogs`, a vendor MADC, or HapApp's fixed-allele-ID output) and writes - one VCF per type. The types are `--vcf target` (the designed variant of each marker, and the - default), `all` (targets plus other base substitutions in the tags) and `haplotype` (whole tag - sequences as alleles). It sets ploidy per sample (`--ploidy`, `--ploidy-file`). The model, - adapted from polyRAD, is a beta-binomial likelihood with a uniform or Hardy-Weinberg prior, - iterated per record. Two-allele records carry `GQ`, `GP` and `PL`, and records with more alleles - carry `MPC`, the lowest - marginal posterior among the called dosages. Coordinates are marker-local. It uses only the - standard library. The tests compare its likelihoods, posteriors, converged allele frequencies - and genotypes with polyRAD 2.0.1 on stored reference values. -- `call --prior structure` is a prior for samples from several subpopulations, a cline or families. - Each sample gets Hardy-Weinberg proportions at its own allele frequency, taken from the samples - most like it across the table's markers. The prior chooses how many of them to use by how well - they predict the sample at markers they were not chosen on, and picks 0 (the `hwe` prior) when - there is no structure. On thirteen tables (simulated groups, clines and families at 8 to 150 - reads, and potato DArTag counts thinned to 10 and 25 reads) it called as many genotypes correctly - as polyRAD's `IteratePopStruct` or up to 0.7 points more, and gained 0.5 to 5.5 points over `hwe` - at 8 to 25 reads. It needs no extra dependency. It cannot yet be combined with - `--allele-bias fit` or `--selfing-rate`. -- `call --allele-bias fit` estimates updog's allelic bias for every record with two alleles and - calls with it. With the other settings equal, the fitted bias is within 0.1% of updog 2.1.7's and - the genotypes are the same, on stored reference values. On two tetraploid alfalfa populations it - raised agreement with published dosages from 90.9% and 90.0% to 92.2% and 91.8% (against 93.0% and - 92.7% for updog). Against SNP-array genotypes of 85 tetraploid potato clones it raised the share - of matching dosages from 92.3% to 96.6% (against 96.7% for updog). The VCF gains `INFO/BIAS`, - and `PREFIX.bias.csv` lists each fit. -- `call --allele-bias auto` fits the bias as `fit` does and calls with it only the records where - it raises the log-likelihood of the read counts by 10 or more over no bias, and calls the - others without one. Against `fit`, on potato it keeps the gain (96.4% against 96.6% of array - dosages matched) and on polyploids pooled from real diploid reads, which have no bias, it - removes the loss (77.6% right for hexaploids at 50 reads, as without a bias, against 69.8%). - Every two-allele record still carries its fitted bias in `INFO/BIAS`, and one called without it - carries the flag `INFO/BIASUNUSED`. The bias table gains `Gain`, `Used` and `Rejected` columns. -- `call --allele-bias auto` does not use a bias that ends at a limit of the search. Samples with - few heterozygotes, such as inbred lines, drive every fit to a limit, where homozygous samples - pass for heterozygous ones. On simulated inbred tetraploids 71.7% of dosages were right with - such biases used and 97.4% without. `call` warns, marks the records in the bias table and lists - them in the QC JSON. `--allele-bias fit` still uses such a bias, and warns when a fifth or more - of the fits end at a limit. -- With `auto`, a record called without its bias gets the same calls as with `--allele-bias off`, - under the overdispersion and background rate estimated without a bias. Before, it was called - under the read model estimated with every bias fitted, which cost 1.9 points on a simulated - hexaploid table where no record kept its bias. -- The read-model estimate with a bias fitted starts from the estimate made without one. Started - from a fixed value, its search could stop early. On that hexaploid table it chose an - overdispersion of 10 where 20 is far more probable. -- `call` writes `MPP` (per sample) and `PMC` (per record) on two-allele records. These are - updog's `maxpostprob` and `prop_mis`, under the names BIGr's `updog2vcf` gives them. `MPP` is the - posterior probability of the most probable genotype, and `PMC` is one minus its mean over the - samples with reads, regardless of the genotype filters. With updog's prior, bias and - overdispersion, `MPP` is within 0.001 of updog 2.1.7's value and `PMC` equal to four decimals, on - stored reference fits. updog counts a sample given a depth of 0 at the largest probability of the - prior, while `call` leaves samples without reads out, as updog does when they are passed as - missing. -- `call` estimates a background-read rate (`--background-rate auto`, the default) together with - the overdispersion. This rate is the share of reads that show an allele they did not come from. - It replaces - polyRAD's contamination rate, which is now 0 unless set, because contamination cannot account for - stray reads of an allele no other sample carries. The estimate gives every dosage its own - probability at each marker, so it does not depend on `--prior` and also works for inbred and - partly inbred samples, and it uses the worker processes. -- Two-allele records are computed for one allele and mirrored for the other, and terms shared by - the samples of a ploidy are computed once. Results are unchanged. -- `call --prior norm` adds updog's `norm` genotype distribution. It gives dosage probabilities a - normal shape, fitted per record and per ploidy, and does not assume random mating. With the same - likelihood, its fitted priors and genotypes match updog 2.1.7 on stored reference values. A - ploidy with fewer than 10 samples uses Hardy-Weinberg proportions. -- `call --selfing-rate` adjusts the Hardy-Weinberg prior for self-fertilisation, for inbred lines - and partly selfing species. Its priors and iterated results match polyRAD's `selfing.rate`. -- `call` estimates the beta-binomial overdispersion from the read counts by maximum likelihood - (`--overdispersion auto`, the default, or a number to set it). On simulated diploid, tetraploid, - hexaploid and mixed-ploidy populations it recovered the simulated value. polyRAD's default of 9 - left 59% of target genotypes uncalled on two real tetraploid alfalfa populations, against 8–11% - with the estimated values. -- New optional command `filter-paralogs`, in its own module, `madcats_paralogs.py`. It filters a - MADC (from `generate`, a vendor DArT MADC with or without its preamble, or HapApp's - fixed-allele-ID output) against a reference genome. It places each marker by its Ref tag - and labels every haplotype `on_target`, `paralog` or `ambiguous`. Paralog and ambiguous rows - are dropped (default) or flagged in a `GenomeLabel` column, while canonical rows are always kept. - It writes a labels sidecar, a markers sidecar and a QC JSON. It needs `mappy` - (`pip install 'madcats[genome]'`), imported only by this command, so MADCATS still has no - required dependencies. On the lettuce 3K DArTag panel the filtered vendor MADC agreed with a - bwa-mem-based prototype for 99.4% of reads. -- `filter-paralogs --clone-positions ignore` keeps CloneID coordinates out of marker placement, for - a genome other than the one the CloneIDs refer to. -- The MADC readers find a header row whose `AlleleID` is not the first cell, as in HapApp's - fixed-allele-ID output, which starts with `Code_version`. -- `validate` and `filter-paralogs` warn when a MADC has negative read counts. Some vendor MADCs - store counts in a signed byte, so counts of 128–255 appear as count − 256, and counts of 256 or - more wrap silently. Both commands report a negative count with a maximum of exactly 127 as this - overflow and record it in their QC reports. Neither command corrects the counts, since the - wrapped ones cannot be recovered. - The lettuce 3K vendor MADC has 90,913 such cells across 1,108 markers. -- For tiled markers, `generate` credits a read that fits several tiles of one amplicon to each of - them, instead of dropping it as a tie. Tiles are markers that read the same tag window, with a - shared or nested left primer and Ref tags that agree except at the markers' variant positions. - `--tile-groups auto|declared|off` (default `auto`) links `LocusGroup` members and detects - undeclared tiles. The design route now also reads `LocusGroup` from the probe design. - Ties across families are still dropped. The QC read accounting counts extra credits as - `tile_extra_credits_*`, so `reads_assigned_full_length` still counts each read once. On the - Lettuce 3K DArTag panel (376 samples), 13 markers in 6 amplicons now get, in one full-panel - run, the same table cells as running each tile in a panel of its own. 11 had zero reads before - (now 6,562–103,852 each), and one pair with identical tags had split its reads 12,299 / 80,306 - (now about 112,400 each). All other markers' rows are unchanged. -- The tile families are part of the per-sample cache key. Panels without tiles keep their caches. +- New command [`call`](../commands/call.md) (experimental), in its own module, `madcats_call.py`. + It calls genotypes from a MADC and writes one VCF per type chosen with `--vcf`, which is + `target` (the default), `all` or `haplotype`. Ploidy can differ between samples. The model is + adapted from polyRAD, with the allelic bias and the `norm` prior adapted from updog, and it uses + only the standard library. +- `call` takes its prior from `--prior` (`hwe`, `norm`, `structure` or `uniform`), with + `--selfing-rate` for inbred and partly selfing samples and `--allele-bias` (`off`, `fit` or + `auto`) for markers that read one allele more readily. It estimates the overdispersion and a + background-read rate from the table unless they are given. +- New optional command [`filter-paralogs`](../commands/filter-paralogs.md) (experimental), in its + own module, `madcats_paralogs.py`. It drops or flags the MADC rows that a reference genome + places at another locus, and writes a labels table, a markers table and a QC JSON. It needs the + `mappy` aligner (`pip install 'madcats[genome]'`), so MADCATS still has no required + dependencies. `--clone-positions ignore` is for a genome other than the one the CloneIDs refer + to. +- For tiled markers, which read the same tag window, `generate` credits a read that fits several + tiles of one amplicon to each of them. Before, it dropped the read as a tie. `--tile-groups` + (`auto`, `declared` or `off`, default `auto`) controls it, and `off` gives the output of 0.13.0. + The tile families are part of the per-sample cache key, so panels without tiles keep their + caches. +- The MADC readers accept HapApp's fixed-allele-ID layout, whose header starts with + `Code_version`, and pass over a `readCountSum` column. +- `validate` and `filter-paralogs` warn when a MADC has negative read counts, which some vendor + MADCs carry from counts stored in a signed byte, and record it in their QC reports. ## 0.13.0 diff --git a/docs/src/tutorials/genotype-calling.md b/docs/src/tutorials/genotype-calling.md index c1f8666..73ed2e9 100644 --- a/docs/src/tutorials/genotype-calling.md +++ b/docs/src/tutorials/genotype-calling.md @@ -100,17 +100,6 @@ Change this when one of these describes your samples: [Which settings to use](../commands/call.md#which-settings-to-use) has the measurements behind each row. -If you come from updog or polyRAD: - -| You used | The same model in `call` | -|----------|--------------------------| -| updog, `norm` | `--prior norm --allele-bias fit`. updog always fits a bias | -| polyRAD, `IterateHWE` | The defaults. polyRAD's own settings are `--overdispersion 9 --contamination-rate 0.001 --background-rate 0` | -| polyRAD, `IteratePopStruct` | `--prior structure`. The method differs, and the purpose is the same | - -`MPP` and `PMC` are updog's `maxpostprob` and `prop_mis`, named as BIGr's `updog2vcf` names them, -and `INFO/BIAS` is updog's bias. BIGr's `filterVCF` filters on all three by these names. - ## 5. Write Other Record Types `--vcf` takes one or more record types, and each goes to its own file: @@ -164,6 +153,34 @@ with SNP-array genotypes for every marker: These figures are from runs with the bias fitted. +## How `call` Relates to polyRAD and updog + +`call` adapts its model from two R packages. Most of it follows +[polyRAD](https://github.com/lvclark/polyRAD) (Clark et al. 2019), which gives it the likelihood +of the read counts, the Hardy-Weinberg prior with its selfing rate, and the repeated estimate of +the allele frequencies. Two parts follow [updog](https://github.com/dcgerard/updog) (Gerard et al. +2018, Gerard and Ferrão 2020). They are the allelic bias (`--allele-bias`) and the `norm` prior +(`--prior norm`). + +`call` is not a port of either package. It estimates one overdispersion and one background-read +rate for the whole table. polyRAD takes a fixed overdispersion and contamination rate, and updog +estimates an overdispersion and a sequencing-error rate at every marker. The search for the bias, +the `auto` rule and the `structure` prior are MADCATS's own. + +If you have used one of the packages, these settings come closest to it. The last column is the +share of genotypes that were the same in four simulated tables (one diploid, two tetraploid and +one hexaploid, with 18,000 genotypes each), with the package at its defaults: + +| Package and model | Closest settings in `call` | Same genotypes | +|-------------------|----------------------------|----------------| +| polyRAD, `IterateHWE` | `--prior hwe --overdispersion 9 --contamination-rate 0.001 --background-rate 0` | All of them. These are polyRAD's settings. `call`'s defaults differ, because they estimate the overdispersion and the background-read rate | +| updog, `norm` | `--prior norm --allele-bias fit` | 99% or more in the diploid and the tetraploids, and 95% in the hexaploid | +| updog, `hw` | `--prior hwe --allele-bias fit` | 99% or more in the diploid and the tetraploids, and 96% in the hexaploid | +| polyRAD, `IteratePopStruct` | `--prior structure` | Not compared. The purpose is the same and the method is different | + +`MPP` and `PMC` are updog's `maxpostprob` and `prop_mis`, and `INFO/BIAS` is updog's bias. They +carry the names BIGr's `updog2vcf` gives them, so BIGr's `filterVCF` filters on all three. + ## Limits - **Allopolyploids.** `call` takes every chromosome copy to be able to carry any allele. A marker From b89188a833ce82985ca0e463ffdf678279cbe013 Mon Sep 17 00:00:00 2001 From: Alexander Sandercock <39815775+alex-sandercock@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:34:21 -0500 Subject: [PATCH 2/2] Docs: a row for an unknown kind of population, and what read depth changes The settings table in the genotype-calling tutorial gains one row: for a mix of sample kinds, or when the kind is not known, add --prior norm (with --allele-bias auto for polyploids). The command page has the measurements for that row. A note under the table, and a fourth point on the command page, say how read depth changes the settings: the prior counts most at low depth, and a bias takes depth to find. The figures come from 162 simulated tables (updog's simulator, 60 markers by 150 samples, three draws of each scenario) at ploidy 2, 4 and 6, in five kinds of population, with and without a bias, and for tetraploids at 10, 25, 60 and 200 reads. The development record has the design, both tables and what they show. --- docs/DEVELOPMENT.md | 121 +++++++++++++++++++++++++ docs/src/commands/call.md | 9 +- docs/src/tutorials/genotype-calling.md | 10 ++ 3 files changed, 139 insertions(+), 1 deletion(-) diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 2f0337f..428897a 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -3418,6 +3418,127 @@ The real data sets with the rules as built, `--prior hwe`: - The limit rule under `auto` rejected 4 records on the potato clones, 25 on the vendor MADC, 10 on smelt and 14 to 23 on the pooled tables. +**`norm` with `auto` for an unknown kind of population, by ploidy and read +depth.** The tutorial and the command page give `--prior norm`, with +`--allele-bias auto` for polyploids, as the choice when the samples are a +mix of kinds or their kind is not known. The measurements behind that row +used updog's simulator (`rgeno` and `rflexdog`, with a sequencing error of +0.005 and an overdispersion of 0.01) for five kinds of population: + +- one random-mating population, +- one F1 family, with both parents drawn at the marker's allele frequency + and at least one of them heterozygous, +- two subpopulations of 75 samples each, whose allele frequencies differ + as under an Fst of 0.15, +- half-inbred samples (updog's `bb` genotypes with an overdispersion of + 0.5), +- inbred lines, with every sample homozygous. + +Each table has 60 markers by 150 samples and each scenario has three +draws, so a row below is the mean over 27,000 genotypes. Diploids have +about 30 reads for each sample and marker, tetraploids 60 and hexaploids +100. A biased panel draws each marker's bias with a standard deviation of +0.5 on the log scale. Every run estimated its own read model and wrote +every genotype (`--min-dp 0 --min-gq 0`). Share of dosages right: + +| Samples | hwe | hwe, `auto` | norm | norm, `auto` | norm with `auto` less hwe with `auto` (range over the draws) | +|---|---|---|---|---|---| +| 4x random mating | 95.0% | 95.0% | 95.0% | 95.0% | -0.04 (-0.08 to +0.01) | +| 4x F1 family | 95.1% | 94.2% | 95.6% | 95.6% | +1.48 (+0.48 to +2.03) | +| 4x two subpopulations | 95.8% | 95.8% | 95.8% | 95.8% | -0.05 (-0.07 to -0.04) | +| 4x half-inbred | 97.0% | 96.6% | 97.3% | 97.3% | +0.74 (+0.51 to +1.20) | +| 4x inbred lines | 99.6% | 99.6% | 99.8% | 99.8% | +0.19 (+0.18 to +0.21) | +| 4x random mating, biased | 81.8% | 94.0% | 81.2% | 93.8% | -0.19 (-0.27 to -0.07) | +| 4x F1 family, biased | 81.3% | 90.8% | 80.6% | 93.4% | +2.68 (+1.37 to +3.77) | +| 4x two subpopulations, biased | 86.9% | 94.0% | 86.6% | 94.2% | +0.22 (+0.06 to +0.39) | +| 4x half-inbred, biased | 87.7% | 84.3% | 89.4% | 92.8% | +8.54 (+0.60 to +23.37) | +| 4x inbred lines, biased | 99.4% | 99.4% | 99.7% | 99.7% | +0.26 (+0.20 to +0.38) | +| 6x random mating | 87.1% | 87.1% | 86.9% | 86.9% | -0.17 (-0.29 to -0.08) | +| 6x F1 family | 88.3% | 88.3% | 89.1% | 89.1% | +0.85 (+0.70 to +1.09) | +| 6x two subpopulations | 89.2% | 88.5% | 89.4% | 89.2% | +0.75 (+0.36 to +1.53) | +| 6x half-inbred | 71.5% | 57.3% | 87.8% | 86.4% | +29.04 (+28.16 to +30.11) | +| 6x inbred lines | 98.8% | 98.8% | 99.4% | 99.4% | +0.60 (+0.52 to +0.70) | +| 6x random mating, biased | 67.3% | 80.8% | 66.7% | 80.4% | -0.43 (-1.10 to +0.09) | +| 6x F1 family, biased | 64.3% | 72.7% | 63.6% | 74.8% | +2.08 (+1.03 to +2.63) | +| 6x two subpopulations, biased | 68.4% | 78.8% | 68.7% | 80.7% | +1.96 (+0.20 to +3.60) | +| 6x half-inbred, biased | 67.7% | 57.1% | 79.3% | 76.2% | +19.10 (+14.70 to +24.69) | +| 6x inbred lines, biased | 98.5% | 98.5% | 99.2% | 99.2% | +0.73 (+0.62 to +0.87) | + +- In diploids `norm` with `auto` was within 0.02 points of `hwe` with + `auto` in every scenario, and within 0.03 in every draw. +- In one random-mating population `norm` with `auto` is 0.04 to 0.19 + points under `hwe` with `auto` in tetraploids and 0.17 to 0.43 under in + hexaploids. Two hexaploid tables of one draw each, in the comparison + of `auto` across kinds of population above, had it 1.2 and 1.4 under. +- In F1 families it is 0.9 to 2.7 points ahead. In two subpopulations it + is within 0.3 points in tetraploids and 0.8 to 2.0 ahead in hexaploids. +- In half-inbred samples `hwe` with `auto` is under `hwe` alone in every + row, by 0.4 and 3.4 points in tetraploids and by 14.2 and 10.6 in + hexaploids. On the hexaploid panel without bias it used a bias at 45 of + 60 markers in each draw, where `norm` used 3 to 7. `norm` with `auto` + is 0.7 to 29 points ahead of `hwe` with `auto`. In hexaploids `norm` + alone is 1.4 and 3.1 points ahead of `norm` with `auto`. +- For inbred lines `auto` used no bias under either prior. `norm` is 0.2 + to 0.3 points ahead of `hwe` in tetraploids and 0.6 to 0.7 in + hexaploids. `--prior hwe --selfing-rate 0.95` is 0.15 to 0.18 points + ahead of `norm` in tetraploids and 0.40 to 0.48 in hexaploids. + +Tetraploids at four read depths, from the same simulator and design, with +`fit` as well. The rows at 60 reads are the tetraploid tables above: + +| Tetraploids | Reads | hwe | hwe, `auto` | hwe, `fit` | norm | norm, `auto` | norm, `fit` | +|---|---|---|---|---|---|---|---| +| Random mating | 10 | 77.4% | 77.4% | 76.5% | 77.2% | 77.2% | 76.1% | +| Random mating | 25 | 88.5% | 88.5% | 88.1% | 88.5% | 88.5% | 88.0% | +| Random mating | 60 | 95.0% | 95.0% | 94.9% | 95.0% | 95.0% | 94.9% | +| Random mating | 200 | 98.5% | 98.5% | 98.4% | 98.5% | 98.5% | 98.5% | +| F1 family | 10 | 77.2% | 76.4% | 71.4% | 79.3% | 79.3% | 77.1% | +| F1 family | 25 | 88.8% | 87.9% | 83.4% | 89.9% | 89.9% | 89.0% | +| F1 family | 60 | 95.1% | 94.2% | 89.9% | 95.6% | 95.6% | 95.6% | +| F1 family | 200 | 98.5% | 98.5% | 97.5% | 98.7% | 98.7% | 98.7% | +| Two subpopulations | 10 | 79.1% | 78.8% | 76.5% | 79.3% | 79.3% | 78.2% | +| Two subpopulations | 25 | 89.7% | 89.6% | 88.6% | 89.7% | 89.7% | 89.2% | +| Two subpopulations | 60 | 95.8% | 95.8% | 95.3% | 95.8% | 95.8% | 95.6% | +| Two subpopulations | 200 | 98.7% | 98.7% | 98.6% | 98.7% | 98.7% | 98.7% | +| Half-inbred | 10 | 79.1% | 70.2% | 62.6% | 84.0% | 83.9% | 82.1% | +| Half-inbred | 25 | 85.4% | 84.9% | 86.8% | 90.7% | 90.7% | 91.8% | +| Half-inbred | 60 | 97.0% | 96.6% | 95.2% | 97.3% | 97.3% | 96.7% | +| Half-inbred | 200 | 99.0% | 99.0% | 98.9% | 99.1% | 99.1% | 99.1% | +| Random mating, biased | 10 | 66.7% | 69.1% | 73.0% | 65.7% | 67.5% | 72.1% | +| Random mating, biased | 25 | 78.3% | 84.6% | 86.9% | 77.6% | 83.7% | 86.9% | +| Random mating, biased | 60 | 81.8% | 94.0% | 94.5% | 81.2% | 93.8% | 94.4% | +| Random mating, biased | 200 | 83.6% | 98.0% | 98.2% | 83.0% | 97.9% | 98.2% | +| F1 family, biased | 10 | 69.3% | 69.9% | 70.1% | 70.0% | 71.7% | 74.4% | +| F1 family, biased | 25 | 78.1% | 82.1% | 82.8% | 78.2% | 84.8% | 87.8% | +| F1 family, biased | 60 | 81.3% | 90.8% | 90.4% | 80.6% | 93.4% | 94.4% | +| F1 family, biased | 200 | 82.6% | 96.0% | 95.5% | 81.6% | 97.7% | 97.9% | +| Two subpopulations, biased | 10 | 72.4% | 73.7% | 76.1% | 72.6% | 73.5% | 77.2% | +| Two subpopulations, biased | 25 | 80.8% | 85.7% | 87.5% | 80.8% | 86.0% | 88.2% | +| Two subpopulations, biased | 60 | 86.9% | 94.0% | 94.9% | 86.6% | 94.2% | 95.3% | +| Two subpopulations, biased | 200 | 86.0% | 97.9% | 98.3% | 85.7% | 98.0% | 98.3% | +| Half-inbred, biased | 10 | 73.8% | 64.9% | 61.4% | 81.0% | 81.2% | 81.7% | +| Half-inbred, biased | 25 | 85.1% | 83.9% | 84.6% | 87.7% | 89.1% | 90.8% | +| Half-inbred, biased | 60 | 87.7% | 84.3% | 82.8% | 89.4% | 92.8% | 93.5% | +| Half-inbred, biased | 200 | 91.2% | 97.9% | 98.5% | 91.4% | 98.4% | 98.8% | + +- The prior counts most at low depth. Without a bias on the panel or in + the run, `norm` is ahead of `hwe` in F1 families by 2.0 points at 10 + reads, 1.1 at 25, 0.5 at 60 and 0.1 at 200, and in half-inbred samples + by 5.0, 5.3, 0.3 and 0.1. In one random-mating population and in two + subpopulations the two priors are within 0.2 points at every depth. +- On a biased panel the cost of `norm` with `auto` in one random-mating + population grows as depth falls. It is 0.1 points under `hwe` with + `auto` at 200 reads, 0.2 at 60, 0.9 at 25 and 1.7 at 10. +- A bias takes depth to find. Of what `fit` gained over `off` under `hwe` + in one random-mating population, `auto` kept 39% at 10 reads, 73% at + 25, 96% at 60 and 98% at 200. In two subpopulations it kept 36%, 73%, + 88% and 96%. +- With `norm` on a biased panel, `fit` is ahead of `auto` by 0.5 to 4.6 + points at 10 reads, 1.8 to 3.2 at 25, 0.6 to 1.1 at 60 and 0.2 to 0.4 + at 200. On a panel without bias `fit` with `norm` loses 1.0 to 2.1 + points at 10 reads and 0.6 or less at 60, and `auto` with `norm` loses + 0.1 or less at every depth. + **`MPP` and `PMC`, updog's two summaries of the posterior.** BIGr's `updog2vcf` writes updog's `maxpostprob` for each sample as `MPP` and its `prop_mis` for each marker as `PMC`, and its `filterVCF` filters on both. diff --git a/docs/src/commands/call.md b/docs/src/commands/call.md index 563448e..dfbcf45 100644 --- a/docs/src/commands/call.md +++ b/docs/src/commands/call.md @@ -51,14 +51,21 @@ Two other priors helped under particular conditions and not otherwise: |--------------|----------|-----------------|------------------| | Families (F1, half-sibs) | `--prior norm` | In simulated tetraploid F1 families at 20 to 60 reads, over five draws, it gained 1.0 point (0.6 to 1.5), and 2.5 with `--allele-bias auto` on a biased panel. With it, `auto` kept no bias that was not there. In two F1 populations at about 10 reads, the error was 0.186 against 0.200, and 0.385 against 0.409 | In alfalfa and potato families at high read depth, it ranged from 0.8 points below `hwe` to 0.2 above. In one population it was up to 0.7 below | | Subpopulations at 60 reads or fewer, on a panel without bias | `--prior structure` | In simulated tetraploids it gained 0.8 to 5.5 points at 8 reads, 0.5 to 2.5 at 25, and 0.6 at 20 to 60. In potato clones of ten families thinned to 10 and 25 reads, it gained 0.9 and 0.7 points | The same potato clones at 300 reads showed no gain. In one F1 family it matched `hwe`. `call` refuses to combine it with a fitted bias or a selfing rate, and on a biased panel `norm` with `auto` was 1.5 points ahead of it | +| A mix of these, or unknown | `--prior norm`, with `--allele-bias auto` for polyploids | In simulated tetraploids at 60 reads and hexaploids at 100, beside `hwe` with `auto`, it gained 0.9 to 2.7 points in F1 families and 0.7 to 29 in half-inbred samples, where `hwe` with `auto` did worse than `hwe` alone. In two subpopulations of hexaploids it gained 0.8 to 2.0 | In one random-mating population it was 0.04 to 0.19 points behind in tetraploids and 0.17 to 0.43 in hexaploids. On a biased panel at low depth it fell further behind there, by 0.9 points at 25 reads and 1.7 at 10. Diploids, and two subpopulations of tetraploids, stayed within 0.3 points. For inbred lines `--selfing-rate 0.95` was up to 0.5 points ahead of it | -Three things to know before changing them: +Four things to know before changing them: - **A bias matters only where there are heterozygotes.** It changes only the read fraction that the model expects a heterozygous genotype to show. Diploids, with one heterozygous class, moved by −0.1 to +0.6 points with it, and tetraploid potato gained 4 to 5. Inbred lines gain nothing. `fit` miscalls a quarter of their genotypes, and `auto` leaves them as without a bias and warns (see [Allelic bias](#allelic-bias)). +- **Read depth changes how much each setting matters.** The prior counts most at low depth, where + a sample's reads leave its dosage uncertain. In simulated tetraploid F1 families without bias, + `norm` was 2.0 points ahead of `hwe` at 10 reads, 1.1 at 25, 0.5 at 60 and 0.1 at 200. A bias + takes depth to find. In one random-mating population on a biased panel, `auto` kept 39% of what + `fit` gained at 10 reads, 73% at 25, 96% at 60 and 98% at 200. Without a bias on the panel, + `fit` lost 0.9 points at 10 reads and 0.2 at 60, and `auto` lost none. - **The other priors and the bias cost time.** `norm` and `structure` take two to five times as long as `hwe`, and a fitted bias seven to eight times. - **The QC JSON counts the records called with the bias and those at a search limit.** diff --git a/docs/src/tutorials/genotype-calling.md b/docs/src/tutorials/genotype-calling.md index 73ed2e9..ab8ca10 100644 --- a/docs/src/tutorials/genotype-calling.md +++ b/docs/src/tutorials/genotype-calling.md @@ -96,6 +96,16 @@ Change this when one of these describes your samples: | Families (F1, half-sibs) | Add `--prior norm` | The default prior assumes random mating. It takes two to five times as long | | Hexaploids, or fewer than about 50 reads, on a panel known to be biased | `--allele-bias fit` | `auto` keeps a bias only where it gains enough, which takes depth, while `fit` uses every bias | | Subpopulations at 25 reads or fewer, on a panel without bias | `--prior structure` | Each sample's prior comes from the samples most like it. You cannot combine it with a bias | +| A mix of these, or you do not know | Add `--prior norm` | It fits the spread of dosages at each marker and assumes no mating pattern. It gains in families and partly inbred samples, and costs little in one random-mating population | + +Read depth changes how much these settings matter: + +- **The prior matters most at low depth**, where a sample's reads leave its dosage uncertain. In + simulated tetraploid families `--prior norm` gained 2.0 points at 10 reads, 0.5 at 60 and 0.1 + at 200. +- **A bias takes depth to find.** In simulated random-mating tetraploids on a biased panel, `auto` + kept 39% of what `fit` gained at 10 reads, 73% at 25 and 96% at 60. Where the panel had no + bias, `fit` lost 0.9 points at 10 reads and 0.2 at 60. [Which settings to use](../commands/call.md#which-settings-to-use) has the measurements behind each row.