Annotating H37Rv against its own reference now gives back every CDS of the AL123456.3 GenBank record at exactly its coordinates, strand and name: 4031 of 4031, against 4020 before. The changes apply to every genome, so results are not directly comparable with 1.0.1.
check_cds.py now reads the reference proteome instead--keep-contig-headers so that its contig names match the FASTA'sdeduplicate_rvs.py deleted every copy of a gene but one, 34 genes on E. coli O157:H7 Sakai, which were then missing from the annotation and from the duplication analysis. Further copies get identifiers of their owncheck_cds.py extended a gene without a stop to the next one in its frame, which after an indel is the next gene's, so that gene was annotated a second time in the same frame (M. bovis Rv0590 over Rv0590A, Rv3373 over Rv3374, Rv2293c over Rv2292c). Such a gene now keeps its extent as a pseudogene, unless the extension only restores residues its reference protein has. ORFs annotated twice fall from 3 to 2 on CDC1551 and from 6 to 3 on M. bovis, and none of those left comes from an extensiontransl_except, and E. coli K-12's GFF3 gives the same 4318 proteins as its GenBank record, 24 of which were missingcount_annot.py filled in a missing category as Unassigned and split it into letters, as it splits JK into J and K, so every gene eggNOG gave no category was counted once in A, D, E, G, I and U and twice in N and S, in *-AMAP.COG.tsv and in the report's COG figures and statistics, where it also shrank every other category's share. A test-profile run with a stand-in for eggNOG that gave none of its 4036 genes a category reported 4036 genes in each of those six and 8072 in N and S. Such a gene now counts once, as NaN, which the report shows as None, and its percentage is of all the genome's genes like every other category'sannotation_qc.py read the first of Is_Pseudogene, pseudogene and is_pseudogene that the master table has, and every master table has Mycobrowser's Is_Pseudogene, empty when Mycobrowser is off. So pseudogene_count in *-AMAP.qc.tsv and the report's QC table was 0 in every organism-agnostic run (E. coli Sakai against K-12, where AMAP called 129 pseudogenes), and in an MTBC run the number of the sample's genes named after one of the 13 that Mycobrowser marks as pseudogenes in H37Rv, whatever AMAP made of them: 13 for CDC1551, where AMAP calls 123 and finds 6 of those 13 intact. It now counts AMAP's own call, the master table's pseudogene column, the same genes the GFF3 marks pseudo=true and the GenBank /pseudocount_annot.py writes a row for each COG or Mycobrowser category a sample has genes in, and the report plotted the rows as they came, so a sample with no gene in a category was left out of that category's figures, of the Dunn tests between lineages run on them, of the overview and of the statistics table, instead of counting 0. Its lineage's box, mean and median were those of the samples that have some, a lineage none of whose samples has any dropped out of the comparison, and the comparison with the median never showed a sample that had lost a category. Each sample now counts 0 in every category another sample of the run has genes in, and the statistics table's Count is every sample of the lineage. A category no sample has keeps its empty figure, and *-AMAP.COG.tsv and *-AMAP.MycoBrowser.tsv are unchangedexport_clean_proteome.py took them from the first of Is_Pseudogene, pseudogene and is_pseudogene that the master table has, which is Mycobrowser's, and looked each protein up by its whole FASTA ID, L4_CDC1551_Rv0001;+;0-1524 where the master table has L4_CDC1551_Rv0001, so it never found one and its log said 0 pseudogenes. Only the proteins with an internal stop codon were left out, and the pseudogenes without one stayed in *-AMAP.clean.faa, which the docs say has none: 42 of the 123 AMAP calls in CDC1551 (a frameshift or lost stop codon, no stop codon within reach, no stop codon as in the reference) and 27 of the 129 in E. coli Sakai against K-12. It now leaves out the genes whose pseudogene is yes, found by the ID before the ;, and on both genomes holds the CDS the GFF3 does not mark pseudo=true, 3891 and 5194, with the proteins the GenBank translates for them. A master table without that column stops it, where a warning let every pseudogene throughcheck_cds.py now reads the reference proteome instead--keep-contig-headers so that its contig names match the FASTA'sdeduplicate_rvs.py deleted every copy of a gene but one, 34 genes on E. coli O157:H7 Sakai, which were then missing from the annotation and from the duplication analysis. Further copies get identifiers of their owncheck_cds.py extended a gene without a stop to the next one in its frame, which after an indel is the next gene's, so that gene was annotated a second time in the same frame (M. bovis Rv0590 over Rv0590A, Rv3373 over Rv3374, Rv2293c over Rv2292c). Such a gene now keeps its extent as a pseudogene, unless the extension only restores residues its reference protein has. ORFs annotated twice fall from 3 to 2 on CDC1551 and from 6 to 3 on M. bovis, and none of those left comes from an extensiontransl_except, and E. coli K-12's GFF3 gives the same 4318 proteins as its GenBank record, 24 of which were missingcount_annot.py filled in a missing category as Unassigned and split it into letters, as it splits JK into J and K, so every gene eggNOG gave no category was counted once in A, D, E, G, I and U and twice in N and S, in *-AMAP.COG.tsv and in the report's COG figures and statistics, where it also shrank every other category's share. A test-profile run with a stand-in for eggNOG that gave none of its 4036 genes a category reported 4036 genes in each of those six and 8072 in N and S. Such a gene now counts once, as NaN, which the report shows as None, and its percentage is of all the genome's genes like every other category'sannotation_qc.py read the first of Is_Pseudogene, pseudogene and is_pseudogene that the master table has, and every master table has Mycobrowser's Is_Pseudogene, empty when Mycobrowser is off. So pseudogene_count in *-AMAP.qc.tsv and the report's QC table was 0 in every organism-agnostic run (E. coli Sakai against K-12, where AMAP called 129 pseudogenes), and in an MTBC run the number of the sample's genes named after one of the 13 that Mycobrowser marks as pseudogenes in H37Rv, whatever AMAP made of them: 13 for CDC1551, where AMAP calls 123 and finds 6 of those 13 intact. It now counts AMAP's own call, the master table's pseudogene column, the same genes the GFF3 marks pseudo=true and the GenBank /pseudoIs_Pseudogene or in AMAP's pseudogene column, lineage by lineage, as "Pseudogenes per Lineage (Mycobrowser annotation)". Is_Pseudogene says which genes are pseudogenes in H37Rv, not in the sample, so CDC1551 came out at 129 where AMAP calls 123 and the QC table says 123, the other 6 being genes Mycobrowser marks that CDC1551 has intact (Rv0947c, Rv1150, Rv2023A, Rv2098c, Rv3021c, Rv3324A), and without Mycobrowser the chart showed AMAP's calls under Mycobrowser's name. It also left out a lineage with none, and added up a lineage's samples, so a lineage with more samples looked as if it had more pseudogenes. REPORT_INPUTS now hands the report every master table row with the sample it is from, and the chart shows how many pseudogenes AMAP called in each sample, by lineage, a sample with none at 0, next to a table of them. The master table explorer shows the note that says why a gene is a pseudogenecount_annot.py writes a row for each COG or Mycobrowser category a sample has genes in, and the report plotted the rows as they came, so a sample with no gene in a category was left out of that category's figures, of the Dunn tests between lineages run on them, of the overview and of the statistics table, instead of counting 0. Its lineage's box, mean and median were those of the samples that have some, a lineage none of whose samples has any dropped out of the comparison, and the comparison with the median never showed a sample that had lost a category. Each sample now counts 0 in every category another sample of the run has genes in, and the statistics table's Count is every sample of the lineage. A category no sample has keeps its empty figure, and *-AMAP.COG.tsv and *-AMAP.MycoBrowser.tsv are unchangedexport_clean_proteome.py took them from the first of Is_Pseudogene, pseudogene and is_pseudogene that the master table has, which is Mycobrowser's, and looked each protein up by its whole FASTA ID, L4_CDC1551_Rv0001;+;0-1524 where the master table has L4_CDC1551_Rv0001, so it never found one and its log said 0 pseudogenes. Only the proteins with an internal stop codon were left out, and the pseudogenes without one stayed in *-AMAP.clean.faa, which the docs say has none: 42 of the 123 AMAP calls in CDC1551 (a frameshift or lost stop codon, no stop codon within reach, no stop codon as in the reference) and 27 of the 129 in E. coli Sakai against K-12. It now leaves out the genes whose pseudogene is yes, found by the ID before the ;, and on both genomes holds the CDS the GFF3 does not mark pseudo=true, 3891 and 5194, with the proteins the GenBank translates for them. A master table without that column stops it, where a warning let every pseudogene throughpseudo=true and Note, /pseudo and /note), and mark the ends a contig cut off (partial=true with start_range or end_range, < or >); the master table and *-AMAP.tsv have them as their last columns/reference from the host, which Docker Desktop refuses and Docker on Linux mounts empty over the container's own. BUNDLED_REFERENCE copies them out first-profile test: VALIDATE_INPUTS still required the sample sheet the test profile does without. 1.0.1 had only been checked with -preview, which runs no task-resume kept results made by older code: Nextflow hashes a task's inputs and script, not the files it runs by path, so after an edit to the classifier or any script under bin/, -resume reused the old results. Every process that runs AMAP code now lists those files in ext code, which puts them in its cache key; edit bin/classify/cases.py and TYPE_CASES and FIX_CDS run again while PATHOTYPR and Bakta stay cachedpathotypr classify, and Nextflow adds any bin/ entry a script names as a bare word to the cache key, so a change in bin/classify/ re-ran PATHOTYPR, Bakta and everything after them. The subcommand is quotednextflow.config, the statements outside the workflow in main.nf, the bare env(SAMPLE_NAME) output and publishDir paths that name a task input. Nextflow 26.10 also refuses the return that ended a --validate_only run. The resource caps are now the resourceLimits directive, which still takes --max_cpus, --max_memory and --max_time from the command line, a profile or a -c config. Nextflow 24.04, where that directive first appeared, is the new minimum--skip_mycobrowser in organism-agnostic mode: a custom reference from another genus turned it on by reassigning the parameter, which current Nextflow ignores with a warning, so the run went ahead with Mycobrowser enabled. It is now worked out in the parameter's defaultbin/classify/__pycache__, and with pathotypr classify naming that directory in PATHOTYPR's script, the first -resume re-ran PATHOTYPR, Bakta and everything after them. Tasks now run with PYTHONDONTWRITEBYTECODE=1--version: --help and the parameter docs have listed it since 1.0.0, but nothing read it, so nextflow run PathoGenOmics-Lab/AMAP --version started a run. It now prints the pipeline and Nextflow versions and exits before any taskCDC1551d, came out as L4_contig_14, and so did every file after it. Two drafts whose first row named the same contig became one sample, the results of one overwriting the other's. A genome is now named <lineage>_<sampleId> from the sample sheet, the lineage of a draft being the one the markers of all its contigs add up to, and the contigs of a draft keep their names instead of the first one taking the sample's. A closed genome whose header is its sample ID comes out as beforeNC_002755.2 Mycobacterium tuberculosis CDC1551, complete sequence) was named with spaces in it, Bakta failed on it three times and the run finished without it, and without failing. Such a genome now takes its sample ID like any other, and so does the test profile's: L4_H37Rv_test, where it was L4_CP110619_H37RvG26689 in L4_G26689.fasta for instance, the input was overwritten with nothing and the run went on without it. The input is now staged in a directory of its ownL4_H37Rv_test was counted as genome H37Rv, the tutorial's Beijing_1 and a Beijing_2 as one genome Beijing in the category counts, the report and the duplication network, an untyped H37Rv_test as lineage H37Rv, and each gene of an untyped sample with no underscore as a genome of its own in the network. PATHOTYPR now says which lineage it named a genome after and every step is handed the sample ID and the lineage. The case summary gains a Sample column and names an untyped sample as it is, not Untyped_<sampleId>, and the report no longer reads the first gene not in the reference as a header, nor stops when every gene has oneVALIDATE_INPUTS staged only the sample sheet and opened the paths written in it from inside its container, which sees only the folder that the work directory and the sheet share. Genomes anywhere else, /data/genomes while running from ~/project for instance, were "FASTA file not found" and the run stopped before anything ran, and a relative path, which the sheet may use, was read from the task directory with or without a container. The genomes are now inputs of the process, each in a directory of its own so that two with the same file name do not collide, and each is read where Nextflow staged itVALIDATE_INPUTS looked for --eggnog_db in its own task directory, where the default eggnog_db, relative to the launch directory, never is, and took a database not built yet for an error, though EGGNOG_DB builds it. It now looks where EGGNOG_DB builds it, staged when it exists, and stops the run only when eggnog_db.ok is there without a complete database, since Nextflow then never runs EGGNOG_DB again--bakta_db reached the tasks as a path the container does not mount. With -profile docker, validation called the database missing and stopped the run, and where nothing validated it, as under -profile test, every sample failed three times and was ignored, and the run ended with nothing annotated and exit 0. A database on the host is now staged into BAKTA_ANNOTATOR, PREPARE_REFERENCE and VALIDATE_INPUTS like any other input; the default inside the container is used as beforedb.sqlite3 that no Bakta database has. It now checks for version.json and bakta.db, which Bakta cannot run without, so their absence stops the run, and names the database's type and version--strict_validation: the documentation said validation warnings could be made to stop the run, but nothing passed the script's --strict on. The new parameter doesEGGNOG_DB built it in a folder it worked out for itself from --eggnog_db, and having no inputs, its container saw that folder only when Nextflow happened to mount it with the work directory and the pipeline's bin/, through a parent folder they share. Otherwise, as when running from /data/project with the pipeline in ~/.nextflow, the database was downloaded and built inside the container and lost with it, and only eggnog_db.ok reached the host: later runs skipped EGGNOG_DB, and ANNOT_EGGNOGMAP_BATCH found no bacteria.dmnd. A database already there without eggnog_db.ok was built again. main.nf now makes the folder on the host and hands it to EGGNOG_DB as an input. A folder left with only eggnog_db.ok needs that file deleted, as validation saysANNOT_EGGNOGMAP_BATCH asked eggNOG-mapper for eggnog_db/bacteria.dmnd and --data_dir eggnog_db, but Nextflow stages a folder under its own name, so with a --eggnog_db called anything else, as the HPC guide's /shared/databases/eggnog or the tutorial's ~/databases/eggnog, EGGNOG_DB built the database and the annotation stopped at "DIAMOND database eggnog_db/bacteria.dmnd not present", with or without a container. The folder is now staged under that one name, whatever it is called. The task's tag and first log line also gave a single sample's FAA size in bytes as the number of samples (batch_1519647_samples under -profile test)BAKTA_ANNOTATOR gave Bakta --ref_annotation, or --bakta_proteins, as a path written into its script, which a container does not see, nor a task directory when it is relative, so the organism-agnostic mode failed under -profile docker: with K-12 as the reference for Sakai, Bakta called the "user proteins file" not valid three times, the sample was ignored and the run ended with exit 0. Likewise --pathotyper_snps and --pathotyper_ref on the host left every genome untyped, as NA_<sampleId>, --mycobrowser_ref left the Mycobrowser categories out, and one --miniprot_ref_* on the host while the others kept their defaults made BUNDLED_REFERENCE fail, so nothing was corrected. A file on the host is now staged into the tasks that read it, each in a directory of its own, and a default is read inside the container as beforeGET_OUTPUTS goes on without it--skip_typing, which every custom reference from another genus needs, it had none and wrote empty files for every sample. It now aligns them onto the sample sheet's genomes, each copied to <sampleId>.fasta first, so that two files of the same name (assembly.fasta from two assemblies) stay apart and a .fna is not passed over-profile test) the section was an empty plot. Every sample is now counted, the section says so when none has any, and each point is drawn at its count, on an axis from 0, where it was jittered up and down by as much as 0.4<name>#0, and the report read the merged matrix with R's default comment character, so its header stopped at the first #. One genome column was dropped and the gene names became row names, so the core/accessory and frequency plots used a single column and the heatmap never appeared. The missing-gene and duplication tables, which carry the reference annotation's gene IDs as given, are now read without a comment character toocomplete (NCBI's "complete genome" and "complete sequence") or circular=true (Unicycler), joins its ends and calls a gene across its origin from its start to its stop, start after end. TYPE_CASES stopped on such a gene: E. coli O157:H7 Sakai with NCBI's headers, whose stcE runs across the origin of plasmid pO157 (92563..2502), failed three times and was left out, and the run finished without failing. Past it, check_cds.py dropped such a gene as invalid coordinates, deduplication and the QC table gave it a negative length, the GFF3 wrote it backwards, the GenBank writer stopped on it and the Panaroo GFF3 spanned it over the whole sequence. It now goes through whole and is written as Bakta writes it: start after end in the tables, join() across the origin in a circular GenBank record, and in the GFF3 its end plus the sequence's length on a region marked Is_circular=true. A gene near the origin of such a sequence is read across it too, so Rop on plasmid pOSAK1, of which Bakta called the first 141 bp, comes out whole. Annotated against K-12, Sakai with NCBI's headers now has 5331 genes in common with the same genome under bare names, which Bakta reads as linear, and differs in stcE and Rop, whole where the linear reading has 804 of stcE's 886 residues and no Rop, and in two phage genes Bakta starts elsewhere on a complete genome<lineage>_<sampleId> and nothing else, so the description Bakta reads a complete, circular replicon from never reached it, whatever the header said, and a gene across the origin of a closed genome came out as two pieces under one name, marked 5' and 3' partial. Only the sequence's ID is replaced now, and the description after it stays, as with --skip_typing. Typed from NCBI's headers, CDC1551, CP110619, AL123456.3 and M. bovis get the same genes as before, in place, name, flags and protein, though Bakta starts a few of its own calls elsewhere on a complete genome (dnaA at 1 instead of 34), so the gene IDs that are Bakta locus tags can be numbered differently. H37Rv turned so that the origin cuts gyrB, gyrA or pknB gives back all 4031 GenBank CDS, where it gave 4030 with the cut gene in two pieces. The GenBank record of such a genome is marked circular, with the description in its definition, and its GFF3 has a region marked Is_circular=true## **48** homologous community pairs detected across all genomes. in a code box. It is now a sentence. The example report's duplication table had columns GET_ALLDUP never writes, so its Network Metrics tab said there were none; the demo data now has the columns GET_ALLDUP writes and the example is rendered againhomologous_communities.tsv: Modularity1 and Modularity2 were computed with the whole community as its only group, which makes them 0 whatever the community, as on all 29 rows of CDC1551 against M. bovis. The two columns are gonehomologous_communities.tsv: with numpy 2, as in the container, Elements1 and Elements2 read {'L': np.int64(4)}, and they counted each protein eggNOG gave no category in category + or -, since its ID keeps the strand FIX_CDS wrote. They are now plain counts, the largest first and ties in alphabetical order rather than in an order that changed from run to run, and a protein without a category counts as -, eggNOG's mark for none-profile: from Nextflow 25.10, whose strict parser applies the profiles in the order -profile names them and lets the last one win, the laptop caps of the container profiles took over from the others: -profile slurm,singularity held every task of a cluster run to the launching machine's CPUs and 16 GB, where -profile singularity,slurm gave it 48 CPUs and 128 GB, -profile test,docker ran the test with 16 GB and -profile highmem,slurm with 128 GB. No profile sets a cap now: they are set after the profiles, from whether highmem, test or a scheduler profile is on, so every order gives what Nextflow 24.04 gave. The parameter docs gave 24 CPUs and 50 GB as the defaults, which no run gothomologous_communities.tsv listed its pairs of communities in the order of a set, which changes with Python's hash seed from one run to the next, and with it which community of a pair came first and the last digit of AvgClustering, an average over the community's proteins that networkx went through in the order of their set: 0.5833333333333333 or 0.5833333333333334 for the same pair of CDC1551 and M. bovis. The pairs now come in the order of the communities, the sample first by name on the left, and the metrics go through the proteins in alphabetical order. Each family in duplications_*_groups.tsv lists its proteins in alphabetical order, where they too came in the order of a set, and each sample's two network drawings no longer start from a random layout and a random partition, so every file both steps write is the same run after runshow(), which also opens the file in a web browser, so a run without a container on a desktop opened a browser tab for every network. They are written as before, without being openedPREPARE_REFERENCE named each reference protein after where its pieces lie, so tagA on plasmid pO157 of E. coli O157:H7 Sakai, join(92527..92721,1..2502) in NCBI's GenBank record, became pO157p01_1-92721, the whole plasmid, and lost its stop; from NCBI's GFF3, which writes it 92527..95223, it came out as its first 65 of 898 residues. Bakta's stcE on the reference, 92563..2502, matched no gene and was listed among the false positives, and the calls were compared with the genes of every sequence at once: seven calls on pO157 and two on the chromosome counted as matched by genes of the other at the same numbers, and the GenBank's 1-92721 hid two more. With Sakai as its own reference, the sample with NCBI's headers came out with tagA as 1..2502, 65 codons short and not marked partial (GenBank), or from Bakta's later start (GFF3), and with bare headers the GFF3 lost the 2502 bp piece altogether. A gene across the origin now ends past the sequence's last base, as GFF3 writes it (pO157p01_92527-95223, with its stop), each protein names its sequence (contig=), Bakta keeps the FASTA's names on the reference, and each call is compared with the genes of its own sequence, on both sides of the origin; calls on a sequence the annotation has no gene on are not judged. Both formats now give the same 5155 proteins and the same 260 false positives, and the sample with NCBI's headers gets tagA exactly, 92527..2502. K-12's 142 false positives are unchangedpipeline_info/params.yml is the run's own: with -resume it came back from the cache, so it held the first run's parameters whatever later runs were given, since its script reads params as a whole, which Nextflow 26.10's default parser leaves out of the task's hash. And all but the first parameter fell outside params:, with a stray YAML line at the end, because Nextflow strips a script's indentation once the script is filled in. DUMP_PARAMS now never comes from the cache, and writes each value as a JSON string under params:--qos=medium under every scheduler, though only SLURM's sbatch takes it, and under -profile sge it took the place of the memory request the sge profile gives every task. The slurm profile now sets it for Bakta--mycobrowser_ref only with the functional annotation, though the interactive report reads it in every run, so under --skip_functional_annotation a wrong path went unwarned and the report came out without Mycobrowser's categoriesenvironment.yml pins eggNOG-mapper 2.1.13, the version the container ships, where it said 2.1.12main.nf's processes, and the four cases, the missing genes and the harder cases (paralogs, small proteins, contig ends, frameshifts, copies) are drawn on genes of CDC1551, M. bovis and E. coli from the benchmark. The tutorial and user guide stay markdown, readable on GitHub, and tools/build_docs.py turns them into pages of the same site, dark by defaulttools/check_site.py runs every drawn window through bin/, compares the thresholds on the pages with the constants in the code, the metro map with main.nf and every option with the pipeline's parameters, so a change that makes the site wrong fails the docs workflow. tools/figure_data.py redraws the windows from a benchmark run--skip_functional_annotation a run still writes AMAP/, with the corrected table and sequences, where the outputs page said it did not; no step writes PATHOTYPR/renamed_fastas.tsv; and the 19-residue toxin the algorithm page names is IbsC (b4665), not an ldr toxintests/benchmark/README.md. A shared link to any page shows a card drawn from the front page's genes--skip_functional_annotation leaves no GFF3 or GenBank, which come after eggNOG; tasks are capped at 16 GB on one machine and at 128 GB under a scheduler profile, not 50 GB; and lineage typing is not skipped on its own for other species, only Mycobrowser istools/pictures.mjs from the front page's own figure. The README's metro map gains the interactive report's stationmain.nf and the interactive report the concept DOI, which always resolves to the latest version. All of them cite the concept DOI now-process.clusterOptionstests/selftest/h37rv_selftest.sh annotates H37Rv with itself inside the container and fails unless every GenBank CDS comes back exactlytests/test_cache_keys.py fails when a process runs AMAP code that is not in its cache key, or when its script names a bin/ entry as a bare word without declaring ittests/benchmark/benchmark.sh scores the correction algorithm on seven genomes whose right answer is known (H37Rv twice, CDC1551 complete and in 40 contigs, M. bovis, E. coli K-12 and Sakai), downloading them and running Bakta and miniprot in the container the first time, and lists every gene that changed since an earlier run. A gene whose sequence changed counts as found when a gene with its name covers half of its own bases there, not half of the stretch it lands on, which an insertion can make far longer (63 kb with a prophage inside for b1004 in Sakai); a gene mostly deleted is not counted. Changed genes found: 50 of 50 in CP110619, 860 of 862 in CDC1551, 1515 of 1515 in M. bovis and 3363 of 3367 in Sakai, and the README says why each of the other six is nottests/test_name_by_lineage.py runs PATHOTYPR's own script, rendered as Nextflow renders it, with a stand-in for pathotypr that writes the summaries pathotypr wrote for CDC1551 complete and in 40 contigs, and fails unless Bakta would read a renamed genome as circular exactly when it reads the original sotests/test_sample_ids.py runs TYPE_CASES, GET_OUTPUTS, GET_COUNT, GET_DUPLICATIONS, REPORT_INPUTS and GET_ALLDUP as Nextflow renders them, chained as the pipeline chains them, for a typed and an untyped run whose sample IDs hold underscores, and then the report's own R code where R is installed, and FINAL_REPORT's own script on a run's tables and on one that leaves every table it can emptytests/test_validate_inputs.py runs VALIDATE_INPUTS's own script, rendered as Nextflow renders it, in a task directory laid out as Nextflow lays one out: genomes in a folder the task cannot see, relative paths, two genomes with the same file name and a sheet of 500 samples; and checks both databases, eggNOG's in each state EGGNOG_DB can leave it intests/test_eggnog_db.py runs EGGNOG_DB's own script, rendered as Nextflow renders it, with its folder staged as Nextflow stages an input and stand-ins for eggNOG-mapper's download and build: the database is built in that folder whatever it is called, one already there is used as it is, and a short bacteria.dmnd is built againtests/test_eggnog_annotation.py runs ANNOT_EGGNOGMAP_BATCH's own script the same way, with a stand-in for emapper.py that looks for the database as eggNOG-mapper does: the database is found whatever its folder is called, a folder called split, where the script writes each sample's annotations, is left as it was, and the tag and log count a single sample as onetests/test_reference_files.py runs the scripts of BAKTA_ANNOTATOR, PATHOTYPR, GET_OUTPUTS and BUNDLED_REFERENCE, rendered as Nextflow renders them, their own Groovy included, with stand-ins for Bakta and pathotypr and each reference file on the host, under a relative path or inside the container; and fails when a process reads a reference file's param other than as what it falls back on when nothing is staged; and checks the Mycobrowser table whenever a step reads ittests/test_pangene_genomes.py runs PANGENE_GENOME and PANGENE's scripts as Nextflow renders them, with stand-ins for miniprot and pangene, and fails unless each sample is aligned onto every other sample's genome and never onto its own, typed or nottests/test_report_tables.py merges two Rtabs as PANGENE writes them and checks that the report reads every table a # can reach without a comment character; where R is installed, as in the container, it also runs the report's pangenome code on the merged matrixtests/test_cog_counts.py counts COG categories as count_annot.py does, for genes in none, in two and in lowercase, then runs the steps after the correction as tests/test_sample_ids.py chains them, with eggNOG giving some genes no category, then no gene one, then some samples no gene in a category another sample has genes in, and reads what the report's own overview, per-COG figures and statistics table show where R is installed: each category in a figure of its own, with every sample, at 0 where it has no gene. The Mycobrowser figures, statistics and ratios to the median are read the same way for a sample with no gene in one of their categoriestests/test_duplication_report.py runs GET_DUPLICATIONS and GET_ALLDUP as Nextflow renders them, chained as the pipeline chains them, with eggNOG giving one copy of a duplicated gene a category, another none and the last no row at all, and checks the columns and COG counts of homologous_communities.tsv; where R is installed, it knits the report's duplication summary on that table and on the one GET_ALLDUP writes when nothing is duplicated. It also fails when a chunk of the report prints markdown without passing it on as markdowntests/test_profile_caps.py fails when a profile sets a cap, or when the order of -profile changes one: it reads nextflow.config and applies its profiles in every order, as both of Nextflow's parsers do, and with Nextflow on PATH also asks nextflow configtests/test_qc_pseudogenes.py runs the steps after the correction as tests/test_sample_ids.py chains them, with FIX_CDS calling some genes pseudogenes and Mycobrowser marking another, with Mycobrowser on and off, and checks that *-AMAP.qc.tsv, REPORT_INPUTS and the report's own QC table count the pseudogenes AMAP called, as many as the GFF3 and the GenBank marktests/test_duplication_reproducible.py runs get_duplications.py and network_genomes.py under three hash seeds, on the Diamond tables of three genomes with a family whose average clustering changes with the order of its proteins, and fails unless every file they write is the same under all three; it also fails when either script opens a web browser, with hits or without, as a stand-in browser that records what it is asked to open tells, and when a network page asks a CDN for its drawing librarytests/test_custom_reference.py extracts a reference whose plasmid has a gene across its origin from a GenBank record and from GFF3 written either way, on either strand, compares Bakta's calls with it sequence by sequence, and runs PREPARE_REFERENCE's own script, rendered as Nextflow renders it, with a stand-in for Bakta that names sequences as Bakta doestests/test_amap_report.py writes the report from four genomes of a real run cut to two stretches of the reference (tests/data/report/, rebuilt by its make_fixture.py), where M. bovis lacks RD4 and a contig end cuts dnaE2, plus a gene across the origin of a circular plasmid, and checks that every gene, and every Bakta call AMAP did not keep, comes back out of the page as it went in, with Python and, where node is installed, with the page's own JavaScript, along with the states, the flags, the correction's steps and what the process hands the scripttests/test_dump_params.py runs DUMP_PARAMS with Nextflow where it is installed, under both parsers, with values holding quotes, $, backticks, a backslash and a line break, and again with -resume and one value changed; everywhere, it fails if the step can come from the cachetests/test_cluster_options.py reads which profile gives Bakta the QoS, and with Nextflow resolves Bakta's and every task's cluster options for each scheduler, in both orders with a container and under both parserstests/test_environment_pins.py fails when environment.yml pins a tool at a version docker/Dockerfile does not installtests/test_citation.py fails when anything cites AMAP with a DOI other than the concept DOI of the manifesttests/test_clean_proteome.py runs the steps after the correction as tests/test_sample_ids.py chains them, with FIX_CDS calling some genes pseudogenes, one with a stop codon in its protein and the others without, and Mycobrowser marking a gene AMAP finds intact, with Mycobrowser on and off, and checks that *-AMAP.clean.faa holds every protein FIX_CDS wrote but the pseudogenes', as it wrote them: the CDS the GFF3 does not mark pseudo=true, with the GenBank's translationstests/selftest/h37rv_selftest.sh annotates H37Rv with itself inside the container and fails unless every GenBank CDS comes back exactlytests/test_cache_keys.py fails when a process runs AMAP code that is not in its cache key, or when its script names a bin/ entry as a bare word without declaring ittests/benchmark/benchmark.sh scores the correction algorithm on seven genomes whose right answer is known (H37Rv twice, CDC1551 complete and in 40 contigs, M. bovis, E. coli K-12 and Sakai), downloading them and running Bakta and miniprot in the container the first time, and lists every gene that changed since an earlier run. A gene whose sequence changed counts as found when a gene with its name covers half of its own bases there, not half of the stretch it lands on, which an insertion can make far longer (63 kb with a prophage inside for b1004 in Sakai); a gene mostly deleted is not counted. Changed genes found: 50 of 50 in CP110619, 860 of 862 in CDC1551, 1515 of 1515 in M. bovis and 3363 of 3367 in Sakai, and the README says why each of the other six is nottests/test_name_by_lineage.py runs PATHOTYPR's own script, rendered as Nextflow renders it, with a stand-in for pathotypr that writes the summaries pathotypr wrote for CDC1551 complete and in 40 contigs, and fails unless Bakta would read a renamed genome as circular exactly when it reads the original sotests/test_sample_ids.py runs TYPE_CASES, GET_OUTPUTS, GET_COUNT, GET_DUPLICATIONS, REPORT_INPUTS and GET_ALLDUP as Nextflow renders them, chained as the pipeline chains them, for a typed and an untyped run whose sample IDs hold underscores, and then the report's own R code where R is installed, and FINAL_REPORT's own script on a run's tables and on one that leaves every table it can emptytests/test_validate_inputs.py runs VALIDATE_INPUTS's own script, rendered as Nextflow renders it, in a task directory laid out as Nextflow lays one out: genomes in a folder the task cannot see, relative paths, two genomes with the same file name and a sheet of 500 samples; and checks both databases, eggNOG's in each state EGGNOG_DB can leave it intests/test_eggnog_db.py runs EGGNOG_DB's own script, rendered as Nextflow renders it, with its folder staged as Nextflow stages an input and stand-ins for eggNOG-mapper's download and build: the database is built in that folder whatever it is called, one already there is used as it is, and a short bacteria.dmnd is built againtests/test_eggnog_annotation.py runs ANNOT_EGGNOGMAP_BATCH's own script the same way, with a stand-in for emapper.py that looks for the database as eggNOG-mapper does: the database is found whatever its folder is called, a folder called split, where the script writes each sample's annotations, is left as it was, and the tag and log count a single sample as onetests/test_reference_files.py runs the scripts of BAKTA_ANNOTATOR, PATHOTYPR, GET_OUTPUTS and BUNDLED_REFERENCE, rendered as Nextflow renders them, their own Groovy included, with stand-ins for Bakta and pathotypr and each reference file on the host, under a relative path or inside the container; and fails when a process reads a reference file's param other than as what it falls back on when nothing is staged; and checks the Mycobrowser table whenever a step reads ittests/test_pangene_genomes.py runs PANGENE_GENOME and PANGENE's scripts as Nextflow renders them, with stand-ins for miniprot and pangene, and fails unless each sample is aligned onto every other sample's genome and never onto its own, typed or nottests/test_report_tables.py merges two Rtabs as PANGENE writes them and checks that the report reads every table a # can reach without a comment character; where R is installed, as in the container, it also runs the report's pangenome code on the merged matrixtests/test_cog_counts.py counts COG categories as count_annot.py does, for genes in none, in two and in lowercase, then runs the steps after the correction as tests/test_sample_ids.py chains them, with eggNOG giving some genes no category, then no gene one, then some samples no gene in a category another sample has genes in, and reads what the report's own overview, per-COG figures and statistics table show where R is installed: each category in a figure of its own, with every sample, at 0 where it has no gene. The Mycobrowser figures, statistics and ratios to the median are read the same way for a sample with no gene in one of their categoriestests/test_duplication_report.py runs GET_DUPLICATIONS and GET_ALLDUP as Nextflow renders them, chained as the pipeline chains them, with eggNOG giving one copy of a duplicated gene a category, another none and the last no row at all, and checks the columns and COG counts of homologous_communities.tsv; where R is installed, it knits the report's duplication summary on that table and on the one GET_ALLDUP writes when nothing is duplicated. It also fails when a chunk of the report prints markdown without passing it on as markdowntests/test_profile_caps.py fails when a profile sets a cap, or when the order of -profile changes one: it reads nextflow.config and applies its profiles in every order, as both of Nextflow's parsers do, and with Nextflow on PATH also asks nextflow configtests/test_qc_pseudogenes.py runs the steps after the correction as tests/test_sample_ids.py chains them, with FIX_CDS calling some genes pseudogenes and Mycobrowser marking another, with Mycobrowser on and off, and checks that *-AMAP.qc.tsv, REPORT_INPUTS and the report's own QC table count the pseudogenes AMAP called, as many as the GFF3 and the GenBank marktests/test_duplication_reproducible.py runs get_duplications.py and network_genomes.py under three hash seeds, on the Diamond tables of three genomes with a family whose average clustering changes with the order of its proteins, and fails unless every file they write is the same under all three; it also fails when either script opens a web browser, with hits or without, as a stand-in browser that records what it is asked to open tells, and when a network page asks a CDN for its drawing librarytests/test_custom_reference.py extracts a reference whose plasmid has a gene across its origin from a GenBank record and from GFF3 written either way, on either strand, compares Bakta's calls with it sequence by sequence, and runs PREPARE_REFERENCE's own script, rendered as Nextflow renders it, with a stand-in for Bakta that names sequences as Bakta doestests/test_amap_report.py writes the report from four genomes of a real run cut to two stretches of the reference (tests/data/report/, rebuilt by its make_fixture.py), where M. bovis lacks RD4 and a contig end cuts dnaE2, plus a gene across the origin of a circular plasmid, and checks that every gene, and every Bakta call AMAP did not keep, comes back out of the page as it went in, with Python and, where node is installed, with the page's own JavaScript, along with the states, the flags, the correction's steps and what the process hands the scripttests/test_dump_params.py runs DUMP_PARAMS with Nextflow where it is installed, under both parsers, with values holding quotes, $, backticks, a backslash and a line break, and again with -resume and one value changed; everywhere, it fails if the step can come from the cachetests/test_cluster_options.py reads which profile gives Bakta the QoS, and with Nextflow resolves Bakta's and every task's cluster options for each scheduler, in both orders with a container and under both parserstests/test_environment_pins.py fails when environment.yml pins a tool at a version docker/Dockerfile does not installtests/test_citation.py fails when anything cites AMAP with a DOI other than the concept DOI of the manifesttests/test_clean_proteome.py runs the steps after the correction as tests/test_sample_ids.py chains them, with FIX_CDS calling some genes pseudogenes, one with a stop codon in its protein and the others without, and Mycobrowser marking a gene AMAP finds intact, with Mycobrowser on and off, and checks that *-AMAP.clean.faa holds every protein FIX_CDS wrote but the pseudogenes', as it wrote them: the CDS the GFF3 does not mark pseudo=true, with the GenBank's translationstests/test_report_pseudogenes.py runs the steps after the correction as tests/test_sample_ids.py chains them, with FIX_CDS calling a different number of pseudogenes in each sample, none in one, and Mycobrowser marking a gene that every sample has intact, with Mycobrowser on and off, and checks that REPORT_INPUTS hands over each master table row with its sample; where R is installed, it runs the report's own pseudogene section on what REPORT_INPUTS made and checks that it draws each sample at the number AMAP called, the QC table's, a sample with none at 0, under a title that does not name MycobrowserBug fixes, most of them found by running the pipeline on real genomes rather than by reading it. Several changed what the pipeline outputs, so results from 1.0.0 are not directly comparable.
COG categories from eggNOG, Mycobrowser functional categories for MTBC, and a comparison against the reference distribution. Each has its own per-category breakdown in counts and percentages, with the statistical tests alongside.
-PE/PPE gene families get their own section, since they are both the hardest part of an MTBC genome to annotate and usually the most interesting. The pseudogenes of each lineage close it.
+PE/PPE gene families get their own section, since they are both the hardest part of an MTBC genome to annotate and usually the most interesting. The pseudogenes close it: how many AMAP called in each sample, lineage by lineage, a sample with none at 0, the same genes the QC table counts and the GFF3 marks pseudo=true. The master table's note column says why each is a pseudogene.
The most represented metabolic pathways, and how they differ between lineages.