[Paururo]

*-AMAP.clean.faa is meant to hold no pseudogenes (the outputs page: "drops pseudogenes and any sequence with an internal stop codon"), but export_clean_proteome.py never left a gene out for being one:

1. It took the flag from the first of Is_Pseudogene, pseudogene and is_pseudogene that the master table has. Every master table has Mycobrowser's Is_Pseudogene (generate_tsv_gff_gbk.py merges an empty Mycobrowser frame when Mycobrowser is off), so it read the reference's Mycobrowser flags or nothing, never AMAP's own pseudogene column. This is the column order Count the pseudogenes AMAP called in the QC table #58 fixed in annotation_qc.py.
2. It looked each record up by the first word of its header, which FIX_CDS writes as L4_CDC1551_Rv0001;+;0-1524, among the master table's simple_ids (L4_CDC1551_Rv0001). Nothing ever matched, and its log said "0 pseudogenes".

Only the proteins with an internal stop codon were left out, so the pseudogenes without one stayed in.

Fix

load_pseudogene_ids reads simple_id and pseudogene and returns the genes whose pseudogene is yes, the same test the GFF3 writer uses for pseudo=true, and each record is matched on its ID before the ;. The internal-stop, trailing-stop and empty checks are unchanged. A master table without those columns now stops the export (pandas' "Usecols do not match columns" error) instead of a warning that let every pseudogene through; the master table GET_OUTPUTS writes always has them.

Measured

GET_OUTPUTS' own rendered script (through tests/test_sample_ids.py's run_task) in paururo/amap:latest, on benchmark FIX_CDS outputs with their Bakta tables and an empty emapper table:

Run	Proteins	AMAP's pseudogenes	clean.faa on main	pseudogenes in it	clean.faa now	pseudogenes in it
CDC1551, Mycobrowser on	4014	123	3933	42	3891	0
CDC1551, Mycobrowser off	4014	123	3933	42	3891	0
Sakai against K-12 (the run behind #58's 129)	5323	129	5221	27	5194	0
Sakai, benchmark run	5334	130	5231	27	5204	0

• CDC1551's 42: 20 "frameshift or lost stop codon", 20 "no stop codon in its frame within N bp", 2 "no stop codon, as in the reference". Sakai's 27: 13 and 14 of the first two.
• Every protein with an internal stop in these runs is a pseudogene (81, 102 and 103), so the log now reads Clean proteome: 3891/4014 proteins exported (123 pseudogenes, 0 internal stops, 0 empty).
• In all four runs the clean proteome is main's minus exactly those pseudogenes, in the same order. Its IDs are the GFF3's CDS without pseudo=true, and its proteins are the GenBank's /translations (3891/3891, 5194/5194, 5204/5204).
• Every other GET_OUTPUTS file (master table, GFF3, GenBank, -AMAP.tsv, QC table, panaroo GFF) is byte-identical to main's.
• With Mycobrowser on, CDC1551's master table marks 13 genes Is_Pseudogene=Yes. AMAP calls 7 of them pseudogenes and finds the other 6 intact, and those 6 stay in the clean proteome.

Tests

tests/test_clean_proteome.py (4 tests, each with Mycobrowser on and off) runs the steps after the correction as tests/test_sample_ids.py chains them, built like tests/test_qc_pseudogenes.py. FIX_CDS calls Rv0002 a pseudogene for a frameshift, Rv0003 (which Bakta missed) for an internal stop, with the * in its protein that FIX_CDS writes, and every copy of the novel gene but the first for having no stop within reach; Mycobrowser marks Rv0001, which AMAP finds intact. The tests check that the master table has both columns, that no pseudogene AMAP called is in *-AMAP.clean.faa, that every other protein is there as FIX_CDS wrote it and in its order, and that the clean proteome is the GFF3's CDS without pseudo=true, with the GenBank's translations.

tests/test_classify.py::TestCleanProteome wrote master tables no run writes (simple_id and Is_Pseudogene only, as True/False) and FASTA IDs without ;strand;start-end. They now have the real shape, test_strips_pseudogenes with Mycobrowser saying the opposite of AMAP, and a new test checks that a master table without pseudogene stops the export and writes nothing.

• Main's export_clean_proteome.py mounted over the branch's: 8 failed, 4 passed (the two master-table tests and the two stop-codon unit tests).
• Mutants: Mycobrowser's column with the IDs split, 8 failed; AMAP's column with whole headers, 7 failed; the union of both columns, 3 failed (the Mycobrowser run's two protein tests and test_strips_pseudogenes).
• Container suite: 669 passed, 22 skipped (691) (main: 660 passed, 22 skipped). Python 3.11 with pandas 3.0.6: test_clean_proteome.py and test_classify.py, 190 passed.
• tools/build_docs.py --check and tools/check_site.py pass on Python 3.11.

Docs

The outputs page now says which pseudogenes the clean proteome drops, those with pseudo=true in the GFF3, as it says for the QC count since #58. CHANGELOG bullets under Unreleased (Correctness and Testing, 691 tests), with changelog.html, outputs.html and the search index rebuilt.