Repository navigation
Fix long-standing LfMerge writing system bugs - #361
Merged
Merged
Conversation
Collaborator
Author
|
Only one project in production could end up with data issues after this PR is merged, because it did the equivalent of having both EDIT: I believe that project can be taken off of hold status now, because the new logic won't cause those data issues. But I'll verify, then remove this comment if it's no longer accurate. |
Collaborator
Author
|
Currently having Claude do code review. Will post results when completed. |
LF stores a writing system tag as it was typed, but LCM canonicalizes a tag when it creates the writing system and thereafter knows it only by that canonical id. TryGet and GetWsFromStr both match the id literally (they are case-insensitive, but not structure-aware), so a tag whose canonical form differs structurally is never found: qaa-x-qaa-v -> qaa-x-v (redundant private-use "qaa" absorbed) th-Thai -> th (suppress-script dropped) LfWsToLcmWs then takes its else branch and calls Create() -- which DOES canonicalize -- so Set() collides with the writing system already there: Unable to set writing system 'qaa-x-kal' because this id already exists. The second test covers the quieter half: LfMultiText resolves each multitext key with GetWsFromStr and skips anything returning 0, so a value keyed by a non-canonical tag is dropped, and the clearing pass then blanks whatever LCM had for that field. testlangproj already contains qaa-x-kal and qaa-Zxxx-x-kal-audio, so these use a non-canonical spelling of an existing writing system rather than adding to the shared fixture project. The qaa-Zxxx-x-kal-AUDIO case already passes (case-only difference) and is a guard against regressing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LF stores a writing system tag exactly as it was typed; LCM canonicalizes it when the writing system is created and thereafter knows it only by that canonical id. GetWsFromStr matches that id literally -- case-insensitively, but not structure-aware -- so a multitext keyed by a tag whose canonical form differs structurally resolves to 0: qaa-x-qaa-v -> qaa-x-v (redundant private-use "qaa" absorbed) th-Thai -> th (suppress-script dropped) WriteToLcmMultiString skips a key that resolves to 0, and its clearing pass then blanks whatever LCM held for that writing system, so the value is not merely dropped on the way in, it is deleted on the way out. Canonicalize the key before the lookup, here and in WsIdAndFirstNonEmptyString. LanguageTags.Canonical guards IetfLanguageTag.Canonicalize with IsValid, which it needs: Canonicalize throws on a tag it cannot parse, and a corpus scan found 48 such strings (custom field names that look like multitext keys). Verified against SIL.WritingSystems: Canonicalize agrees with the id liblcm itself stores for all 1251 valid tags in that corpus, and is idempotent on all of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TryGet matched the writing system id literally while Create canonicalized it, so a tag whose canonical form differs -- "qaa-x-qaa-v" becomes "qaa-x-v", because the private-use section repeats the "qaa" language subtag -- was not found, was created under its canonical id, and collided with the writing system already there: Unable to set writing system 'qaa-x-v' because this id already exists. GetOrSet does the lookup and the creation in one step and canonicalizes for both, so the two halves can no longer disagree. It returns true when it found an existing writing system and false when it created one, and a created one is already in the manager, so the separate Set call goes away and the two branches collapse into one path plus a conditional vernacular/analysis assignment. This implements only half of the TODO it replaces. The other half -- "use the ILgWritingSystemFactory interface and remove one point of FW 8-to-9 API incompatibility" -- does not follow: GetOrSet is declared on WritingSystemManager, not on ILgWritingSystemFactory, which offers only get_Engine, GetWsFromStr, GetStrFromWs, GetIcuLocaleFromWs and GetWritingSystems. The #if FW8_COMPAT blocks therefore stay, and the comment now records why rather than leaving a TODO that cannot be done as written. The vernacular comparison keeps comparing the tag as LF spells it against languageCode as LF spells it. Both are LF's own strings, so a project whose languageCode is itself non-canonical still matches its own input system. Braces added to the single-statement Abbreviation guard, which reads as indentation-scoped without them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LF has no vernacular/analysis flag on a writing system; the distinction is conventional, encoded in which config fields use it. LfWsToLcmWs treated exactly one writing system as vernacular -- whichever equalled the project's languageCode -- and filed every other one as analysis. A project whose lexeme field uses more than one writing system therefore lost the distinction: apltw2016's lexeme field uses qaa-x-IPA-dupl1 alongside th, and the phonetic one was made analysis. Take the vernacular writing systems from the fields that are vernacular by convention instead: citationForm, lexeme, etymology, senses.fields.examples.fields.sentence That list was already in MongoConnection, which applies the same convention in the other direction when it writes input systems back out to the config. Rather than have a second copy to keep in step by hand, it moves to MagicStrings.LfVernacularConfigFieldPaths and both directions read it from there. MongoConnection keeps its field name, so its six use sites are unchanged; the type widens to IReadOnlyList so the shared instance is not handed out as mutable. The example sentence path is nested, so the paths are walked a segment at a time. They are spelled as Mongo sees them, where each level of nesting goes through a "fields" document; in the mapped classes that is the Fields dictionary being indexed, so those segments are skipped. Where the config names no vernacular input systems at all, the project's language code is still used, so a project with a config LfMerge cannot read behaves as it did before. Tags are compared case-insensitively, matching LCM's own writing system lookups. This replaces the "needs to be a bit fuzzier" TODO in that method. Note the assignment still only happens for a writing system LfMerge has just created; one that already exists in the FieldWorks project keeps whatever vernacular/analysis membership it has there. That is unchanged behaviour, and it is why the new tests cover the derivation directly: creating a writing system in the shared test project cannot be undone afterwards, since WritingSystemManager has no removal API. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some older projects had a missing or null DirtySR field, which was causing Mongo to throw an error because we never told it to treat null values as 0 for that int field. It's a simple one-line fix to treat missing values as 0, but treating null values as 0 requires a custom serialization helper. Still simple enough, thankfully.
The previous commit took the vernacular writing systems from a fixed list of config fields -- citationForm, lexeme, etymology and the example sentence -- shared with MongoConnection. Checking that against the 2026-07-06 corpus showed one list cannot be right for every project: 741 projects configure etymology with analysis writing systems and only 378 with vernacular ones, so naming etymology vernacular adds "en" to the vernacular list of hundreds of projects that have no English headwords. Classify per project from its own config instead. The lexeme and citation form fields are vernacular by definition and the sense definition and gloss are analysis by definition, so they anchor the decision; every other field carrying input systems -- etymology, the example sentence, custom fields -- is assigned by which anchor its writing systems already appear in. Only a writing system exclusive to one anchor counts as evidence: "en" is in both in many projects and so says nothing about any other field's role. This keeps what the fixed list already got right over the original languageCode rule. flh-flex has languageCode "flh-x-ortho", a writing system only its etymology field uses, so its real lexeme writing system "flh-x-cm" was classified as analysis; khg-yl-flex has the bare tag "khg" while its lexeme uses "khg-CN-x-Yangla" and "khg-fonipa"; odo has languageCode "th", which is not among its writing systems at all, leaving its 24,460 "en" headwords as analysis and the project with no vernacular writing system whatsoever. All three still come out right. Writing systems that overlap neither anchor are treated as vernacular and reported. 11 projects need this; grc-vie-flex is the clearest, a Koine Greek dictionary glossed in English and Vietnamese whose etymologies are in Hebrew and Aramaic -- neither vernacular nor analysis. The FieldWorks fields those feed are vernacular-typed, so calling them analysis would leave the data unreachable, whereas a spare vernacular writing system is inert. Since the two directions no longer share a rule, MongoConnection gets its own list back and MagicStrings.LfVernacularConfigFieldPaths goes away. MongoConnection answers a different question -- which config fields to write the vernacular writing systems into -- and that divergence is now deliberate rather than drift. languageCode survives as a last-resort fallback, for a config LfMerge cannot read at all: without it such a project would end up with no vernacular writing system, which is worse than the rule this replaces. Note this only affects writing systems LfWsToLcmWs creates. A project already synced under an earlier rule keeps the membership its FieldWorks project has. The tests move with the logic, keeping their coverage of the union, case-insensitive matching, the nested example sentence path, apltw2016's second lexeme writing system and the fallbacks. Two of them had described configs that do not match how Mongo spells them, with definition and gloss at the entry level rather than under senses; the anchor paths are strict about that, so those configs are corrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SetCustomFieldData had no case for CellarPropertyType.MultiUnicode, so every multilingual text custom field fell through to `default: return false` and its data was dropped. The other direction has handled MultiUnicode all along (ConvertLcmToMongoCustomField line 363), so LfMerge could read one of these fields out of FieldWorks but never write one back in. That asymmetry is invisible in a log. The name lookup succeeds -- the field exists in LCM, so it is not among the remainingFieldNames that produce "Custom field X from LF skipped" -- and the default case returned false without saying anything, carrying a TODO asking for exactly the warning it never got. Migrating thai_jay is what turned it up: its "Top 10 list" field arrived in FieldWorks present but empty for all 537 entries, with nothing in 10,801 lines of LfMerge output to explain why. The comment above the switch is the likely origin, since it lists the types to implement and leaves MultiUnicode out; it is wrong, and is corrected here. This matters more than a missing type usually would, because MultiUnicode is the type every text custom field in a Language Forge project has. LF has never offered a single-string custom field, so nothing LF owns arrives as CellarPropertyType.String -- the one text type that did work. Alternatives LF does not have are cleared rather than left in place. That is safe here specifically because the export writes every alternative LCM holds, via LfMultiText.FromMultiITsString: an alternative missing from LF is one a user deleted there, not one LF never saw. It also matches what LfMultiText.WriteToLcmMultiString does for the built-in multitext fields. Values that have not changed are left untouched, as the String case does, so they stay out of the .fwdata XML and out of the Mercurial commit. MultiString is deliberately not added. GetCustomFieldData cannot produce one either, so an import path for it would be untested code for a shape LfMerge never emits; it now reaches the new warning instead of disappearing. The default case logs a warning, which is what the TODO asked for and what the LCM-to-Mongo direction has always done. A field type going unimplemented is worth a line in the log rather than silence. Tests cover the four cases against the test project's Cust_Single_Line_All, which really is MultiUnicode with English and French alternatives: both alternatives written, a previously emptied field written (the migration case), an alternative LF dropped cleared rather than left stale, and an unidentified writing system skipped. All four fail on the unmodified converter, each showing the field still holding its original "Some custom text". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude determined that this warning is no longer needed; there are indeed some LF fields (usually on the sense level) that have empty but non-null multitexts, but they truly are empty and there is no loss of data involved, so no need for a spammy log warning.
The MultiUnicode case added to SetCustomFieldData looked each LF key up with GetWsFromStr as LF spells it. LCM knows a writing system only by its canonical id, so a non-canonical key such as "fr-Latn" (canonically "fr") resolved to 0 and was skipped as unidentified. The clearing pass then blanked that writing system's alternative, since nothing from LF had claimed it: the LF value was not merely dropped, the LCM value was deleted. This is the same bug 0c70604 fixed in LfMultiText.WriteToLcmMultiString for the built-in multitext fields; canonicalize the key here the same way. The new test fails without the fix, French having been cleared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ClassifyVernacularWritingSystems returned only a vernacular set, and LfWsToLcmWs filed every new writing system in exactly one list: vernacular if the set held it, analysis otherwise. That got three shapes wrong. A field with a vernacular-only writing system had ALL its writing systems made vernacular, including ones exclusive to the analysis anchors: an etymology configured as [seh, pt] made Portuguese vernacular-only, so a new pt writing system was never added as analysis and pt glosses had nowhere to go. The field's evidence now decides the role of the field, not of every writing system in it; an analysis-only writing system stays analysis. A writing system in both anchors -- "en" in lexeme and gloss alike -- was vernacular, and so never analysis. The classification now returns an analysis set as well, and a new writing system is added to every list it belongs in. A writing system in a field overlapping neither anchor was made vernacular only, on the reasoning that a spare vernacular writing system is inert. It is not: a French-only notes custom field left French out of the analysis list, so an analysis-typed FieldWorks field would not offer it. With no evidence either way such a writing system is now both, which serves vernacular-typed fields (the etymology form, the example sentence) and analysis-typed ones alike. The integration test fails without the LfWsToLcmWs change, the new writing system missing from the current analysis list. This commit also removes the BOM from ConvertMongoToLcmLexicon.cs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
0c70604 and c91a6c7 canonicalized LF's tags at three GetWsFromStr call sites, but the other places that resolve a tag LF supplied still matched it literally, so each had the non-canonical-tag bug those commits fixed: - BestStringAndWsFromMultiText compared LF's keys with ws.Id. A key such as "fr-Latn" never matched French, so a single-string lexicon field took the text of whichever writing system came next in the search order, or the first key in the document, instead of the preferred one. - The String custom field case resolved the key it picked literally, and gave up on the whole field when that returned 0, so the LF edit never reached LCM. - A span's lang attribute resolved to 0, and building the TsString then threw ArgumentOutOfRangeException ("The specified writing system code is invalid"). - A multi-paragraph's input system and the option list item writing system were resolved literally too. LanguageTags.WsIdFromLfTag is now the one way to resolve an LF tag: the canonical tag first, then the tag as LF spells it, for a writing system LCM might hold under a non-canonical id. Keys are still resolved at lookup rather than rewritten when the multitext is read in, since LF shows text under the tags its own config spells, and rewritten keys would not be shown. LfMultiText gains BestStringAndWsId, which resolves each key to a handle before comparing it with the search order, and whose fallback skips values in writing systems LCM does not know. BestStringAndWsFromMultiText and the String custom field case both use it. The MultiUnicode custom field case re-implemented WriteToLcmMultiString, and the two copies had already drifted: that is how the copy came to lack the canonical lookup that c91a6c7 added. Both now call LfMultiText.WriteToLcm, which reaches LCM through delegates, since a custom field has only ISilDataAccess and not an IMultiAccessorBase. The built-in fields gain two behaviours the custom field copy had: an unchanged alternative is left alone, keeping it out of the .fwdata XML and the Mercurial commit, and a null value is treated as empty rather than throwing. The String custom field and span tests fail without the fix; the others test the new methods directly. All 219 LCM tests pass. The 21 Mercurial and synchronize tests that fail do so because hg init and hg serve cannot run in this environment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ClassifyVernacularWritingSystems decides whether a writing system is vernacular
or analysis from the project config alone, and the config is weaker evidence
than it looks: Language Forge routinely offers every input system in every
field, so a tag configured for the lexeme field may have no headword in it
anywhere. Scanning the 2026-07-06 corpus, 750 (project, writing system) pairs
are configured somewhere but hold no text at all.
Counting the text says what a writing system is really being used for. This
adds the counting pass on its own; nothing consumes it yet.
Fields are keyed by the dotted path the config spells them with, nesting through
"fields" ("senses.fields.examples.fields.sentence"), so the counts line up with
the paths classification already works in. Every built-in multitext field is
counted, not just the ones classification consults today.
Deleted entries are skipped: they are tombstones Language Forge keeps so the
deletion can be replayed, and their text is not evidence that anyone is using
that writing system. Whitespace counts as empty. Tags are matched
case-insensitively, as every LCM writing system lookup is, because the config
and the lexicon do not always agree on the case of a tag.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The config says which writing systems a field OFFERS, and Language Forge routinely offers every input system in every field. ClassifyVernacularWriting- Systems believed it, so a project whose lexeme field lists "en" alongside the vernacular got "en" classified vernacular even when not one headword is written in it. That is the shape behind most of the corpus's unresolved writing systems: 168 of the 323 have lexeme and gloss offering exactly the same input systems, so neither anchor has any exclusive tag and every other field becomes unresolved. Where the lexicon has text to show for a writing system, that text now decides, and the config only answers for writing systems with nothing written in them. By share, not by presence. A few strays are not evidence of a second role: in brb-flex-2022 the phonetic writing system has 4,030 headwords and one stray string in an example reference, and FieldWorks has it as vernacular only, while in cmo-khm-flex the vernacular has 7,815 vernacular strings against 300 in glosses and is likewise vernacular only. The one writing system those projects really do list as both sits at 28%, so 10% is a round number in the middle of the gap. What no text can settle, language often can. A writing system nothing else has placed still has a language, and the other writing systems spelling that language have usually been placed already: a phonetic or audio spelling of the vernacular is vernacular too, and a regional spelling of the analysis language is analysis. The language of a tag, for this purpose, is its language subtag plus any private-use subtags that name the language rather than a variant of it. Script and region come out, so "en" and "en-GB" are one language. The private-use subtags have to stay, because under "qaa" -- the code for a language with no code of its own, which Language Forge projects lean on heavily -- they ARE the language: "qaa-x-kal" and "qaa-x-hbo" are no more the same language than "fr" and "de". The phonetic markers, the audio marker and the "dupl1" suffixes FieldWorks appends all come out, so "qaa-x-kal", "qaa-fonipa-x-kal" and "qaa-Zxxx-x-kal-audio" are one language. A language both roles already use settles nothing, for the same reason a writing system in both anchors is no evidence of a field's role. Checked against the four FieldWorks repositories on hand, whose LanguageProject.langproj records the real vernacular and analysis lists: this gets 18 of 18 writing systems right, where the config-only rule gets 15 right and 2 wrong (cmo-khm-flex's km-KH and xkk-flex-2022's en, both wrongly vernacular). Across the 2026-07-06 corpus it takes the projects with at least one unresolved writing system from 323 to 69, and the unresolved (project, writing system) pairs from 419 to 71. A writing system the config forgot but the lexicon uses is now classified too, rather than falling through as neither. Counting means a second streaming pass over the Mongo lexicon before any writing system is created. GetLexicon yields from a cursor, so the lexicon does not all sit in memory at once. WithNoAnalysisAnchorEveryOtherFieldIsUnresolved covered a phonetic spelling of the lexeme's own language, which language affinity now settles, so it grows a writing system of an unrelated language to keep making its original point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…vidence Only the lexeme, citation form, definition and gloss counted as fields whose role FieldWorks fixes. Four more do: an example sentence is vernacular-typed, and an example translation, an entry note and a literal meaning are analysis- typed. Counting the 2026-07-06 corpus for each field, the projects whose text there is exclusively one role against exclusively the other: example sentence 332 vernacular : 5 analysis example translation 1 vernacular : 320 analysis entry note 0 vernacular : 255 analysis literal meaning 4 vernacular : 136 analysis Taking them as evidence takes the projects with at least one unresolved writing system from 69 to 61, and still gets all 18 writing systems in the four FieldWorks repositories on hand right. The pronunciation field is deliberately left out although it looks vernacular at 114 projects to 16. That is 12% analysis, above the share a single writing system may hold in the other role and still count as playing one role alone, so projects evidently use the field for more than one thing and it cannot be trusted to settle anything. The field paths are now the constants LfWritingSystemUsage already defines, so the counting pass and the classification cannot drift apart on a spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The field loop takes its evidence as it stood before the loop began, so a field holding only writing systems the loop itself has settled looks like evidence of nothing, and everything in it goes back into doubt. A writing system settled by one custom field can therefore be dragged into doubt by another. A second pass settles those again from the company they keep: a custom field holding the vernacular and one writing system nothing has placed is being used for vernacular content, so the unplaced one is vernacular too. Only unanimous company counts, for the same reason a writing system in both anchors is no evidence of a field's role. This takes the projects with at least one unresolved writing system from 61 to 57, and still gets all 18 writing systems in the four FieldWorks repositories on hand right. The pass deliberately does not chain: the company is counted once, before any of it is settled, so a writing system whose only company was itself in doubt stays unresolved rather than inheriting a second-hand answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The field loop's no-evidence branch put every writing system in a field into doubt, including ones something had already placed. Before the companions pass that cost nothing but a log line, since a doubtful writing system goes in both lists anyway. With it, it strips a role. Take a config whose lexeme offers [seh, en], whose gloss and definition offer [en], and whose general note offers [en], with nothing written in en yet. The anchors correctly put en in both lists. The note holds nothing exclusive to one role, so the loop also put en in doubt; the companions pass then found that the only single-role company en keeps is seh, in the lexeme field, and removed en from the analysis list. A new "en" writing system was created vernacular only, and English glosses and definitions lost their analysis writing system. A field with no evidence now puts in doubt only the writing systems nothing else has placed. One already placed keeps its place. In the 2026-07-06 corpus five writing systems were losing a role this way. The fix also stops flagging the 47 empty projects whose "en" is offered in every field, both anchors included, so the projects with a writing system in doubt go from 57 to 10. It still gets all 18 writing systems in the four FieldWorks repositories on hand right. Two tests described the old behaviour. AWritingSystemInBothAnchorsIsNotEvidence expected en itself to be in doubt; it now gives the etymology a second, unplaced writing system to keep making its point. The companions tests relied on a writing system being dragged back into doubt, which no longer happens, so they now list the doubtful field first -- fields are visited in config order -- and the "company is itself unsettled" case uses two writing systems with no other company at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BestStringAndWsFromMultiText ended with a "last-ditch" fallback: take the first
non-empty value whatever writing system it is in. Once 74e1684 gave it a
fallback that already takes the first value in any writing system LCM has, the
last-ditch branch was reached only when no key resolved at all, and then it
always returned writing system 0. Building a string in writing system 0 throws
("The specified writing system code is invalid"), and the per-entry catch in
RunConversion then abandons the entry half-written: every field converted
before the throw is kept, and every field after it is skipped.
Returning null instead, as the review suggested, would be worse. Every caller
assigns the result straight into the LCM property, so null clears the field,
deleting what LCM holds on account of text that could not be placed.
So the field now keeps its current value, and the log says which writing
systems' text had nowhere to go. Null, clearing the field, still means what it
always meant: LF holds no text there. The callers pass the current value in.
The picture caption had the same hole: a caption in no writing system LCM has
came back as writing system 0 and threw in GetOrCreatePictureByGuid. It now
starts the picture with an empty caption in the default analysis writing
system, and SetMultiStringFrom fills in every alternative that can be placed.
WsIdAndFirstNonEmptyString, which fed both, is gone.
The new test fails without this: the entry is abandoned at the scientific name,
so the sociolinguistics note converted after it never arrives.
The etymology-source line changed here is compiled only for DBVERSION_7000068,
which does not build against the current SIL.LCModel for reasons unrelated to
this change (ILexEntry.EtymologyOA and ILexEtymology.Source are gone), so it is
unverified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LCM knows a writing system by its Id, which is whatever its .ldml file is
named, and also has its LanguageTag, which is always canonical. In older
projects the two differ. brb-flex-2022 holds "km-Khmr-KH.ldml", whose identity
is km plus KH, so LCM holds it as Id "km-Khmr-KH" with LanguageTag "km-KH"; so
do spt-flex's "hi-Deva-IN" (hi-IN) and "spt-Tibt-x-tib" (spt-x-tib). LfMerge
writes LF's multitext keys by Id and the project's input systems by
LanguageTag, so LF can hand back either spelling.
WsIdFromLfTag tried the canonical form first, which deletes data in the
brb-flex-2022 layout. That project also holds a second, unused writing system
whose Id is "km-KH". Text keyed by the real writing system's Id, "km-Khmr-KH",
canonicalized to "km-KH" and went to the unused one, and WriteToLcm's clearing
pass would then blank the real writing system's text, as missing from LF, in
every entry LF sent back. (This branch has never run against that project, and
LF has never edited it, so nothing has been lost.)
It now tries, in turn:
1. The tag as LF spells it, as an Id -- the spelling LfMerge itself exports.
2. Its canonical form, as an Id.
3. Its canonical form as a LanguageTag, which finds spt-flex's "hi-Deva-IN"
for the "hi-IN" in LF's config, where before it found nothing and the
edit was dropped. Should several writing systems share the LanguageTag,
the one created first wins.
The two new tests reproduce those layouts and fail without this: the first
resolves to the unused writing system, the second to nothing.
Trying the exact Id first is never worse than develop, which only ever tried
it. One consequence is worth knowing. cmo-khm-flex's LF entries still carry a
few stale keys spelled "km-Khmr-kh", from before the project's Khmer writing
system was renamed "km-KH"; they resolve, as they do on develop, to the unused
"km-Khmr-KH" writing system LfMerge created in 2022 rather than to "km-KH".
Sending them to the writing system in the project's lists instead needs the
LanguageProject, which this function does not have, and needs a rule for which
of two keys resolving to one writing system wins, which does not exist yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfWsToLcmWs looked each LF input system up with GetOrSet, which looks a tag up only as an Id, and created a writing system when that found nothing. LfMerge exports input systems by LanguageTag, while LCM knows a writing system by its Id, and in older projects the two differ: spt-flex's input system "hi-IN" is the writing system LCM holds as "hi-Deva-IN". On the next sync GetOrSet would not find it, would create a second writing system "hi-IN" and add it to the current lists. WsIdFromLfTag, which every multitext key goes through, then matches that duplicate exactly, so text typed into LF's hi-IN field goes to a writing system FieldWorks shows as new and empty rather than to hi-Deva-IN. The input system is now found the way every other LF tag is found, with WsIdFromLfTag, and a writing system is created only when that finds nothing. The new test fails without this: the duplicate appears among the project's writing systems. This was already the case on develop, which also used GetOrSet here. The LanguageTags comment claiming GetOrSet made the resolver unnecessary for LfWsToLcmWs was wrong, and is corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After applying an input system's abbreviation and direction, LfWsToLcmWs called wsManager.Replace(ws). ws is already the manager's own object, so the edits are in place without it, and Replace can only do nothing or do harm. It looks the writing system up by LanguageTag, which in older projects is not the Id LCM knows it by, and then: - If no writing system has that LanguageTag as its Id, Replace falls through to Set(ws), which gives the writing system a fresh handle mid-sync. That happens whenever an input system is spelled by an Id that is not its LanguageTag, as in projects LfMerge set up under FieldWorks 8, which exported input systems by Id: to judge from those exports, aeb-ra-flex holds "aeb-Arab" (LanguageTag "aeb") and nod-wlist-flex holds "th-Thai" (LanguageTag "th"). - If another writing system has it -- brb-flex-2022's unused "km-KH" beside the real "km-Khmr-KH" -- Replace removes that one from the manager and gives its handle to ws. The call is gone. The new test covers both layouts, and fails without this because the writing system's handle changes; the eviction in the second layout is from reading Replace, since the test stops at the handle. The test also checks that the input system's abbreviation is still applied. This was already the case on develop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfMerge spelled a writing system two ways in Language Forge. Every multitext key comes from GetStrFromWs, which gives the writing system's Id. The project's input systems, its languageCode, the config's field lists at initial clone and the custom-field config came from its LanguageTag, which is always canonical, while the Id is whatever the .ldml file is named. In older projects the two differ: brb-flex-2022's Khmer is Id "km-Khmr-KH" with LanguageTag "km-KH", and che-flex's vernacular is Id "ce-la" with LanguageTag "ce-LA". LF's editor matches a field's input systems against an entry's keys exactly and case-sensitively, so it showed all of that text as empty: in the 2026-07-06 mongodump, twelve projects are affected, che-flex's 18,423 headwords among them. All of them are now spelled by GetStrFromWs too, through LfTagOf, so the input systems and the keys cannot disagree. FieldWorks 8 builds did the same, by Id; the three #if FW8_COMPAT blocks here existed only to choose between the two spellings, and FW8_COMPAT is never defined, so they are gone. Spelling by Id also agrees with LCM, which looks every writing system up by Id, so a key LF hands back resolves at the first step of WsIdFromLfTag. The project records already spelled by LanguageTag need re-spelling once, since LfMerge rewrites the input systems and custom-field config on every sync but built-in field config only at initial clone. That is done outside LfMerge, by a mongosh script. The new test fails without this: the input system is exported as "nl-BE" while the gloss is keyed "nl-Latn-BE". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A project's current vernacular, analysis and pronunciation lists are stored as strings of Ids. If one names an Id LCM has no writing system for, the export threw KeyNotFoundException before writing anything to Language Forge: LINQ's Select and ToList read a list through its indexer or CopyTo, and liblcm's versions of both throw for such an Id. Only its enumerator tolerates one, by yielding null. The three lists are now read with foreach, and an Id with no writing system is left out with a warning. This makes the pronunciation list safe, which nothing else in the export reads. A vernacular list in that state would still fail, inside liblcm's own GetWritingSystemList when the custom-field config is written; FLEx would hit the same thing, so such a project is unlikely to exist. The new test fails without this, with the KeyNotFoundException above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LanguageOf looked for the private-use "x" from the second subtag on, so a tag that begins with it, such as "x-kal", came out as just "x". Every such tag then counted as one language, and language affinity could place "x-hbo" in the same role as "x-kal". The 2026-10-01 corpus has two tags like this, "x-IPA" and "x-v". The search now starts at the first subtag, so "x-kal" comes out "x-kal" and "x-kal-dupl1" still matches it. The new test fails without this: "x-hbo" is classified with "x-kal" instead of being left unresolved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A custom field whose type the LF to LCM direction does not implement, such as an owning collection or sequence, logged its warning for every entry, sense or example that had it, so a project with 10,000 senses logged it 10,000 times on every sync. That is the same kind of log spam d90c0b9 removed. The converter now remembers which fields it has reported and warns about each once. One converter is made per transfer, so that is once per field per sync. The new test fails without this: three entries give three warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A config field that is not an anchor is classified by the writing systems it
offers that belong to one role only. A field offering a vernacular-only and an
analysis-only writing system side by side took the vernacular branch, only
because that branch was checked first, and the other writing systems it offered
became vernacular. That is the opposite of the rule ResolveByCompanions follows,
where only unanimous company counts.
Such a field is now evidence of neither role. Writing systems it offers that
nothing else has placed are left unresolved, as for a field overlapping neither
role, and the companions and language-affinity passes then decide them where
they can.
In the 2026-10-01 corpus 166 projects have 1,381 fields like this, but only 4
projects and 5 writing systems change, measured with a simulation of this
classifier. Compared with the roles FLEx gives them (the inputSystems record
LfMerge exports):
- kam-flex: "es" and "fr", in a wordlist field, were vernacular and are now
both; FLEx has them as analysis.
- mni-flex: "zh" was vernacular and is now both, as FLEx has it.
- msea-thai-flex: "th-x-latin" was vernacular and is now both; FLEx has it as
vernacular.
- xin-flex: "xin", which LCM does not have yet, was vernacular and is now
analysis, through language affinity with its analysis writing system
"xin-GT-fonipa". It holds only pronunciation text, so vernacular was the
better answer there; the project's writing systems look mislabelled.
Only xin-flex is affected in practice: the others already exist in LCM, and
classification only decides the lists for a writing system LCM does not have.
The new test fails without this: "fr" is classified vernacular-only instead of
left unresolved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FieldWorks draws its pronunciation writing systems from the vernacular ones: liblcm's InitializePronunciationWritingSystems considers no others. But the classifier ignored the pronunciation field's text altogether, since the field is not an anchor, so a writing system used only for pronunciations got whatever role its company or language suggested. xin-flex's "xin", which holds 226 pronunciations and nothing else, became analysis-only after 7b76f8f. A writing system with no text in the anchor fields but some in the pronunciation field is now vernacular. That only adds the role, it does not settle it, so the config can still add analysis: the field stays out of the anchors, because 12% of the projects writing in it write analysis there, and a stray pronunciation string should not take an analysis role away. Measured with a simulation of this classifier on the 2026-10-01 corpus, against the roles FLEx gives the 564 writing systems LfMerge's export records: 516 agree, up from 515, and none gets worse. Projects with an unresolved writing system go from 10 to 7. 18 writing systems in 18 projects change, 8 of them in Send/Receive projects, nearly all phonetic ones becoming vernacular. In dtlat-flex, Swedish, configured for the gloss with a single pronunciation string, becomes both rather than vernacular only. This is the part of the four-set design measured best. The rest of it -- "both" only for a genuinely split writing system, with a tie-break for one offered in both anchor groups with no text -- lost five roles FLEx gives, and adding to FLEx's pronunciation list is left out, since LfMerge only reads it. ThePronunciationFieldIsNotEvidenceOfAnything asserted that pronunciation text leaves a writing system unresolved; it is now ThePronunciationFieldIsNotAnAnchor, for a writing system merely offered there. The two new tests fail without this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BestStringAndWsId, which the single-string lexicon fields, picture captions and the String custom field go through, took the first non-empty value in document order when two keys named one writing system. WriteToLcm, which writes the multitext fields, takes the key spelled as the project's config spells the writing system, since that is the one LF's editor shows. The two could pick different spellings' text for the same keys, and the single-string fields could pick the export's hidden, stale value over an LF user's edit. BestStringAndWsId now chooses each writing system's key with the same rule, from the same configured spellings, before it leaves out empty values, so a value the user has cleared under the config's spelling is not replaced by the hidden one either. The 2026-10-01 corpus holds no two keys naming one writing system in any single-string field, so this changes nothing there today. The two new tests fail with the old choice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpanStrToTsString gave a run whose span lang resolves to no LCM writing system
writing system 0, and building the string then threw ArgumentOutOfRangeException
("The specified writing system code is invalid"). Inside WriteToLcm that came
after earlier alternatives had been written and before the clearing pass, so
the entry's per-entry catch left it half-written -- the failure c66141f fixed
for single-string fields whose text could not be placed. 8900058 resolved span
langs through WsIdFromLfTag, which fixed non-canonical spellings of writing
systems LCM has, but not this.
Such a run's text is now kept in the string's own writing system, losing only
its language mark, and a warning names the lang and the text.
In the 2026-10-01 corpus 29 of the 83,743 span langs in Send/Receive projects
name no writing system those projects have, in five projects, and none is in an
entry edited since its project last synced.
The new test fails without this, with the ArgumentOutOfRangeException.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LcmWsToLfWs set each input system's VernacularWS and AnalysisWS flags with the current lists' own Contains, which compares Ids exactly. LCM resolves the lists' Ids ignoring case, and LfTagsOf reads them that way too, so a list spelling an Id in another case put the writing system in the list LF is given but flagged its input system as not in it. testlangproj's vernacular list names "qaa-Zxxx-x-kal-AUDIO" for "qaa-Zxxx-x-kal-audio"; a TODO in TransferLcmToMongoActionTests asked why that one was not flagged, and worked around it. The flags now come from the handles of the writing systems the lists yield, read the same way as LfTagsOf reads them. The TODO and its workaround are gone, and that test now checks every vernacular writing system is flagged. Language Forge does not read the flags, so nothing it shows changes. The new test fails without this: the flag is false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveByCompanions' summary gave as its example a writing system the field loop has already placed: one sharing a custom field with the vernacular alone. What the pass settles is a writing system offered only in fields that said nothing of their role, whose company there another field has since placed. It also makes one pass, so a writing system whose only company is another one in doubt keeps both roles; the summary now says so. The header of ConvertMongoToLcmWritingSystemsTests said LCM's lookups match an Id literally and that the tests add nothing to the fixture. The lookups ignore case, as the file itself says further down, and many tests now add writing systems of their own, each under a tag no other test uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A single-string field keeps its value when none of LF's text can be placed, and logged that the text "has nowhere to go, since LCM has none" of its writing systems. That is one reason. The other is that the text is all under keys passed over for an empty key naming the same writing system: the config's spelling, cleared by an LF user or filled in empty by LF's editor beside the export's value under the Id. LCM has that writing system, so the log named the wrong cause. Each key with text left out is now reported with its own reason, and the field still keeps its value, since nothing here tells a clear from the editor's empty value. The String custom field, which said nothing at all, reports the same way. The choice of one key per writing system, which WriteToLcm and BestStringAndWsId each made, is now made in one place, ChooseKeys, and KeysPassedOver gives the keys it leaves out. The Mongo test double kept its entries by round-tripping them through LfMerge's serializer, which drops a multitext holding an empty value, so no test could hand the converter LF's empty values. It now keeps entries as documents, as Mongo does; LfMerge's own writes still go through its serializer, as they do against Mongo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every test of choosing between two keys for one writing system handed LfMultiText a set of configured spellings built by hand. Had ConfiguredTags returned nothing, or a writer stopped being given the set, those tests would still have passed while a sync quietly went back to writing the later key. This test runs a whole sync with the config's spelling added to the field's input systems and put first in the entry, so that going on key order alone writes the export's value instead. It covers the built-in multitext fields (picture captions share their writer), the single-string fields, and the multitext and string custom fields. With ConfiguredTags made to return an empty set, all four cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test's writing system, "kal-fonipa", is a phonetic spelling of the vernacular, so language affinity made it vernacular only whether or not it had pronunciation text; with the pronunciation rule removed from ClassifyByText, the test still passed. It now uses "xyz-fonipa", a language the project has nowhere else, and runs both with and without the pronunciation text: vernacular only with it, both and unresolved without. With the rule removed, the case with pronunciation text fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e new MongoToLcm_InputSystemNewToLcm_CountsTheLexiconToo added a single new input system, so it expected two reads of the lexicon whether or not the classification was kept after the first new writing system. Dropping that, and counting the whole lexicon again for every new writing system, would have gone unnoticed. The test now adds two and still expects two reads; without the cache it sees three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field loop's comment, and the docstring of the test covering it, said "en" is not made doubtful by turning up in "a note". The entry's note is an analysis anchor, which the loop skips; the field meant, and the one the test uses, is a sense's general note. LogTextNotPlaced's messages are right only when nothing was written, which both its callers ensure but its doc did not say. SetLastSyncedDate's comment still compared it with DeepCopy, which the double no longer uses for entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing defines FW8_COMPAT, and LfMerge has not built against FieldWorks 8 for years, so the #if branches naming IWritingSystemManager and IWritingSystem were never compiled. Only the FieldWorks 9 types are left, and with them goes the comment explaining why the #if blocks were there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfMerge now exports every writing system by its LCM Id. It rewrites the inputSystems record and custom-field config on every sync, but the built-in fields' config only at initial clone, so projects cloned while LfMerge spelled input systems by LanguageTag keep a config LF's editor cannot match against the lexicon's Id-spelled keys. migrate-input-systems-to-lcm-ids.js re-spells, for the fifteen projects the 2026-07-06 and 2026-10-01 mongodumps show are affected, every config inputSystems array, the inputSystems record and languageCode, each mapping checked against the project's FieldWorks repository. In che-flex, whose lexicon is still keyed by the Id its Chechen writing system had before FieldWorks moved the text to "ce-Latn", it also renames the lexicon's keys in place, leaving entries' dates alone. It is a dry run unless DRY_RUN=false, leaves an inputSystems entry alone where the new spelling is already there, and changes nothing when run a second time. Run it before the new LfMerge build first syncs these projects, and keep them on HOLD from before it runs until that build is deployed: the old build's Replace call would evict or re-register writing systems keyed by Id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmunn
force-pushed
the
bugfix/canonical-writing-system-tags
branch
from
October 6, 2026 02:46
fffeef6 to
499b2ea
Compare
Collaborator
Author
|
I don't think we have CodeRabbit configured for this repo, but in case that's changed: @CodeRabbit fullreview |
rmunn
changed the title
migrate-input-systems-to-lcm-ids.js is run once against Language Forge's Mongo before this build first syncs the projects it lists, and is then of no further use, so it does not belong in LfMerge's long-lived code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Devin raised a few points, all of which were invalid (Claude could see actual data while Devin was just theorizing about it). Both Claude and Devin are now in agreement in finding no more major issues. I will now test the migration script against a dev-machine copy of the repos that need it, and report my results tomorrow. |
Collaborator
Author
|
Self-review done, and I finally think I understand the logic. Relying on Claude's summaries was just not working; I had to read the actual code. I have not yet read the unit tests that Claude generated, but the logic LGTM. Time to ask for actual review now. |
hahn-kev
approved these changes
Nov 9, 2026
The current way of creating the installation tarball is brittle: any changes to the file list of Mercurial will result in the files not being copied. Instead, we will just copy everything from the Mercurial NuGet package, then ensure that the `hg` and `chg` scripts are world-executable so that the www-data user can run them. This makes it far less brittle against Mercurial version updates.
A never-synced project's entries count as edited in LF only if they carry a user reference (#362), because LfMerge clears those on every entry it writes. A LIFT import into LF leaves none either, so on an LF project's first Send/Receive -- as when one is migrated -- every imported entry was taken to be LfMerge's own and left out. The transfer back to Mongo then marked each of them deleted, since FieldWorks did not have it: 12,638 of a test project's 14,370 entries, and 47,827 entries across the projects due to be migrated. FieldWorks cannot hold a newer version of an entry it does not have, so in a never-synced project such an entry is now sent. Entries FieldWorks has are still left to FieldWorks, as #362 intended. The one case this changes besides migration is an entry LfMerge wrote on an initial clone and FLEx then deleted, before any sync was recorded: it now comes back instead of staying deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Commit 2e748f8 contains the fix to a subtle bug introduced in #362 that only affected migration projects: if they had been imported into LF from a LIFT file, then there was no author data, and #362 began using author data to verify whether a project had been edited in LF. This would have resulted in LIFT entries that were imported but never changed being incorrectly seen as deleted in FW. Verified by testing the migration scripts from sillsdev/web-languageforge#1859 locally: the non-SR projects that started out as LIFT imports would have had some entries incorrectly deleted without that commit, but with that commit added, they get migrated correctly. |
Collaborator
Author
|
I have now tested the entire migration locally, and everything worked. It's time to merge these so we can deploy the bugfix to production. |
This was referenced Nov 12, 2026
rmunn
added a commit
that referenced
this pull request
Nov 12, 2026
…orks (#365) * Send a never-synced project's missing entries only to an empty FieldWorks #361 made LfMerge send every entry a never-synced project's FieldWorks lacks, so that an LF project's first Send/Receive -- as when one is migrated -- would not lose the entries a LIFT import into LF left without user references. But it did that for every never-synced project, and the ones whose FieldWorks already has entries were cloned from it: the entries it lacks were since deleted in FLEx, or dropped when its repository was begun afresh. Their next sync would bring those entries back, undoing what #362 was for. Three never-synced projects on the live server are like that, with 664 such entries between them. So the missing entries are now sent only when FieldWorks had no entries at all before the conversion began, which is how a migration's new LexBox project starts. Every other never-synced project is back to how #362 left it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Log each never-synced entry taken to be deleted in FLEx In a never-synced project whose FieldWorks has entries, an LF entry with no user references that FieldWorks lacks is taken to have been deleted in FLEx since the initial clone, and the transfer back to Mongo marks it deleted. The data cannot check that: an entry imported into LF from LIFT after the clone looks just the same, and would be marked deleted too. That is deliberate, but it should not pass unnoticed. Each such entry is now logged by GUID and name as it is left out, so that one which turns out to be LF's own can be found and restored -- marking it deleted leaves it in Mongo. The comment there now says this is an assumption, and names the case it gets wrong. After a sync nothing is logged: an entry FieldWorks lacks then was certainly deleted in FLEx. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
For the record, the migration has been run in production and the new LfMerge build has been deployed. So there's nothing else that needs to be done here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When LfMerge started supporting FieldWorks 9.x, it started using the
.LanguageTagproperty of writing systems instead of.Idas it haddone in FW 8. That proved to be a mistake, which I only caught once I
started testing the migration path of non-SR projects. The
.Idproperty is whatever the writing system's
.ldmlfile is named (oftenwhat the user entered), whereas
.LanguageTagis a canonicalized formof the tag, and Language Forge stores writing systems the way the user
entered them. So if the user entered something like
qaa-x-qaa-v,LfMerge looked for
qaa-x-qaa-v, didn't find FieldWorks'qaa-x-v, andtried to create it, producing errors like "Unable to set writing system
'qaa-x-v' because this id already exists."
At first I thought the solution was to canonicalize writing systems
before storing them, so that
qaa-x-qaa-vwould disappear. But the LFdata still would have had
qaa-x-qaa-vall over the place, and thatwould have resulted in a massive data migration. So instead:
WsIdFromLfTagbelow), soqaa-x-qaa-vfindsqaa-x-v. A writingsystem is only created when none is found, and
Replaceis no longercalled.
.Idwhen writing toLF: multitext keys (as it always did), the project's
inputSystemsrecord, and custom-field config.
Because LfMerge writes the LF field config only during an initial clone,
15 existing S/R projects still have config spelled by LanguageTag and
need a one-time migration script run on LF's Mongo. The script contains
project codes, so it is not in this PR; instead, we will copy it to
production, run it, and delete it. Those 15 projects must be put on HOLD
until after the script has run. Then once this PR is merged and deployed
they can be taken off HOLD.
LfMerge now has a single helper function,
WsIdFromLfTag, which alluses of writing systems go through. It first looks up the
.Idvaluethat LF has; if that's not found, then it tries the canonical form. If a
project has data in both forms (e.g. if it had
en-USanden-Latn-USwriting systems, and both had data) then a warning is logged and the
spelling from the LF config is used.
Other fixes
A few other bugfixes got folded into this PR as well:
field LF shows) are now written to FLEx. Previously MultiUnicode was
missing from LfMerge's LF-to-LCM handler, so those edits were silently
dropped, in S/R projects too. (LfMerge still does not create custom
fields that exist in LF but not in FLEx.)
project is migrated, entries that FieldWorks does not have are now
sent to FieldWorks. Since Fix incorrect handling of fresh clone #362, such entries counted as LF edits only
if they carried a user reference, and entries from a LIFT import carry
none, so they were marked deleted instead (47,827 entries across the
projects due to be migrated).
dropped silently: keys and span
langs naming no writing systemFieldWorks has, single-string fields left unchanged, and custom fields
of types LfMerge cannot write (once per sync).
dirtySRfield was null or missing in many non-SR projects, whichLfMerge was not designed to handle. LfMerge's Mongo deserializer now
treats a null/missing
dirtySRfield as 0.#if FW8_COMPATblocks have been removed, since LfMerge no longersupports FW 8 and nothing defined
FW8_COMPAT.from a hard-coded file list, which had fallen out of date with
Mercurial 6.5.
How to review
CustomMongoSerializerForInt32.csandLfLexEntry.csfirst.This is the bugfix for the
dirtySRfield being null in some older,non-SR projects.
LanguageTags.cs, whoseWsIdFromLfTagreplaces the old,inconsistently-used
GetWsFromStr.LfWritingSystemUsage.cs, which counts where each writingsystem's text actually is, and
ClassifyVernacularWritingSystemsinConvertMongoToLcmLexicon.cs, which uses those counts to decidevernacular and analysis. (See explanation below for rationale.)
Some code explanations
This used to be in the "How to review" section but it got verbose. I've
split out the code explanations below.
GetWsFromStrto convert a writing system ID ("en-Latn") to a writing system ID (int),
but we were inconsistent about it. The new static function
LanguageTags.WsIdFromLfTagshould be used everywhere now; if there'ssomewhere where that statement isn't true (a use of GetWsFromStr that
has survived, for example), that's a bug in this PR.
LF writing system configuration to figure out which writing systems were
vernacular or analysis, and the logic was simple. Is it associated with
the
lexemefield? Vernacular. Anything else? Analysis. But what wefound was that many older LF projects had configurations that put things
like
enin the lexeme field, making it end up counted as vernaculareven though the
lexeme.envalues were always empty. So I asked Claudeto write some code to check the actual project data for how often
writing systems are used as vernacular and analysis, and it came up with
LfWritingSystemUsage.cs. This code is only called when there are
writing systems that are new in LF and not in FW, meaning existing
Send/Receive projects (where we don't allow creating new writing sytems
in LF) will be unaffected, and only the non-SR projects that we plan to
migrate to LexBox are going to end up running this code.