From 8031155f88250ace62920ca124a36877dffcbe39 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:20:02 -0500 Subject: [PATCH 01/13] docs/formats/findings.md lists every finding id the library can report, and a test keeps it complete. Co-Authored-By: Claude Opus 5.5 --- docs/formats/findings.md | 145 ++++++++++++++++++ .../FormatDocsTests.cs | 41 +++++ 2 files changed, 186 insertions(+) create mode 100644 docs/formats/findings.md create mode 100644 tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs diff --git a/docs/formats/findings.md b/docs/formats/findings.md new file mode 100644 index 0000000..cf6d5e1 --- /dev/null +++ b/docs/formats/findings.md @@ -0,0 +1,145 @@ +# Findings + +A **finding** is what a reader reports about its input: a stable id, a severity, a place and, where one exists, the one exact fix. +This page lists every id the library can report. `FindingCatalogueTests` checks that every id in the source is listed here. + +- [FieldWorks Markdown](fieldworks-markdown.md) findings are `fwmd.*` and come from `MarkdownConverter.Parse` and `ParseInline`. +- [Record file](record-files.md) findings are `fwmd.yaml.*` (the YAML of the frame) and `fwmd.record.*` (what the frame says against the + schema). `RecordFileReader` also reports the FieldWorks Markdown ids for text inside rich values and the body, placed in the file. +- The LCModel adapter's ids are `textediting.lcm.*`. They are not `MarkdownDiagnostic`s; see [the last section](#lcmodel-adapter). + +Every fixture case cited below is in `tests/conformance` (Markdown) or `tests/conformance/records` (record files). + +## How a finding looks + +| Member | Meaning | +|---|---| +| `Id` | The id, as listed below. | +| `Severity` | `Error`, `Warning` or `Suggestion`. This page calls a suggestion **information**: the text is read as it means, and the writer spells it canonically next time. | +| `Line`, `Column`, `EndLine`, `EndColumn` | 1-based. Columns count UTF-16 code units. The end is exclusive. | +| `Start`, `End` | UTF-16 offsets in `Source`, the text as written. | +| `Observed` | The text the finding is about. | +| `Replacement` | The one exact replacement of `Observed` that fixes it, or null when the intent is not clear. | +| `Choices` | The candidates, when there is no single safe fix (an ambiguous link, an unknown writing system). | +| `Nearest` | Names close to the one written (a style, a writing system, a field). | + +A text with an error is still read as far as it can be. `HasErrors` says whether any finding is an error. + +**Hosts add and change findings.** The ids below are the library's. A host's checker can give its own id and severity: +`IWritingSystemChecker.CheckWritingSystem`, `IStyleChecker.CheckStyle` and `IObjectResolver` return a `MarkdownProblem` with an optional +id and severity, and an `INotationChecker` returns `LanguageDiagnostic`s whose `Code` is the id. `MarkdownParseResult.CreateDiagnostic` +makes a placed finding for any other check a host runs. + +## FieldWorks Markdown: syntax + +| Id | Severity | Meaning | Usual fix | +|---|---|---|---| +| `fwmd.syntax.unclosed-mark` | error | An attribute block has no closing `}`, or a div has no closing `:::`. Fixtures: `02-attributes.json` "a block with no closing brace is an error with the closing brace as the fix"; `04-structure.json` "a div that is never closed is an error". | Add the `}` (the finding's fix) or the closing `:::` line. | +| `fwmd.syntax.unknown-attribute` | error | A key or class the notation does not have (`{wss=ceb}`, `{.vern}`), the old bare `{v}`, or a key written as a class (`{.ws}`). Fixture: `02-attributes.json` "an unknown key is an error with a hint" and the three cases after it. | Use a key from the [attribute tables](fieldworks-markdown.md#attributes), or the class `.v`. | +| `fwmd.syntax.duplicate-attribute` | error | A key appears twice, two keys set one property, or `.v` and `ws=` both name a writing system. Fixtures: `02-attributes.json` "a repeated key is an error", ".v and ws together name two writing systems". | Keep one. | +| `fwmd.syntax.malformed-attribute` | error | An attribute block that breaks the grammar (`{}`, `{#id}`, a bad escape), a value that does not fit its key (`bold=yes`), a unit key without its base key, `url=` that is not a `silfw://` URL on a link to a project object, or text after a `:::` line's block. In a plain FieldWorks Markdown file it also covers the front matter: `fwmd` that is not a whole number, an empty or list default writing system, a tab in the indentation, a line that is not a key and a value. Fixtures: `02-attributes.json` "a value that does not fit its key is malformed"; `06-front-matter.json` "a version that is not a number is malformed"; `11-hardening.json` "a tab in the front matter's indentation is malformed, because Pandoc refuses it". | Write the value the key takes (the message says which). In the front matter, indent with spaces. | +| `fwmd.syntax.unsupported-markup` | error | Pandoc reads a construct that has no counterpart in FieldWorks text: raw HTML, a code span, a thematic break, a reference definition, a nested list, a div in a div, an unescaped block marker in a styled div, math, a citation, an inline note, strikeout, heading attributes, fancy list markers, line blocks, grid tables, definition lines, a table caption, a `:::` line straight after a paragraph line, a `[]{.empty}` that is not the whole paragraph, and a pipe table when the host names no table styles. See [Unsupported Pandoc constructs](fieldworks-markdown.md#unsupported-pandoc-constructs). | The fix escapes the construct so that it reads as text (`\$x\$`, `\@`, `a\. palit`), or adds the blank line a div fence needs. Raw HTML, code and a thematic break have no fix: escape or remove them. | +| `fwmd.syntax.too-deep` | error | A block, line or table is nested too deeply or is too large for the parser (Markdig's limit). One finding; the lines are kept as plain text. Fixture: `08-checks.json` "text nested deeper than the reader allows is one finding and its lines are kept as text". | Flatten the nesting or split the table. | +| `fwmd.version.unsupported` | error | The front matter says `fwmd:` with a version newer than 1. Nothing else is read. Fixture: `06-front-matter.json` "a newer version is refused with a finding and nothing is read". | Read the file with a newer library. | +| `fwmd.prop.unknown-number` | warning | A `pN` or `sN` key whose number is not a FieldWorks text property. The value is kept as written. Fixture: `02-attributes.json` "a property number that is not known is kept with a warning". | Check the number. Nothing to do if it is meant. | +| `fwmd.record.record-file` | error | `MarkdownConverter.Parse` was given a [record file](record-files.md): the front matter has a top-level `record:` key. One finding on that line and nothing else is read, so a host cannot write back only the body (owner decision R-9). Fixture: `06-front-matter.json` "a record file is not read as FieldWorks Markdown: one finding on its record line, and nothing else (owner decision R-9)". | Read the file with `RecordFileReader`. | + +## FieldWorks Markdown: blocks and fields + +| Id | Severity | Meaning | Usual fix | +|---|---|---|---| +| `fwmd.heading.too-deep` | error | A heading deeper than `###` has no paragraph style. Fixture: `04-structure.json` "a heading deeper than three has no style". | Use `###`, or a div with the style. | +| `fwmd.table.ragged` | error | A row has another number of cells than the header, or the delimiter row has another width. Fixtures: `04-structure.json` "a row with another number of cells than the header is an error", "a delimiter row of another width than the header is an error with the delimiter row as the fix". | Add or remove cells. For the delimiter row the fix is one `---` per header cell. | +| `fwmd.table.alignment` | error | A delimiter row with a colon (`:---:`). Alignment has nowhere to be stored. Fixture: `04-structure.json` "a column alignment has nowhere to be stored". | `---` (the fix). | +| `fwmd.table.tab-in-cell` | error | A tab inside a cell, raw or as ` `. In storage the tab is the cell separator. Fixtures: `04-structure.json` "a tab inside a cell cannot be stored", "a raw tab inside a cell cannot be stored either, although it would read as a space". | Use spaces. | +| `fwmd.field.block-syntax` | error | `ParseInline` (a field of one paragraph, such as a multi-string field) met block syntax: a heading, a list, a div or a blank line. Fixtures: `08-checks.json` "a multi-string field takes inline grammar only: a heading is an error"; `11-hardening.json` "a multi-string field is one paragraph: a blank line is an error that asks for a hard line break". | Keep the text inline. For a line break, end the line with `\`. | + +The table findings need table styles (`MarkdownStyleNames.TableHeader` and `TableRow`). Without them a pipe table is +`fwmd.syntax.unsupported-markup`. + +## FieldWorks Markdown: checks against the project + +These need the host's interfaces in `MarkdownOptions`. With the defaults (no project) they are never reported. + +| Id | Severity | Meaning | Usual fix | +|---|---|---|---| +| `fwmd.ws.unknown` | error | A writing system the project does not have: after `ws=`, in an empty-paragraph marker, as a front-matter default, or (in a record file) as a key of a per-writing-system field. Needs `IWritingSystemMapper.IsKnown` to say no. `Choices` lists the project's writing systems when the host implements `IKnownWritingSystems`. Fixtures: `08-checks.json` "a writing system the project does not have is an error with the project's systems as choices"; `11-hardening.json` "a default writing system in the front matter that the project lacks is as unknown as one after ws=". | Pick one of the choices. | +| `fwmd.ws.unmarked-vernacular` | information | Unmarked text that looks vernacular. **The library never reports it.** The id is reserved for a host's checker, which makes the finding with `CreateDiagnostic`. | `[text]{.v}` if it is vernacular; otherwise nothing. | +| `fwmd.style.unknown` | error | A `custom-style=` the project does not have, as a character or a paragraph style. Needs `MarkdownOptions.Styles`. `Nearest` names close styles when the host implements `IKnownStyles`. Fixtures: `08-checks.json` "a character style the project does not have is an error with the nearest names", "a paragraph style the project does not have is an error with the nearest names". | Use the nearest name. | +| `fwmd.link.not-found` | error | A link path matches no object; a `silfw://` link has neither a GUID nor a path; a `motif://` link into another project has only a path. Fixtures: `05-links.json` "a path that matches nothing is an error"; `11-hardening.json` "a silfw link with neither a GUID nor a path is an error and the host is not asked". | Correct the path, or add `guid=`. | +| `fwmd.link.ambiguous` | error | A path matches more than one object (a homograph written without its number). `Choices` lists each match with its GUID. Fixture: `05-links.json` "a path that matches several objects is an error with a choice for each". | Add the homograph number (`palit(2)`) or `guid=`. | +| `fwmd.link.broken` | warning | The link is marked `.broken`, or its `guid=` is not an object of this project (with the default `UnknownGuidPolicy.MarkBroken`). The text is kept. Fixtures: `05-links.json` "a GUID this project does not know is a warning, and the link is kept for an agent's text", "a link marked broken is a warning, and is written with the flag while the target stays deleted". | Relink to an object that exists, or keep it broken. | +| `fwmd.link.guid-mismatch` | error | The URL names one GUID and `guid=` another; or an external address has `guid=`. Fixtures: `05-links.json` "a URL that names one GUID and guid= another is an error, not a guess"; `11-hardening.json` "an external address with guid= is an error, not a link whose address is dropped". | Remove the wrong one. | +| `fwmd.link.relative-without-base` | error | A link relative to the file, read with no `MarkdownOptions.LinkBase`. Fixture: `05-links.json` "a link relative to the file needs a base". | Give the reader a base, or write a root path. | +| `fwmd.link.stale-path` | information | The GUID resolves, but the object is now at another path. The next write has the current path. Fixture: `05-links.json` "the GUID is the identity: an out-of-date path is information and the next write has the current path". | None needed. | +| `fwmd.link.rebound` | information | With `UnknownGuidPolicy.RebindByPath` (interactive paste), an unknown GUID was bound to the one object at the link's path. Fixture: `05-links.json` "an interactive paste binds an unknown GUID by the one object at its path and says so". | None needed. | +| `fwmd.link.url-rebuilt` | information | `guid=` names another object than the stored `url=`, so the URL was rebuilt from the GUID. Fixture: `05-links.json` "an editor who changes guid= makes the stored URL stale: it is rebuilt from the new GUID". | None needed. | +| `fwmd.picture.unknown` | error | A picture the project does not hold. Fixture: `05-links.json` "a picture that is not in the project is an error". | Correct the path or the GUID. | + +## Record files: YAML + +All cases are in `records/02-traps.json`. A fix that "removes" something replaces it with nothing. + +| Id | Severity | Meaning | Usual fix | +|---|---|---|---| +| `fwmd.yaml.syntax` | error | YAML that does not parse. Nothing more is read. Only the id and the start line are stable; where the error sits is YamlDotNet's. Case: "YAML that does not parse is a syntax error, and nothing is read". | Correct the YAML. No fix is offered. | +| `fwmd.yaml.tab` | error | A tab in the indentation of a front-matter line. The rest is read with each tab as a space. Case: "a tab after the indentation's spaces is an error too". | Two spaces per tab. The fix is offered only when the file then reads to the same tree. | +| `fwmd.yaml.line-separator` | error | A raw U+2028, U+2029 or U+0085. One YAML reader breaks lines at them and another does not. Case: "a raw line separator in a rich block is an error whose fix is its character reference". | In a rich value, `
` (the fix). Elsewhere, no fix. | +| `fwmd.yaml.duplicate-key` | error | A key the mapping already has. The first is read. Case: "a duplicate key is an error; the first is read". | Remove the later pair (the fix; none when the pair shares a line). | +| `fwmd.yaml.anchor` | error | An anchor (`&x`). Case: "an anchor is an error whose fix removes it". | Remove it. | +| `fwmd.yaml.alias` | error | An alias (`*x`), usually a bound root such as `*palit` written without quotes. Case: "a bound root written plain is a YAML alias: an error whose fix quotes it". | `"*palit"`. | +| `fwmd.yaml.tag` | error | A tag (`!!str`, `!x`). The schema types every value. Case: "a tag is an error whose fix removes it". | Remove it. | +| `fwmd.yaml.merge-key` | error | A merge key (`<<`). It is not read. Case: "a merge key is an error whose fix removes it, since it is not read". | Remove its lines (no fix when it shares a line). | +| `fwmd.yaml.folded` | error | A folded scalar (`>`), which rewrites line ends. Cases: "folded rich text rewrites line ends: an error whose fix is the literal block", "folded plain text: an error whose fix is the quoted string". | The value as it reads, spelled canonically. | +| `fwmd.yaml.second-document` | error | A second YAML document: a `---` line with text after it, or a YAML block in the body (Pandoc merges it into the metadata). It is not read. Case: "a --- line with text after it starts a second YAML document: an error". | Remove it. | +| `fwmd.yaml.too-deep` | error | YAML nested deeper than 64 levels. The deeper parts are not read. Case: "YAML nested past the reader's limit is an error". | Flatten it. | +| `fwmd.yaml.comment-dropped` | information | A comment. It is dropped: the project has nowhere to keep it. Case: "comments are dropped with information whose fix removes them". | Remove it. | +| `fwmd.yaml.unquoted` | information | A plain (unquoted) scalar where the schema expects text. It is read as the text it spells: `no` is "no", `1.10` is "1.10", `null` is "null". Case: "the Norway problem: a plain no in a text field is the text no, with information". | The value double-quoted (for a rich value, a block). | +| `fwmd.yaml.scalar-style` | information | A value in a style that is not canonical: single quotes, other escapes, a quoted number or Boolean, a leading zero, an uppercase GUID, a needlessly quoted key, a rich value in the wrong kind of block. Cases: "a single-quoted value is read; it is written double-quoted", "012 in an integer field is twelve, written 12". | The canonical spelling. | +| `fwmd.yaml.flow` | information | A flow collection (`{...}`, `[...]`, JSON). It is read; the writer writes block style. Case: "flow collections and JSON are read, with information". | None needed. | +| `fwmd.yaml.empty` | information | An empty value (`key:`, `~`, `""`, `[]`, `{}`). It means the field is empty, which a canonical file says by leaving the key out. Case: "~ is an empty value, which is left out". | Remove the line, when the key has a line of its own. | +| `fwmd.yaml.key-order` | information | Keys out of the schema's order (`guid` not first, fields out of order). Case: "keys out of the schema's order are read, with information". | None needed. | + +## Record files: the record + +All cases are in `records/03-schema.json` unless another file is named. + +| Id | Severity | Meaning | Usual fix | +|---|---|---|---| +| `fwmd.record.not-a-record` | error | The text is not a record file: no front matter, no closing line, or no `fwmd` or `record` key. Case: "plain FieldWorks Markdown (no record key) is not a record file". | Read it with `MarkdownConverter.Parse`. | +| `fwmd.record.unknown-key` | error | A front-matter key a record file does not have (owner decision R-5). Case: "an unknown front-matter key is an error in a record file". | Remove it, or correct its spelling. | +| `fwmd.record.unknown-kind` | error | A `record:` kind the schema does not have. Case: "a record kind the schema does not have". | Use a kind of the schema. | +| `fwmd.record.schema-mismatch` | error | `schema:` names another schema than the reader's. Case: "a schema other than the reader's". | The reader's schema id (the fix). | +| `fwmd.record.unknown-field` | error | A key an object of its type does not have. `Nearest` names close fields. Case: "an unknown field, with the nearest name". | Use the field's name. | +| `fwmd.record.wrong-shape` | error | A mapping, list or scalar where the schema has another shape. Case: "a value of the wrong shape: a list where alternatives are expected, a string where an object is". | Write the shape the field takes. | +| `fwmd.record.not-an-integer` | error | Not a whole number in an integer field. Case: "not a whole number in an integer field". | A whole number. | +| `fwmd.record.not-a-boolean` | error | Not `true` or `false` in a Boolean field. For `yes`, `no`, `on`, `off` and other YAML 1.1 words the fix is the Boolean they mean. File `02-traps.json`, case "yes in a Boolean field is an error whose fix is true". | `true` or `false`. | +| `fwmd.record.bad-guid` | error | Not a GUID in a GUID field. Case: "not a GUID in a GUID field". | A GUID, lowercase with hyphens. | +| `fwmd.record.duplicate-guid` | error | One GUID on two objects of the file. Case: "one GUID on two objects". | Give each object its own GUID. | +| `fwmd.record.not-a-reference` | error | A reference that is not one FieldWorks Markdown link. Case: "a reference that is not a link". | One link with `guid=`, double-quoted. | +| `fwmd.record.plain-field-markup` | error | Formatting or another writing system in a plain-text value written as a block. A block holds plain text as FieldWorks Markdown only so that it can carry control characters. Case: "formatting in a plain text field written as a block is an error". | Remove the markup, or escape it (`\*`). | +| `fwmd.record.unwritable` | error | A value or body the reader can read but the writer cannot spell: a rich value whose first line starts with a space, or that holds a tab, a control character or a line separator; a front-matter value or a writing-system key with a control character. File `04-unwritable.json`, case "a quoted rich value with a leading space reads, but cannot be written: an error whose fix spells what was read". | What was read, written again (the fix), where that reads back the same. | +| `fwmd.record.bad-root` | error | A `root:` that is not a project path, or whose segments after the record's do not walk the schema (a sequence needs an index). Cases: "a root that does not walk the schema", "a root with a sequence and no index". | Correct the path. | +| `fwmd.record.outside-root` | error | A key in `record-data:` outside the field a partial file's `root:` names. Case: "a key outside the field the root names". | Remove it, or widen the root. | +| `fwmd.record.unexpected-body` | error | Text below the front matter of a record kind with no body field. Case: "text below the front matter of a record with no body field". | Move the text into a field, or remove it. | +| `fwmd.record.body-in-frame` | error | The body field written in `record-data:`. Its value is the body (owner decision R-6). Case: "the body field in record-data is an error". | Move the value below the front matter. | +| `fwmd.record.block-not-allowed` | error | A table, list or heading in a rich field whose block features do not allow it (ADR 0033). One finding per block. File `05-blocks.json`, case "a table in a field that allows no blocks is an error, and its fix is one plain paragraph per row". | The block as plain paragraphs, flattened as paste does (the fix; none when the value is not a literal block). | +| `fwmd.record.notation` | error | The default id of a host notation checker's finding that gives no code of its own. Case: "a notation finding with no code takes the default id". | What the checker's message says. | + +A host's notation checker usually gives its own codes. The fixtures' stub checker uses `fwmd.env.unknown-class` +(`03-schema.json`, "a notation the host's checker refuses: its finding is placed in the file, through the quoting"). That id belongs to +the fixture, not to the library. + +## LCModel adapter + +`SIL.TextEditing.LCModel` reports each property it leaves out of an `ITsString` or StyleRules because liblcm cannot load it back (ADR +0016, amended). These are **not** `MarkdownDiagnostic`s. Each is a `DroppedProperty` passed to an optional `Action` +callback of `TsStringConversion.ToTsString`, `ToStyleRules`, `ToTextParagraphs` and `TsStringClipboard.FragmentToXml`, or to +`TsStringClipboardForm.OnDropped`. A `DroppedProperty` has the property `Number`, `ParagraphIndex`, `RunIndex`, `Reason`, `Detail` and +`Message`; it has no line, column or severity. With no callback the property is dropped silently. The run model's property bag keeps it +either way. + +| Id | Meaning | Usual fix | +|---|---|---| +| `textediting.lcm.unstorable-property` | A property, or one value of it, that liblcm refuses was left out (`Reason` is `PropertyNotLoadable` or `ValueNotLoadable`). | Nothing in the text. A host that must keep the property needs its own storage. | +| `textediting.lcm.replaced-property` | A refused `rightToLeft` (128) value in a paragraph's bag was stored as the paragraph's direction instead (`Reason` is `ValueReplaced`). The data is kept, in another form. | None needed. | diff --git a/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs b/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs new file mode 100644 index 0000000..db46a01 --- /dev/null +++ b/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs @@ -0,0 +1,41 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; + +namespace SIL.TextEditing.Markdown.Tests +{ + /// Keeps the format documentation in docs/formats honest: every finding id the source can report is in the catalogue (findings.md). It reads the + /// repository from disk, so it finds the ids of every project (the LCModel adapter's too) without referencing it. + public class FormatDocsTests + { + // A whole id in a string literal: "fwmd.syntax.unclosed-mark", "textediting.lcm.unstorable-property". A prefix such as "fwmd.table." ends in a dot and is + // not an id. + static readonly Regex IdLiteral = new Regex("\"((?:fwmd|textediting)\\.[a-z0-9.-]*[a-z0-9])\"", RegexOptions.CultureInvariant); + + static string RepositoryRoot() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir != null && !File.Exists(Path.Combine(dir.FullName, "SIL.TextEditing.sln"))) dir = dir.Parent; + return dir?.FullName ?? throw new InvalidOperationException("SIL.TextEditing.sln is not above " + AppContext.BaseDirectory); + } + + static string ReadText(string path) => File.ReadAllText(path).Replace("\r\n", "\n"); + + static string FormatsDirectory => Path.Combine(RepositoryRoot(), "docs", "formats"); + + [Fact] + public void EveryFindingIdInTheSourceIsInTheCatalogue() + { + string catalogue = ReadText(Path.Combine(FormatsDirectory, "findings.md")); + var ids = new SortedSet(StringComparer.Ordinal); + foreach (string path in Directory.GetFiles(Path.Combine(RepositoryRoot(), "src"), "*.cs", SearchOption.AllDirectories)) + foreach (Match m in IdLiteral.Matches(ReadText(path))) ids.Add(m.Groups[1].Value); + Assert.NotEmpty(ids); + var missing = ids.Where(id => catalogue.IndexOf("`" + id + "`", StringComparison.Ordinal) < 0).ToList(); + Assert.True(missing.Count == 0, "docs/formats/findings.md does not list: " + string.Join(", ", missing)); + } + } +} From 0ad01102a550fff0357eecba74997a435a1ab735 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:22:34 -0500 Subject: [PATCH 02/13] docs/formats/fieldworks-markdown.md documents the FieldWorks Markdown profile, and a test checks its examples against their fixtures. Co-Authored-By: Claude Opus 5.5 --- docs/formats/fieldworks-markdown.md | 519 ++++++++++++++++++ .../FormatDocsTests.cs | 36 +- 2 files changed, 553 insertions(+), 2 deletions(-) create mode 100644 docs/formats/fieldworks-markdown.md diff --git a/docs/formats/fieldworks-markdown.md b/docs/formats/fieldworks-markdown.md new file mode 100644 index 0000000..4623997 --- /dev/null +++ b/docs/formats/fieldworks-markdown.md @@ -0,0 +1,519 @@ +# FieldWorks Markdown + +FieldWorks Markdown is the Markdown form of FieldWorks rich text. It is a strict profile of Pandoc Markdown: every file is valid Pandoc +Markdown, and Pandoc reads it to the same structure as this library (ADR 0028). It carries everything a FieldWorks paragraph holds: +writing systems, styles, every text property, links and pictures. + +| | | +|---|---| +| File name | `*.fw.md` (`MarkdownFrontMatter.FileExtension`). Only a hint: the front matter decides. | +| Clipboard id | `FieldWorks Markdown` (`ClipboardForms.FieldWorksMarkdownId`); see [Clipboard](clipboard.md). | +| Version | `fwmd: 1` (`MarkdownFrontMatter.CurrentVersion`). | +| API | `MarkdownConverter.ToMarkdown`, `Parse`, `Canonicalize`, `ToInlineMarkdown`, `ParseInline`, `AreEquivalent` in `SIL.TextEditing.Markdown`. | +| Defined by | The conformance fixtures in `tests/conformance` (their README explains the case format) and ADRs 0011, 0018, 0028 and 0029. The parser does not define the format. | + +**Read lenient, write strict.** The reader takes any spelling Pandoc reads to the same structure. The writer writes one canonical spelling +and reads every paragraph back to check it. A non-canonical spelling is not an error. `Canonicalize` gives the canonical form of a text. + +**The examples** on this page are copied from the fixtures; each names its file and case. A test (`FormatDocsTests`) checks that they +still match. A case with no front matter of its own is read after the fixtures' standard front matter: `analysis` is `en` and +`vernacular` is `ceb`. So unmarked text is English and `.v` is Cebuano. + +## A whole file + +A file is YAML front matter, a blank line, then the paragraphs. Each paragraph is one line. Blocks are separated by one blank line. Line +ends are `\n`. + + +```markdown +--- +fwmd: 1 +analysis: "en" +vernacular: "ceb" +writingSystems: + en: + lang: "en" + dir: "ltr" + font: "Charis SIL" + ceb: + lang: "ceb" + dir: "ltr" + font: "Charis SIL" + ar: + lang: "ar" + dir: "rtl" + font: "Scheherazade New" + ceb-x-old: + lang: "ceb" + dir: "ltr" + font: "Doulos SIL" +--- + +[a]{ws=ar}[b]{ws=ceb-x-old}c +``` + +## Front matter + +| Key | Value | +|---|---| +| `fwmd` | The format version, a whole number. `1` today. A newer version is refused with `fwmd.version.unsupported`, and nothing is read. | +| `analysis` | The writing system of unmarked text. | +| `vernacular` | The writing system `.v` stands for. | +| `writingSystems` | Each writing system the text uses, the two defaults first: its id as the key, then `lang` (language tag), `dir` (`ltr` or `rtl`) and `font`. | + +- The writer generates it (`MarkdownOptions.WriteFrontMatter`, default on). People do not maintain it by hand. +- Values are written double-quoted. The reader also takes bare and single-quoted values (`06-front-matter.json`, "values may be bare or + single-quoted and are written double-quoted"). +- The file's `analysis` and `vernacular` win over the host's defaults. The project's own writing-system definitions win over `writingSystems`. +- Front matter with no `fwmd` key is some other YAML and is ignored. Text with no front matter takes the host's defaults + (`MarkdownOptions.WritingSystems`). +- A top-level `record:` key makes the file a [record file](record-files.md). `Parse` refuses it with `fwmd.record.record-file`. +- A byte order mark at the start is ignored. +- Problems in the front matter are `fwmd.syntax.malformed-attribute`: `fwmd` that is not a number, an empty or list default, a tab in the + indentation (Pandoc refuses it). + +## Runs and writing systems + +A run is text in one writing system with one set of properties. Unmarked text is the `analysis` writing system. A span `[...]{...}` +gives its text other attributes. + + +```markdown +The prefix [mang-]{.v}. +``` + +- `.v` means the `vernacular` writing system. `ws=` names any other writing system of the project, such as `ws=ceb-x-old`. +- `.v` and `ws=` together are `fwmd.syntax.duplicate-attribute`. +- The writer writes `.v` for the default vernacular, `ws=` for any other non-analysis writing system, and nothing for the analysis. +- An inner span overrides an outer one. The result is always flat runs (`02-attributes.json`, "an inner span overrides an outer one, and + the result is flat runs"). +- Bold and italic set to `on` are written with stars when the run starts and ends with a letter or digit and has no other attribute. + Otherwise they are attributes: + + +```markdown +[ x]{bold=on}; [y.]{bold=on} [(z)]{italic=on} +``` + +- `^x^` and `~x~` are read as superscript and subscript, and written as `[x]{superscript=super}` and `[x]{superscript=sub}`. + +## Attributes + +An attribute block follows `]` or `)` with no space. It holds classes and `key=value` pairs, separated by one space. + +```text +block = "{" item { " " item } "}" ; "{}" is an error +item = class | pair +class = "." ALPHA { ALPHA | DIGIT | "-" | "_" } +pair = key "=" value +key = ALPHA { ALPHA | DIGIT | "-" } +value = bare | quoted +bare = 1*( ALPHA | DIGIT | "-" | "_" | "." | ":" | "#" | "/" ) +quoted = DQUOTE { char | "\" DQUOTE | "\\" } DQUOTE +``` + +(The grammar is quoted from `AttributeScanner` in `src/SIL.TextEditing.Markdown/Notation/AttributeBlock.cs`.) + +### Classes + +| Class | Meaning | +|---|---| +| `.v` | The run is in the default vernacular writing system. | +| `.name` | A link whose text is the target's current name. | +| `.example` | A link to a text segment: a glossed example. | +| `.broken` | A link whose target was deleted. | +| `.empty` | An empty paragraph's marker, `[]{.empty}`. | +| `.nul` | A NUL character, `[]{.nul}`. CommonMark cannot carry it. | +| `.surrogate` | An unpaired surrogate, `[]{.surrogate x=D800}`. CommonMark cannot carry it. | +| `.nostyle` | A link run with no character style. | + +The writer writes classes in this order: `.v .name .example .broken .empty .nul .surrogate .nostyle`. + +### Canonical order + +The writer writes, in this order: the classes; `ws=`; `custom-style=`; a link's `guid=`, `tool=` and `url=`; the named properties in +alphabetical order (ignoring case); `pN` by number; `sN` and `sNx` by number. The reader takes any order. + + +```markdown +[x]{italic=on bold=off ws=ceb-x-old} +``` + +is written + + +```markdown +[x]{ws=ceb-x-old bold=off italic=on} +``` + +### Property keys + +Every text property has a key, so nothing is lost. A property with a FieldWorks number and no named key is `pN` (an integer) or `sN` +(a string). + +| Key | Number | Values | +|---|---|---| +| `italic`, `bold` | 2, 3 | `off`, `on`, `invert` | +| `superscript` | 4 | `off`, `super`, `sub` | +| `underline` | 5 | `none`, `dotted`, `dashed`, `single`, `double`, `strikethrough`, `squiggle` | +| `align` | 17 | `leading`, `left`, `center`, `right`, `trailing`, `justify` | +| `spellcheck` | 26 | `normal`, `donotcheck`, `forcecheck` | +| `editable` | 155 | `no`, `yes`, `semi` | +| `fontsize`, `offset` | 6, 7 | millipoints; add `fontsizeUnit=rel` (or `offsetUnit=rel`) for a relative value | +| `firstIndent`, `leadingIndent`, `trailingIndent`, `spaceBefore`, `spaceAfter`, `tabDef`, `MarginTop`, `padLeading`, `padTrailing`, `padTop`, `padBottom`, `borderTop`, `borderBottom`, `borderLeading`, `borderTrailing` | 18–23, 50, 134–141 | millipoints, with a `...Unit` key as above. `marginTop` is read too. | +| `lineHeight` | 24 | millipoints, with `lineHeightUnit=rel` and `lineHeightType=exact` (default `atLeast`) | +| `forecolor`, `backcolor`, `undercolor`, `paracolor`, `borderColor` | 8, 9, 10, 25, 142 | `#rrggbb`, `transparent` or a colour name (`black`, `red`, `green`, `yellow`, `blue`, `magenta`, `cyan`, `white`) | +| `rightToLeft`, `directionDepth`, `bulNumScheme`, `bulNumStartAt`, `keepWithNext`, `keepTogether`, `widowOrphan` | 128, 129, 143, 144, 148, 149, 162 | a whole number | +| `fontFamily`, `tabList`, `tags`, `fontVariations`, `bulNumTxtBef`, `bulNumTxtAft` | strings 1, 4, 5, 130, 145, 146 | text | +| `custom-style` | string 133 | a character style on a span, a paragraph style on a div | +| `pN` | integer N | `value:variant`, such as `p150=1:default`. The variant is `default`, `mpt`, `rel`, `enum`, `ninch` or a number. | +| `sN` | string N | text | +| `sNx` | string N | the text as hex: four uppercase digits per UTF-16 code unit | + +- A named key is used only when it spells the value exactly. Otherwise the writer uses `pN`. A relative font size is written + `fontsize=150 fontsizeUnit=rel`; `p6=150:rel` is read too. +- A string with a control character or a lone surrogate is written as `sNx`, whatever its key: + + +```markdown +[]{s6x=000500610062} +``` + +- A number FieldWorks does not define is kept, with the warning `fwmd.prop.unknown-number`. +- A paragraph's direction (`Paragraph.Rtl`) is `rightToLeft=1` on its div. + +### Quoting + +A value is bare when it holds only letters, digits and `- _ . : # /`. Any other value is in double quotes. Inside quotes, `\"` and `\\` +are the only escapes. A `}`, a space or an `=` inside quotes is text. + + +```markdown +[x]{fontFamily="a\"b\\c d}e=f'g"} +``` + +- An ampersand is written `&` only where it would read as a character reference. +- A control character in a value that is not a string property (such as a style name) is a character reference: ` `. +- A value that starts with white space has that character written as a reference (` `), which Pandoc needs. +- The reader also takes single quotes, extra spaces and character references, as Pandoc does. The writer never writes them. +- Attribute values are never normalised. + +## Styles + +`custom-style=` names a style, as Pandoc's Word writer does. On a span it is a character style. On a div it is a paragraph style. + +- The writer leaves out `custom-style=` for the `Hyperlink` style on a link, because a link implies it. `.nostyle` marks a link with no + character style. +- The paragraph styles of the block syntax default to the names a new FieldWorks project has (`MarkdownStyleNames`): + +| Markdown | Paragraph style | +|---|---| +| a plain paragraph | `Normal` | +| `#`, `##`, `###` | `Heading 1`, `Heading 2`, `Heading 3` | +| `- ` | `Bulleted List` | +| `1. ` | `Numbered List`. The number is not stored; the writer numbers from 1. | +| `> ` | `Block Quote` | +| a pipe table | the host's table header and row styles. They have no default. | +| a div | the style in its `custom-style=` | + +- A heading deeper than `###` is `fwmd.heading.too-deep`. +- With `MarkdownOptions.Styles` set, a style the project lacks is `fwmd.style.unknown`. + + +```markdown +- a +- b + +1. c +2. d +``` + +## Empty and special paragraphs + +An empty paragraph has no run, so its marker names its writing system: + + +```markdown +[]{.empty} + +[]{.v .empty} + +[]{.empty ws=ceb-x-old} + +[]{.empty ws=ar} +``` + +- A styled empty paragraph is a div around the marker (`04-structure.json`, "an empty paragraph with a style is a div around the marker"). +- The marker must be the whole paragraph. `[]{.empty} more` is `fwmd.syntax.unsupported-markup`. +- `[]{.nul}` is a NUL and `[]{.surrogate x=D800}` an unpaired surrogate. They are the two characters CommonMark cannot carry. +- An embedded object with data the notation has no form for is a span over U+FFFC with the data in `s6x` (see above). + +## Divs + +A paragraph with another style, or with paragraph properties, is in a Pandoc div. One div holds a maximal run of adjacent paragraphs +with the same style and properties. Paragraphs inside are separated by blank lines. + + +```markdown +::: {custom-style=Example} +a + +b +::: + +::: {custom-style=Example spaceBefore=6000} +c +::: +``` + +- A `Normal` paragraph with properties is a div with no `custom-style=`. A heading with properties is a div with the heading style. +- Inside a styled div, block markers are text and must be escaped (`\#`). An unescaped one is `fwmd.syntax.unsupported-markup`, with + the escape as the fix. +- A div inside a div has no counterpart: `fwmd.syntax.unsupported-markup`. +- A `:::` line straight after a paragraph line is text in Pandoc. The fix adds a blank line before it. +- A soft line break in a paragraph is one space. + +## Tables + +A pipe table is one paragraph per row: the header row in the host's header style, the body rows in its row style, and a tab between +cells. [Tables](tables.md) has the details and the planned design. + + +```markdown +| Form | Gloss | +|---|---| +| [palit]{.v} | change | +| [mamalit]{.v custom-style="Emphasized Text"} | buy \| sell | +``` + +## Links + +Every reference to a FieldWorks object is a Markdown link to the object's **project path**, with `guid=`. The GUID is the identity. The +path finds the object and is refreshed when the text is written again. + + +```markdown +See [palit](/lexicon/palit(2)){.name guid=7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d}. +``` + +`.name` says that the link text is the target's current name. + +### The five forms + +The reader takes all five and converts between them without loss. `MarkdownOptions.LinkForm` picks the one the writer writes. Here is the +same link in each form. + +| Form | `MarkdownLinkForm` | Use | +|---|---|---| +| A | `RootPath` (default) | Agents, source mode, the clipboard. | +| B | `FileRelative` | A full export to files. Needs `LinkBase`, the path of the file being written. | +| C | `SilfwUrl` | A back-link that opens FieldWorks. | +| D | `MotifUrl` | A back-link that opens Motif. Needs `ProjectName`. | +| E | `ObjectAlias` | `/objects/`. Also used for any target with no path. | + + +```markdown +[palit](/lexicon/palit(2)){.name guid=7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d} +``` + + +```markdown +[palit](../../lexicon/palit(2).fw.md){.name guid=7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d} +``` + + +```markdown +[palit](silfw://localhost/link?database%3dthis%24%26tool%3dlexiconEdit%26guid%3d7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d%26tag%3d){.name} +``` + + +```markdown +[palit](motif://projects/Sena3/lexicon/palit(2)?guid=7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d){.name} +``` + + +```markdown +[palit](/objects/7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d){.name} +``` + +Form B was written with `LinkBase` `/lists/GDR/GDR-47.fw.md`, and form D with `ProjectName` `Sena3`. + +### Project paths + +- The top folders are `lexicon`, `texts`, `grammar`, `notebook`, `lists`, `reversals`, `objects` and `files` (`ProjectPath.TopFolders`). + A path under any other folder, with no GUID, is an external address. +- A homograph number is in parentheses: `palit(2)`. The reader also takes `palit₂`. +- Siblings that share a name get eight GUID digits: `Epenthesis(77e41a02)`. +- A path is written in NFC and compared under canonical equivalence, case-sensitively. +- A `/`, `%`, `?`, `#` or `\` in a name, an unbalanced parenthesis, and a name that itself ends in a number in parentheses are + percent-encoded. A destination with a space is in angle brackets: + + +```markdown +[Noun (common)](){.name guid=00000009-0000-4000-8000-000000000009} +``` + +- A fragment names a field and a writing system: `/lexicon/palit(2)#definition.en`. It is reserved: read, kept and written back, not + resolved. +- The host resolves paths through `IProjectPathResolver` (`MarkdownOptions.Paths`). With none, paths and `guid=` are kept as written. + +### Stored URLs and tools + +- FieldWorks stores a link as a `silfw://` URL. When the stored URL is not the one the writer would rebuild (extra parameters, another + project), it is kept in `url=`: + + +```markdown +[palit](/lexicon/palit(2)){.name guid=7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d url="silfw://localhost/link?database%3dthis%24%26tool%3dlexiconEdit%26guid%3d7d2c9a10-3e4f-4a5b-8c6d-7e8f9a0b1c2d%26server%3dx%26tag%3d"} +``` + +- `tool=` names the FLEx tool when the path does not give it. +- An unknown GUID makes the link broken (`fwmd.link.broken`) unless `MarkdownOptions.UnknownGuids` is `RebindByPath`, which an + interactive paste uses (`fwmd.link.rebound`). + +### External links + +An external link is a plain Markdown link and takes the same attributes as any run. + + +```markdown +[a](https://x.org/){.v bold=on}[b](https://x.org/){.nostyle}[c](https://x.org/){custom-style="Emphasized Text"} +``` + +An address that cannot stand as a destination is written `[x](<>){url="..."}`. + +## Pictures and glossed examples + +A picture is an image link to its file path, with its GUID. The alt text is the caption. + + +```markdown +![A hut](/files/pictures/hut.jpg){guid=1f2e3d4c-5b6a-4978-8695-a4b3c2d1e0f9} +``` + +A glossed example is a link to a text segment, with the class `.example`. The full interlinear is not in the Markdown. + + +```markdown +[(1)](/texts/Kuwento/3.2){.example guid=5b6c0e2a-1d3f-4c8e-9a71-0f2b3c4d5e6f} +``` + +## Escapes + +Grammar prose is full of brackets and symbols. Most of them need no escape: + + +```markdown +[ng] [+nasal] {a,b} /a/ C -> [+nasal] a_b -um- ~ -in- x@y.org +``` + +The writer escapes a character only where Pandoc would read it as markup: + +| Character | Escaped | +|---|---| +| `\`, `` ` ``, `*` | always | +| `_` | where it could open or close emphasis | +| `<` | where it could start a tag, an autolink or a comment | +| `&` | where it could read as a character reference | +| `$`, `^`, `~` | where another one follows in the paragraph; `^` also before `[` | +| `@` | unless a letter or digit comes before it (a citation) | +| `[`, `]` | a literal pair before `(`, `[` or `{`; an unbalanced bracket in a span; a pair nested deeper than eight | +| `!` | before `[` | +| `\|` | in a table row, every `\|` that is not a cell separator | +| a block marker at the start of a paragraph | `#`, `>`, `+`, `-`, `\|`, `:`, `~`, `%` (first line), a list number's `.` or `)`, `Table:` | + + +```markdown +\# not a heading + +\> not a quote + +\+ not a list + +\- not a list + +-not a list either + +\-- + +\| not a table +``` + +White space and line breaks: + +- Leading, trailing and repeated spaces are ` `. A no-break space, a tab and a form feed outside a table are numeric references. +- U+FFFC outside an object run is ``. +- A hard line break (U+2028) is `\` at the end of the line. At the start or end of a paragraph, or next to another, it is `
`. +- A literal backslash at the end of a paragraph is `\\`. +- On read, a raw tab is a space (as in Pandoc), two trailing spaces are a hard line break, and an escape that is not needed is fine. + +## Unicode: NFD and NFC + +- `Parse` and `ParseInline` return run text in NFD, the form FieldWorks stores (ADR 0018). `MarkdownOptions.NormalizeText = false` keeps + the text as written. +- Attribute values (`s6` object data, `url=`, `guid=`, `sN`, `pN`) are never normalised. +- `ToMarkdown` writes the text as stored. `AreEquivalent` treats two texts as equal when they are equal in NFD. +- Link paths are written in NFC. +- Finding offsets, lines and columns refer to the text as written (`Source`). +- Plain text the field puts on the clipboard is NFC (see [Clipboard](clipboard.md)). + +## Fields of one paragraph + +A multi-string field (an abstract, a gloss) holds one paragraph. `ToInlineMarkdown` writes it with no front matter, and `ParseInline` +reads it. Block syntax there (a heading, a list, a div, a blank line) is `fwmd.field.block-syntax`. A line break is `\` at the end of a +line. + +## Limits + +| Limit | What happens | +|---|---| +| Headings | Three levels. A deeper one is `fwmd.heading.too-deep`. | +| Nesting | Markdig's depth limit. Deeper text is one `fwmd.syntax.too-deep` finding, and its lines are kept as plain text. | +| Table size | Markdig reads a pipe table only while rows × (columns + 1) stays under about 10,200. A bigger table is `fwmd.syntax.too-deep` on read, and divs on write. | +| Homograph numbers | 1 to 999. | +| Slow inputs | A few inputs are slow in Markdig itself: tens of thousands of `[a](`, a table of about 4,000 rows, a list nested a thousand deep. A host that reads untrusted text should bound its size. | +| Writer | `ToMarkdown` throws `MarkdownWriteException` when a paragraph cannot be written so that it reads back the same. For text this should not happen; the usual cause is a host's `IObjectMapper` that does not invert itself. | + +## Unsupported Pandoc constructs + +Pandoc reads these, and FieldWorks text has no counterpart. Each is `fwmd.syntax.unsupported-markup`. Where an escape keeps the text, +the finding's fix is that escape. + +- Raw HTML, code spans, thematic breaks (`***`) and link reference definitions. +- Nested lists, and a div inside a div. +- Math (`$x$`), citations (`@doe`), inline notes (`^[note]`), strikeout (`~~x~~`) and an empty subscript (`~~`). +- Heading attributes (`# Title {#id}`). +- Fancy list markers (`a.`, `(1)`, `II.`, `A.` followed by two spaces, `(@)`, `#.`), line blocks (`| x`), grid tables, definition lines + (`: meaning`, `~ meaning` after a term) and a `Table:` caption next to a table. +- A pipe table when the host names no table styles. + +Pandoc's own rules decide, not a broader escape rule. Prose such as `M. Dixon`, `A. Consonants`, `I. Introduction`, `Table: the forms +are given in Appendix B.` or `#palit#` is plain text in both and is never reported (`12-pandoc-prose.json`). A wrapped line that starts +with `#`, `-`, `+`, `>` or `1.` stays in its paragraph, as in Pandoc. + +## Relation to Pandoc + +FieldWorks Markdown is read as Pandoc reads it with these extensions off: + +```text +pandoc -f markdown-smart-implicit_figures-implicit_header_references-task_lists +``` + +- `PandocAgreementTests` reads every fixture, its canonical form and generated documents with that reader (`-t json`), and requires the + same paragraphs. CI pins the Pandoc version. Locally the test skips when Pandoc is missing. +- Pandoc constructs the format uses: YAML metadata, bracketed spans with attributes, links and images with attributes, fenced divs, + `custom-style`, headings, lists, block quotes and pipe tables. Pandoc's `^x^` and `~x~` are read, not written. +- The library itself reads with Markdig.Signed 0.37.0 and a notation extension of its own. It moves to Markdig 1.x only if every + fixture passes unchanged (ADR 0028). + +## For integrators + +- Everything project-aware reaches the converter through `MarkdownOptions`: `WritingSystems` (`IWritingSystemMapper`, optionally + `IKnownWritingSystems`, `IWritingSystemChecker`, `IWritingSystemDetails`), `StyleNames` and `Styles` (`IStyleCatalog`, optionally + `IKnownStyles`, `IStyleChecker`), `Objects` (`IObjectMapper`, optionally `IObjectResolver`) and `Paths` (`IProjectPathResolver`). All are + optional. The defaults need no project. +- `MarkdownParseResult.Objects` lists the links and pictures the parser built, with their ranges. `CreateDiagnostic` makes a placed + finding for a host check. +- Every finding id is in [Findings](findings.md). +- A net48 host needs binding redirects for Markdig.Signed's dependencies (System.Memory, System.Buffers, + System.Runtime.CompilerServices.Unsafe). The test project's `app.config` is the model. diff --git a/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs b/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs index db46a01..e97e883 100644 --- a/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs +++ b/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs @@ -3,18 +3,24 @@ using System.IO; using System.Linq; using System.Text.RegularExpressions; +using SIL.TextEditing.Markdown.Tests.Conformance; using Xunit; namespace SIL.TextEditing.Markdown.Tests { - /// Keeps the format documentation in docs/formats honest: every finding id the source can report is in the catalogue (findings.md). It reads the - /// repository from disk, so it finds the ids of every project (the LCModel adapter's too) without referencing it. + /// Keeps the format documentation in docs/formats honest: every finding id the source can report is in the catalogue (findings.md), and every + /// example marked with a fixture comment is, character for character, the text of that fixture case. Both read the repository from disk, so they find the + /// ids of every project (the LCModel adapter's too) without referencing it. public class FormatDocsTests { // A whole id in a string literal: "fwmd.syntax.unclosed-mark", "textediting.lcm.unstorable-property". A prefix such as "fwmd.table." ends in a dot and is // not an id. static readonly Regex IdLiteral = new Regex("\"((?:fwmd|textediting)\\.[a-z0-9.-]*[a-z0-9])\"", RegexOptions.CultureInvariant); + // before a fenced block. The field is markdown, canonical or file. + static readonly Regex FixtureComment = new Regex(@"\n```[a-z]*\n(?.*?)\n```", + RegexOptions.Singleline | RegexOptions.CultureInvariant); + static string RepositoryRoot() { var dir = new DirectoryInfo(AppContext.BaseDirectory); @@ -37,5 +43,31 @@ public void EveryFindingIdInTheSourceIsInTheCatalogue() var missing = ids.Where(id => catalogue.IndexOf("`" + id + "`", StringComparison.Ordinal) < 0).ToList(); Assert.True(missing.Count == 0, "docs/formats/findings.md does not list: " + string.Join(", ", missing)); } + + [Fact] + public void EveryMarkedExampleIsTheTextOfItsFixtureCase() + { + int checkedExamples = 0; + var problems = new List(); + foreach (string doc in Directory.GetFiles(FormatsDirectory, "*.md")) + { + foreach (Match m in FixtureComment.Matches(ReadText(doc))) + { + checkedExamples++; + string file = m.Groups["file"].Value, name = m.Groups["name"].Value, field = m.Groups["field"].Value; + string where = Path.GetFileName(doc) + ": " + file + " \"" + name + "\" " + field; + string path = Path.Combine(RepositoryRoot(), "tests", "conformance", file.Replace('/', Path.DirectorySeparatorChar)); + if (!File.Exists(path)) { problems.Add(where + ": no such fixture file"); continue; } + var found = Json.Arr(Json.Parse(ReadText(path))).Select(Json.Obj).FirstOrDefault(c => Json.Str(c["name"]) == name); + if (found == null) { problems.Add(where + ": no such case"); continue; } + // A canonical of null means the input is canonical. + string? text = Json.OptStr(found, field) ?? (field == "canonical" && found.ContainsKey("canonical") ? Json.OptStr(found, "markdown") ?? Json.OptStr(found, "file") : null); + if (text == null) { problems.Add(where + ": the case has no " + field); continue; } + if (text.TrimEnd('\n') != m.Groups["text"].Value) problems.Add(where + ": the example differs from the fixture"); + } + } + Assert.True(problems.Count == 0, string.Join("\n", problems)); + Assert.True(checkedExamples > 0, "No example in docs/formats is marked with its fixture."); + } } } From e227c1189aa64600ba934852bb4f3a03b6648a68 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:25:32 -0500 Subject: [PATCH 03/13] docs/formats/record-files.md documents the record-file frame, its schema and its JSON Schema export, and the docs test checks fixture excerpts too. Co-Authored-By: Claude Opus 5.5 --- docs/formats/record-files.md | 357 ++++++++++++++++++ .../FormatDocsTests.cs | 21 ++ 2 files changed, 378 insertions(+) create mode 100644 docs/formats/record-files.md diff --git a/docs/formats/record-files.md b/docs/formats/record-files.md new file mode 100644 index 0000000..25586df --- /dev/null +++ b/docs/formats/record-files.md @@ -0,0 +1,357 @@ +# Record files + +A record file holds one FieldWorks record (a lexical entry, a notebook record, a list item), or part of one, as one text file (ADR 0032). +The YAML front matter holds the tree of objects and fields. Every rich value in it is [FieldWorks Markdown](fieldworks-markdown.md). For a +record kind with a main long-text field, that field is the Markdown body below the front matter. + +| | | +|---|---| +| File name | `*.fw.md`, like any FieldWorks Markdown file. A top-level `record:` key is what makes it a record file. | +| API | `RecordFileReader.Read(text, schema, options)` and `RecordFileWriter.Write(document, schema)` in `SIL.TextEditing.Markdown.Records`. | +| Schema | A `RecordSchema`, which is data. `RecordSchema.ToJsonSchema()` publishes it as a JSON Schema for editors. | +| Defined by | The fixtures in `tests/conformance/records` (README "Record files"), ADRs 0032 and 0033, owner decisions R-2, R-5 to R-10 and Q79. | +| Status | Shipped: the frame, the schema, the findings and the fixtures (slice R1a). Planned: export from liblcm, diffs and shape checks, and the schema built from liblcm metadata (`SIL.TextEditing.Project`, ADR 0031). | + +`MarkdownConverter.Parse` refuses a record file with one finding, `fwmd.record.record-file`, so a host cannot write back only its body. +Read it with `RecordFileReader`. + +The examples are copied from the fixtures, and `FormatDocsTests` checks them. They use the fixtures' schema `test.schema.json` +(`records/schemas.json`, entry `test`) unless they say otherwise. + +## A whole record + +A notebook record. Its `description` field is the body. Its `conclusions` field holds paragraphs, so it is a `|` block in the frame. + + +```markdown +--- +fwmd: 1 +record: "notebook/record" +schema: "test.schema.json" +root: "/notebook/Kaon survey 2026-05" +analysis: "en" +vernacular: "ceb" +writingSystems: + en: + lang: "en" + dir: "ltr" + font: "Charis SIL" + ceb: + lang: "ceb" + dir: "ltr" + font: "Charis SIL" +record-data: + guid: "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + title: |- + Kaon survey, May 2026 + type: "[Observation](/lists/record-types/Observation){guid=5c6d7e8f-9a0b-4c1d-8e2f-3a4b5c6d7e8f}" + date-of-event: "2026-05-03" + researchers: + - "[J. Cruz](){guid=6d7e8f9a-0b1c-4d2e-9f3a-4b5c6d7e8f9a}" + conclusions: | + The northern speakers use [kaon]{.v} only for cooked rice. + + The southern speakers use it for any meal. +--- + +## Observations + +On 3 May the speakers used [kaon]{.v} for every meal. +``` + +## The frame + +The front matter has these keys, and no others (owner decision R-5). The writer writes them in this order. + +| Key | Value | +|---|---| +| `fwmd` | The format version, `1`. A newer one is `fwmd.version.unsupported`. | +| `record` | The record kind, a kind of the schema: `"lexicon/entry"`. Required. | +| `schema` | The schema's id, which is also the JSON Schema's `$id`: `"test.schema.json"`. Another id is `fwmd.record.schema-mismatch`. | +| `root` | What the file holds, as a project path. Absent: the whole record. See [Partial files](#partial-files). | +| `base` | The export the file was made from, as text (a digest, such as `"sha256:9f2c41d0"`). | +| `analysis`, `vernacular`, `writingSystems` | As in [FieldWorks Markdown](fieldworks-markdown.md#front-matter). They say what unmarked text and `.v` mean in every rich value. | +| `record-data` | The record's objects and fields. | + +Any other key is `fwmd.record.unknown-key`. That rule holds in record files only; a plain FieldWorks Markdown file ignores keys it does +not know. + +### Objects and fields + +- `record-data` is the root object. Its keys are the fields of the root type, in the schema's order. +- An object of a type with GUIDs has `guid:` first: lowercase, with hyphens, double-quoted. +- An owned object is a mapping. A sequence is a list. A field per writing system is a mapping from writing-system id to value. +- A reference is one FieldWorks Markdown link with `guid=`, double-quoted. +- Integers and Booleans are plain. A Boolean is only `true` or `false`. +- Empty values are left out. An empty object in a list cannot be written; the writer refuses it. + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +record-data: + guid: "abcdef00-1111-4222-8333-444455556666" + citation-form: + ceb: "x" + homograph-number: 3 + do-not-use-for-parsing: true +--- +``` + +## Quoting + +The writer writes block style, two-space indentation and `\n` line ends. It never writes anchors, aliases, tags, flow collections or +comments. + +- **Strings are double-quoted.** `\"` and `\\` are the only escapes. Every other character, YAML's special ones included, is itself: + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +record-data: + citation-form: + ceb: "a \"b\" c\\d: #e &f *g !h |i >j {k} [l] %m @n `o" +--- +``` + +- **Keys are bare** when they are a letter followed by letters, digits, `-`, `_` and `/`, and are not a word YAML 1.1 reads as a Boolean or + null. So the Norwegian writing system `no` is written `"no"` + (`records/01-frame.json`, "a partial file whose root names a plain field: record-data holds that field only; the writing system no is a + quoted key"). +- **The schema types every value**, not YAML. A plain `no` in a text field is the text "no", with the information `fwmd.yaml.unquoted`: + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +record-data: + citation-form: + ceb: no +--- +``` + + It is written: + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +record-data: + citation-form: + ceb: "no" +--- +``` + +- **Characters no canonical file holds raw:** C0 and C1 controls (tab included), DEL, U+2028, U+2029, U+FEFF, U+FFFE and U+FFFF. One YAML + reader breaks lines at U+2028 and another does not. Plain text that holds one of them is written as a `|-` block of one paragraph of + inline FieldWorks Markdown, which spells it as a character reference: + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +record-data: + citation-form: + ceb: |- + pa lit + computed: + headword: |- + a\ + b +--- +``` + + A plain-text value written as a block may not hold formatting (`fwmd.record.plain-field-markup`). + +## Rich text: `|` and `|-` + +A rich value is FieldWorks Markdown, written as a literal block in place (owner decisions R-6 and R-8). The block header follows the field: + +| Field | Header | Pandoc reads it as | Read with | +|---|---|---|---| +| A rich field of paragraphs (an StText) | `\|` (clip) | blocks | `MarkdownConverter.Parse` | +| A rich field of one paragraph (a MultiString or String) | `\|-` (strip) | inlines | `MarkdownConverter.ParseInline` | +| Plain text that needs a block (above) | `\|-` | inlines | `ParseInline` | + +- A value never ends in a line end, so the clip block's one final line end belongs to the header, not to the value. +- Unmarked text in an alternative of a per-writing-system field means that alternative's writing system. In a field in one writing system + it means the field's default (`analysis` or `vernacular`, from the file's front matter). `.v` is the file's vernacular. +- Block syntax in a field of one paragraph is `fwmd.field.block-syntax`. +- The reader also takes a quoted rich value, another block header (`|+`, `|2-`) or a field of one paragraph in a `|` block, with the + information `fwmd.yaml.scalar-style` and the canonical block as the fix. A plain (unquoted) rich value gets `fwmd.yaml.unquoted`. A + folded block (`>`) is an error, `fwmd.yaml.folded`. + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +record-data: + comment: + en: | + a note +--- +``` + +Here `comment` is a field of one paragraph, so the writer writes `|-`. + +## The body + +- A record kind whose schema names a **body field** writes that field as the Markdown body: one blank line after the front matter, ending + with a line end. In the fixtures, `notebook/record` has `description` and `lists/item` has `discussion`. `lexicon/entry` has none. +- The body field is never in `record-data` (`fwmd.record.body-in-frame`). +- Text below the front matter of a kind with no body is `fwmd.record.unexpected-body`. Blank lines are not a body. +- The body is trimmed of spaces, tabs and line ends only. A no-break space at its end is text. +- A YAML block in the body is `fwmd.yaml.second-document`: Pandoc would merge it into the metadata. + +## Partial files + +`root:` names what the file holds. Agents write partial files by default (owner decision Q62). + +- The first `PathSegments` segments of the path name the record (2 for `lexicon/entry`: `/lexicon/palit(2)`). +- Each further segment is a field of an owned object, followed by a 1-based index for a sequence: `/lexicon/palit(2)/senses/1`. + `record-data` then holds that object's fields. +- A fragment names a field, and after a dot one writing system: `#gloss`, `#definition.en`. + - A fragment that names one rich value (a field in one writing system, or one alternative) makes that value the body. + - Any other fragment makes `record-data` hold that field only. +- A root that does not walk the schema is `fwmd.record.bad-root`. A key outside the field the root names is `fwmd.record.outside-root`. + + +```markdown +--- +fwmd: 1 +record: "lexicon/entry" +schema: "test.schema.json" +root: "/lexicon/palit(2)/senses/1#definition.en" +analysis: "en" +vernacular: "ceb" +--- + +to pay money for [something]{.v} +``` + +## Read lenient, write strict + +- **Information** (`fwmd.yaml.unquoted`, `scalar-style`, `flow`, `empty`, `key-order`, `comment-dropped`): the file is read as it means, + and the writer spells it canonically. Comments are dropped, because the project has nowhere to keep them. +- **Errors** for what YAML readers disagree on or what would lose data: tabs in indentation, raw line separators, duplicate keys, anchors, + aliases, tags, merge keys, folded blocks, a second document, nesting deeper than 64, and YAML that does not parse. +- **Read, then write, is total.** A value the reader can read but the writer cannot spell is an error, `fwmd.record.unwritable`, whose + fix spells what was read. A file read with no error can always be written, and reads back the same. +- Findings inside a rich value or the body keep the FieldWorks Markdown ids, placed in the file. Every id is in [Findings](findings.md). +- `RecordReadOptions.Markdown` gives the project's writing systems, styles and objects for the rich values. `RecordReadOptions.Notations` + is the host's `INotationChecker` for notation fields. +- Not checked yet: whether a reference resolves, its target's class, collection order, deletes and moves (R2). + +## Block features + +A rich field of paragraphs says which of tables, lists and headings it allows (ADR 0033). The default is all three. A field of one +paragraph allows none. A block a field does not allow is `fwmd.record.block-not-allowed`, one finding per block, with a fix that flattens +it as paste does. This case uses the schema `blocks` and the project `records-tables`, which names table styles: + + +```markdown +--- +fwmd: 1 +record: "notebook/record" +schema: "blocks.schema.json" +record-data: + plain: | + Intro. + + | Name | Gloss | + |---|---| + | palit | buy | +--- +``` + +The fix replaces the table's lines with `Name | Gloss`, a blank line and `palit | buy`. + +## The schema is data + +A `RecordSchema` describes object types and record kinds. Hosts build one with `RecordSchemaBuilder`. The planned +`SIL.TextEditing.Project` will build it from liblcm metadata. The fixtures give schemas as JSON in `records/schemas.json`, in a form the +conformance README documents: + + +```json + { + "name": "plain", + "kind": "rich", + "paragraphs": true, + "blocks": [] + }, +``` + +| Field kind | Builder | Value in the file | +|---|---|---| +| `Text` | `Text`, `MultiText`, `Texts` | double-quoted plain text | +| `Rich` | `Rich`, `MultiRich` (`paragraphs`, `blocks`) | FieldWorks Markdown in a `\|` or `\|-` block | +| `Notation` | `Notation`, `MultiNotation`, `Notations` | double-quoted text the host's checker reads (an environment, a feature structure) | +| `Integer`, `Boolean`, `Guid` | `Integer`, `Boolean`, `Guid` | plain number, `true`/`false`, quoted lowercase GUID | +| `Reference` | `Reference`, `References` | one FieldWorks Markdown link with `guid=`, double-quoted | +| `Object` | `Object`, `Objects` | a mapping of the object type's fields | +| `Sequence` | `Objects`, `References`, `Texts`, `Notations` | a list | + +- An object type has fields in canonical order and `HasGuid` (default true). +- A record kind has a name (`record:`), a root type, `PathSegments` and an optional body field (`RecordSchemaBuilder.Kind`). +- A text field in one writing system has a default: `RecordWritingSystem.Analysis` or `Vernacular`. +- A field name has no `.`, `/`, `#`, space or control character, because those delimit field paths (`senses/1/definition.en`). + +## JSON Schema export + +`RecordSchema.ToJsonSchema()` writes a JSON Schema (draft-07) of the front matter, for editors and checkers (owner decision R-7). The same +schema always gives the same bytes. `records/test.schema.json` is the fixtures' `test` schema, byte for byte. + +- `$id` is the schema's id. `record` is an enum of the kinds. `schema` is a constant. Unknown front-matter keys are not allowed. +- One definition per object type. For each kind, an `if`/`then` gives `record-data` its type when `root` is absent or names the record + itself. A kind with a body uses a `--frame` definition that leaves the body field out. +- Every text value has `contentMediaType`: + +| Value | `contentMediaType` | +|---|---| +| Plain text | `text/plain` | +| Rich text of paragraphs | `text/x-fieldworks-markdown` | +| Rich text of one paragraph, and a reference | `text/x-fieldworks-markdown-inline` | +| A notation | `text/x-fieldworks-`, such as `text/x-fieldworks-environment` | + +- A rich field of paragraphs that does not allow every block kind has `x-fieldworks-blocks`, the kinds it allows (`[]`, `["lists"]`, ...). + A field that allows all has none. + + +```json + "plain": { + "type": "string", + "contentMediaType": "text/x-fieldworks-markdown", + "x-fieldworks-blocks": [] + }, + "lists-only": { + "type": "string", + "contentMediaType": "text/x-fieldworks-markdown", + "x-fieldworks-blocks": ["lists"] + }, +``` + +VS Code's YAML extension does not validate Markdown front matter by itself. Until a FieldWorks extension exists, Motif's checker and its +command line validate record files against the JSON Schema (owner decision R-10). + +## What the tests hold the format to + +- `RecordConformanceTests` runs every case: the document read, the findings, and the canonical text written back byte for byte. +- `RecordGateTests`: YamlDotNet's representation model reads each canonical front matter to the same tree; Pandoc (with the flags in + [Relation to Pandoc](fieldworks-markdown.md#relation-to-pandoc)) reads each canonical file, its rich values and its body as the reader + does; a whole-record canonical file validates against the published JSON Schema. +- `RecordFuzzTests` mutates the canonical cases (seeded). The reader never throws, and a mutated file it reads without an error writes and + reads back without one. diff --git a/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs b/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs index e97e883..de10d98 100644 --- a/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs +++ b/tests/SIL.TextEditing.Markdown.Tests/FormatDocsTests.cs @@ -21,6 +21,10 @@ public class FormatDocsTests static readonly Regex FixtureComment = new Regex(@"\n```[a-z]*\n(?.*?)\n```", RegexOptions.Singleline | RegexOptions.CultureInvariant); + // before a fenced block that is part of that fixture file, as it stands. + static readonly Regex ExcerptComment = new Regex(@"\n```[a-z]*\n(?.*?)\n```", + RegexOptions.Singleline | RegexOptions.CultureInvariant); + static string RepositoryRoot() { var dir = new DirectoryInfo(AppContext.BaseDirectory); @@ -69,5 +73,22 @@ public void EveryMarkedExampleIsTheTextOfItsFixtureCase() Assert.True(problems.Count == 0, string.Join("\n", problems)); Assert.True(checkedExamples > 0, "No example in docs/formats is marked with its fixture."); } + + [Fact] + public void EveryMarkedExcerptIsPartOfItsFixtureFile() + { + var problems = new List(); + foreach (string doc in Directory.GetFiles(FormatsDirectory, "*.md")) + { + foreach (Match m in ExcerptComment.Matches(ReadText(doc))) + { + string file = m.Groups["file"].Value; + string path = Path.Combine(RepositoryRoot(), "tests", "conformance", file.Replace('/', Path.DirectorySeparatorChar)); + if (!File.Exists(path)) problems.Add(Path.GetFileName(doc) + ": " + file + ": no such fixture file"); + else if (ReadText(path).IndexOf(m.Groups["text"].Value, StringComparison.Ordinal) < 0) problems.Add(Path.GetFileName(doc) + ": " + file + ": the excerpt is not in the file"); + } + } + Assert.True(problems.Count == 0, string.Join("\n", problems)); + } } } From 0f201f640f8f12f1b9a96a32ae6b46b18562d0a5 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:25:42 -0500 Subject: [PATCH 04/13] docs/formats/clipboard.md documents the clipboard forms, paste order, writing-system rule and backends, with Word HTML/RTF and tables marked planned. Co-Authored-By: Claude Opus 5.5 --- docs/formats/clipboard.md | 153 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 docs/formats/clipboard.md diff --git a/docs/formats/clipboard.md b/docs/formats/clipboard.md new file mode 100644 index 0000000..7de5335 --- /dev/null +++ b/docs/formats/clipboard.md @@ -0,0 +1,153 @@ +# Clipboard + +A field copies a selection in several **clipboard forms** at once and pastes the best one the clipboard holds (ADR 0029). The host +registers the forms. The field itself depends on neither liblcm nor the Markdown library. + +## The forms + +| Form | Id | Clipboard format name | Payload | Provided by | Default paste order | "Copy as" | Status | +|---|---|---|---|---|---|---|---| +| FieldWorks TsString | `TsString` (`ClipboardForms.TsStringId`) | `TsString` | a host object: FieldWorks' .NET-serialised `TsStringWrapper` | the FieldWorks host | 1st | no | The id ships. The host writes the form and a clipboard backend that can store an object. | +| FieldWorks Markdown | `FieldWorks Markdown` (`ClipboardForms.FieldWorksMarkdownId`) | chosen by the form | text: [FieldWorks Markdown](fieldworks-markdown.md) with its front matter | the host, around `MarkdownConverter` | 2nd | yes | The id ships; no form class ships yet. The Markdown library is to provide one. | +| Plain text | `text/plain` (`ClipboardForms.PlainTextId`) | the system's text format | text: the paragraphs' text joined with `\n`, in NFC | built into the field | 3rd | always | Shipped. | +| TsString XML | `SIL.TextEditing.TsStringXml` (`TsStringClipboard.XmlFormatName`) | `SIL.TextEditing.TsStringXml` | text: the XML of one `ITsString` | `TsStringClipboardForm` in `SIL.TextEditing.LCModel` | not in it; the host adds the id | no | Shipped. | +| Runs dump | `SIL.TextEditing.Runs.v1` | `SIL.TextEditing.Runs.v1` (`Interchange.ClipboardFormat`) | text: `Interchange` XML | `InterchangeClipboardForm` in `SIL.TextEditing` | not in it | yes | Shipped, for tests and debugging only. | + +### FieldWorks TsString + +FieldWorks puts a .NET-serialised `TsStringWrapper` on the clipboard under `TsString`. That class lives in FieldWorks' own SimpleRootSite +assembly. This library neither references nor copies it, and nothing here writes under `TsString`. A FieldWorks host registers its own +form with the id and format name `TsString`: + +- its `Write` builds the wrapper from `TsStringClipboard.FragmentToXml`; +- its `Read` unwraps it and calls `TsStringClipboard.FromXmlToFragment`; +- its host clipboard backend stores and reads the object, since Avalonia's clipboard cannot. + +A TsString copy of several paragraphs joins them with `"; "` and pastes as one paragraph, as FieldWorks does. With TsString first in +the paste order, a copy of several paragraphs from one field to another loses its paragraph breaks unless the FieldWorks Markdown form is +tried first. + +### FieldWorks Markdown + +The payload is a whole FieldWorks Markdown text **with its front matter**, so unmarked text never takes its writing system from the +target project's defaults. Read it with `MarkdownConverter.Parse` and write it with `MarkdownConverter.ToMarkdown`. FieldWorks Markdown +is a back end first (agents, files, drafting); on the clipboard it is an extra form, not the default. + +### Plain text + +Every copy puts the plain text on the clipboard: the paragraphs' text joined with `\n`, in NFC (ADR 0018). The field stores NFD; text +meant for other programs is NFC. The rich forms keep the stored form. + +### TsString XML + +`TsStringClipboardForm` carries the XML of one `ITsString`, written by liblcm's own `TsStringSerializer`. It is **not** FieldWorks' wire +format, so it uses its own id and format name, `SIL.TextEditing.TsStringXml`, and can sit beside a host's `TsString` form. + +- Copy joins the paragraphs with `WriteParagraphSeparator` (default `"; "`, as FieldWorks does). +- Paste gives one paragraph, as FieldWorks does, unless `PasteParagraphSeparator` is set; then it splits there. +- A writing system the host's handles do not know keeps its id during the read, so the paste rule below can map it. +- A property liblcm cannot load back is left out and reported to `OnDropped` (see [Findings](findings.md#lcmodel-adapter)). +- The id is not in the default paste order. A host that wants it adds it. + +### Runs dump (tests and debugging) + +`Interchange.ToXml` writes every run, writing system and property, with nothing lost. ADR 0029 retired it from the user's clipboard; +`InterchangeClipboardForm` exists so that tests and the playground can carry formatting without liblcm or the Markdown library. Its +shape, from `src/SIL.TextEditing/Interchange.cs`: + +| Element | Attributes | +|---|---| +| `Runs` | the root | +| `Para` | `style`, `rtl` (`1` or `0`), `defws`, then one attribute per property: `i_=":"` for an integer property, `s_=""` for a string property | +| `Run` | `ws`, `objdata` when the run has object data, and the property attributes; the text is the content | + +A backslash, a control character, a lone surrogate and U+FFFC are escaped as `\\` and `\uXXXX` in text and string values, so any XML +reader gets the exact UTF-16 back. `Interchange.Json` is a JSON dump of a document for exact comparison in tests; it is not a clipboard +form. + +## Copy + +- `RichTextField.CopyAsync` asks every registered form for its payload and puts them all on the clipboard with the plain text. +- A form that returns null, returns an empty string or throws is left out. The others still go. +- The backend reports the format names it could not store. The field raises `ClipboardFormsNotStored` with them. + +## Paste + +`RichTextField.PasteAsync` tries the ids of `ClipboardForms.PasteOrder` in turn and uses the first form that the clipboard holds and that +reads. + +- The default order is FieldWorks' own: `TsString`, `FieldWorks Markdown`, `text/plain`. +- The host may replace or edit the list. An id with no registered form is skipped. A form whose id is not in the list is never read. +- A payload a form cannot read (null, or an exception) is skipped, so paste falls through to the next form. +- The editor normalises the pasted text to the form the host stores (`Document.Normalization`; NFD for FieldWorks). +- A block the field does not allow is flattened, and the field raises `BlocksFlattened` (ADR 0033). + +## The writing-system rule + +Text in a writing system the target project lacks takes the writing system of the selection it replaces, as FieldWorks does +(`ClipboardWritingSystems.MapMissing`). + +- The field asks `RichTextField.KnowsWritingSystem`. By default that is its `WritingSystems` registry, and an empty registry accepts every + writing system. +- After a paste that mapped something, the field raises `WritingSystemsMapped` with the missing ids and the target, so the host can show a + notice. +- A run with no writing-system id takes the target's too, but is not listed as missing. +- The rule runs for every rich form. Plain text has no writing system; it is pasted as typed text is. + +## Copy as + +`RichTextField.CopyAsAsync(formId)` puts one form on the clipboard as text: the plain text (`text/plain`), or a form whose +`OffersCopyAs` is true, such as FieldWorks Markdown. Only a form whose payload is a string can be copied this way. `CopyAsAsync` returns +false for any other, and for a form with nothing to say. `ClipboardForms.CopyAsForms` lists the forms a menu offers. + +## Backends + +The field reads and writes the clipboard only through `IClipboardBackend` (`RichTextField.ClipboardBackend`): + +| Member | Does | +|---|---| +| `SetAsync(plainText, forms)` | Replaces the clipboard with the plain text and the named payloads. Returns the names it could not store. | +| `GetAsync(formatName, kind)` | The payload under a name, read as the form's `ClipboardPayloadKind`, or null. Must not throw on data it does not understand. | +| `GetTextAsync()` | The plain text, or null. | + +- Payloads pass through untouched. A form says what it holds (`PayloadKind`): `Text` (a string), `Bytes` (a `byte[]`) or `Object` (a + host object). The backend reads a format back as that kind and never guesses text from bytes. +- The default backend is Avalonia's clipboard of the field's window (`AvaloniaClipboardBackend`). It stores strings and `byte[]`s. It + cannot store a host object and reports it as not stored. +- FieldWorks' WinForms host plugs in its own backend, so it can keep the clipboard itself and store the `TsStringWrapper` object. + +## Planned: Word HTML and RTF + +> **Planned. Nothing in this section is built.** It records owner decisions and research proposals so that integrators know what is coming. + +**Decided by the owner** (2026-11-05, `decisions-2026-11-05-records.md`): + +| Decision | | +|---|---| +| R-3 | Word-compatible copy and paste, "as much as that will work". This reverses ADR 0029's "No HTML or RTF" (amended by ADR 0031). TsString stays first on paste between FieldWorks fields. | +| Q65 | Copy writes HTML and RTF. Paste reads HTML only. Reading RTF is decided after testing with Word. | +| Q66 | Paste from Word keeps meaning (emphasis, links, styles, language, writing system) and drops fonts, sizes and colours. A host setting allows full formatting. | +| Q67 | Writing systems travel as character styles named `