From 1e2bd21f2373fd719ad0f39d19ad4dd8cd7f4de1 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:41:38 -0500 Subject: [PATCH 1/6] docs/reference holds the library's reference material (run model, editing, clipboard, spelling, rendering, Avalonia versions, LCModel, Markdown, limitations), moved from the README. Co-Authored-By: Claude Opus 5.5 --- docs/reference/README.md | 17 +++ docs/reference/avalonia-versions.md | 58 ++++++++ docs/reference/clipboard.md | 39 ++++++ docs/reference/editing.md | 118 ++++++++++++++++ docs/reference/lcmodel.md | 48 +++++++ docs/reference/limitations.md | 28 ++++ docs/reference/markdown.md | 113 +++++++++++++++ docs/reference/projects.md | 83 +++++++++++ docs/reference/rendering.md | 105 ++++++++++++++ docs/reference/run-model.md | 56 ++++++++ .../spelling-and-language-service.md | 132 ++++++++++++++++++ 11 files changed, 797 insertions(+) create mode 100644 docs/reference/README.md create mode 100644 docs/reference/avalonia-versions.md create mode 100644 docs/reference/clipboard.md create mode 100644 docs/reference/editing.md create mode 100644 docs/reference/lcmodel.md create mode 100644 docs/reference/limitations.md create mode 100644 docs/reference/markdown.md create mode 100644 docs/reference/projects.md create mode 100644 docs/reference/rendering.md create mode 100644 docs/reference/run-model.md create mode 100644 docs/reference/spelling-and-language-service.md diff --git a/docs/reference/README.md b/docs/reference/README.md new file mode 100644 index 0000000..e0adf0e --- /dev/null +++ b/docs/reference/README.md @@ -0,0 +1,17 @@ +# Reference + +How SIL.TextEditing works, for developers who host it or change it. The repository began as the native-editor prototype; what it still lacks is in [Known limitations](limitations.md) and the README's [Roadmap](../../README.md#roadmap). The formats it reads and writes are in +[docs/formats](../formats/README.md); the decisions behind it are in [docs/adr](../adr). + +| Page | What it covers | +|---|---| +| [Projects and tests](projects.md) | Every project in detail, the test projects, the opt-in real-clipboard tests, the build environment. | +| [The run model](run-model.md) | `Run`, `Paragraph`, `RunList`, the property bag, change notices, threading, work counters. | +| [Editing, keyboards and host hooks](editing.md) | FieldWorks editing semantics, selection and the word rule, block features, keyboard switching, find, context menu, scrolling. | +| [Clipboard in the field](clipboard.md) | The field's side of the clipboard forms and the FieldWorks interop hooks. | +| [Spell checking and the language service](spelling-and-language-service.md) | Dictionaries, squiggles, background checking, what a host calls after a change, the notation box, completion. | +| [Rendering](rendering.md) | The properties drawn, justification, line limits, trailing spaces, very long paragraphs, graphemes, colours and themes. | +| [Two Avalonia versions](avalonia-versions.md) | Avalonia 11.3.17 and 12.1.2 from one source tree, the per-version differences, the Avalonia faults the layout works around. | +| [The LCModel adapter](lcmodel.md) | `TsStringConversion`, the TsString XML form, FieldWorks' own TsString wire format. | +| [The Markdown library](markdown.md) | Integrating `MarkdownConverter` and record files: host interfaces, robustness, AOT, net48 binding redirects, tests. | +| [Known limitations](limitations.md) | Accessibility, input, undo. | diff --git a/docs/reference/avalonia-versions.md b/docs/reference/avalonia-versions.md new file mode 100644 index 0000000..008b781 --- /dev/null +++ b/docs/reference/avalonia-versions.md @@ -0,0 +1,58 @@ +# Two Avalonia versions + +One source tree serves both Avalonia generations (ADR 0009). + +| Target | Avalonia | Who pins it | Runs on | +|---|---|---|---| +| netstandard2.0 | 11.3.17 | FieldWorks | net48, inside WinForms | +| net10.0 | 12.1.2 | Motif | net10.0 | + +The versions are set per target framework in `Directory.Packages.props`. + +## Per-version differences + +There is currently no `#if` in the source for the Avalonia API: + +- The spike's one per-version spot (the clipboard API) is not needed, because Avalonia 11.3.17 already has the `DataTransfer` API that 12 + uses. +- `FontFeature` parsing is written without `FontFeatureCollection.Parse` (absent in 11.3) for both. + +Keep any future per-version difference small, behind `#if NET10_0_OR_GREATER`, and list it here. The differences now: + +- **`SpellScanner`** asks `CharUnicodeInfo` for the category of a code point directly on net10.0 and through a one-character string on + netstandard2.0 (no overload there). That only costs an allocation for a letter outside the basic plane. +- **`ScriptPlacement`** reads a font's OS/2 and hhea tables with `IGlyphTypeface.TryGetTable` on netstandard2.0. Avalonia 12 gives no + access to tables, so on net10.0 it reads them from the font file's own table directory: once for each font; in a font collection from the + member whose family name is the typeface's; and nothing is read whole. + +## Packages used by the samples only + +Avalonia.Desktop, Avalonia.Themes.Fluent and Avalonia.Win32.Interoperability are used by the samples only; the libraries do not reference +them. The playground tests load the Fluent theme so the sample's buttons and pickers have templates. + +## Avalonia faults, pinned by tests + +Tests pin these faults. When such a test starts failing, remove the workaround, not the test. + +- **Right-to-left hit testing on 11.3.17.** Hit testing in a right-to-left paragraph is wrong on 11.3.17 (fixed in 12.1.2). +- **Lines of several runs, both versions.** On both 11.3.17 and 12.1.2 the caret, hit testing and selection of a left-to-right run are + wrong on a right-to-left paragraph's line of two or more runs, and likewise on a left-to-right paragraph's line of two or more runs with + a right-to-left run in it (on 12 some of its ranges throw). + - The layout works around these first two by reading such a line from its runs as they are drawn (`LineInfo.UsesRuns`: caret x, hit + testing, visual stops and selection boxes from each run's glyphs). + - It is not behind `#if`, because 12 has the second fault too (ADR 0027, which supersedes the 11.3-only scope of ADR 0015). +- **Inline objects in right-to-left text** are mis-ordered on both. `RichTextField.ObjectAsGlyph` is the workaround. +- **Wrapped Arabic before trailing spaces, 12 only.** The spaces a wrapped line ends with after a run of Arabic in a left-to-right paragraph + come with glyph clusters numbered against the run's direction, and answer distances and hits that do not add up (a click at the end of + the line lands one space short). `RunSlot` places such a run by counting its spaces (`ClusterOrderTests` names the fault). +- **Zero width joiner, 11.3 only.** The formatter throws "Cannot split: requested length N consumes entire run" for some sequences with a + zero width joiner, such as one after a space before Arabic text in a left-to-right paragraph (`ZwjAfterSpaceTests` names it). + - The layout catches the refusal and lays the paragraph out again with the joiners swapped for word joiners, then the other format + characters, then (for a space followed by a mark, the other way the formatter refuses) the space swapped for a dotted circle so the mark + is still drawn, and as a last resort every format character and mark swapped for a word joiner. The model, the offsets and the grapheme + boundaries are untouched. + - Two costs of the retry, on 11.3 only: the no-break space it puts before a mark does not hang at a line end the way a space does, and the + dotted circle is drawn where the model has a space (the caret and the offsets still treat it as the space). +- **Empty ranges, 12 only.** Asking for the boxes of a range that covers no width (a right-to-left mark alone before the spaces a line ends + with) throws "Covered length must be greater than zero". The layout catches it, in the boxes of a range and in the spaces a justified line + stretches, and answers that such a range has no box (`RlmWrapTests` names it). diff --git a/docs/reference/clipboard.md b/docs/reference/clipboard.md new file mode 100644 index 0000000..601416d --- /dev/null +++ b/docs/reference/clipboard.md @@ -0,0 +1,39 @@ +# Clipboard in the field (ADR 0029) + +The clipboard forms, the paste order, the writing-system rule, "Copy as" and the backend interface are documented in +[docs/formats/clipboard.md](../formats/clipboard.md). This page is the field's side, in brief. + +- The host registers the clipboard forms the field copies and pastes. The field takes no dependency on liblcm or the Markdown converter. +- `RichTextField.ClipboardForms` holds `IClipboardForm`s: an id, a display name, a clipboard format name, `Write` and `Read` over an + `object` payload (a string, a `byte[]` or a host object), and a `PayloadKind` saying which, so the backend reads the format back as that + and never guesses text from bytes. The clipboard forms types (`IClipboardForm`, `ClipboardForms`) and the paste writing-system rule live in + the UI-free `SIL.TextEditing`. +- The clipboard itself is behind `RichTextField.ClipboardBackend` (`IClipboardBackend`), which is Avalonia's clipboard unless the host sets + its own. Payloads pass through it untouched. The default backend stores strings and `byte[]`s, each read back as the kind the form + declares. A host object needs a host backend, and `RichTextField.ClipboardFormsNotStored` tells the host which forms the default backend + could not store. See [Backends](../formats/clipboard.md#backends). +- Plain text is built in. `CopyAsAsync` returns false for a form whose payload is not text. +- **Copy** puts every registered form on the clipboard and the plain text in NFC. A form that throws or has nothing to say is left out. +- **Paste** tries `ClipboardForms.PasteOrder` in turn, by default TsString, FieldWorks Markdown, plain text (FieldWorks' own preference), + and uses the first form the clipboard holds and that reads. The list is the host's to change; a form that is not in it is never read. +- **Writing systems the target lacks** take the writing system of the selection the text replaces, as FieldWorks does. The field asks + `RichTextField.KnowsWritingSystem` (by default: its `WritingSystems` registry, and a registry with no entries accepts everything), and + raises `WritingSystemsMapped` after the paste so the host can show a notice. +- **Copy as** (`RichTextField.CopyAsAsync(formId)`) puts one form on the clipboard as text: the plain text, or a form that offers it + (`IClipboardForm.OffersCopyAs`, such as FieldWorks Markdown). The playground's context menu has a "Copy as" submenu. +- `SIL.TextEditing.Runs.v1` is no longer on the clipboard. `Interchange` stays as a test and debug dump, and `InterchangeClipboardForm` + lets tests and the playground carry formatting without liblcm or the Markdown library. + +## FieldWorks interop + +FieldWorks interop is the host's job and uses two hooks, the backend and a host-object payload. They are described in +[The LCModel adapter](lcmodel.md#it-is-not-fieldworks-tsstring-wire-format) and +[Clipboard: FieldWorks TsString](../formats/clipboard.md#fieldworks-tsstring). + +A FieldWorks host also registers the FieldWorks Markdown form. The Markdown library is to provide it (issue #168); only the registration +point and the ids `ClipboardForms.FieldWorksMarkdownId` and `ClipboardForms.TsStringId` exist today. + +## The real clipboard + +`tests/SIL.TextEditing.RealClipboard.Tests` puts bytes and Arabic text through the default backend and the real Windows clipboard. See +[Projects and tests](projects.md#test-projects). diff --git a/docs/reference/editing.md b/docs/reference/editing.md new file mode 100644 index 0000000..9e29913 --- /dev/null +++ b/docs/reference/editing.md @@ -0,0 +1,118 @@ +# Editing, keyboards and host hooks + +How `RichTextField` (in `src/SIL.TextEditing.Avalonia`) edits, and the hooks a host uses. The field does layout with Avalonia's +`TextFormatter`, the caret, selection, hit testing, bidi caret stops, the IME client, and copy, cut, paste and "Copy as" through the host's +clipboard forms ([Clipboard](clipboard.md)). Editing itself is `DocumentEditor`, in the UI-free model. + +## FieldWorks editing semantics (ADR 0018) + +- **Arrow keys are logical by paragraph direction.** Left goes backwards in a left-to-right paragraph and forwards in a right-to-left one; + Right is the reverse. Inside mixed-direction text the caret may therefore move the other way on screen. Shift+arrows extend the same way, + Ctrl+arrows go by word in logical order, Up/Down and Home/End are unchanged. The caret is drawn at the visual place of its logical + offset (trailing edge of the cluster passed after a forward step, leading edge of the next cluster after a backward step). +- **Backspace** removes one code point (a surrogate pair is one); an embedded object goes whole. +- **Delete** removes one base character and the combining marks (Mn, Mc, Me) after it. +- **Normalisation.** `Document.Normalization` (also `RichTextField.Normalization`) is a nullable `NormalizationForm`, null by default (text + is inserted as it arrives). A host sets `NormalizationForm.FormD` for FieldWorks: typed text, IME commits and pasted text (plain or rich) + are converted before insertion and the caret lands after the converted text. Plain text the field puts on the clipboard is always NFC; + the rich forms keep the stored form. + +## Selection + +Selection by mouse is done: click with bidi and soft-wrap affinity, Shift+click, drag across paragraphs, double-click word, triple-click +paragraph. So is selection by keyboard: Shift+arrows, Ctrl+arrows by word, Home/End by visual line, Ctrl+Home/End, Ctrl+A, Up/Down with the +preferred x. + +- Left/Right and Ctrl+Left/Right are logical. Home/End in a right-to-left paragraph are the logical line start and end. +- **The word rule** (`WordBreaks`) is deliberately simple: runs of letters, digits and marks, with an apostrophe between letters inside the + word. It has no dictionary, so Thai and Chinese are not split, and it is not UAX #29 word breaking. +- The word rule is also the host's. One rule (`IWordRules`: which code points form words in a writing system, and which join the words on + each side of them) serves the double-click, Ctrl+Left and Right, the whole-word find, the word of a context request and the spell checker. + Set `RichTextField.WordRules`, or the rule of `SpellChecking` is used. So a writing system that makes a hyphen a letter makes "foo-bar" one + word everywhere. +- The double-click (and a drag that goes by words) also stops where the writing system or the spell-check property changes, and where two + different character styles, resolved by the host's `IStyleResolver`, differ in either. A style that only changes bold or colour keeps the + word, and with no resolver a style name never splits it. This is what FieldWorks does, so a word that runs across such a change is + selected a piece at a time. The other uses of the rule take the whole word. + +## Block features (ADR 0033) + +Which of **tables, lists and headings** a field may hold is `RichTextField.BlockFeatures` (forwards to `DocumentEditor.BlockFeatures`), a +`BlockFeatures` flags value (`None`, `Tables`, `Lists`, `Headings`, `All`) beside `SingleParagraph`. The default is `All`, so nothing changes +until a host turns a kind off (owner decision Q79). When a kind is off: + +- **Commands that make it are refused.** The editor has no style command (the host applies a paragraph style as an undoable delta), so the + host asks `RichTextField.IsParagraphStyleAllowed(styleName)` before it applies one and hides what is off. The playground has a switch for + each kind, hides the styles of a kind that is off in its style box and refuses them in `ApplyParagraphStyle`. +- **A paste, and so a Markdown paste, flattens it.** A table becomes one plain paragraph per row with the cells joined by a space, a bar and + a space (`a | b`), a list item or a heading becomes a plain paragraph, and `RichTextField.BlocksFlattened` tells the host how many of each + (the same way `WritingSystemsMapped` does). Blocks the document already holds are kept, undo restores them exactly, and Enter in one + starts a plain paragraph. +- **How a paragraph is a table, list or heading is decided in one place**, `IBlockRecognizer` (`RichTextField.BlockRecognizer`). The + default, `StyleBlockRecognizer`, goes by the host's paragraph style names; `MarkdownStyleNames.ToBlockRecognizer()` makes one from the + names the Markdown reader uses (a host that pastes Markdown gives the field the recogniser made from its Markdown options). Today's table + is a tab between cells in a paragraph of a table style; a later storage form is another recogniser. + +A **record schema's** rich field has the same flags (`Rich(name, paragraphs, ws, blocks)`): `All` for a field of paragraphs and `None` for a +field of one paragraph. `RecordFileReader` reports `fwmd.record.block-not-allowed` for each disallowed block, with a fix that flattens it as +paste does, and the JSON Schema carries `x-fieldworks-blocks` where a field restricts them +([record files](../formats/record-files.md#block-features)). The defaults computed from liblcm (all on for a multi-paragraph field, off +for `Text.Contents` and Scripture) belong to the planned `SIL.TextEditing.Project`; see ADR 0033. + +## Keyboards (ADR 0008) + +The field never switches keyboards; the host does. Subscribe to `RichTextField.ActiveWsChanged` (old and new writing system) and switch the +OS keyboard or Keyman to the new one. + +- The field raises it only while it has focus, as FieldWorks' views set the keyboard only with focus. A change made while it is unfocused + is not reported; the focus-gain event below carries the result. +- It is raised once per change, never when the writing system stays the same: + - when the caret moves into text of another writing system; + - when a selection's first writing system changes (a selection takes the writing system of the first text it covers, as in FieldWorks); + - when an edit or undo leaves the caret in another one; + - when the host or the writing-system toolbar calls `Editor.SetActiveWs`. +- It is also raised, with old and new equal, when the field gains focus, so the host can switch to the keyboard of the writing system at + the caret even though nothing changed in this field. + +### Focus in and out + +What FieldWorks does, and what a host does to match: + +- On Windows, FieldWorks sets the keyboard on focus gain only when focus comes from a window of the same process. So a keyboard the user + picked in the tray or in another application is kept when focus returns without a click (Alt+Tab, the tray). +- A click into a view makes a selection, and the selection sets the keyboard for it, so a click does set the keyboard. +- On Linux FieldWorks sets it on every focus gain. +- When a view loses focus to a control elsewhere in the same application that is not a view, it restores the default keyboard. Leaving the + application is left to the operating system. + +The field cannot tell where focus came from: Avalonia's `GotFocusEventArgs` gives the navigation method and key modifiers, not the source. +So it raises the focus-gain event every time and the host decides: switch only if the previously focused window was its own, otherwise +leave the keyboard alone. The field's event follows changes of writing system, so a click into text of the same writing system gives no +event of its own. + +To restore the default keyboard, subscribe to the field's `LostFocus` and switch when focus has moved to a control of your own application +that is not a field. The field raises nothing when it loses focus. + +Avalonia gives the field `LostFocus` when its window is deactivated and `GotFocus` when it is activated again, on both lines. This was +checked in real windows with a second window of the same application; switching to another application was not driven. So a reactivation +gives one focus-gain event. The event arguments cannot tell that from any other focus gain, so a host that must tell uses its window's +`Activated`. + +## Host hooks (ADR 0021, ADR 0025) + +The field owns one text; the window around it belongs to the host. These are the hooks: + +- **Find and replace.** The host's dialog builds a `FindOptions` (text, match case, match diacritics, whole word, regular expression, + writing system, character style). `Editor.TryFind` searches from the selection (next or previous, optionally wrapping), `Editor.FindAll` + lists the matches, `Editor.ReplaceMatch`, `ReplaceCurrent` and `ReplaceAll` replace as single undo steps. `RichTextField.Find` and + `SelectMatch` also select the match, mark it as the current match (`CurrentMatch`) and ask the host to scroll; `SetFindHighlight` + highlights every match while the dialog is open. Text is compared as NFD, so composed and decomposed forms find each other. +- **Context menu.** `RichTextField.ContextMenuRequested` fires for a right click, the Menu key and Shift+F10 with a `ContextInfo`: the word + and its range, its writing system, the run and its properties, any link or embedded object, the selection and an (empty, for now) list of + spelling suggestions. The host builds the menu. Use the event or Avalonia's `ContextMenu` on the field; with both, the event wins. +- **Paging and scrolling.** PageUp and PageDown (with Shift) move the caret by the field's visible height (`VisibleBounds`, or `PageHeight` + when the host sets it); a drag past the edge extends the selection and asks for scrolling. The field never scrolls anything itself: it + raises `ScrollRequested` (reason, the part to show, the distance) and a scroll viewer round the field answers when no handler marks it + handled. Scrolling inside a long field is therefore the host's. +- **Keys a host routes itself.** See [the language service](spelling-and-language-service.md#keys-a-host-routes-itself) for + `RichTextField.WantsKey`. diff --git a/docs/reference/lcmodel.md b/docs/reference/lcmodel.md new file mode 100644 index 0000000..e06fe79 --- /dev/null +++ b/docs/reference/lcmodel.md @@ -0,0 +1,48 @@ +# The LCModel adapter + +`src/SIL.TextEditing.LCModel` (netstandard2.0, net10.0) is the LCModel adapter (ADR 0016). It references `SIL.LCModel.Core`. + +**It is not trimmable or AOT-compatible, and it is deliberately not in `tests/AotCheck`.** It is for FieldWorks' and Motif's LCModel-side +host code, not Motif's GUI. + +## Conversion + +`TsStringConversion` maps `ITsString`, and an IStText-shaped list of paragraphs, to runs and a `Document` and back. It carries every int +property (with its variation) and every string text property by number, ObjData and NamedStyle included. + +- The host supplies the writing-system handle mapping (`IWritingSystemHandles`) and the paragraph keys. +- A property liblcm cannot load back is left out of the `ITsString` and reported, not thrown; the run model's bag keeps it. The callback, + the `DroppedProperty` shape and the finding ids are in [Findings: LCModel adapter](../formats/findings.md#lcmodel-adapter). These + refusals retire once FieldWorks pins a liblcm with the fixes (issue #159). + +## Clipboard XML + +`TsStringClipboard` reads and writes the XML of one `ITsString` through LCModel's own `TsStringSerializer`. + +- A multi-paragraph copy joins paragraphs with "; " (a parameter), as FieldWorks' own copy does. +- The serializer needs no ICU setup (Hangul and composition-exclusion strings tested, net48 and net10.0). +- `TsStringClipboardForm` is that XML as a clipboard form a host registers on a field (ADR 0029). It has its own id and format name, + `SIL.TextEditing.TsStringXml`, so it can sit beside a host's "TsString" form, and the host adds the id to the paste order if it wants it. + Its details are in [Clipboard: TsString XML](../formats/clipboard.md#tsstring-xml). + +## It is not FieldWorks' "TsString" wire format + +FieldWorks stores a .NET-serialised `TsStringWrapper` (a class in its own SimpleRootSite assembly, which this package does not reference or +copy) under "TsString". Nothing here writes raw XML under "TsString", and an unreadable payload under any name is skipped so paste falls +through to the next form. + +FieldWorks interop is the host's job and uses two hooks: + +1. `RichTextField.ClipboardBackend` (`IClipboardBackend`: `SetAsync(plain, formats)`, `GetAsync(formatName)`, `GetTextAsync()`; Avalonia's + clipboard by default) lets FieldWorks' WinForms host keep the clipboard itself, so it can put the serialised `TsStringWrapper` under + "TsString" and read it with `GetData(...) as TsStringWrapper`, as it does today. +2. A form's payload is an `object` (a string, a `byte[]` or a host object) that passes through the backend untouched. So FieldWorks + registers its own `IClipboardForm` with the id and format name "TsString" whose `Write` returns the wrapper object (built from + `TsStringClipboard.FragmentToXml`) and whose `Read` unwraps it (`TsStringClipboard.FromXmlToFragment`). + +See [Clipboard: FieldWorks TsString](../formats/clipboard.md#fieldworks-tsstring). + +## Tests + +`tests/SIL.TextEditing.LCModel.Tests` (net48, net10.0): round trips of the adapter against real `TsStrFactory` strings, edits that leave +undrawn properties alone, and the clipboard XML. To test against a local liblcm build, see `AGENTS.md`. diff --git a/docs/reference/limitations.md b/docs/reference/limitations.md new file mode 100644 index 0000000..f7e778e --- /dev/null +++ b/docs/reference/limitations.md @@ -0,0 +1,28 @@ +# Known limitations + +What the library does not do today. Planned work is in the README's [Roadmap](../../README.md#roadmap). + +## Accessibility + +- The field gives screen readers what FieldWorks gives them today (ADR 0017): through Avalonia's automation peers it reports a name set by + the host, an Edit role, its text as the value, its read-only state and its bounds. +- There is no UI Automation `TextPattern` (caret, selection, ranges and attributes). It is deferred indefinitely (issue #8). +- Inside a net48 `WinFormsAvaloniaControlHost`, Avalonia's UIA subtree was empty in the spike (unexplained). FieldWorks' WinForms host needs + its own small fix so that the Avalonia accessibility tree is visible at all (ADR 0017). + +## Input + +- Real IME and Keyman interaction are untested. The headless tests drive the IME client directly. +- Missing: scrolling inside long fields (the host scrolls; see [host hooks](editing.md#host-hooks-adr-0021-adr-0025)), drag and drop + (issue #5), touch. +- The word rule is deliberately simple and not UAX #29 word breaking; Thai and Chinese are not split. See + [Selection](editing.md#selection). + +## Undo + +Undo is a host stand-in (`SimpleUndoStack`); no LCModel glue exists. See [The run model](run-model.md#the-host-owns-undo). + +## Avalonia faults + +The layout works around several Avalonia faults, on one or both versions. They and their costs are listed in +[Avalonia versions](avalonia-versions.md#avalonia-faults-pinned-by-tests). diff --git a/docs/reference/markdown.md b/docs/reference/markdown.md new file mode 100644 index 0000000..7820828 --- /dev/null +++ b/docs/reference/markdown.md @@ -0,0 +1,113 @@ +# The Markdown library + +`src/SIL.TextEditing.Markdown` (netstandard2.0, net10.0) is the Markdown core (ADRs 0011 and 0028). `MarkdownConverter` writes the run model +as FieldWorks Markdown and reads it back. `SIL.TextEditing.Markdown.Records` reads and writes record files (ADR 0032). + +The formats themselves are documented in [docs/formats](../formats/README.md). This page holds what an integrator needs beyond them. + +## The format, in brief + +FieldWorks Markdown is a strict profile of Pandoc Markdown. Each point links to its full description. + +- YAML [front matter](../formats/fieldworks-markdown.md#front-matter): `fwmd: 1`, the default analysis and vernacular writing systems, and + each writing system's language tag, direction and font. A newer version is refused with `fwmd.version.unsupported`. +- [Flags as classes](../formats/fieldworks-markdown.md#classes) (`.v .name .example .broken .empty .nul .surrogate .nostyle`, in that + order), `custom-style=` for [styles](../formats/fieldworks-markdown.md#styles), and + [empty and special paragraphs](../formats/fieldworks-markdown.md#empty-and-special-paragraphs) (`[]{.empty}`, `[]{.nul}`, + `[]{.surrogate x=D800}`; a styled empty paragraph is a div around it). +- [Pandoc divs](../formats/fieldworks-markdown.md#divs) (`::: {custom-style=Example}`, paragraphs separated by blank lines) and + [pipe tables](../formats/tables.md) (one paragraph per row, cells separated by a tab). +- [Attributes](../formats/fieldworks-markdown.md#attributes): `sNx` hex for a string property with a control character or a lone + surrogate, [minimal quoting](../formats/fieldworks-markdown.md#quoting), the canonical order, a relative font size as + `fontsize=150 fontsizeUnit=rel`, and `^x^` and `~x~` read as superscript and subscript. +- [Links are project paths](../formats/fieldworks-markdown.md#links), such as `[palit](/lexicon/palit(2)){.name guid=...}`, in any of + [five forms](../formats/fieldworks-markdown.md#the-five-forms). `MarkdownOptions.LinkForm` picks the one written; the reader takes all + and converts without loss. Homographs are `palit(2)`, siblings that share a name get eight-digit GUID suffixes, and paths are written in + NFC and compared under canonical equivalence. A `#field.ws` fragment is reserved. +- [Pictures and glossed examples](../formats/fieldworks-markdown.md#pictures-and-glossed-examples): + `![caption](/files/pictures/hut.jpg){guid=...}` and `[(1)](/texts/Kuwento/3.2){.example guid=...}`. +- [Unicode](../formats/fieldworks-markdown.md#unicode-nfd-and-nfc): `Parse` and `ParseInline` return run text as NFD, the form the editor + stores (ADR 0018, issue 75), each run on its own. `MarkdownOptions.NormalizeText = false` turns it off. `AreEquivalent` treats two texts + as equal when they are equal as NFD. +- [Unsupported Pandoc constructs](../formats/fieldworks-markdown.md#unsupported-pandoc-constructs) are `fwmd.syntax.unsupported-markup` + findings, decided by Pandoc's own rules. +- [Limits](../formats/fieldworks-markdown.md#limits): the depth limit, the table limit and Markdig's slow paths. +- Every finding id is in [Findings](../formats/findings.md). + +## Read lenient, write strict + +- **The reader** takes any spelling Pandoc reads to the same structure: attributes in any order, single quotes, setext headings, `+` + bullets, `<...>` around a destination. +- **The writer** writes only the canonical one, by the escape table of the spec and not by asking a parser. It then reads every paragraph + back and falls back to a safer spelling only if the canonical one does not read back. + +## Host interfaces + +- `ProjectPath` is the path grammar. The host turns paths into GUIDs through `IProjectPathResolver` and object data through + `IObjectMapper`. `DefaultObjectMapper` reads and writes `silfw://` links with no host. +- Everything project-aware reaches the core through these (the full list is in + [For integrators](../formats/fieldworks-markdown.md#for-integrators)). The checks that need a project stay with the host. +- The parser reports the format's findings, all `fwmd.*` (syntax, table, heading, version, link, picture, style, writing system), with line + and column. Finding and object offsets, lines and columns refer to `Source`, the text as written. +- It lists the objects it built with their ranges (`MarkdownParseResult.Objects`) and makes located findings for the host + (`CreateDiagnostic`). +- A Motif link read with no `ProjectName` keeps the project it names. + +## Robustness + +- `Parse` returns findings for any text. Input past Markdig's depth limit is a `fwmd.syntax.too-deep` finding and its lines are kept as + plain text. +- **Table limit.** A table too big for Markdig is one `fwmd.syntax.too-deep` finding on `Parse` and divs on `ToMarkdown`, never an + exception (the numbers are in [Limits](../formats/fieldworks-markdown.md#limits)). +- A table whose header cells are all empty is written as divs, because Pandoc reads such a header as no header. +- **Table styles.** `MarkdownStyleNames.TableHeader` and `TableRow` have no default. Without them a pipe table is an + `fwmd.syntax.unsupported-markup` finding and table-style paragraphs are written as divs ([Tables](../formats/tables.md)). +- A raw tab in a table cell is `fwmd.table.tab-in-cell`. +- **Markdig's own slow paths** are not guarded by the library: a run of `[a](` repeated tens of thousands of times is quadratic, reading a + table of about 4,000 rows takes seconds, and a list nested a thousand levels deep is slow before it reaches the depth limit. A host that + reads untrusted text should bound its size. The writer's own paths (many ampersands, many table rows) are linear. + +## Record files (ADR 0032) + +`SIL.TextEditing.Markdown.Records` reads and writes a whole record, or part of one, as one file: + +- YAML front matter in a strict profile, parsed with YamlDotNet 18.1.0's parser layer; +- every value typed by a `RecordSchema` that is data; +- findings `fwmd.yaml.*` and `fwmd.record.*`; +- FieldWorks Markdown in every rich value and, for a kind that has one, the body. + +`RecordSchema.ToJsonSchema()` publishes the schema for editors. The fixtures are in `tests/conformance/records`. The format is in +[Record files](../formats/record-files.md). + +## Dependencies, trimming and AOT + +- No libpalaso, liblcm or Avalonia reference. +- Trimmable and AOT-compatible on net10.0. +- `tests/AotCheck` proves it: `dotnet publish tests/AotCheck -r win-x64`, warnings are errors. Locally add `-p:NuGetAudit=false` while + issue 104 is open, and put vswhere on `PATH` (issue #164). It roots this library and Markdig whole, then runs `MarkdownCheck.cs` with the + public API only: every block and inline form, the front matter, the five link forms against a host that implements every interface, the + findings of each family, the abstract (`ParseInline`, `ToInlineMarkdown`), and two hosts. +- AotCheck runs with invariant globalization, where `string.Normalize` does not decompose, so the NFD import runs there but its result is + not asserted. A host that needs NFD must not turn invariant globalization on. + +## net48 binding redirects + +Markdig.Signed 0.37.0 brings System.Memory 4.5.5 (assembly version 4.0.1.2), System.Buffers (4.0.3.0) and +System.Runtime.CompilerServices.Unsafe (4.0.4.1). A net48 host needs a binding redirect to those versions for each. The `app.config` of +`tests/SIL.TextEditing.Markdown.Tests` is the model. FieldWorks has them already. + +## Tests + +`tests/SIL.TextEditing.Markdown.Tests` (net48, net10.0): + +- **Conformance.** The fixtures in `tests/conformance` (JSON, documented in its README: Markdown in, expected paragraphs and findings out, + and model to canonical Markdown) are run by `ConformanceTests`. They and the spec define the format. +- **Pandoc agreement.** `PandocAgreementTests` reads every fixture, its canonical form and 150 generated documents + (`TEXTEDITING_PANDOC_SEEDS`) with `pandoc -f markdown-smart-implicit_figures-implicit_header_references-task_lists -t json` and requires + the same paragraphs. Pandoc is pinned in the CI workflow. Where it is missing the tests skip, except in CI (`CI` or `GITHUB_ACTIONS` + set), where they fail. +- **Random round trips.** 600 seeded random models and 600 random texts per run (`TEXTEDITING_MARKDOWN_SEEDS` raises the count) check the + round trip and the stability of the canonical form. +- Also: tests of the findings' positions, normalisation, size limits and the host interfaces. +- **Format docs.** `FormatDocsTests` checks that every finding id is in `docs/formats/findings.md` and that the examples in docs/formats + still match their fixtures. diff --git a/docs/reference/projects.md b/docs/reference/projects.md new file mode 100644 index 0000000..84a4090 --- /dev/null +++ b/docs/reference/projects.md @@ -0,0 +1,83 @@ +# Projects and tests + +The README has the one-line table. This page has the detail. + +## Libraries + +### `src/SIL.TextEditing` (netstandard2.0, net10.0) + +The UI-free core. No Avalonia, no liblcm. + +- The run model: `Run`, `Paragraph`, `Document`, `PropertyBag` ([The run model](run-model.md)). +- Editing: `DocumentEditor`. +- Paragraph-key rules: `Paragraph.Key`, opaque, null until the host assigns one. +- Clipboard forms (`IClipboardForm`, `ClipboardForms`) and the paste writing-system rule ([Clipboard](clipboard.md)). +- The debug interchange dump, and grapheme helpers. +- The language-service interface, `ITextLanguageService` ([Language service](spelling-and-language-service.md#language-service-adr-0030)). + +### `src/SIL.TextEditing.Avalonia` (netstandard2.0, net10.0) + +`RichTextField`: layout with Avalonia's `TextFormatter`, caret, selection, hit testing, bidi caret stops, IME client, copy, cut, paste and +"Copy as" through the host's clipboard forms. See [Editing](editing.md), [Rendering](rendering.md) and +[Spell checking and the language service](spelling-and-language-service.md). + +### `src/SIL.TextEditing.Markdown` (netstandard2.0, net10.0) + +FieldWorks Markdown and record files. See [The Markdown library](markdown.md). + +### `src/SIL.TextEditing.LCModel` (netstandard2.0, net10.0) + +The `ITsString` adapter and the TsString XML clipboard form. See [The LCModel adapter](lcmodel.md). + +## Samples + +Samples do not ship. + +- **`samples/SIL.TextEditing.Playground`** (netstandard2.0, net10.0): the shared playground UI. A `RichTextField`, scenario picker, + writing-system bar, format and undo buttons, block-feature switches (tables, lists, headings; ADR 0033), live model inspector and event + log. Both hosts and the integration tests use it. +- **`samples/Playground.Desktop`** (net48, net10.0): plain Avalonia desktop app showing the playground. Avalonia 11.3.17 on net48, Avalonia + 12 on net10.0 (Motif's shape). +- **`samples/Host48`** (net48): WinForms form hosting the playground through `WinFormsAvaloniaControlHost` between two WinForms TextBoxes, + Avalonia 11.3.17: FieldWorks' shape. + +The playground loads ready-made documents (Latin and Arabic, Indic and Thai shaping, NFD and surrogates, inline pictures, unknown +properties, an environment box with a language service, 200 paragraphs, many fields) and shows what the model holds in a live inspector. + +## Test projects + +| Project | Targets | What it tests | +|---|---|---| +| `tests/SIL.TextEditing.Tests` | net48, net10.0 | Unit tests for the model. | +| `tests/SIL.TextEditing.Avalonia.Tests` | net48, net10.0 | Avalonia.Headless tests for the surface (Skia, real shaping). | +| `tests/SIL.TextEditing.SecondDispatcher.Tests` | net10.0 | The tests that make a dispatcher on a second thread (Avalonia 12 allows it). | +| `tests/SIL.TextEditing.RealClipboard.Tests` | net48, net10.0 | Bytes and Arabic text through the default clipboard backend and the real Windows clipboard. Opt-in. | +| `tests/SIL.TextEditing.LCModel.Tests` | net48, net10.0 | The adapter ([details](lcmodel.md#tests)). | +| `tests/SIL.TextEditing.Markdown.Tests` | net48, net10.0 | Conformance, Pandoc agreement, random round trips ([details](markdown.md#tests)). | +| `tests/SIL.TextEditing.Playground.Tests` | net48, net10.0 | Avalonia.Headless integration tests that drive the playground the way a person would. | +| `tests/AotCheck` | net10.0 | Publishes the Markdown library native AOT and runs it ([details](markdown.md#dependencies-trimming-and-aot)). | + +### SecondDispatcher + +`Dispatcher.UIThread` is process-wide, so the tests that make a dispatcher on a second thread run in a test host of their own and cannot +change it for the other tests (issue #72). + +### RealClipboard + +It runs in a real Avalonia window (Avalonia.Desktop), not Avalonia.Headless. + +- **Opt-in.** Skipped unless `TEXTEDITING_REAL_CLIPBOARD=1` (and on any system but Windows), because it overwrites the clipboard of whoever + runs it and needs an interactive desktop. +- On GitHub's windows-latest runner the net48 test host crashed (cause not found; Avalonia 11.3 Win32 in a non-interactive session is the + suspect) while net10.0 passed. +- When it runs, the two target frameworks take turns through a `Global\` mutex, and a turn or a test over 60 s fails. +- If the Win32 platform cannot start, the tests say so and end. + +## Build environment + +Windows with the .NET 10 SDK and the .NET Framework 4.8 targeting pack. net48 builds and tests run only on Windows. + +- From WSL, run the same commands through `powershell.exe`. +- The samples are built by the solution. +- The headless tests use the Windows fonts Segoe UI, Nirmala UI, Leelawadee UI and Constantia. +- A person tests with the checklist in [docs/manual-testing.md](../manual-testing.md). diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md new file mode 100644 index 0000000..8bab3d7 --- /dev/null +++ b/docs/reference/rendering.md @@ -0,0 +1,105 @@ +# Rendering + +What the field draws, and how. For the per-version Avalonia faults the layout works around, see +[Avalonia versions](avalonia-versions.md#avalonia-faults-pinned-by-tests). + +## Paragraph and character properties + +Drawn, with the host's `IStyleResolver` supplying named styles (ADR 0004). + +**Character properties:** + +- weight (any 100 to 900, invert), italic; +- foreground and background colour; +- underline type and colour: dotted, dashed, single, double, strikethrough, and the squiggle, which the layout draws itself as a zigzag + below the baseline; +- superscript and subscript: sized, and raised or lowered, as the font's own OS/2 table says, with FieldWorks' two thirds where it gives + nothing. The ratios are applied to the size of the text beside it, which is a default; +- font size (absolute, relative, named), font family and per-run font features; +- baseline offset: drawn by drawing the run ourselves, shifted. The line's slot makes room for it and the caret, selection and squiggle + follow it. + +**Paragraph properties:** alignment, first-line and hanging indents, margins, space before and after, line height, paragraph colour, +borders and padding, a line limit, bullets and numbers. + +Pagination properties and the rest are carried untouched. Lengths in points are drawn as device-independent pixels, as font sizes are. + +A writing system the host did not register (`RichTextField.WritingSystems`) is drawn in Segoe UI at 10 point, FieldWorks' own default size. + +## Justified paragraphs + +A justified paragraph is stretched by the field itself (Avalonia's justifier is internal). + +- Each line but the last shares its slack equally among its interior spaces, so it reaches both edges of the text area. +- The last line stays at the leading edge, as in FieldWorks. +- The caret, selection, mouse and run backgrounds follow the stretched text. +- Each stretched line is drawn piece by piece, with its underlines and strikethrough drawn by the field as lines across the whole stretched + run, so a dotted or dashed pattern is not restarted or scaled. + +## Line limit + +Text a line limit hides stays in the model but cannot be reached. + +- The caret, Right, End, Up/Down and mouse stop at the end of the last line shown, as FieldWorks puts an insertion point in truncated text + at the last visible position. +- Typing, Delete, Enter, paste and composition are refused at that point, and Backspace at the start of the next paragraph does not join + it onto the hidden end. +- Select All and Copy still take the hidden text, because the selection model holds it. +- As in FieldWorks, the last visible line of a cut-off paragraph ends with "..." in the paragraph style's character format, on the trailing + side of the text (left of it in a right-to-left paragraph). Room is kept for it when the text area is wide enough for one beside text. +- The caret and hit testing stay on the text, and a hard line break that ends the paragraph is not cut-off text. + +## Spaces a wrapped line ends with + +They hang past the line's text, outside the alignment (a line is aligned by its text). The caret across them moves one space width per +space, as in FieldWorks, even past the text area. + +- The control draws nothing outside itself, so a caret that would be there is held at the control's edge: the End caret at the edge, the + carets of the spaces before it 2 px apart inside it. +- When the text itself reaches the edge (a justified line, a right-aligned one, a line that just fits) there is no room for those carets to + show apart: they crowd on the text's last letters. +- A host that wants each caret seen gives the text area room on the side the spaces hang to, with the paragraph's trailing indent + (property 20: the side lines end on, in either direction) of a few spaces' width. A margin around the control does not help, as the field + cannot draw there. +- The mouse follows the carets as they are drawn: a click at one gives it, and a click beyond the End caret gives the End caret. + +## Very long paragraphs + +Runs that look alike are one stretch of text to Avalonia's formatter, and a paragraph's ranges are found by halving, so a layout grows with +the runs. + +- Avalonia's formatter itself (about nine tenths of a layout's time, on both versions) carries what is left of the text from each line to + the next, so a paragraph of tens of thousands of runs costs more than its length: 64,000 runs lay out in about 2.0 s on 11.3 and 1.6 s + on 12. +- From 16,000 runs (after runs that look alike are one), a paragraph of Latin text (letters, their marks, punctuation) that is not right to + left is formatted line by line from a window of its text instead (`ParagraphLayout.FormatWindowed`). It gives the same lines and carets, + but for the kerning of a pair across a line start (widths move by a fraction of a pixel), in about 1.1 s on 11.3 and 0.46 s on 12 for + those 64,000 runs. +- A paragraph with any other script is formatted whole, as the directions of right-to-left text are resolved for all of it at once and a + neutral character at a line start keeps the fallback font of the text before it (CJK, Thai). + +## Graphemes + +`GraphemeClusters` is `StringInfo` plus Indic conjuncts (GB9c, six scripts; .NET 10 lacks it). + +- The netstandard2.0 build also fuses Extend, SpacingMark, emoji ZWJ sequences and flags per UAX #29, with approximated property tables. +- Caret stops in the field come from the shaper, except on a right-to-left paragraph's line of several runs (`LineInfo.UsesRuns`). There + the layout steps by this class's grapheme boundaries and measures each on the run's glyphs (with the approximated tables on + netstandard2.0). + +## Colours and themes + +Until the host sets `Foreground` (text, and a border with no colour of its own) and `Background` (the page), the field takes them from the +theme: + +- the text colour the caret uses and the theme's window colour; +- in a Windows high contrast theme, the system's window text and window colours, as FieldWorks does; +- the selection takes the system highlight, and selected text the system's highlight text colour, whatever colour its runs have. + +The text keeps one default brush that the field changes in place when the theme or the system's colours change, so a theme switch redraws +the text and does not lay out the paragraphs again. + +**The contract.** A colour the host sets is kept as it is under every theme, high contrast included. So a host that sets one sets the other, +or neither: a host that sets only `Foreground = Black` on a dark theme gets black text on the theme's dark page. `ResetForeground()` and +`ResetBackground()` go back to following the theme. A run's own colours (text property 8 and 9) are the document's and do not change with +the theme. diff --git a/docs/reference/run-model.md b/docs/reference/run-model.md new file mode 100644 index 0000000..06dd3c6 --- /dev/null +++ b/docs/reference/run-model.md @@ -0,0 +1,56 @@ +# The run model + +`src/SIL.TextEditing` holds the model. It has no Avalonia and no liblcm reference. + +## Runs are shaped like ITsString + +A `Run` is shaped like one run of FieldWorks' `ITsString`: + +- text; +- a writing-system id; +- an opaque property bag (`PropertyBag`) keyed by FieldWorks property numbers; +- object data (`ObjData`) for inline objects. + +Ws and ObjData are fields of `Run`, not bag entries (ITsString holds them as properties 1 and 6). The Ws is a language-tag string, not +an integer handle. The [LCModel adapter](lcmodel.md) maps both. The editor never interprets a bag; it carries it through every edit. + +The other model types are `Paragraph`, `Document` and `PropertyBag`. A paragraph's key (`Paragraph.Key`) is opaque and null until the host +assigns one (ADR 0006). + +## The host owns undo + +The editor keeps no undo stack. The host owns it, as FieldWorks' LCModel does (ADR 0003). The playground uses a stand-in, +`SimpleUndoStack`; no LCModel undo glue exists yet. + +## `RunList` + +`Paragraph.Runs` is a `RunList`, not a `List`. + +- It tells its paragraph whenever a run is added, removed or replaced, and each run tells it when its text changes. +- So `Paragraph.Length`, `RunBefore`, `RunAfter` and `Spans` are answered from run starts the paragraph keeps current (a binary search, not + a walk over the runs), however a host changes the runs. +- A host may edit `Run.Text` or the list directly. +- `RunList` has `Add`, `Insert`, `RemoveAt`, `RemoveRange`, `AddRange`, `Clear`, an indexer and `IEnumerable`. Use LINQ for + `ConvertAll`, `FindIndex` and the like. +- To replace the runs, assign a `new RunList(runs)` to `Paragraph.Runs`. +- `Paragraph.Runs = list` also takes a `List` directly. The paragraph adopts a new `RunList` over the same runs, not the list itself: + the `Run` objects are shared, but a later `list.Add` is not seen by the paragraph. + +## Changes in place + +A change to a run's `PropertyBag` in place (`SetInt`, `SetStr`, or a write to `IntProps` or `StrProps`) is seen too. The bag tells the +paragraphs that hold it, so `ChangeStamp` moves and a field shows the change. See +[what a host calls after a change](spelling-and-language-service.md#what-a-host-calls-after-a-change) for the rest. + +## Rules + +- `Run.Text = null` throws `ArgumentNullException` (use an empty string). So do `Run.Props = null` and `Paragraph.Props = null`. +- Looping over `Paragraph.Runs` while the list is added to, removed from or has a run replaced throws `InvalidOperationException`, as it + does for a `List`. Changing a run's text or properties meanwhile is fine. +- `Paragraph.Text` is safe to read from any thread while nothing changes the paragraph. The model itself is edited on the UI thread only + (ADR 0024). + +## Work counters + +The work counters tests use (`TextReads`, `ElementReads`) exist in a build that defines `TEXTEDITING_COUNTERS` (every configuration but +Release) and cost nothing in a Release build. diff --git a/docs/reference/spelling-and-language-service.md b/docs/reference/spelling-and-language-service.md new file mode 100644 index 0000000..7db71e1 --- /dev/null +++ b/docs/reference/spelling-and-language-service.md @@ -0,0 +1,132 @@ +# Spell checking and the language service + +Both are host services the field asks when idle. Neither writes anything into the document. + +## Spell checking (ADR 0020, ADR 0026) + +The host gives the field its dictionaries. Set `RichTextField.SpellChecking` to an `ISpellCheckHost` that: + +- returns an `ISpellChecker` (`Check`, `Suggest`, `Add`; `ISpellIgnorer` adds `Ignore` when a dictionary has it) for each writing system; +- says which characters form words and which join them; +- raises `DictionariesChanged` when a dictionary changes. + +Derive from `SpellCheckHostBase` to write only `GetChecker`: it has the default word rules and `RaiseDictionariesChanged()` (any thread). +The same word rules (`IWordRules`) serve the double-click, Ctrl+arrows and whole-word find; `RichTextField.WordRules` gives them without +spell checking (see [the word rule](editing.md#selection)). + +### What the field draws + +The field checks the paragraphs in view when idle, a few words at a time, and draws squiggles: + +- red: a misspelled word, or a word with an object in it; +- blue: a word that mixes writing systems or has unchecked text inside. + +The rules are in `SpellScanner` (UI-free). Words break at characters that are not word-forming. Do-not-check text and text that may not be +edited (unless forced) are skipped. The word of a context request and the squiggled word are one definition (an underscore ends both, as in +FieldWorks). + +### What a host's menu calls + +`SpellingSuggestionsAt`, `ApplySpellingSuggestion`, `AddToDictionary`, `IgnoreWord` and `RestartSpelling` are what a host's context menu and +dictionary changes call. `ApplySpellingSuggestion` also takes runs, for "insert missing spaces" and, with `WithWholeWordIn`, for "make the +whole word writing system X" in a word with several writing systems. + +### A checker that fails + +A checker that throws raises `SpellCheckerFailed` (written to `Trace` once per writing system when nothing handles it) and is left alone; it +does not end the application. A host that repairs the dictionary raises `DictionariesChanged` (or calls `RestartSpelling`), which tries that +writing system again. + +### Slow dictionaries + +A dictionary that is slow can be asked off the UI thread. Its checker also implements `IBackgroundSpellChecker`: `Check` may then run on a +worker, and the field finds the words and applies the answers on its own thread, and drops an answer for a paragraph that has changed since. + +`CheckSpellingStep` then returns false while a batch is out, and `IsSpellingComplete` says when everything in view is checked. A loop of +`while (!field.CheckSpellingStep()) { }` on the UI thread therefore never ends with such a checker (the answer is applied by the dispatcher +the loop blocks). Let the dispatcher run and wait for `IsSpellingComplete`, or use a checker without the interface where a synchronous check +is wanted. + +The playground's "Spell checking" scenario shows it with a short English and French word list. + +### What a host calls after a change + +The field restarts checking itself on an edit through the editor, `Apply`, `ExternalReplaceParagraph`, a change of `IsReadOnly` on the +field or its editor, a word added or ignored, and a `DictionariesChanged` notice. + +After a host changes a paragraph directly it calls nothing. A paragraph tells the field when any of these change (`Paragraph.ChangeStamp` +moves): + +- its runs; +- a run's text, writing system, object data or property bag (replaced, or changed in place with `SetInt`, `SetStr` or a write to + `IntProps` or `StrProps`); +- its style, direction, default writing system or property bag. + +The field then measures and draws again, lays out that paragraph again (and only that one, keyed by the stamp), numbers the list again, drops +the marks of that paragraph and checks it again when idle. A notice from another thread is marshalled to the field's own, but the model is +not thread-safe (ADR 0024): the edit itself must still not overlap a layout, so a host edits on the UI thread. + +What a host does call: + +- `RichTextField.RefreshFromDocument()` for what a paragraph cannot tell: the list of paragraphs (`Document.Paragraphs`) changed, or a + paragraph was replaced in it. It restarts checking too. +- `InvalidateAllLayouts()` when a style resolver's answers change (not the resolver itself, which the field takes again when it is set), + and `RestartSpelling()` when what is checked changed. +- `DictionariesChanged` after repairing a failed dictionary. + +Setting `DocumentEditor.IsReadOnly` directly tells automation as `RichTextField.IsReadOnly` does. + +## Language service (ADR 0030) + +A box whose text is a notation (the first is a FieldWorks phonological environment, `/ [V] _ #`) takes a host **language service**. + +`RichTextField.LanguageService` is an `ITextLanguageService` (in `SIL.TextEditing`, no Avalonia or liblcm). It is shaped like the +spell-check host and carries Language Server Protocol data: zero-based line and character in UTF-16 units, ranges, severities 1 to 4, +completion items. So a VS Code adapter and agents' findings can use the same answers. + +- `AnalyzeAsync` gives tokens (a kind string and a range) and diagnostics (range, severity, id, message, an optional exact fix) together. +- `CompleteAsync`, `HoverAsync` and `CanInsert` do the rest. +- `Changed` tells every field to ask again. +- Every call that can take time is a `Task`, so Motif's backend can answer from another process. +- `TextLanguageService` is a base class for one that answers at once. + +The field serves a field of one paragraph, and `SingleLine` (single-paragraph and no hard line break) is what a notation box sets. + +### How answers are applied + +The field asks when idle and applies an answer only to the paragraph and stamp it was asked about (as spelling does). After an edit it +carries the results the edit left alone, so nothing flickers. + +- **Tokens are chips.** A `TokenStyleMap` (per kind, a fill and an outline for the light theme and the dark one) is drawn behind plain text, + so copy, Markdown and undo see only text. A high contrast theme gets none. +- **Diagnostics are squiggles**, drawn with the spelling marks' drawing (red error, blue warning, grey information; a hint is not drawn). + They are never written into the document: no run, bag or property 5 changes. +- Hovering sets the tooltip from the diagnostics' messages and the service's help (`GetLanguageHoverAsync`). +- `ContextInfo.Diagnostics` and `ApplyDiagnosticFix` give the host's context menu a fix as one undo step. +- `CanInsert(kind)` enables a host's Insert commands. + +### Completion + +Ctrl+Space, or a trigger character the service names, opens a popup left-aligned under the caret. + +- The popup is a `List` of named `ListItem` rows: no theme needed, the keyboard stays in the field, the wheel scrolls it. +- It closes when the word or class being completed is done, so Enter and Tab reach the host again. +- Keys: Up, Down, PageUp, PageDown, Enter or Tab (accepts, and neither reaches the host), Esc. +- Typing asks again. A caret move, a selection, losing focus, read-only or an input method composing closes it. +- Where no popup can be shown the state and `CompletionChanged` still work. +- The field's peer is an expand/collapse provider for the popup and gives the worst diagnostic as its item status. + +### Keys a host routes itself + +A host that routes keys itself (WinForms `IsInputKey` and `ProcessCmdKey`, a dialog's default and cancel buttons) asks +`RichTextField.WantsKey(key, modifiers)` and leaves the keys it wants to the field: Ctrl+Space, and Enter, Tab, Esc and the arrows while the +popup is open. `samples/Host48` shows it. + +### Timing and threads + +`LanguageDebounce` waits after the last edit before asking, and `BackgroundTextLanguageService` runs the analysis on the thread pool. + +### The sample + +The playground's **Environment** scenario is the sample (`SampleEnvironmentService`, over a made-up phonology; not shipped). The real +service that reads a project's phonology comes later, in a project that references liblcm. From 6cd9bb0c7c898775ff4a3a21a212492bb2ff0d1a Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:41:38 -0500 Subject: [PATCH 2/6] The README introduces the project: pitch, screenshots, what sets it apart with evidence, quick start, projects, docs map and roadmap. Co-Authored-By: Claude Opus 5.5 --- README.md | 411 ++++++++++++++---------------------------------------- 1 file changed, 106 insertions(+), 305 deletions(-) diff --git a/README.md b/README.md index 33dfd65..3def032 100644 --- a/README.md +++ b/README.md @@ -1,324 +1,125 @@ # SIL.TextEditing -A rich text editor for [Avalonia](https://avaloniaui.net/) in which **every run carries its FieldWorks writing -system**. The model is shaped like FieldWorks' `ITsString`: runs with a writing-system id, an opaque property bag -keyed by FieldWorks property numbers, and object data for inline objects. Ws and ObjData are fields of `Run`, not bag -entries (ITsString holds them as properties 1 and 6), and the Ws is a language-tag string, not an integer handle; the -LCModel adapter maps them. The editor keeps no undo stack; the host owns -it, as FieldWorks' LCModel does. - -**Status: bootstrap.** This is the native-editor prototype made into a repository. It is not released, the API will -change, and the list under "Known gaps" is the work still to do. - -**Formats:** every format the library reads or writes (FieldWorks Markdown, record files, the clipboard forms) and every finding id is -documented in [docs/formats](docs/formats/README.md). - -## Projects - -| Project | Targets | What it is | -|---|---|---| -| `src/SIL.TextEditing` | netstandard2.0, net10.0 | UI-free run model (`Run`, `Paragraph`, `Document`, `PropertyBag`), editing (`DocumentEditor`), paragraph-key rules (`Paragraph.Key`, opaque, null until the host assigns one), clipboard forms (`IClipboardForm`, `ClipboardForms`), the paste writing-system rule, the debug interchange dump, grapheme helpers. No Avalonia, no liblcm. | -| `src/SIL.TextEditing.Avalonia` | netstandard2.0, net10.0 | `RichTextField`: layout with Avalonia's `TextFormatter`, caret, selection, hit testing, bidi caret stops, IME client, copy, cut, paste and "Copy as" through the host's clipboard forms. | -| `src/SIL.TextEditing.LCModel` | netstandard2.0, net10.0 | The LCModel adapter (ADR 0016). `TsStringConversion` maps `ITsString`, and an IStText-shaped list of paragraphs, to runs and a `Document` and back, carrying every int (with its variation) and string text property by number, ObjData and NamedStyle included. `TsStringClipboard` reads and writes the XML of one `ITsString` through LCModel's own `TsStringSerializer`. A multi-paragraph copy joins paragraphs with "; " (a parameter), as FieldWorks' own copy does. `TsStringClipboardForm` is that XML as a clipboard form a host registers on a field (ADR 0029), with its own id and format name `SIL.TextEditing.TsStringXml` (so it can sit beside a host's "TsString" form, and the host adds the id to the paste order if it wants it). **It is not FieldWorks' "TsString" wire format**: FieldWorks stores a .NET-serialised `TsStringWrapper` (a class in its own SimpleRootSite assembly, which this package does not reference or copy) under "TsString". FieldWorks interop is the host's job and uses two hooks. (1) `RichTextField.ClipboardBackend` (`IClipboardBackend`: `SetAsync(plain, formats)`, `GetAsync(formatName)`, `GetTextAsync()`; Avalonia's clipboard by default) lets FieldWorks' WinForms host keep the clipboard itself, so it can put the serialised `TsStringWrapper` under "TsString" and read it with `GetData(...) as TsStringWrapper`, as it does today. (2) A form's payload is an `object` (a string, a `byte[]` or a host object) that passes through the backend untouched, so FieldWorks registers its own `IClipboardForm` with the id and format name "TsString" whose `Write` returns the wrapper object (built from `TsStringClipboard.FragmentToXml`) and whose `Read` unwraps it (`TsStringClipboard.FromXmlToFragment`). Nothing here writes raw XML under "TsString", and an unreadable payload under any name is skipped so paste falls through to the next form. The serializer needs no ICU setup (Hangul and composition-exclusion strings tested, net48 and net10.0). The host supplies the writing-system handle mapping (`IWritingSystemHandles`) and the paragraph keys. References `SIL.LCModel.Core`. **Not trimmable or AOT-compatible, and deliberately not in `tests/AotCheck`: it is for FieldWorks' and Motif's LCModel-side host code, not Motif's GUI.** | -| `tests/SIL.TextEditing.Tests` | net48, net10.0 | Unit tests for the model. | -| `tests/SIL.TextEditing.Avalonia.Tests` | net48, net10.0 | Avalonia.Headless tests for the surface (Skia, real shaping). | -| `tests/SIL.TextEditing.SecondDispatcher.Tests` | net10.0 | The tests that make a dispatcher on a second thread (Avalonia 12 allows it). `Dispatcher.UIThread` is process-wide, so they run in a test host of their own and cannot change it for the other tests (issue #72). | -| `tests/SIL.TextEditing.RealClipboard.Tests` | net48, net10.0 | Bytes and Arabic text through the default clipboard backend and the real Windows clipboard, in a real Avalonia window (Avalonia.Desktop), not Avalonia.Headless. **Opt-in**: skipped unless `TEXTEDITING_REAL_CLIPBOARD=1` (and on any system but Windows), because it overwrites the clipboard of whoever runs it and needs an interactive desktop: on GitHub's windows-latest runner the net48 test host crashed (cause not found; Avalonia 11.3 Win32 in a non-interactive session is the suspect) while net10.0 passed. When it runs, the two target frameworks take turns through a `Global\` mutex, and a turn or a test over 60 s fails. If the Win32 platform cannot start the tests say so and end. | -| `tests/SIL.TextEditing.LCModel.Tests` | net48, net10.0 | Round trips of the adapter against real `TsStrFactory` strings, edits that leave undrawn properties alone, and the clipboard XML. | -| `src/SIL.TextEditing.Markdown` | netstandard2.0, net10.0 | The Markdown core (ADRs 0011 and 0028): `MarkdownConverter` writes the run model as FieldWorks Markdown and reads it back. FieldWorks Markdown is a strict profile of Pandoc Markdown: YAML front matter (`fwmd: 1`, the default analysis and vernacular writing systems, and each writing system's language tag, direction and font; a newer version is refused with `fwmd.version.unsupported`), flags as classes (`.v .name .example .broken .empty .nul .surrogate .nostyle`, in that order), `custom-style=` for styles, `[]{.empty}` for an empty paragraph (a styled one is a div around it), `[]{.nul}` and `[]{.surrogate x=D800}` for the two characters CommonMark cannot carry, Pandoc divs (`::: {custom-style=Example}`, paragraphs separated by blank lines), pipe tables (one paragraph per row, cells separated by a tab), `sNx` hex for a string property with a control character or a lone surrogate, minimal quoting (a value is bare when it holds only letters, digits and `- _ . : # /`, otherwise double-quoted with only `\"` and `\\`; `&` only where an ampersand would read as a character reference; `\|` for every pipe that is not a cell separator), attributes in the canonical order, a relative font size as `fontsize=150 fontsizeUnit=rel`, and `^x^` and `~x~` read as superscript and subscript. **Links are project paths**: `[palit](/lexicon/palit(2)){.name guid=...}` in any of five forms (a root path, a path relative to the file, a `silfw://` or `motif://` back-link, `/objects/`; `MarkdownOptions.LinkForm` picks the one written, the reader takes all and converts without loss), with `palit(2)` homographs, eight-digit GUID suffixes for siblings that share a name, paths written in NFC and compared under canonical equivalence, a reserved `#field.ws` fragment, pictures as `![caption](/files/pictures/hut.jpg){guid=...}` and glossed examples as `[(1)](/texts/Kuwento/3.2){.example guid=...}`. `ProjectPath` is the path grammar; the host turns paths into GUIDs through `IProjectPathResolver` and object data through `IObjectMapper`; `DefaultObjectMapper` reads and writes `silfw://` links with no host. **Read lenient, write strict:** the reader takes any spelling Pandoc reads to the same structure (attributes in any order, single quotes, setext headings, `+` bullets, `<...>` around a destination); the writer writes only the canonical one, by the escape table of the spec and not by asking a parser, then reads every paragraph back and falls back to a safer spelling only if the canonical one does not read back. **Unicode:** `Parse` and `ParseInline` return run text as NFD, the form the editor stores (ADR 0018, issue 75), each run on its own (attribute values and destinations are never touched; `MarkdownOptions.NormalizeText = false` turns it off); finding and object offsets, lines and columns refer to `Source`, the text as written. `ToMarkdown` writes the text as stored, and `AreEquivalent` treats two texts as equal when they are equal as NFD. Everything project-aware reaches the core through `IWritingSystemMapper` (optionally `IKnownWritingSystems`, `IWritingSystemChecker`, `IWritingSystemDetails` for the front matter), `MarkdownStyleNames`/`IStyleCatalog` (optionally `IKnownStyles`, `IStyleChecker`), `IObjectMapper` (optionally `IObjectResolver`) and `IProjectPathResolver`; the parser reports the format's findings, all `fwmd.*` (syntax, table, heading, version, link, picture, style, writing system), with line and column, lists the objects it built with their ranges (`MarkdownParseResult.Objects`) and makes located findings for the host (`CreateDiagnostic`); the checks that need a project stay with the host. `Parse` returns findings for any text: input past Markdig's depth limit is a `fwmd.syntax.too-deep` finding and its lines are kept as plain text. **Table limit:** Markdig reads a pipe table only while rows x (columns + 1) stays under about 10,200; a bigger table is one `fwmd.syntax.too-deep` finding on `Parse` and divs on `ToMarkdown`, never an exception. A table whose header cells are all empty is written as divs, because Pandoc reads such a header as no header. **Markdig's own slow paths:** a few inputs are slow in Markdig itself and the library does not guard them: a run of `[a](` repeated tens of thousands of times is quadratic, reading a table of about 4,000 rows takes seconds, and a list nested a thousand levels deep is slow before it reaches the depth limit; a host that reads untrusted text should bound its size. The writer's own paths (many ampersands, many table rows) are linear. **Table styles:** `MarkdownStyleNames.TableHeader` and `TableRow` have no default; without them a pipe table is an `fwmd.syntax.unsupported-markup` finding and table-style paragraphs are written as divs. **Pandoc constructs:** hand-written math, citations, inline notes, strikeout, heading attributes and the block constructs Pandoc has and the notation lacks (fancy list markers such as `a.` and `(1)`, line blocks, grid tables, definitions after a term line, a caption next to a table) are `fwmd.syntax.unsupported-markup` findings with the escape as the fix, found by Pandoc's own rules and not by the writer's broader escape rule: grammar prose such as `M. Dixon`, `A. Consonants`, `Table: the forms` or `#palit#` is plain text in both and is never reported. A wrapped line that starts with `#`, `-`, `+`, `>` or `1.` stays in its paragraph, as in Pandoc, in the document and in a div. A Motif link read with no `ProjectName` keeps the project it names, and a raw tab in a table cell is still `fwmd.table.tab-in-cell`. **Record files** (ADR 0032): `SIL.TextEditing.Markdown.Records` reads and writes a whole record or part of one as one file, YAML front matter in a strict profile (parsed with YamlDotNet 18.1.0's parser layer, every value typed by a `RecordSchema` that is data, findings `fwmd.yaml.*` and `fwmd.record.*`) with FieldWorks Markdown in every rich value and, for a kind that has one, the body; `RecordSchema.ToJsonSchema()` publishes the schema for editors; the fixtures are in `tests/conformance/records`. No libpalaso, liblcm or Avalonia reference. Trimmable and AOT-compatible on net10.0. `tests/AotCheck` (`dotnet publish tests/AotCheck -r win-x64`, warnings are errors; locally add `-p:NuGetAudit=false` while issue 104 is open) roots this library and Markdig whole, then runs `MarkdownCheck.cs` with the public API only: every block and inline form, the front matter, the five link forms against a host that implements every interface, the findings of each family, the abstract (`ParseInline`, `ToInlineMarkdown`), and two hosts. AotCheck runs with invariant globalization, where `string.Normalize` does not decompose, so the NFD import runs there but its result is not asserted; a host that needs NFD must not turn invariant globalization on. **net48 binding redirects:** Markdig.Signed 0.37.0 brings System.Memory 4.5.5 (assembly version 4.0.1.2), System.Buffers (4.0.3.0) and System.Runtime.CompilerServices.Unsafe (4.0.4.1), so a net48 host needs a binding redirect to those versions for each (the app.config of `tests/SIL.TextEditing.Markdown.Tests` is the model); FieldWorks has them already. | -| `tests/SIL.TextEditing.Markdown.Tests` | net48, net10.0 | The conformance fixtures in `tests/conformance` (JSON, documented in its README: Markdown in, expected paragraphs and findings out, and model to canonical Markdown; they and the spec define the format), run by `ConformanceTests`; `PandocAgreementTests` reads every fixture, its canonical form and 150 generated documents (`TEXTEDITING_PANDOC_SEEDS`) with `pandoc -f markdown-smart-implicit_figures-implicit_header_references-task_lists -t json` and requires the same paragraphs. Pandoc is pinned in the CI workflow; where it is missing the tests skip, except in CI (`CI` or `GITHUB_ACTIONS` set), where they fail. Also: 600 seeded random models and 600 random texts per run (`TEXTEDITING_MARKDOWN_SEEDS` raises the count) that check the round trip and the stability of the canonical form, and tests of the findings' positions, normalisation, size limits and the host interfaces. | -| `samples/SIL.TextEditing.Playground` | netstandard2.0, net10.0 | The shared playground UI: a `RichTextField`, scenario picker, writing-system bar, format and undo buttons, block-feature switches (tables, lists, headings; ADR 0033), live model inspector and event log. Both hosts and the integration tests use it. | -| `samples/Playground.Desktop` | net48, net10.0 | Plain Avalonia desktop app showing the playground: Avalonia 11.3.17 on net48, Avalonia 12 on net10.0 (Motif's shape). | -| `samples/Host48` | net48 | WinForms form hosting the playground through `WinFormsAvaloniaControlHost` between two WinForms TextBoxes, Avalonia 11.3.17: FieldWorks' shape. | -| `tests/SIL.TextEditing.Playground.Tests` | net48, net10.0 | Avalonia.Headless integration tests that drive the playground the way a person would. | - -One source tree serves both Avalonia generations. The netstandard2.0 target compiles against **Avalonia 11.3.17** -(what FieldWorks pins; it runs on net48) and the net10.0 target against **Avalonia 12.1.2** (what Motif pins). The -versions are set per target framework in `Directory.Packages.props`. There is currently no `#if` in the source: the spike's one per-version spot (the clipboard API) is not needed, -because Avalonia 11.3.17 already has the `DataTransfer` API that 12 uses, and `FontFeature` parsing is written -without `FontFeatureCollection.Parse` (absent in 11.3) for both. Keep any future per-version difference small, behind -`#if NET10_0_OR_GREATER`, and list it here. The differences now: `SpellScanner` asks `CharUnicodeInfo` for the category of a code point directly on net10.0 and through a -one-character string on netstandard2.0 (no overload there), which only costs an allocation for a letter outside the basic plane; and `ScriptPlacement` reads a font's -OS/2 and hhea tables with `IGlyphTypeface.TryGetTable` on netstandard2.0 and, as Avalonia 12 gives no access to tables, from the font file's own table directory on net10.0 (once for each font; in a font collection from the member whose family name is the typeface's, and nothing is read whole). - -Avalonia.Desktop, Avalonia.Themes.Fluent and Avalonia.Win32.Interoperability are used by the samples only; the -libraries do not reference them. The playground tests load the Fluent theme so the sample's buttons and pickers have -templates. - -## The run model - -`Paragraph.Runs` is a `RunList`, not a `List`: it tells its paragraph whenever a run is added, removed or replaced, and each -run tells it when its text changes, so `Paragraph.Length`, `RunBefore`, `RunAfter` and `Spans` are answered from run starts the -paragraph keeps current (a binary search, not a walk over the runs) however a host changes the runs. A host may edit `Run.Text` or -the list directly. `RunList` has `Add`, `Insert`, `RemoveAt`, `RemoveRange`, `AddRange`, `Clear`, an indexer and `IEnumerable` -(use LINQ for `ConvertAll`, `FindIndex` and the like); assign a `new RunList(runs)` to `Paragraph.Runs` to replace the runs. A -change to a run's `PropertyBag` in place (`SetInt`, `SetStr`, or a write to `IntProps` or `StrProps`) is seen too: the bag tells the paragraphs that -hold it, so `ChangeStamp` moves and a field shows the change (see the spelling section for what a host still calls). - -`Paragraph.Runs = list` takes a `List` directly (the paragraph adopts a new `RunList` over the same runs, not the list itself: the `Run` objects are shared, but a later `list.Add` is not seen by the paragraph). `Run.Text = null` throws -`ArgumentNullException` (use an empty string); so do `Run.Props = null` and `Paragraph.Props = null`. Looping over `Paragraph.Runs` while the list is added to, removed from or has a run -replaced throws `InvalidOperationException`, as it does for a `List`; changing a run's text or properties meanwhile is fine. -`Paragraph.Text` is safe to read from any thread while nothing changes the paragraph (the model itself is edited on the UI thread only). -The work counters tests use (`TextReads`, `ElementReads`) exist in a build that defines `TEXTEDITING_COUNTERS` (every configuration but -Release) and cost nothing in a Release build. - -## Build and test - -Windows with the .NET 10 SDK and the .NET Framework 4.8 targeting pack (net48 builds and tests run only on Windows): +**Rich text editing for Avalonia in which every run of text knows its writing system.** + +SIL.TextEditing is the rich text field for [FieldWorks](https://software.sil.org/fieldworks/) and Motif, and for any +[Avalonia](https://avaloniaui.net/) app that edits multilingual text. Each run carries its own writing system, so Latin, Arabic and +Devanagari share one paragraph and each keeps its own font, spelling dictionary and keyboard. The run model is shaped like FieldWorks' +`ITsString`, and an adapter maps it to and from liblcm. One source tree builds against Avalonia 11.3 for FieldWorks (.NET Framework 4.8, +inside WinForms) and Avalonia 12 for Motif (.NET 10). Text can be written as FieldWorks Markdown, a strict profile of Pandoc Markdown that +Pandoc reads the same way the library does. The host owns the data and the undo stack; the field edits what it is given. + +**Status: unreleased. The API will change.** Everything outside the [Roadmap](#roadmap) is built and tested today. + +![Latin and Arabic in one paragraph, with one selection running across the direction changes](docs/images/bidi-selection.png) + +| | | +|---|---| +| ![Hindi, Tamil and Thai shaping](docs/images/indic-thai-shaping.png) | ![Inline pictures in left-to-right and right-to-left text](docs/images/inline-pictures.png) | +| [Hindi, Tamil and Thai, shaped, with a conjunct selected as one cluster](docs/images/README.md) | [Inline pictures in left-to-right and right-to-left text](docs/images/README.md) | +| ![Runs with unknown properties and the live inspector](docs/images/unknown-properties-inspector.png) | ![Spelling squiggles](docs/images/spelling-squiggles.png) | +| [Properties the editor does not know survive every edit](docs/images/README.md) | [Spelling per writing system; blue where a word mixes two](docs/images/README.md) | +| ![The phonological-environment box](docs/images/environment-box.png) | ![Headings, lists and tables with their switches](docs/images/block-features.png) | +| [A notation box with a language service: chips and diagnostics](docs/images/README.md) | [Tables, lists and headings, switched on or off per field](docs/images/README.md) | +| ![The playground inside a WinForms form](docs/images/host-winforms-fieldworks-shape.png) | ![Composed and decomposed text and emoji](docs/images/nfd-surrogates.png) | +| [Inside WinForms on .NET Framework 4.8: FieldWorks' shape](docs/images/README.md) | [NFD, stacked marks and surrogate pairs](docs/images/README.md) | + +## Why it's different + +- **A writing system on every run.** A `Run` holds text, a writing-system id, a property bag keyed by FieldWorks property numbers, and + object data, as an `ITsString` run does. The editor carries properties it does not understand through every edit. The LCModel adapter + round-trips real liblcm strings (ADR 0016, 111 tests per target). +- **Both Avalonia lines from one source tree.** Avalonia 11.3.17 on net48 inside WinForms, and Avalonia 12.1.2 on net10.0, with two small + per-version differences, [listed](docs/reference/avalonia-versions.md#per-version-differences) ([ADR 0009](docs/adr/0009-two-avalonia-lines.md)). The same surface tests run on both: 777 on + net48, 778 on net10.0. +- **FieldWorks editing behaviour.** Arrow keys are logical by paragraph direction, text is stored as NFD, Backspace and Delete work as + in FieldWorks ([ADR 0018](docs/adr/0018-fieldworks-editing-semantics.md)). Complex shaping, inline objects, and screen reader support + at FieldWorks' level ([ADR 0017](docs/adr/0017-screen-reader-gate.md)). +- **FieldWorks Markdown that Pandoc agrees with.** Read lenient, write strict. 667 tests run Pandoc 3.9.0.2 itself over every fixture + and generated documents and require the same structure ([ADR 0028](docs/adr/0028-fieldworks-markdown-is-a-pandoc-profile.md)). +- **Records as files.** A whole record as one file: a YAML frame typed by a schema that is data, FieldWorks Markdown in the rich values, + and a JSON Schema for editors ([ADR 0032](docs/adr/0032-record-files-yaml-frame.md)). +- **The host stays in charge.** The host owns data and undo ([ADR 0003](docs/adr/0003-host-owns-data-and-undo.md)), and supplies the + dictionaries, keyboards, clipboard forms, language services and the blocks a field allows. +- **Ready for native AOT.** The net10.0 libraries other than the LCModel adapter are marked trimmable and AOT-compatible, and + `tests/AotCheck` publishes the Markdown library native AOT and runs it. + +The whole suite passes with 0 failures on net48 and net10.0. Per target, as of 2026-11-06 (main 7ffd8f6 with PRs #165 and #166 merged, Release): + +| Tests | Model | Avalonia | Markdown | LCModel | Playground | SecondDispatcher | +|---|---|---|---|---|---|---| +| net48 | 416 | 777 | 3841 | 111 | 304 | — | +| net10.0 | 416 | 778 | 3841 | 111 | 304 | 3 | + +## Quick start + +Windows, the .NET 10 SDK and the .NET Framework 4.8 targeting pack: ``` dotnet build SIL.TextEditing.sln -dotnet test SIL.TextEditing.sln # both target frameworks of both test projects +dotnet test SIL.TextEditing.sln ``` -From WSL, run the same commands through `powershell.exe`. The samples are built by the solution. The headless tests use -the Windows fonts Segoe UI, Nirmala UI, Leelawadee UI and Constantia. - -## Try it +Try it: ``` run desktop10 # Avalonia desktop window on .NET 10, Avalonia 12 run desktop48 # the same window on .NET Framework 4.8, Avalonia 11.3.17 run winforms48 # WinForms form: TextBox, playground, TextBox (FieldWorks' shape) +run desktop10 --scenario "Inline picture (ORC)" ``` -`run.cmd` and `run.ps1` build the host and start it; add `--scenario "Inline picture (ORC)"` to open a scenario. -The playground loads ready-made documents (Latin and Arabic, Indic and Thai shaping, NFD and surrogates, inline -pictures, unknown properties, an environment box with a language service, 200 paragraphs, many fields) and shows what the model holds in a live inspector. A -person tests with the checklist in [docs/manual-testing.md](docs/manual-testing.md). - -## FieldWorks editing semantics (ADR 0018) - -- **Arrow keys are logical by paragraph direction.** Left goes backwards in a left-to-right paragraph and forwards in a - right-to-left one; Right is the reverse. Inside mixed-direction text the caret may therefore move the other way on - screen. Shift+arrows extend the same way, Ctrl+arrows go by word in logical order, Up/Down and Home/End are unchanged. - The caret is drawn at the visual place of its logical offset (trailing edge of the cluster passed after a forward step, - leading edge of the next cluster after a backward step). -- **Backspace** removes one code point (a surrogate pair is one); an embedded object goes whole. -- **Delete** removes one base character and the combining marks (Mn, Mc, Me) after it. -- **Normalisation.** `Document.Normalization` (also `RichTextField.Normalization`) is a nullable `NormalizationForm`, null by - default (text is inserted as it arrives). A host sets `NormalizationForm.FormD` for FieldWorks: typed text, IME commits - and pasted text (plain or rich) are converted before insertion and the caret lands after the converted text. Plain - text the field puts on the clipboard is always NFC; the rich forms keep the stored form. - -## Clipboard (ADR 0029) - -The host registers the clipboard forms the field copies and pastes; the field takes no dependency on liblcm or the Markdown -converter. `RichTextField.ClipboardForms` holds `IClipboardForm`s (an id, a display name, a clipboard format name, `Write` and -`Read` over an `object` payload: a string, a `byte[]` or a host object, and a `PayloadKind` saying which, so the backend reads the format back as that and never guesses text from bytes). The clipboard itself is behind `RichTextField.ClipboardBackend` (`IClipboardBackend`), which is Avalonia's -clipboard unless the host sets its own; payloads pass through it untouched (the default backend stores strings and `byte[]`s, each read back as the kind the form declares; a host object needs a host backend, and `RichTextField.ClipboardFormsNotStored` tells the host which forms the default backend could not store). `CopyAsAsync` returns false for a form whose payload is not text. Plain text is built in. -FieldWorks interop is the host's job and uses two hooks. (1) `RichTextField.ClipboardBackend` (`IClipboardBackend`: `SetAsync(plain, formats)`, `GetAsync(formatName)`, `GetTextAsync()`; Avalonia's clipboard by default) lets FieldWorks' WinForms host keep the clipboard itself, so it can put the serialised `TsStringWrapper` under "TsString" and read it with `GetData(...) as TsStringWrapper`, as it does today. (2) A form's payload is an `object` (a string, a `byte[]` or a host object) that passes through the backend untouched, so FieldWorks registers its own `IClipboardForm` with the id and format name "TsString" whose `Write` returns the wrapper object (built from `TsStringClipboard.FragmentToXml`) and whose `Read` unwraps it (`TsStringClipboard.FromXmlToFragment`). A FieldWorks host also registers the FieldWorks Markdown form (the Markdown library will provide it; only the registration point and the ids -`ClipboardForms.FieldWorksMarkdownId` and `ClipboardForms.TsStringId` exist here). - -- **Copy** puts every registered form on the clipboard and the plain text in NFC. A form that throws or has nothing to say is left out. -- **Paste** tries `ClipboardForms.PasteOrder` in turn, by default TsString, FieldWorks Markdown, plain text (FieldWorks' own preference), - and uses the first form the clipboard holds and that reads. The list is the host's to change; a form that is not in it is never read. -- **Writing systems the target lacks** take the writing system of the selection the text replaces, as FieldWorks does. The field asks - `RichTextField.KnowsWritingSystem` (by default: its `WritingSystems` registry, and a registry with no entries accepts everything), and - raises `WritingSystemsMapped` after the paste so the host can show a notice. -- **Copy as** (`RichTextField.CopyAsAsync(formId)`) puts one form on the clipboard as text: the plain text, or a form that offers it - (`IClipboardForm.OffersCopyAs`, such as FieldWorks Markdown). The playground's context menu has a "Copy as" submenu. -- `SIL.TextEditing.Runs.v1` is no longer on the clipboard. `Interchange` stays as a test and debug dump, and - `InterchangeClipboardForm` lets tests and the playground carry formatting without liblcm or the Markdown library. - -A writing system the host did not register (`RichTextField.WritingSystems`) is drawn in Segoe UI at 10 point, FieldWorks' own default size. - -## Block features (ADR 0033) +`run.cmd` and `run.ps1` build the host and start it. The playground opens ready-made scenarios (bidi, Indic and Thai shaping, NFD, inline pictures, unknown properties, spelling, the +environment box, long documents) with a live model inspector. The Pandoc tests need `pandoc` on `PATH`. -Which of **tables, lists and headings** a field may hold is `RichTextField.BlockFeatures` (forwards to `DocumentEditor.BlockFeatures`), a `BlockFeatures` flags -value (`None`, `Tables`, `Lists`, `Headings`, `All`) beside `SingleParagraph`. The default is `All`, so nothing changes until a host turns a kind off (owner -decision Q79). When a kind is off: - -- **Commands that make it are refused.** The editor has no style command (the host applies a paragraph style as an undoable delta), so the host asks - `RichTextField.IsParagraphStyleAllowed(styleName)` before it applies one and hides what is off. The playground has a switch for each kind, hides the styles of - a kind that is off in its style box and refuses them in `ApplyParagraphStyle`. -- **A paste, and so a Markdown paste, flattens it.** A table becomes one plain paragraph per row with the cells joined by a space, a bar and a space (`a | b`), a list item or a heading - becomes a plain paragraph, and `RichTextField.BlocksFlattened` tells the host how many of each (the same way `WritingSystemsMapped` does). Blocks the - document already holds are kept, undo restores them exactly, and Enter in one starts a plain paragraph. -- **How a paragraph is a table, list or heading is decided in one place**, `IBlockRecognizer` (`RichTextField.BlockRecognizer`). The default, - `StyleBlockRecognizer`, goes by the host's paragraph style names; `MarkdownStyleNames.ToBlockRecognizer()` makes one from the names the Markdown reader uses - (a host that pastes Markdown gives the field the recogniser made from its Markdown options). Today's table is a tab between cells in a paragraph of a table - style; a later storage form is another recogniser. - -A **record schema's** rich field has the same flags (`Rich(name, paragraphs, ws, blocks)`): `All` for a field of paragraphs and `None` for a field of one -paragraph. `RecordFileReader` reports `fwmd.record.block-not-allowed` for each disallowed block, with a fix that flattens it as paste does, and the JSON Schema -carries `x-fieldworks-blocks` where a field restricts them. The defaults computed from liblcm (all on for a multi-paragraph field, off for `Text.Contents` and -Scripture) belong to the planned `SIL.TextEditing.Project`; see ADR 0033. - -## Keyboards (ADR 0008) - -The field never switches keyboards; the host does. Subscribe to `RichTextField.ActiveWsChanged` (old and new writing system) -and switch the OS keyboard or Keyman to the new one. The field raises it only while it has focus, as FieldWorks' views set the -keyboard only with focus; a change made while it is unfocused is not reported, and the focus-gain event below carries the result. -It is raised once per change, never when the writing system stays the same, when the caret moves into text -of another writing system, when a selection's first writing system changes (a selection takes the writing system of the first -text it covers, as in FieldWorks), when an edit or undo leaves the caret in another one, and when the host or the writing-system -toolbar calls `Editor.SetActiveWs`. It is also raised, with old and new equal, when the field gains focus, so the host can -switch to the keyboard of the writing system at the caret even though nothing changed in this field. - -**Focus in and out (what FieldWorks does, and what a host does to match).** On Windows, FieldWorks sets the keyboard on focus -gain only when focus comes from a window of the same process, so a keyboard the user picked in the tray or in another -application is kept when focus returns without a click (Alt+Tab, the tray). A click into a view makes a selection, and the -selection sets the keyboard for it, so a click does set the keyboard. (On Linux FieldWorks sets it on every focus gain.) When a -view loses focus to a control elsewhere in the same application that is not a view, it restores the default keyboard; leaving -the application is left to the operating system. The field cannot tell where focus came from (Avalonia's `GotFocusEventArgs` -gives the navigation method and key modifiers, not the source), so it raises the focus-gain event every time and the host -decides: switch only if the previously focused window was its own, otherwise leave the keyboard alone. The field's event -follows changes of writing system, so a click into text of the same writing system gives no event of its own. To restore the -default keyboard, subscribe to the field's `LostFocus` and switch when focus has moved to a control of your own application -that is not a field. The field raises nothing when it loses focus. Avalonia gives the field `LostFocus` when its window is deactivated and `GotFocus` when it is activated again, on both lines (checked in real windows with a second window of the same application; switching to another application was not driven), so a reactivation gives one focus-gain event; the event arguments cannot tell that from any other focus gain, so a host that must uses its window's `Activated`. - -## Host hooks (ADR 0021, ADR 0025) - -The field owns one text; the window around it belongs to the host. These are the hooks: - -- **Find and replace.** The host's dialog builds a `FindOptions` (text, match case, match diacritics, whole word, regular - expression, writing system, character style). `Editor.TryFind` searches from the selection (next or previous, optionally - wrapping), `Editor.FindAll` lists the matches, `Editor.ReplaceMatch`, `ReplaceCurrent` and `ReplaceAll` replace as - single undo steps. `RichTextField.Find` and `SelectMatch` also select the match, mark it as the current match - (`CurrentMatch`) and ask the host to scroll; `SetFindHighlight` highlights every match while the dialog is open. Text - is compared as NFD, so composed and decomposed forms find each other. -- **Context menu.** `RichTextField.ContextMenuRequested` fires for a right click, the Menu key and Shift+F10 with a - `ContextInfo`: the word and its range, its writing system, the run and its properties, any link or embedded object, the - selection and an (empty, for now) list of spelling suggestions. The host builds the menu. Use the event or Avalonia's - `ContextMenu` on the field; with both, the event wins. -- **Paging and scrolling.** PageUp and PageDown (with Shift) move the caret by the field's visible height - (`VisibleBounds`, or `PageHeight` when the host sets it); a drag past the edge extends the selection and asks for - scrolling. The field never scrolls anything itself: it raises `ScrollRequested` (reason, the part to show, the distance) - and a scroll viewer round the field answers when no handler marks it handled. - -## Spell checking (ADR 0020, 0026) - -The host gives the field its dictionaries: set `RichTextField.SpellChecking` to an `ISpellCheckHost` that returns an -`ISpellChecker` (`Check`, `Suggest`, `Add`; `ISpellIgnorer` adds `Ignore` when a dictionary has it) for each writing system, says which -characters form words and which join them, and raises `DictionariesChanged` when a dictionary changes. Derive from -`SpellCheckHostBase` to write only `GetChecker`: it has the default word rules and `RaiseDictionariesChanged()` (any thread). The same word rules (`IWordRules`) serve the double-click, Ctrl+arrows and whole-word find; `RichTextField.WordRules` gives them without spell checking. The -field checks the paragraphs in view when idle, a few words at a time, and draws red squiggles (misspelled, or a word with an object in it) and blue -ones (a word that mixes writing systems or has unchecked text inside). The rules are in `SpellScanner` (UI-free): words -break at characters that are not word-forming, and do-not-check text and text that may not be edited (unless forced) are -skipped. `SpellingSuggestionsAt`, `ApplySpellingSuggestion`, `AddToDictionary`, `IgnoreWord` and `RestartSpelling` are -what a host's context menu and dictionary changes call (`ApplySpellingSuggestion` also takes runs, for "insert missing spaces" and, with -`WithWholeWordIn`, for "make the whole word writing system X" in a word with several writing systems). The word of a context request and the squiggled word -are one definition (an underscore ends both, as in FieldWorks). A checker -that throws raises `SpellCheckerFailed` (written to `Trace` once per writing system when nothing handles it) and is left alone; it does not end -the application. A host that repairs the dictionary raises `DictionariesChanged` (or calls `RestartSpelling`), which tries that writing system again. The playground's "Spell checking" scenario shows it with a -short English and French word list. -A dictionary that is slow can be asked off the UI thread: its checker also implements `IBackgroundSpellChecker` (`Check` may then run on a worker; the field finds the words and applies the answers on its own thread, and drops an answer for a paragraph that has changed since). -`CheckSpellingStep` then returns false while a batch is out, and `IsSpellingComplete` says when everything in view is checked. A loop of -`while (!field.CheckSpellingStep()) { }` on the UI thread therefore never ends with such a checker (the answer is applied by the dispatcher the loop blocks): -let the dispatcher run and wait for `IsSpellingComplete`, or use a checker without the interface where a synchronous check is wanted. - -What a host calls (the field restarts checking itself on an edit through the editor, `Apply`, `ExternalReplaceParagraph`, a change of -`IsReadOnly` on the field or its editor, a word added or ignored, and a `DictionariesChanged` notice): nothing, after it changes a paragraph -directly. A paragraph tells the field when its runs, a run's text, writing system, object data or property bag (replaced, or changed in place with -`SetInt`, `SetStr` or a write to `IntProps` or `StrProps`), or its style, direction, default writing system or property bag, change -(`Paragraph.ChangeStamp` moves): the field measures and draws again, lays out that paragraph again (and only that one, keyed by the stamp), numbers -the list again, drops the marks of that paragraph and checks it again when idle. A notice from another thread is marshalled to the field's own, but the model is not thread-safe (ADR 0024): the edit itself must still not overlap a layout, so a host edits on the UI thread. -`RichTextField.RefreshFromDocument()` is for what a paragraph cannot tell: the list of paragraphs (`Document.Paragraphs`) changed, a paragraph -replaced in it. It restarts checking too. When a style resolver's answers change (not the resolver itself, which the field takes again when it is set), `InvalidateAllLayouts()`, and -`RestartSpelling()` when what is checked changed. After repairing a failed dictionary, raise `DictionariesChanged`. Setting -`DocumentEditor.IsReadOnly` directly tells automation as `RichTextField.IsReadOnly` does. - -## Language service (ADR 0030) - -A box whose text is a notation (the first is a FieldWorks phonological environment, `/ [V] _ #`) takes a host **language service**: -`RichTextField.LanguageService` is an `ITextLanguageService` (in `SIL.TextEditing`, no Avalonia or liblcm), shaped like the spell-check host and with -Language Server Protocol data (zero-based line and character in UTF-16 units, ranges, severities 1 to 4, completion items), so a VS Code adapter and -agents' findings can use the same answers. `AnalyzeAsync` gives tokens (a kind string and a range) and diagnostics (range, severity, id, message, an -optional exact fix) together; `CompleteAsync`, `HoverAsync` and `CanInsert` do the rest; `Changed` tells every field to ask again; every call that can take -time is a `Task`, so Motif's backend can answer from another process. `TextLanguageService` is a base class for one that answers at once. The field serves a -field of one paragraph, and `SingleLine` (single-paragraph and no hard line break) is what a notation box sets. - -The field asks when idle and applies an answer only to the paragraph and stamp it was asked about (as spelling does); after an edit it carries the results -the edit left alone, so nothing flickers. **Tokens are chips**: a `TokenStyleMap` (per kind, a fill and an outline for the light theme and the dark one) is -drawn behind plain text, so copy, Markdown and undo see only text, and a high contrast theme gets none. **Diagnostics are squiggles** drawn with the -spelling marks' drawing (red error, blue warning, grey information; a hint is not drawn) and are never written into the document: no run, bag or property 5 -changes. Hovering sets the tooltip from the diagnostics' messages and the service's help (`GetLanguageHoverAsync`); `ContextInfo.Diagnostics` and -`ApplyDiagnosticFix` give the host's context menu a fix as one undo step; `CanInsert(kind)` enables a host's Insert commands. **Completion**: Ctrl+Space, or a -trigger character the service names, opens a popup left-aligned under the caret (a `List` of named `ListItem` rows, no theme needed, the keyboard stays in the field, the wheel scrolls it; it closes when the word or class being completed is done, so Enter and Tab reach the host again); Up, Down, PageUp, -PageDown, Enter or Tab (accepts, and neither reaches the host), Esc; typing asks again, and a caret move, a selection, losing focus, read-only or an input method -composing closes it. Where no popup can be shown the state and `CompletionChanged` still work. **A host that routes keys itself** (WinForms `IsInputKey` and `ProcessCmdKey`, a dialog's default and cancel buttons) asks `RichTextField.WantsKey(key, modifiers)` and leaves the keys it wants (Ctrl+Space, and Enter, Tab, Esc and the arrows while the popup is open) to the field; `samples/Host48` shows it. The field's peer is an expand/collapse provider for the popup and gives the worst diagnostic as its item status. `LanguageDebounce` waits after the last edit before asking, and `BackgroundTextLanguageService` runs the analysis on the thread pool. The playground's **Environment** scenario is the sample -(`SampleEnvironmentService`, over a made-up phonology; not shipped); the real service that reads a project's phonology comes later, in a project that -references liblcm. - -## Colours and themes - -Until the host sets `Foreground` (text, and a border with no colour of its own) and `Background` (the page), the field takes them from the -theme: the text colour the caret uses and the theme's window colour, and in a Windows high contrast theme the system's window text and window -colours, as FieldWorks does (the selection takes the system highlight, and selected text the system's highlight text colour, whatever colour its runs have). The text keeps one default brush that the field changes in place when -the theme or the system's colours change, so a theme switch redraws the text and does not lay out the paragraphs again. - -A colour the host sets is kept as it is under every theme, high contrast included. So a host that sets one sets the other, or neither: a host -that sets only `Foreground = Black` on a dark theme gets black text on the theme's dark page. `ResetForeground()` and `ResetBackground()` go back -to following the theme. A run's own colours (text property 8 and 9) are the document's and do not change with the theme. - -## Known gaps +## Projects -- **Accessibility**: no UIA `TextPattern`; the field is an Edit with a name and a value only. Inside a net48 - `WinFormsAvaloniaControlHost` Avalonia's UIA subtree was empty in the spike (unexplained). -- **Input**: real IME and Keyman interaction are untested (the headless tests drive the IME client directly). Missing: - scrolling inside long fields (the host scrolls; see Host hooks), drag and drop, touch. Selection by mouse (click - with bidi and soft-wrap affinity, Shift+click, drag across paragraphs, double-click word, triple-click paragraph) and - keyboard (Shift+arrows, Ctrl+arrows by word, Home/End by visual line, Ctrl+Home/End, Ctrl+A, Up/Down with the - preferred x) is done. The word rule (`WordBreaks`) is deliberately simple: runs of letters, digits and marks, with - an apostrophe between letters inside the word; no dictionary, so Thai and Chinese are not split, and it is not UAX #29 - word breaking. It is also the host's: one rule (`IWordRules`: which code points form words in a writing system, and which join the words on - each side of them) serves the double-click, Ctrl+Left and Right, the whole-word find, the word of a context request and the spell checker - (set `RichTextField.WordRules`, or the rule of `SpellChecking` is used), so a writing system that makes a hyphen a letter makes "foo-bar" one word - everywhere. The double-click (and a drag that goes by words) also stops where the writing system or the spell-check property changes (and where two different character styles, resolved by the host's `IStyleResolver`, differ in either; a style that only changes bold or colour keeps the word, and with no resolver a style name never splits it) - as FieldWorks does, so a word that runs across such a change is selected a piece at a time; the other uses of the rule take - the whole word. Left/Right and Ctrl+Left/Right are logical. Home/End in a right-to-left paragraph are the logical line start and end. -- **Avalonia faults, pinned by tests**: hit testing in a right-to-left paragraph is wrong on 11.3.17 (fixed in 12.1.2), and on - both 11.3.17 and 12.1.2 the caret, hit testing and selection of a left-to-right run are wrong on a right-to-left paragraph's line - of two or more runs, and likewise on a left-to-right paragraph's line of two or more runs with a right-to-left run in it (on 12 some of its ranges throw); an inline object in right-to-left text is mis-ordered on both. The layout works around the first two by - reading such a line from its runs as they are drawn (`LineInfo.UsesRuns`: caret x, hit testing, visual stops and selection boxes - from each run's glyphs). It is not behind `#if`, because 12 has the second fault too (ADR 0027, which supersedes the 11.3-only scope of ADR 0015). - `RichTextField.ObjectAsGlyph` is the workaround for the third. On 12 only, the spaces a wrapped line ends with after a run of Arabic in a - left-to-right paragraph come with glyph clusters numbered against the run's direction and answer distances and hits that do not add up - (a click at the end of the line lands one space short); `RunSlot` places such a run by counting its spaces (`ClusterOrderTests` names the - fault and fails when 12 stops doing it). - On 11.3 only, the formatter throws "Cannot split: requested length N consumes entire run" for some sequences with a zero width joiner, such as one - after a space before Arabic text in a left-to-right paragraph (`ZwjAfterSpaceTests` names it and fails when 11.3 stops doing it). The layout - catches the refusal and lays the paragraph out again with the joiners swapped for word joiners, then the other format characters, then (for a space followed by a mark, the other way the formatter refuses) the space swapped for a dotted circle so the mark is still drawn, and as a last resort every format character and mark swapped for a word joiner (the model, the offsets and the grapheme boundaries are untouched). Two costs of the retry, on 11.3 only: the no-break space it puts before a mark - does not hang at a line end the way a space does, and the dotted circle is drawn where the model has a space (the caret and the offsets still treat it as the space). - On 12 only, asking for the boxes of a range that covers no width (a right-to-left mark alone before the spaces a line ends with) throws "Covered length must - be greater than zero"; the layout catches it, in the boxes of a range and in the spaces a justified line stretches, and answers that such a range has no box (`RlmWrapTests` names it and fails when 12 stops doing it). -- **Paragraph and character properties**: drawn, with the host's `IStyleResolver` supplying named styles (ADR 0004): weight (any 100 to - 900, invert), italic, foreground and background colour, underline type and colour (dotted, dashed, single, double, - strikethrough, and the squiggle, which the layout draws itself as a zigzag below the baseline), superscript and subscript (sized, and raised or lowered, as the font's own OS/2 table says, with FieldWorks' two thirds where it gives nothing; the ratios are applied to the size of the text beside it, which is a default), font size (absolute, relative, named), font - family and per-run font features; for a paragraph, alignment, first-line and hanging indents, margins, space before - and after, line height, paragraph colour, borders and padding, a line limit, bullets and numbers. A run's baseline offset is drawn by drawing the run - ourselves, shifted: the line's slot makes room for it and the caret, selection and squiggle follow it. A justified paragraph is stretched by - the field itself (Avalonia's justifier is internal): each line but the last shares its slack equally among its interior - spaces, so it reaches both edges of the text area; the last line stays at the leading edge, as in FieldWorks. The caret, selection, - mouse and run backgrounds follow the stretched text, and each stretched line is drawn piece by piece, with its underlines and strikethrough drawn by the field as lines across the whole stretched run (so a dotted or dashed pattern is not restarted or scaled). Pagination properties and the rest are carried untouched. - Lengths in points are drawn as device-independent pixels, as font sizes are. - Text a line limit hides stays in the model but cannot be reached: the caret, Right, End, Up/Down and mouse stop at the end of - the last line shown, as FieldWorks puts an insertion point in truncated text at the last visible position. Typing, Delete, Enter, paste - and composition are refused at that point, and Backspace at the start of the next paragraph does not join it onto the hidden end. - Select All and Copy still take the hidden text, because the selection model holds it. As in FieldWorks, the last - visible line of a cut-off paragraph ends with "..." in the paragraph style's character format, on the trailing side of the text - (left of it in a right-to-left paragraph), with room kept for it when the text area is wide enough for one beside text; the - caret and hit testing stay on the text, and a hard line break that ends the paragraph is not cut-off text. -- **Spaces a wrapped line ends with**: they hang past the line's text, outside the alignment (a line is aligned by its text), and the - caret across them moves one space width per space, as in FieldWorks, even past the text area. The control draws nothing outside itself, so - a caret that would be there is held at the control's edge (the End caret at the edge, the carets of the spaces before it 2 px apart inside - it). When the text itself reaches the edge (a justified line, a right-aligned one, a line that just fits) there is no room for those - carets to show apart: they crowd on the text's last letters. A host that wants each caret seen gives the text area room on the side the - spaces hang to, with the paragraph's trailing indent (property 20: the side lines end on, in either direction) of a few spaces' width; a margin around the control does not help, as - the field cannot draw there. The mouse follows the carets as they are drawn: a click at one gives it, and a click beyond the End caret - gives the End caret. -- **Very long paragraphs**: runs that look alike are one stretch of text to Avalonia's formatter, and a paragraph's ranges are found by halving, so a layout grows with the runs. Avalonia's - formatter itself (about nine tenths of a layout's time, on both versions) carries what is left of the text from each line to the next, so a paragraph of tens of thousands of runs - costs more than its length: 64,000 runs lay out in about 2.0 s on 11.3 and 1.6 s on 12. From 16,000 runs (after runs that look alike are one) a paragraph of Latin text (letters, their marks, punctuation) that is not right to left - is formatted line by line from a window of its text instead (`ParagraphLayout.FormatWindowed`), which gives the same lines and carets, but for the kerning of a pair across a line start (widths move by a fraction of a pixel), in about 1.1 s on 11.3 and 0.46 s on 12 for those 64,000 runs; a paragraph with any other script is formatted whole, as the directions of right-to-left text are resolved for all of it at once and a neutral character at a line start keeps the fallback font of the text before it (CJK, Thai). -- **Graphemes**: `GraphemeClusters` is `StringInfo` plus Indic conjuncts (GB9c, six scripts; .NET 10 lacks it). The - netstandard2.0 build also fuses Extend, SpacingMark, emoji ZWJ sequences and flags per UAX #29, with approximated - property tables; caret stops in the field come from the shaper, except on a right-to-left paragraph's line of several runs - (`LineInfo.UsesRuns`), where the layout steps by this class's grapheme boundaries and measures each on the run's glyphs (with the - approximated tables on netstandard2.0). -- **Undo** is a host stand-in (`SimpleUndoStack`); no LCModel glue exists. +| Project | What it is | +|---|---| +| `src/SIL.TextEditing` | The UI-free run model, editing, clipboard forms and paste rules. No Avalonia, no liblcm. | +| `src/SIL.TextEditing.Avalonia` | `RichTextField`, the control. | +| `src/SIL.TextEditing.Markdown` | FieldWorks Markdown and record files. Trimmable and AOT-compatible. | +| `src/SIL.TextEditing.LCModel` | The `ITsString` adapter and the TsString XML clipboard form. | +| `samples/SIL.TextEditing.Playground` | The shared playground UI both hosts and the integration tests use. | +| `samples/Playground.Desktop` | The playground in a plain Avalonia window, net48 and net10.0. | +| `samples/Host48` | The playground inside a WinForms form, net48: FieldWorks' shape. | +| `tests/*` | Model, surface, Markdown, LCModel, playground, second-dispatcher, real-clipboard and AOT tests. | + +Libraries target netstandard2.0 and net10.0. Details: [Projects and tests](docs/reference/projects.md). + +## Documentation + +- [Reference](docs/reference/README.md): the run model, editing, clipboard, spelling, the language service, rendering, the two Avalonia + versions, the LCModel adapter, the Markdown library, known limitations. +- [Formats](docs/formats/README.md): FieldWorks Markdown, record files, the clipboard forms, tables, and every finding id. +- [Decisions](docs/adr): the architecture decision records. +- [Feature inventory](docs/feature-inventory.md): FieldWorks' editing features and where each stands here. +- [Manual testing](docs/manual-testing.md): the checklist a person runs. +- [Glossary](GLOSSARY.md): the words this project uses. +- [Screenshots](docs/images/README.md): every image, and how to regenerate it. + +## Roadmap + +Planned, not built: + +- **Word-compatible copy and paste.** HTML and RTF out, HTML in, in a new `SIL.TextEditing.WordInterchange` project + ([ADR 0031](docs/adr/0031-shared-editing-and-interchange-layer.md), + [clipboard plan](docs/formats/clipboard.md#planned-word-html-and-rtf)). +- **Real tables** in the field ([#152](https://github.com/sillsdev/SIL.TextEditing/issues/152), [tables plan](docs/formats/tables.md#planned-real-tables-in-the-field)). +- **`SIL.TextEditing.Project`**: LCModel-backed record I/O and the record schema built from liblcm metadata + ([ADR 0031](docs/adr/0031-shared-editing-and-interchange-layer.md), [ADR 0033](docs/adr/0033-block-features.md)). +- **The `fwmd` command-line checker** ([#161](https://github.com/sillsdev/SIL.TextEditing/issues/161)). +- **The FieldWorks Markdown clipboard form** ([#168](https://github.com/sillsdev/SIL.TextEditing/issues/168)). +- **Editor integration**: an inline WYSIWYG and Markdown source mode, and a VS Code extension with a language server. Proposed ADR 0034, + awaiting owner decisions. +- **Retiring the liblcm refusals** once FieldWorks pins a liblcm with its fixes [#410](https://github.com/sillsdev/liblcm/issues/410), [#413](https://github.com/sillsdev/liblcm/issues/413) and [#414](https://github.com/sillsdev/liblcm/issues/414) ([#159](https://github.com/sillsdev/SIL.TextEditing/issues/159)). + +## Contributing + +Read [AGENTS.md](AGENTS.md) first: the rules for layering, packages, tests and commits apply to people and agents alike. Known limitations +are in [docs/reference/limitations.md](docs/reference/limitations.md). ## Licence -MIT, copyright SIL Global. See `LICENSE`. FieldWorks code is LGPL and is not copied here; see `AGENTS.md`. +MIT, copyright SIL Global. See [LICENSE](LICENSE). FieldWorks code is LGPL and is not copied here: FieldWorks behaviour is re-implemented +from documentation, standards, tests and observed behaviour ([ADR 0010](docs/adr/0010-mit-clean-room.md)). From e4159369d708d48574157162311ae4d6dfbdb0c7 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:41:38 -0500 Subject: [PATCH 3/6] AGENTS.md states the rules for headless tests, counted work, local liblcm, Pandoc and AotCheck, format docs, screenshots and merging. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2118ebf..444fc5e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ # Agent rules for SIL.TextEditing -- **Read first.** `GLOSSARY.md` for the words, `docs/adr/` for the decisions. Don't reverse an ADR silently: propose a - new one that supersedes it. +- **Read first.** `GLOSSARY.md` for the words, `docs/adr/` for the decisions, `docs/reference/` for how it works. Don't reverse an ADR + silently: propose a new one that supersedes it. - **Licence (ADR 0010).** This repository is MIT. FieldWorks code is LGPL. Never copy FieldWorks code here, not even edited. Re-implement FieldWorks behaviour from documentation, standards, tests and observed behaviour. Property numbers, run shapes, paragraph-key rules and clipboard formats are facts; the code is ours. @@ -9,9 +9,11 @@ code around them is still written from scratch. - **Layering.** `src/SIL.TextEditing` is UI-free and has no Avalonia or liblcm reference. The shared-layer split by dependency is in ADR 0031. `src/SIL.TextEditing.Avalonia` holds `RichTextField` and everything that touches Avalonia. Samples are samples and do not ship; `samples/SIL.TextEditing.Playground` is the one UI both hosts and `tests/SIL.TextEditing.Playground.Tests` use, so a new feature gets a playground control and a test that drives it. + - `src/SIL.TextEditing.Markdown` has no Avalonia, libpalaso or liblcm reference. + - `src/SIL.TextEditing.LCModel` references liblcm and is not trimmable or AOT-compatible; keep it out of `tests/AotCheck`. - **One source tree, two Avalonia versions.** netstandard2.0 builds against Avalonia 11.3.17 (FieldWorks, net48); net10.0 builds against Avalonia 12 (Motif). Versions live in `Directory.Packages.props` only. A per-target - difference goes behind `#if NET10_0_OR_GREATER`, is small, and is listed in the README. + difference goes behind `#if NET10_0_OR_GREATER`, is small, and is listed in `docs/reference/avalonia-versions.md`. - **Packages.** Only SIL.WritingSystems (libpalaso, ADR 0002; referenced by the Avalonia and LCModel projects), Avalonia (and its Headless, Skia, Desktop, Themes.Fluent and Win32.Interoperability packages; Themes.Fluent and Win32.Interoperability are samples-only; Desktop is for samples and test projects that need a real window, never for a shipped library), SIL.LCModel.Core (FieldWorks' pin; referenced by `SIL.TextEditing.LCModel`, ADR 0016) and SIL.LCModel (FieldWorks' pin, hosts may float up; referenced by the planned `SIL.TextEditing.Project` only, ADR 0031), Markdig.Signed 0.37.0 (referenced by the `SIL.TextEditing.Markdown` project only, ADRs 0011 and 0028), YamlDotNet 18.1.0 (its parser layer only, never the reflection serializer; referenced by the `SIL.TextEditing.Markdown` project only, ADR 0032) and the xunit test stack. Ask before adding anything else. - **Warnings are errors, nullable is on.** `IsTrimmable` and `IsAotCompatible` stay on for net10.0 libraries, except `SIL.TextEditing.LCModel` and the planned `SIL.TextEditing.Project` (liblcm is not trim-safe; neither host publishes trimmed or native AOT, ADR 0031). @@ -20,4 +22,19 @@ property numbers, and `ObjData`. The editor never interprets a bag; it carries it through every edit. - **Tests.** Build and test on Windows (`dotnet test SIL.TextEditing.sln`): net48 only runs there. A test that pins a known Avalonia fault says so in its name; when it starts failing, remove the workaround, not the test. + - Avalonia tests that create controls run inside `Headless.Run`. + - No wall-clock assertions. Assert counted work (counters such as `RecordWork.Steps`, `TextReads`). + - Pandoc tests skip silently when `pandoc` is not on `PATH`, unless `CI` (or `GITHUB_ACTIONS`) is set; then they fail. Set `CI` locally + to be sure they ran. + - `tests/AotCheck` needs `vswhere` on `PATH` (issue #164), and `-p:NuGetAudit=false` locally while issue #104 is open. +- **Local liblcm.** Pack the liblcm stack into `H:\caches\localnugetpackages` with a stamped version, then + `dotnet test -p:SilLcmVersion=` (see `Directory.Packages.props`). The refusal sweep tests are expected to fail against a fixed + liblcm. +- **Format docs.** + - Every `fwmd.*` finding id needs a conformance fixture case (`EveryPublicFindingIdIsGuardedOrNamedAsHostOnly`; host-only ids are named there) and an entry in `docs/formats/findings.md` + (`FormatDocsTests`). + - Examples in `docs/formats` carry fixture markers and are tested; change the fixture and the doc together. +- **Screenshots.** `TEXTEDITING_SCREENSHOTS=1` turns on the generator (`docs/images/README.md`). Regenerate the images when the UI changes + visibly. +- **Merging.** Merge only when every check passes. Run `gh pr checks N --json bucket` and check every bucket, not just the exit code. - **Commits.** One concern per commit, a one-sentence message saying what is now true. From 6adbb975358f2b632e2b1a7522ccf0d0bb3410dc Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:41:39 -0500 Subject: [PATCH 4/6] Comments and docs that pointed at README sections now point at docs/reference. Co-Authored-By: Claude Opus 5.5 --- docs/feature-inventory.md | 20 +++++++++---------- docs/manual-testing.md | 2 +- src/SIL.TextEditing.Avalonia/RichTextField.cs | 2 +- .../SurfaceBasicsTests.cs | 2 +- .../ThemeColourTests.cs | 2 +- .../ScaleTests.cs | 2 +- 6 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/feature-inventory.md b/docs/feature-inventory.md index 85044ea..2117f9b 100644 --- a/docs/feature-inventory.md +++ b/docs/feature-inventory.md @@ -52,7 +52,7 @@ Engine entry: `VwPropertyStore::put_IntProperty` (V/VwPropertyStore.cpp:1177, ca | ktptBackColor (9) | :1368; DRAW run background (UniscribeSegment DrawText, :1012-1014 comments) | NO | YES (`RunTextProperties.Back`, painted by `ParagraphLayout.DrawBackgrounds` below the selection; not under the spaces a wrapped line ends with, as FieldWorks' line-end segments leave them out of their width) | S | | ktptBaseWs (16) | :1403; CARRY mostly: `m_wsBase` seeds the font size for relative line height (:1470) | NO | NO (carried) | S | | ktptFontFamily string (1) incl. markup ``, ``, ``, `` | :1577; DRAW; markup resolved by V/../AppCore/FwStyledText.cpp:48-51,1467 | PARTIAL: FA/DetailModel run FontFamily; per-run font display FA/DetailRichTextChrome.cs:30-60 | PARTIAL: carried (ParagraphLayout uses the ws font only, :181) | M | -| ktptFontVariations string (130) OpenType/Graphite feature list, concatenated | :1595; DRAW features parsed V/lib/UniscribeSegment.cpp:241 (`TryParseFontFeatureRecords`), Graphite V/lib/GraphiteSegment.cpp | NO | PARTIAL: ws features only, per-run NOT read (README "Paragraph and character properties") | M | +| ktptFontVariations string (130) OpenType/Graphite feature list, concatenated | :1595; DRAW features parsed V/lib/UniscribeSegment.cpp:241 (`TryParseFontFeatureRecords`), Graphite V/lib/GraphiteSegment.cpp | NO | PARTIAL: ws features only, per-run NOT read (docs/reference/rendering.md) | M | | ktptWsStyle string (156): ws-dependent character-style overrides, merged by `ComputeWsStyleInheritance` | :1580-1594, :2120-2160 (explains order); applied when the ws is known; CARRY in string, RESOLVE at draw | NO (lossy: read-only) | NO | L | | ktptNamedStyle string (133): paragraph or character style name; applied first, may bring ws styles, drop caps hack | :1609-1630; DRAW via the style sheet | PARTIAL: NamedStyle on run, apply FA/FwFieldControls.cs:342-429 | PARTIAL: carried as CharStyle/NamedStyle numbers, no stylesheet resolution (TE/M/PropertyBag.cs `TextPropertyNumbers`) | L | | ktptCharStyle (2), ktptParaStyle (3): old style carriers | ignored (:1642 default); CARRY | NO | PARTIAL (constants only) | S | @@ -119,10 +119,10 @@ Dispatch: key down -> `EditingHelper.OnKeyDown` (SRS/EditingHelper.cs:648) -> `C | Feature | Legacy source | FA | TE | Size | |---|---|---|---|---| | Left/Right by character, logical by default, RTL-aware | V/VwSelection.cpp:7923 `LeftArrow`, :7945 `RightArrow`, :8013 `PhysicalArrow`, :8244 `BackwardArrow` | PARTIAL: own bidi nav FA/DetailBidirectionalTextNavigation (used FwFieldControls.cs:156-225) | PARTIAL: visual movement instead (`RichTextField.MoveVisualLeftRight`) | M | -| "Clumping": arrow skips a whole cluster/ligature unless registry `ArrowByCharacter`=True or env `FW_ARROWBYCHARACTER` | V/VwRootBox.cpp:1384-1403, used :1517-1560 | NO (TextBox default) | PARTIAL: shaper caret stops (`ParagraphLayout.VisualStops`; on a right-to-left paragraph's line of several runs, grapheme boundaries measured on each run's glyphs, README), no override switch | S | +| "Clumping": arrow skips a whole cluster/ligature unless registry `ArrowByCharacter`=True or env `FW_ARROWBYCHARACTER` | V/VwRootBox.cpp:1384-1403, used :1517-1560 | NO (TextBox default) | PARTIAL: shaper caret stops (`ParagraphLayout.VisualStops`; on a right-to-left paragraph's line of several runs, grapheme boundaries measured on each run's glyphs, docs/reference/rendering.md), no override switch | S | | F7 / F8 = strict left/right arrow by character (physical) | V/VwRootBox.cpp (kecF7/kecF8 cases); SRS/EditingHelper.cs:658-659 | NO | NO | S | | Shift+arrows extend selection | V/VwSelection.cpp:9241 `ShiftUpArrow`, ShiftLeft/Right | YES (TextBox + FA nav 191-208) | YES (`RichTextField.HandleKey`) | S | -| Ctrl+Left/Right by word (logical) | V/VwSelection.cpp:9492 `ControlLeftArrow`; word boundary :11199 `FindWordBoundary` (ICU per ws) | NO (Ctrl keys skipped, FwFieldControls.cs:160-162) | PARTIAL: `WordBreaks`, a simple rule that is the host's per writing system (`IWordRules`, `RichTextField.WordRules`; the spell checker's rule), no dictionary (README, ADR 0026) | M | +| Ctrl+Left/Right by word (logical) | V/VwSelection.cpp:9492 `ControlLeftArrow`; word boundary :11199 `FindWordBoundary` (ICU per ws) | NO (Ctrl keys skipped, FwFieldControls.cs:160-162) | PARTIAL: `WordBreaks`, a simple rule that is the host's per writing system (`IWordRules`, `RichTextField.WordRules`; the spell checker's rule), no dictionary (docs/reference/editing.md, ADR 0026) | M | | Ctrl+Up / Ctrl+Down = previous / next paragraph start | V/VwSelection.cpp:9706 `ControlUpArrow`, :9817 `ControlDownArrow` | NO | NO | S | | Up/Down keep a preferred X (`m_xdPos`) across lines | V/VwSelection.cpp:8487 `UpArrow`, VwRootBox.cpp:1431; `UpOneLine`/`DownOneLine` | PARTIAL (TextBox) | YES (RichTextField MoveVertical) | S | | Home/End: visual line; Ctrl+Home/End: start/end of view; logical in RTL | V/VwSelection.cpp:8777 `HomeKey`, :9908 `ControlEndKey`, :9969 `ControlHomeKey`; EditingHelper.cs:1142-1190 scrolls | PARTIAL | YES (`RichTextField.HandleKey`, Home and End; End on a wrapped line puts the caret after the line's trailing space, as in FieldWorks, and draws it one space width per hanging space past the text, held inside the control; where the control gives no room the carets across two or more spaces are held a default 2 px apart (`ParagraphLayout.MinCaretStep`) so each arrow step shows, except where the text itself ends at the edge, which a host margin cures; a click among those spaces goes to the nearest caret as drawn) but RTL Home/End logical | S | @@ -131,13 +131,13 @@ Dispatch: key down -> `EditingHelper.OnKeyDown` (SRS/EditingHelper.cs:648) -> `C | Tab / Shift+Tab: to next/previous editable place in the root box; S_FALSE = leave the view (next slice/record); Ctrl+Tab = last editable; Ctrl+Shift+Tab = first | V/VwRootBox.cpp:~1672-1705; V/VwSelection.cpp:10213 `TabKey`, `ShiftTabKey`; `FindEditablePlaceOnLine`; SRS `IsInputKey` 4907 (`AcceptsTab`) | PARTIAL: one tab stop per field, group logic FA/DetailValueStopGroup.cs:124-149 | YES: next/previous editable place, Ctrl+Tab last, Ctrl+Shift+Tab first, leave only when none (`DocumentEditor.TryGetEditablePlace` in src/SIL.TextEditing/DocumentEditor.Editability.cs; `RichTextField.HandleKey` in src/SIL.TextEditing.Avalonia/RichTextField.cs) | M | | Arrow past the last position of a field leaves the field (S_FALSE: previous/next field) | V/VwRootBox.cpp:~1517-1700 ("Need to move to the previous/next field") | YES: Left/Right at text edges, Up/Down move between rows (FA/DetailValueStopGroup.cs:136-149) | YES: LeaveRequested at the edges (`RichTextField.HandleKey` in src/SIL.TextEditing.Avalonia/RichTextField.cs) | S | | Caret placement across non-editable text: IP moves to nearest editable place | V/VwSelection.cpp:12669 `FindClosestEditableIP`; V/VwRootBox.cpp MouseUp ~2370-2400 | NO | YES: caret moves to the nearest editable place within the paragraph (`DocumentEditor.NearestEditable` in src/SIL.TextEditing/DocumentEditor.Editability.cs) | M | -| Mouse click places IP (associates with previous/next char, bidi/line-end affinity) | V/VwRootBox.cpp:2108 `MouseDown`, :1771 `MakeSelAt`; SRS/SimpleRootSite.cs:4449 `OnMouseDown`, 5407 `CallMouseDown` | YES (TextBox) | YES (affinity: ParagraphLayout HitTestDetailed 350-373; a right-to-left paragraph's line of several runs is read from its runs, because Avalonia's hit testing is wrong there on 11.3 and 12, README; a click beyond a line that ends in a hard break goes to the line's visual end in the paragraph's direction) | M | +| Mouse click places IP (associates with previous/next char, bidi/line-end affinity) | V/VwRootBox.cpp:2108 `MouseDown`, :1771 `MakeSelAt`; SRS/SimpleRootSite.cs:4449 `OnMouseDown`, 5407 `CallMouseDown` | YES (TextBox) | YES (affinity: ParagraphLayout HitTestDetailed 350-373; a right-to-left paragraph's line of several runs is read from its runs, because Avalonia's hit testing is wrong there on 11.3 and 12, docs/reference/avalonia-versions.md; a click beyond a line that ends in a hard break goes to the line's visual end in the paragraph's direction) | M | | Shift+click extends | V/VwRootBox.cpp:2274 `MouseDownExtended`; SimpleRootSite.cs:4545 | YES | YES (`RichTextField.OnPointerPressed`) | S | | Drag selects; auto-scroll when dragging above/below the view | V/VwRootBox.cpp:2253 `MouseMoveDrag`; SRS/SimpleRootSite.cs:5482-5540 (10 px per event, 30 px zone) | PARTIAL (TextBox) | YES: drag selects; past the visible edge the selection goes only just beyond it and the field raises `ScrollRequested` (10 px a move near the edge, 30 beyond, no timer); the host scrolls (`RichTextField.DragPoint`, ADR 0025) | M | | Double-click selects word; word growth `GrowToWord` and `ExpandToWord` | V/VwRootBox.cpp:2193 `MouseDblClk`; V/VwSelection.cpp:7066, 12366 | YES (TextBox) | YES (`RichTextField.OnPointerPressed`) | S | | Triple-click selects paragraph | NOT in legacy (no triple-click handler in V/VwRootBox.cpp, `kmeTriple` grep empty) | YES (TextBox: line) | YES (`RichTextField.OnPointerPressed`) -- extra | S | | Caret blink (`FlashInsertionPoint`, timer) | V/VwRootBox.cpp:1753; SRS/SimpleRootSite.cs:3702 `OnTimer` | YES (TextBox) | NO (static caret, `RichTextField.Render`) | S | -| Insertion point shape: bidi split caret (primary + secondary), I-beam cursor rotated for vertical text | V/lib/UniscribeSegment.cpp:1946 `DrawInsertionPoint`, 2001 `PositionsOfIP`; EditingHelper.cs:1460 `SetCursor`, SRS `IBeamCursor` | PARTIAL | PARTIAL: one caret, bidi caret stops (README) | M | +| Insertion point shape: bidi split caret (primary + secondary), I-beam cursor rotated for vertical text | V/lib/UniscribeSegment.cpp:1946 `DrawInsertionPoint`, 2001 `PositionsOfIP`; EditingHelper.cs:1460 `SetCursor`, SRS `IBeamCursor` | PARTIAL | PARTIAL: one caret, bidi caret stops (docs/reference/editing.md) | M | | Selection shading (range drawn, not in-segment inversion) incl. disjoint bidi ranges | V/lib/UniscribeSegment.cpp:2159 `DrawRange`, 2192 `PositionOfRange` | YES | YES (`ParagraphLayout.SelectionRects`, drawn by `RichTextField.Render`) | M | | Selection state when the view loses focus (disabled/out-of-focus look) | V/VwRootBox.cpp:2474 `get_SelectionState`, 2419 `Activate`; SRS/SimpleRootSite.cs:5693 `GetNonFocusedSelectionState`, 6340 `Activate` | YES (TextBox) | NO | S | | Scroll selection into view (options, one line of margin, warm-up of lazy boxes) | SRS/SimpleRootSite.cs:1612 `ScrollSelectionIntoView`, 3173-3510 `MakeSelectionVisible`; lazy boxes V/VwLazyBox.cpp | YES (ScrollViewer) | PARTIAL: the field raises `ScrollRequested` after caret moves, with a line of margin, and a scroll viewer round the field answers by default; no warm-up of lazy boxes and no scrolling inside the field (the host scrolls the view, ADR 0021) | L | @@ -186,7 +186,7 @@ Dispatch: key down -> `EditingHelper.OnKeyDown` (SRS/EditingHelper.cs:648) -> `C | Activate the keyboard of the ws at the selection when the selection changes | SRS/EditingHelper.cs:2877 `SetKeyboardForWs(ws)`, 2915, 2932-2965 `SetKeyboardForSelection`; ws from `SelectionHelper.GetFirstWsOfSelection` | YES: `WireWritingSystemKeyboard` FA/FwFieldControls.cs:715,880 (on focus) | PARTIAL: ActiveWs + `ActiveWsChanged` event (`DocumentEditor.ActiveWs`); host must switch keyboard | M | | On keyboard (input language) change, switch the ws of the IP or of the range (may re-tag the range) | SRS/SimpleRootSite.cs:115 `OnInputLanguageChanged`, 5588 `HandleKeyboardChange`, 5642 `GetPossibleWritingSystemsToSelectByInputLanguage`, 6767 `GetWSForInputMethod`; `m_wsPending` cleared by nav keys (EditingHelper.cs:1126) | NO | PARTIAL: `SetActiveWs` override (`DocumentEditor.SetActiveWs`) | M | | On focus lost switch back to the default (UI) keyboard unless focus moves to another view; 0.5 s guard for keyboard-switch focus flicker (LT-19289) | SRS/EditingHelper.cs:2966, 3007-3062 `LostFocus`, `ShouldRestoreKeyboardSwitchingTo`; 3063 `GotFocus` | NO | NO | S | -| Keyman: type-ahead disabled; keyboard activation per ws | SRS/EditingHelper.cs:843-870 (FWR-2205); libpalaso `Keyboard.Controller` | PARTIAL: "ordinary IME/Keyman composition" claim FA/DetailModel.cs:442 (UNVERIFIED) | UNVERIFIED (manual test 4 in docs/manual-testing.md; README: "real IME and Keyman interaction untested") | M | +| Keyman: type-ahead disabled; keyboard activation per ws | SRS/EditingHelper.cs:843-870 (FWR-2205); libpalaso `Keyboard.Controller` | PARTIAL: "ordinary IME/Keyman composition" claim FA/DetailModel.cs:442 (UNVERIFIED) | UNVERIFIED (manual test 4 in docs/manual-testing.md; docs/reference/limitations.md: "real IME and Keyman interaction are untested") | M | | IBus on Linux | SRS/IbusRootSiteEventHandler.cs (653 lines) | NO | NO | M (Linux only; may be out of scope) | | TSF composition (Windows): ITextStoreACP implementation: text, formatted text, selection, attrs, text extent, ACP from point, insert at selection, compositions start/update/end, `TerminateAllCompositions` | V/VwTextStore.cpp:207-2810 (e.g. GetText 698, SetText 938, GetFormattedText 994, GetTextExt 1228, InsertTextAtSelection 1373, OnSelectionChange 1669, OnStartComposition 2467, TerminateAllCompositions 2008) | N/A (Avalonia TextInputMethodClient via TextBox) | PARTIAL: `InputMethodClient` (nested in RichTextField.cs): preedit shown inline, not in model, surrounding text, selection | L | | Composition display attributes (underline kinds, colours per clause) | V/VwTextStore.cpp:2260-2420 `DoDisplayAttrs` (maps TF_LS_* to kunt*, falls back to swapped colours) | N/A | PARTIAL: preedit always underlined (`ParagraphLayout.MakeProperties`), in the character formatting the committed text will get (`DocumentEditor.TypingProperties`) | M | @@ -339,9 +339,9 @@ U+FFFC (ORC) carries `ktptObjData`: first char = kodt type, rest = payload (KRN: | Paragraph direction (ktptRightToLeft) and ws direction; base direction affects alignment (Leading/Trailing) | V/VwPropertyStore.cpp:1357; alignment keyed to direction | PARTIAL | PARTIAL: Rtl flag; alignment not applied | M | | Direction depth per segment | V/lib/UniscribeSegment.cpp:1335 `get_DirectionDepth`, 1353 `SetDirectionDepth` | N/A | N/A | S | | Logical vs physical arrows; F7/F8 | see group 3; SRS/SimpleRootSite.cs:2433 `ComplexKeyBehavior` | PARTIAL | PARTIAL (visual instead of logical) | M | -| Ctrl+arrow word moves in RTL (logical) | V/VwSelection.cpp:9492 | NO | PARTIAL (logical, README) | S | +| Ctrl+arrow word moves in RTL (logical) | V/VwSelection.cpp:9492 | NO | PARTIAL (logical, docs/reference/editing.md) | S | | Home/End in RTL | V/VwSelection.cpp:8777 `HomeKey(fLogical)`, `PhysicalHomeOrEnd` | PARTIAL | PARTIAL (logical only) | S | -| Bidi caret: two-position IP at direction boundaries, caret affinity | V/lib/UniscribeSegment.cpp:2001 `PositionsOfIP` | NO | YES (affinity, CurrentHit; README; stops, caret, mouse and selection of a right-to-left paragraph's multi-run line are read from its runs, `LineInfo.UsesRuns`) | M | +| Bidi caret: two-position IP at direction boundaries, caret affinity | V/lib/UniscribeSegment.cpp:2001 `PositionsOfIP` | NO | YES (affinity, CurrentHit; docs/reference/avalonia-versions.md; stops, caret, mouse and selection of a right-to-left paragraph's multi-run line are read from its runs, `LineInfo.UsesRuns`) | M | | Inline object inside RTL text positioned correctly | Uniscribe | N/A | PARTIAL: Avalonia fault, workaround `ObjectAsGlyph` | M | | Valid insertion points (no IP inside a cluster/ligature) | V/lib/UniscribeSegment.cpp:1508 `IsValidInsertionPoint`, 2477 `FindValidIPBackward`/`Forward` | YES | YES (shaper stops; grapheme boundaries on a right-to-left paragraph's line of several runs) | M | | Combining marks and NFC/NFD: layout in NFC, offsets mapped back | V/lib/NfcOffsetMap.h; UniscribeSegment.cpp:1605-1700 `OffsetInNfc`, `OffsetToOrig` | N/A | PARTIAL: no normalisation; offsets are UTF-16 of stored text | M | @@ -359,10 +359,10 @@ U+FFFC (ORC) carries `ktptObjData`: first char = kodt type, rest = payload (KRN: | MSAA root: name, role TEXT/PANE/GRAPHIC etc., value, state, parent/child navigation, hit test, location, select, focus | V/VwAccessRoot.cpp:122 `accNavigate`, 178-215 children, 267 `get_accName`, 280 `get_accRole`, 301 `get_accValue`, 337 `accLocation`, 363 `accHitTest`; boxes expose cells/rows/tables (:13-20) | PARTIAL: AutomationId/Name on box and field (FA/FwFieldControls.cs:~110-140), tests OwnedControlAutomationConventionTests | PARTIAL: Edit peer, name "Rich text field", value empty (`SetValue` no-op, IsReadOnly true) `RichTextField.FieldPeer` | M | | Accessible name settable by host | SRS/SimpleRootSite.cs:3724 `SetAccessibleName`; V/VwRootBox.cpp:5240 | YES (AutomationProperties.SetName) | NO (hard-coded name) | S | | Wrapper for WinForms (`AccessibleObject` bridging IAccessible) | SRS/AccessibilityWrapper.cs:66-264 | N/A | N/A | S | -| UIA TextPattern (caret, selection, ranges, attributes) | disabled in legacy (WM_GETOBJECT comment block: FWR-2874/3045) | via Avalonia TextBox: YES | NO (README "Known gaps"; empty UIA subtree inside WinFormsAvaloniaControlHost unexplained) | L | +| UIA TextPattern (caret, selection, ranges, attributes) | disabled in legacy (WM_GETOBJECT comment block: FWR-2874/3045) | via Avalonia TextBox: YES | NO (docs/reference/limitations.md; empty UIA subtree inside WinFormsAvaloniaControlHost unexplained) | L | | Screen reader reads caret line / word on move | via IAccessible text events UNVERIFIED | via TextBox | NO | L | | On-screen keyboard compatibility | FWR-2874 note | UNVERIFIED | UNVERIFIED | S | -| High-contrast / system colours for text and selection | V/VwColor.cpp, kclr* constants | YES (theme) | PARTIAL: the selection and caret brushes, and the text, border and page defaults (when the host has not set `Foreground` or `Background`), come from theme or system resources and are re-resolved on a theme change; in a Windows high contrast theme the text, page and selection are the system window text, window and highlight colours, and selected text and its underlines the highlight text colour (`SystemColours`; the selected part is drawn from a second layout of the paragraph in that one colour, `ParagraphLayout.InColour`), as FieldWorks uses the window text colour; a theme or system colour change redraws without a new layout (the default text colour is one brush changed in place); a host-set colour is kept, and `ResetForeground`/`ResetBackground` go back to the theme (README "Colours and themes") | S | +| High-contrast / system colours for text and selection | V/VwColor.cpp, kclr* constants | YES (theme) | PARTIAL: the selection and caret brushes, and the text, border and page defaults (when the host has not set `Foreground` or `Background`), come from theme or system resources and are re-resolved on a theme change; in a Windows high contrast theme the text, page and selection are the system window text, window and highlight colours, and selected text and its underlines the highlight text colour (`SystemColours`; the selected part is drawn from a second layout of the paragraph in that one colour, `ParagraphLayout.InColour`), as FieldWorks uses the window text colour; a theme or system colour change redraws without a new layout (the default text colour is one brush changed in place); a host-set colour is kept, and `ResetForeground`/`ResetBackground` go back to the theme (docs/reference/rendering.md, "Colours and themes") | S | | Keyboard-only operation of every command | Edit/Format menus | PARTIAL | PARTIAL | M | --- diff --git a/docs/manual-testing.md b/docs/manual-testing.md index fec56c5..5df83b7 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -167,7 +167,7 @@ only by the playground and the tests) and plain text. A FieldWorks host also cop | 9.3 | *Long document*: press Ctrl+End, type, press Ctrl+Home. | The view follows the caret; typing is not slower than in a short document; the inspector shows a window around the caret. | | | | | | 9.4 | *Many fields* with 1000 fields (Fields picker, then pick the scenario again): scroll and type in a field far down. | The window stays usable. Record how long it takes to load and whether scrolling stutters. | | | | | | 9.5 | Select all (Ctrl+A) in the *Long document* and press B. | It finishes in a reasonable time; Undo restores it in one step. | | | | | -| 9.6 | Switch Windows between light, dark and a high contrast theme (Settings, Accessibility, Contrast themes) while the playground is open, with the fields' Foreground and Background not set. | The text and page follow at once, without a pause: in high contrast the page is the system window colour, the text the window text colour and a selection the highlight colour. Record anything unreadable. A field whose host set both colours keeps them (the host's choice, README "Colours and themes"). | | | | | +| 9.6 | Switch Windows between light, dark and a high contrast theme (Settings, Accessibility, Contrast themes) while the playground is open, with the fields' Foreground and Background not set. | The text and page follow at once, without a pause: in high contrast the page is the system window colour, the text the window text colour and a selection the highlight colour. Record anything unreadable. A field whose host set both colours keeps them (the host's choice, docs/reference/rendering.md, "Colours and themes"). | | | | | ## 10. Mouse and keyboard selection (all hosts) diff --git a/src/SIL.TextEditing.Avalonia/RichTextField.cs b/src/SIL.TextEditing.Avalonia/RichTextField.cs index cbaa6ad..527902a 100644 --- a/src/SIL.TextEditing.Avalonia/RichTextField.cs +++ b/src/SIL.TextEditing.Avalonia/RichTextField.cs @@ -263,7 +263,7 @@ public IBlockRecognizer BlockRecognizer /// writing system at the caret, because the keyboard then in effect belongs to whatever had focus before and the /// writing system here may not have changed. While the field lacks focus it raises nothing, as FieldWorks' views do not /// set the keyboard without focus: a change made then (a host setting the caret in a background field, say) is reported - /// once, as the focus-gain event, with the writing system the field has by then. See the README, "Keyboards", for what + /// once, as the focus-gain event, with the writing system the field has by then. See docs/reference/editing.md, "Keyboards", for what /// a host does on focus in and out. /// /// On Windows, FieldWorks sets the keyboard on focus gain only when focus comes from a window of the same process, so diff --git a/tests/SIL.TextEditing.Avalonia.Tests/SurfaceBasicsTests.cs b/tests/SIL.TextEditing.Avalonia.Tests/SurfaceBasicsTests.cs index 611346a..83d94f1 100644 --- a/tests/SIL.TextEditing.Avalonia.Tests/SurfaceBasicsTests.cs +++ b/tests/SIL.TextEditing.Avalonia.Tests/SurfaceBasicsTests.cs @@ -186,7 +186,7 @@ public void Automation_peer_is_an_edit_with_a_name_and_the_text_value() => Headl Assert.Equal("Rich text field", peer.GetName()); var value = (global::Avalonia.Automation.Provider.IValueProvider)peer; Assert.StartsWith("Hello bonjour world", value.Value); - // Gap: no TextPattern provider yet (see README). + // Gap: no TextPattern provider yet (see docs/reference/limitations.md). }); } } diff --git a/tests/SIL.TextEditing.Avalonia.Tests/ThemeColourTests.cs b/tests/SIL.TextEditing.Avalonia.Tests/ThemeColourTests.cs index 15cc9ae..426794f 100644 --- a/tests/SIL.TextEditing.Avalonia.Tests/ThemeColourTests.cs +++ b/tests/SIL.TextEditing.Avalonia.Tests/ThemeColourTests.cs @@ -211,7 +211,7 @@ public void Resetting_the_foreground_or_the_background_goes_back_to_following_th [Fact] public void A_host_that_sets_only_the_foreground_keeps_the_themes_page() => Headless.Run(() => { - // The contract (README): a colour the host sets is kept, and the other follows the theme, so a black foreground on a dark theme is + // The contract (docs/reference/rendering.md): a colour the host sets is kept, and the other follows the theme, so a black foreground on a dark theme is // black on the dark page. A host that sets one sets the other, or resets it. var (window, field) = Themed(); window.RequestedThemeVariant = ThemeVariant.Dark; diff --git a/tests/SIL.TextEditing.Markdown.Tests/ScaleTests.cs b/tests/SIL.TextEditing.Markdown.Tests/ScaleTests.cs index e03e3a8..47dca4a 100644 --- a/tests/SIL.TextEditing.Markdown.Tests/ScaleTests.cs +++ b/tests/SIL.TextEditing.Markdown.Tests/ScaleTests.cs @@ -256,7 +256,7 @@ public void TwentyThousandRunsInOneParagraphWithoutMarksAreRead() } namespace SIL.TextEditing.Markdown.Tests { - /// The writer's own paths are linear: only Markdig's slow paths (a quadratic [a]( run, deep lists, a table read) are left, and the README says so. + /// The writer's own paths are linear: only Markdig's slow paths (a quadratic [a]( run, deep lists, a table read) are left, and docs/reference/markdown.md says so. /// Counted, not timed (issue 141): the work is read from , and four times the input must be about four times the work, not sixteen. public class WriterGrowthTests { From e30cd3d6ea77bdf1e4df4621f0c6651dec83de86 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:44:53 -0500 Subject: [PATCH 5/6] The reference lists every per-version #if and says how the LCModel adapter reports a dropped property, and the README's Avalonia claim matches. Co-Authored-By: Claude Opus 5.5 --- README.md | 4 ++-- docs/reference/avalonia-versions.md | 14 ++++++++++---- docs/reference/lcmodel.md | 2 +- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 3def032..9432191 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ Pandoc reads the same way the library does. The host owns the data and the undo - **A writing system on every run.** A `Run` holds text, a writing-system id, a property bag keyed by FieldWorks property numbers, and object data, as an `ITsString` run does. The editor carries properties it does not understand through every edit. The LCModel adapter round-trips real liblcm strings (ADR 0016, 111 tests per target). -- **Both Avalonia lines from one source tree.** Avalonia 11.3.17 on net48 inside WinForms, and Avalonia 12.1.2 on net10.0, with two small - per-version differences, [listed](docs/reference/avalonia-versions.md#per-version-differences) ([ADR 0009](docs/adr/0009-two-avalonia-lines.md)). The same surface tests run on both: 777 on +- **Both Avalonia lines from one source tree.** Avalonia 11.3.17 on net48 inside WinForms, and Avalonia 12.1.2 on net10.0, with a few small + per-version differences, [listed](docs/reference/avalonia-versions.md#per-version-differences) ([ADR 0009](docs/adr/0009-two-avalonia-lines.md)). The surface tests run on both: 777 on net48, 778 on net10.0. - **FieldWorks editing behaviour.** Arrow keys are logical by paragraph direction, text is stored as NFD, Backspace and Delete work as in FieldWorks ([ADR 0018](docs/adr/0018-fieldworks-editing-semantics.md)). Complex shaping, inline objects, and screen reader support diff --git a/docs/reference/avalonia-versions.md b/docs/reference/avalonia-versions.md index 008b781..2c054ae 100644 --- a/docs/reference/avalonia-versions.md +++ b/docs/reference/avalonia-versions.md @@ -11,19 +11,25 @@ The versions are set per target framework in `Directory.Packages.props`. ## Per-version differences -There is currently no `#if` in the source for the Avalonia API: +Most of the code needs no per-version branch: - The spike's one per-version spot (the clipboard API) is not needed, because Avalonia 11.3.17 already has the `DataTransfer` API that 12 uses. - `FontFeature` parsing is written without `FontFeatureCollection.Parse` (absent in 11.3) for both. -Keep any future per-version difference small, behind `#if NET10_0_OR_GREATER`, and list it here. The differences now: +Keep any future per-version difference small, behind `#if NET10_0_OR_GREATER`, and list it here. The differences now (every one is behind +`#if NET10_0_OR_GREATER`): -- **`SpellScanner`** asks `CharUnicodeInfo` for the category of a code point directly on net10.0 and through a one-character string on - netstandard2.0 (no overload there). That only costs an allocation for a letter outside the basic plane. +- **`SpellScanner`** (`Spelling.cs`) asks `CharUnicodeInfo` for the category of a code point directly on net10.0 and through a + one-character string on netstandard2.0 (no overload there). That only costs an allocation for a letter outside the basic plane. - **`ScriptPlacement`** reads a font's OS/2 and hhea tables with `IGlyphTypeface.TryGetTable` on netstandard2.0. Avalonia 12 gives no access to tables, so on net10.0 it reads them from the font file's own table directory: once for each font; in a font collection from the member whose family name is the typeface's; and nothing is read whole. +- **`ClusterFuser`** adds only the Indic conjunct rule (GB9c) to `StringInfo` on net10.0. On netstandard2.0 it also fuses Extend, + SpacingMark, emoji ZWJ sequences and regional-indicator flags itself (see [Graphemes](rendering.md#graphemes)). +- **`RichTextField`'s dispatcher.** Avalonia 12 lets a control live on a dispatcher other than the main UI thread's, so on net10.0 the + field uses its own `Dispatcher` (and the typing-pause timer is created on it); Avalonia 11 has only `Dispatcher.UIThread`. +- **`RichTextField.ContextMenu.cs`** has one extra `using Avalonia.Input` on net10.0. ## Packages used by the samples only diff --git a/docs/reference/lcmodel.md b/docs/reference/lcmodel.md index e06fe79..7c47720 100644 --- a/docs/reference/lcmodel.md +++ b/docs/reference/lcmodel.md @@ -11,7 +11,7 @@ host code, not Motif's GUI. property (with its variation) and every string text property by number, ObjData and NamedStyle included. - The host supplies the writing-system handle mapping (`IWritingSystemHandles`) and the paragraph keys. -- A property liblcm cannot load back is left out of the `ITsString` and reported, not thrown; the run model's bag keeps it. The callback, +- A property liblcm cannot load back is left out of the `ITsString`, and reported to an optional callback (with none, it is dropped silently); the run model's bag keeps it either way. The callback, the `DroppedProperty` shape and the finding ids are in [Findings: LCModel adapter](../formats/findings.md#lcmodel-adapter). These refusals retire once FieldWorks pins a liblcm with the fixes (issue #159). From 120965150c1db684d41791cd496bff72db3f8692 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 6 Nov 2026 06:56:33 -0500 Subject: [PATCH 6/6] The README, reference and AGENTS.md state the Pandoc, AOT, test-count, Copy as, screen-reader and CI=true facts as they are on main, and the feature inventory says per-run font features are read. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 6 +++--- README.md | 29 +++++++++++++++++------------ docs/feature-inventory.md | 2 +- docs/reference/clipboard.md | 5 +++-- docs/reference/markdown.md | 8 ++++---- docs/reference/projects.md | 2 +- 6 files changed, 29 insertions(+), 23 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 444fc5e..af605b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,10 +24,10 @@ known Avalonia fault says so in its name; when it starts failing, remove the workaround, not the test. - Avalonia tests that create controls run inside `Headless.Run`. - No wall-clock assertions. Assert counted work (counters such as `RecordWork.Steps`, `TextReads`). - - Pandoc tests skip silently when `pandoc` is not on `PATH`, unless `CI` (or `GITHUB_ACTIONS`) is set; then they fail. Set `CI` locally - to be sure they ran. + - Pandoc tests skip silently when `pandoc` is not on `PATH`, unless `CI=true` (or `GITHUB_ACTIONS`) is set; then they fail. Set `CI=true` locally + to be sure they ran (`CI=1` does nothing). - `tests/AotCheck` needs `vswhere` on `PATH` (issue #164), and `-p:NuGetAudit=false` locally while issue #104 is open. -- **Local liblcm.** Pack the liblcm stack into `H:\caches\localnugetpackages` with a stamped version, then +- **Local liblcm.** Pack the liblcm stack with a stamped version into a local NuGet source registered in your user NuGet.Config (e.g. a folder such as `H:\caches\localnugetpackages`), then `dotnet test -p:SilLcmVersion=` (see `Directory.Packages.props`). The refusal sweep tests are expected to fail against a fixed liblcm. - **Format docs.** diff --git a/README.md b/README.md index 9432191..96b9410 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ **Rich text editing for Avalonia in which every run of text knows its writing system.** -SIL.TextEditing is the rich text field for [FieldWorks](https://software.sil.org/fieldworks/) and Motif, and for any +SIL.TextEditing is being built as the rich text field for [FieldWorks](https://software.sil.org/fieldworks/) and Motif, and for any [Avalonia](https://avaloniaui.net/) app that edits multilingual text. Each run carries its own writing system, so Latin, Arabic and Devanagari share one paragraph and each keeps its own font, spelling dictionary and keyboard. The run model is shaped like FieldWorks' `ITsString`, and an adapter maps it to and from liblcm. One source tree builds against Avalonia 11.3 for FieldWorks (.NET Framework 4.8, @@ -33,23 +33,28 @@ Pandoc reads the same way the library does. The host owns the data and the undo per-version differences, [listed](docs/reference/avalonia-versions.md#per-version-differences) ([ADR 0009](docs/adr/0009-two-avalonia-lines.md)). The surface tests run on both: 777 on net48, 778 on net10.0. - **FieldWorks editing behaviour.** Arrow keys are logical by paragraph direction, text is stored as NFD, Backspace and Delete work as - in FieldWorks ([ADR 0018](docs/adr/0018-fieldworks-editing-semantics.md)). Complex shaping, inline objects, and screen reader support - at FieldWorks' level ([ADR 0017](docs/adr/0017-screen-reader-gate.md)). -- **FieldWorks Markdown that Pandoc agrees with.** Read lenient, write strict. 667 tests run Pandoc 3.9.0.2 itself over every fixture - and generated documents and require the same structure ([ADR 0028](docs/adr/0028-fieldworks-markdown-is-a-pandoc-profile.md)). + in FieldWorks ([ADR 0018](docs/adr/0018-fieldworks-editing-semantics.md)). Complex shaping and inline objects. Screen readers get FieldWorks' MSAA level (name, role, value) + through Avalonia's peers; in WinForms the host needs a small fix first ([ADR 0017](docs/adr/0017-screen-reader-gate.md)). +- **FieldWorks Markdown that Pandoc agrees with.** Read lenient, write strict. 664 tests run Pandoc 3.9.0.2 itself: 334 fixture + cases (every one without an error finding, bar a few excluded with a stated reason), 200 generated documents and record files and the 5 + link forms must read to the same structure; 124 more check that what Pandoc reads as another construct is reported; one pins the version + ([ADR 0028](docs/adr/0028-fieldworks-markdown-is-a-pandoc-profile.md)). - **Records as files.** A whole record as one file: a YAML frame typed by a schema that is data, FieldWorks Markdown in the rich values, and a JSON Schema for editors ([ADR 0032](docs/adr/0032-record-files-yaml-frame.md)). - **The host stays in charge.** The host owns data and undo ([ADR 0003](docs/adr/0003-host-owns-data-and-undo.md)), and supplies the dictionaries, keyboards, clipboard forms, language services and the blocks a field allows. -- **Ready for native AOT.** The net10.0 libraries other than the LCModel adapter are marked trimmable and AOT-compatible, and - `tests/AotCheck` publishes the Markdown library native AOT and runs it. +- **Ready for native AOT.** `SIL.TextEditing`, `SIL.TextEditing.Avalonia` and `SIL.TextEditing.Markdown` are trimmable and AOT-compatible on + net10.0 (the LCModel adapter is not). CI's `tests/AotCheck` publishes all three native AOT, rooted whole so any trim or AOT warning + fails, and runs the Markdown converter, record files and the libpalaso writing-system path in the published exe. -The whole suite passes with 0 failures on net48 and net10.0. Per target, as of 2026-11-06 (main 7ffd8f6 with PRs #165 and #166 merged, Release): +The whole suite passes with 0 failures on net48 and net10.0. Passing tests per target, Release, main 7e77466 (2026-11-06), Pandoc on PATH. +Opt-in tests that skip by default are not counted: 11 screenshot generators (`TEXTEDITING_SCREENSHOTS`) and 3 real-clipboard tests +(`TEXTEDITING_REAL_CLIPBOARD`). | Tests | Model | Avalonia | Markdown | LCModel | Playground | SecondDispatcher | |---|---|---|---|---|---|---| -| net48 | 416 | 777 | 3841 | 111 | 304 | — | -| net10.0 | 416 | 778 | 3841 | 111 | 304 | 3 | +| net48 | 416 | 777 | 3843 | 111 | 304 | — | +| net10.0 | 416 | 778 | 3843 | 111 | 304 | 3 | ## Quick start @@ -110,8 +115,8 @@ Planned, not built: ([ADR 0031](docs/adr/0031-shared-editing-and-interchange-layer.md), [ADR 0033](docs/adr/0033-block-features.md)). - **The `fwmd` command-line checker** ([#161](https://github.com/sillsdev/SIL.TextEditing/issues/161)). - **The FieldWorks Markdown clipboard form** ([#168](https://github.com/sillsdev/SIL.TextEditing/issues/168)). -- **Editor integration**: an inline WYSIWYG and Markdown source mode, and a VS Code extension with a language server. Proposed ADR 0034, - awaiting owner decisions. +- **Editor integration**: an inline WYSIWYG and Markdown source mode, and a VS Code extension with a language server. The plan and a proposed ADR + are awaiting owner decisions ([#170](https://github.com/sillsdev/SIL.TextEditing/issues/170)). - **Retiring the liblcm refusals** once FieldWorks pins a liblcm with its fixes [#410](https://github.com/sillsdev/liblcm/issues/410), [#413](https://github.com/sillsdev/liblcm/issues/413) and [#414](https://github.com/sillsdev/liblcm/issues/414) ([#159](https://github.com/sillsdev/SIL.TextEditing/issues/159)). ## Contributing diff --git a/docs/feature-inventory.md b/docs/feature-inventory.md index 2117f9b..111c5f5 100644 --- a/docs/feature-inventory.md +++ b/docs/feature-inventory.md @@ -52,7 +52,7 @@ Engine entry: `VwPropertyStore::put_IntProperty` (V/VwPropertyStore.cpp:1177, ca | ktptBackColor (9) | :1368; DRAW run background (UniscribeSegment DrawText, :1012-1014 comments) | NO | YES (`RunTextProperties.Back`, painted by `ParagraphLayout.DrawBackgrounds` below the selection; not under the spaces a wrapped line ends with, as FieldWorks' line-end segments leave them out of their width) | S | | ktptBaseWs (16) | :1403; CARRY mostly: `m_wsBase` seeds the font size for relative line height (:1470) | NO | NO (carried) | S | | ktptFontFamily string (1) incl. markup ``, ``, ``, `` | :1577; DRAW; markup resolved by V/../AppCore/FwStyledText.cpp:48-51,1467 | PARTIAL: FA/DetailModel run FontFamily; per-run font display FA/DetailRichTextChrome.cs:30-60 | PARTIAL: carried (ParagraphLayout uses the ws font only, :181) | M | -| ktptFontVariations string (130) OpenType/Graphite feature list, concatenated | :1595; DRAW features parsed V/lib/UniscribeSegment.cpp:241 (`TryParseFontFeatureRecords`), Graphite V/lib/GraphiteSegment.cpp | NO | PARTIAL: ws features only, per-run NOT read (docs/reference/rendering.md) | M | +| ktptFontVariations string (130) OpenType/Graphite feature list, concatenated | :1595; DRAW features parsed V/lib/UniscribeSegment.cpp:241 (`TryParseFontFeatureRecords`), Graphite V/lib/GraphiteSegment.cpp | NO | PARTIAL: OpenType features from the writing system and per run (property 130 in a run's or style's bag is read and appended to them, `StyleResolver`), drawn through Avalonia; no Graphite (ADR 0022) (docs/reference/rendering.md) | M | | ktptWsStyle string (156): ws-dependent character-style overrides, merged by `ComputeWsStyleInheritance` | :1580-1594, :2120-2160 (explains order); applied when the ws is known; CARRY in string, RESOLVE at draw | NO (lossy: read-only) | NO | L | | ktptNamedStyle string (133): paragraph or character style name; applied first, may bring ws styles, drop caps hack | :1609-1630; DRAW via the style sheet | PARTIAL: NamedStyle on run, apply FA/FwFieldControls.cs:342-429 | PARTIAL: carried as CharStyle/NamedStyle numbers, no stylesheet resolution (TE/M/PropertyBag.cs `TextPropertyNumbers`) | L | | ktptCharStyle (2), ktptParaStyle (3): old style carriers | ignored (:1642 default); CARRY | NO | PARTIAL (constants only) | S | diff --git a/docs/reference/clipboard.md b/docs/reference/clipboard.md index 601416d..3ce2d9f 100644 --- a/docs/reference/clipboard.md +++ b/docs/reference/clipboard.md @@ -19,8 +19,9 @@ The clipboard forms, the paste order, the writing-system rule, "Copy as" and the - **Writing systems the target lacks** take the writing system of the selection the text replaces, as FieldWorks does. The field asks `RichTextField.KnowsWritingSystem` (by default: its `WritingSystems` registry, and a registry with no entries accepts everything), and raises `WritingSystemsMapped` after the paste so the host can show a notice. -- **Copy as** (`RichTextField.CopyAsAsync(formId)`) puts one form on the clipboard as text: the plain text, or a form that offers it - (`IClipboardForm.OffersCopyAs`, such as FieldWorks Markdown). The playground's context menu has a "Copy as" submenu. +- **Copy as** (`RichTextField.CopyAsAsync(formId)`) puts one form on the clipboard as text: the plain text, or any registered form whose + payload is a string, such as FieldWorks Markdown (planned, #168). `IClipboardForm.OffersCopyAs` only decides which forms + `ClipboardForms.CopyAsForms` lists for a menu. The playground's context menu has a "Copy as" submenu. - `SIL.TextEditing.Runs.v1` is no longer on the clipboard. `Interchange` stays as a test and debug dump, and `InterchangeClipboardForm` lets tests and the playground carry formatting without liblcm or the Markdown library. diff --git a/docs/reference/markdown.md b/docs/reference/markdown.md index 7820828..3f2004d 100644 --- a/docs/reference/markdown.md +++ b/docs/reference/markdown.md @@ -102,10 +102,10 @@ System.Runtime.CompilerServices.Unsafe (4.0.4.1). A net48 host needs a binding r - **Conformance.** The fixtures in `tests/conformance` (JSON, documented in its README: Markdown in, expected paragraphs and findings out, and model to canonical Markdown) are run by `ConformanceTests`. They and the spec define the format. -- **Pandoc agreement.** `PandocAgreementTests` reads every fixture, its canonical form and 150 generated documents - (`TEXTEDITING_PANDOC_SEEDS`) with `pandoc -f markdown-smart-implicit_figures-implicit_header_references-task_lists -t json` and requires - the same paragraphs. Pandoc is pinned in the CI workflow. Where it is missing the tests skip, except in CI (`CI` or `GITHUB_ACTIONS` - set), where they fail. +- **Pandoc agreement.** `PandocAgreementTests` reads every fixture case without an error finding (bar a few excluded with a stated + reason), its canonical form and 150 generated documents (`TEXTEDITING_PANDOC_SEEDS`; the record tests add a third as many record files) with `pandoc -f markdown-smart-implicit_figures-implicit_header_references-task_lists -t json` and requires + the same paragraphs. Pandoc is pinned in the CI workflow. Where it is missing the tests skip, except in CI (`CI=true` or + `GITHUB_ACTIONS` set), where they fail. - **Random round trips.** 600 seeded random models and 600 random texts per run (`TEXTEDITING_MARKDOWN_SEEDS` raises the count) check the round trip and the stability of the canonical form. - Also: tests of the findings' positions, normalisation, size limits and the host interfaces. diff --git a/docs/reference/projects.md b/docs/reference/projects.md index 84a4090..a0ef2c7 100644 --- a/docs/reference/projects.md +++ b/docs/reference/projects.md @@ -55,7 +55,7 @@ properties, an environment box with a language service, 200 paragraphs, many fie | `tests/SIL.TextEditing.LCModel.Tests` | net48, net10.0 | The adapter ([details](lcmodel.md#tests)). | | `tests/SIL.TextEditing.Markdown.Tests` | net48, net10.0 | Conformance, Pandoc agreement, random round trips ([details](markdown.md#tests)). | | `tests/SIL.TextEditing.Playground.Tests` | net48, net10.0 | Avalonia.Headless integration tests that drive the playground the way a person would. | -| `tests/AotCheck` | net10.0 | Publishes the Markdown library native AOT and runs it ([details](markdown.md#dependencies-trimming-and-aot)). | +| `tests/AotCheck` | net10.0 | Publishes the three net10.0 libraries native AOT and runs the Markdown, record and writing-system checks ([details](markdown.md#dependencies-trimming-and-aot)). | ### SecondDispatcher