Skip to content

fix(tui): preserve graphemes and terminal display columns - #14919

Merged
TheHypnoo merged 3 commits into
PerryTS:mainfrom
wsafight:fix/tui-unicode-cell-width
Nov 20, 2026
Merged

TheHypnoo merged 3 commits into
PerryTS:mainfrom
wsafight:fix/tui-unicode-cell-width

Conversation

@wsafight

@wsafight wsafight commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Measure and paint perry/tui text in terminal columns while retaining complete extended grapheme clusters. This fixes CJK/emoji layout, combining-mark updates, right-edge clipping, and stale columns after wide-to-narrow replacements.

Changes

  • Use the existing unicode-segmentation dependency and the already-locked unicode-width 0.2.2 for one consistent non-CJK width policy in layout and painting.
  • Store complete multi-scalar graphemes in Rust-owned shared text, reserve explicit continuation cells, and clear an entire overlapping span before painting over it.
  • Emit grapheme leaders only; track cursor advancement in display columns and erase obsolete trailing cells. Update both TUI guide pages and add unit regressions beside the implementation.
  • Overwriting part of a wide grapheme leaves blanks that keep its colors and style, and repainting an unchanged multi-scalar grapheme shares the previous frame's text instead of allocating.
  • TypeScript APIs are unchanged. The Rust implementation's Cell becomes Clone rather than Copy and carries grapheme/width metadata; the former 16-byte-cell claim no longer applies.

Scope: no multiline/wrapping implementation, configurable ambiguous-width policy, cross-widget grapheme joining, or widget-lifetime changes. Leading zero-width graphemes without a base are omitted. Unicode/width-policy differences between terminal emulators remain possible and are documented.

Related issue

n/a. Checked all open upstream PRs and searched TUI/Unicode/grapheme/width topics before starting; none overlaps this fix. Widget ownership remains the separate #11919 workstream.

Test plan

  • Red/green: six initial regressions fail against the unchanged implementation (42 passed, 6 failed, 1 ignored in a harness importing actual TUI modules); after the fix, the expanded module harness has 56 passed, 0 failed, 1 ignored benchmark.
  • CARGO_HOME=<task-cache> CARGO_TARGET_DIR=<task-target> cargo check --locked --offline --profile perry-dev -p perry-runtime --tests — passes on macOS arm64, pinned nightly-2026-10-04, with pre-existing warnings.
  • Real runtime TUI unit-test binary: 98 passed, 0 failed, 2 ignored, plus three repeat runs. Includes input bytes/GC custody, Unix reader PTY lifecycle, async pumping, and new Unicode regressions. The ignored cases are a manual benchmark and a subprocess helper explicitly exercised by its parent.
  • External terminal-state replay harness importing the changed real modules: 42,632 Unicode/ASCII/clipping frame transitions across terminal widths 1–8 match a fresh full-frame render.
  • Changed-source rustfmt --check, git diff --cached --check, python3 scripts/check_test_registration.py, and python3 scripts/tokio_inventory.py --offline pass.
  • python3 scripts/lock_no_downgrade.py --vs origin/main — no resolved package/version changes, only the new runtime dependency edge to an already-locked crate.
  • RUST_TEST_THREADS=1 cargo test --profile perry-dev -p perry-runtime --lib tui:: -- --test-threads=1 on macOS arm64 after rebasing onto current main (no shim needed since fix(runtime): declare the ELF image symbols on Linux only, so perry-runtime's lib tests link on macOS #14931): 101 passed, 0 failed, 1 ignored.
  • CARGO_NET_OFFLINE=true SKIP_COMPILE_GATES=1 BASE_SHA=origin/main bash scripts/run_lint_gates.sh — 17/142 fail, 2 CI-only arms skipped, compile tier explicitly skipped. A clean detached clone at base 1bf3c0e8 runs the same runner with BASE_SHA=HEAD and reports the same 17 failing commands, with no new failures. Existing failures cover release-pipeline fixtures, rustup/CARGO_HOME and missing cargo-xwin, unrelated formatting/file-size drift, address/string/GC inventories, WASM ABI currency, and debt/global-sink ledgers. No full-lint pass is claimed.
  • The early upstream e2e-scoped failure reproduces byte-for-byte on pristine 1bf3c0e8: python3 scripts/ci_e2e_scope.py --self-test expects the snap_image_literal_order suite twice. This PR changes neither the scope script nor that suite.
  • Upstream Linux cargo-test executes all TUI unit tests: 98 passed, 0 failed, 2 ignored. The job subsequently fails in the CLI build-cache audit, not a TUI test. See the PR job.
  • Rebased onto current main; CI is re-running. scripts/check_file_size.sh fails locally on files this PR does not touch (same on main).
  • Gap/GC triage is not a full clean bill of health. Of 74 gap regressions collected so far, 67 have exactly matching case status and failure detail on f54495f1 CI (runs 44091250261 and 44097228628). Seven remain unclassified because main's third shard receives a runner shutdown before reaching them: test_gap_v8_heap_snapshot_4916, test_gap_stack_runtime_frames, test_gap_typed_array_number_key_plain_site, test_gap_dyncode_direct_eval_patches_methods.cjs, test_gap_native_namespace_enumeration_nskeys.cjs, test_gap_net_constructor_class_identity, and test_gap_native_payload_zlib_one_shots. Both failing GC matrix subjects (repsel_p4a3_numarray_barriers and gc_namespace_and_computed_dispatch_rooting) also fail in the exact main-parent rerun. No snapshot or debt baseline is raised here to hide these failures.
  • Windows/WASM execution and a full static-wrapper/native-program build are not locally verified. The initial upstream run has completed with execution failures, including wasi-components stopping at the stale Rust-host fixture-path assertion. A passing build is not a passing execution test.

CI repair follow-ups

Confirmed CI defects found during this review are being fixed independently: #14928 removes the duplicate e2e suite registration (the upstream e2e-scoped job now passes on that PR); #14929 adds the two missing compiler cache inputs (all 53 actual CLI cache tests pass locally); #14930 upgrades ZIP to the patched 9.0.5 release (official RustSec audit exits 0 and all 22 updater tests pass locally); #14932 fixes WASI guest-host fixture preparation (the new regression fails before the fix and the eight-test qualification suite passes afterward). These PRs do not establish that the remaining gap/GC/inventory failures are resolved.

Upstream verification: cargo-test is green on #14929, and security-audit / security-audit is green on #14930. Together with #14928's green e2e-scoped result, these are observed CI outcomes, not just local predictions.

Checklist

  • No workspace version bump, no CHANGELOG.md edits, no raised debt baselines.
  • One logical runtime fix, implementation-level tests and user-visible documentation.
  • Added changelog.d/14919-tui-unicode-columns.md using the assigned PR number.

Summary by CodeRabbit

  • Bug Fixes
    • Terminal text now renders Unicode graphemes correctly, including CJK characters, emoji, flags, and combining marks. Wide characters occupy the appropriate number of columns, and text that cannot fit at the right edge is clipped cleanly.
    • Updating or moving wide text now clears leftover columns, while preserving the existing cell styling.
  • Documentation
    • Updated the terminal UI guides to explain display-width behavior, clipping, and handling of zero-width graphemes.

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The TUI now measures text and paints graphemes using terminal-column widths. The renderer emits changed graphemes and handles continuation cells. Tests and documentation cover Unicode widths, clipping, overwrites, and buffer updates.

Changes: Unicode TUI rendering

Layer / File(s) Summary
Measure text in terminal columns
crates/perry-runtime/Cargo.toml, crates/perry-runtime/src/tui/layout.rs, docs/src/tui/widgets.md
Text-node widths and horizontal sibling positions use grapheme display widths. The widget documentation describes the width policy and clipping behavior.
Paint graphemes into grid cells
crates/perry-runtime/src/tui/cell.rs, crates/perry-runtime/src/tui/mod.rs, docs/src/tui/overview.md
Cells store grapheme text and width. Grid painting handles continuation cells, clipping, and clearing overwritten graphemes.
Render changed graphemes
crates/perry-runtime/src/tui/render.rs, changelog.d/14919-tui-unicode-columns.md
The renderer skips continuation cells, emits grapheme text, and advances by cell width. Tests cover Unicode changes and buffer updates; the changelog records the change.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Grid
  participant RendererDiff
  participant Terminal
  participant FrontBuffer
  Grid->>RendererDiff: Provide changed back-buffer cells
  RendererDiff->>Terminal: Emit grapheme text and cursor movement
  RendererDiff->>FrontBuffer: Copy back buffer during flush
Loading

Possibly related PRs

Suggested reviewers: raliclo

Merge Risk: Minimal · up to 51e35

This change makes perry/tui measure and paint text by terminal display columns while keeping complete grapheme clusters. I found no concrete merge-blocking defect. The CI failures shown are not attributable to this PR.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "fix(tui): preserve graphemes and terminal display columns" matches the main change. The changeset stores complete grapheme clusters in the TUI cell grid, measures text in terminal display c…
Description check Passed The description follows the repository template. It has a Summary, Changes, Related issue ("n/a" with context), Test plan, and Checklist. It leaves the Type of change, Platforms, and Screenshots secti…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "fix(tui): preserve graphemes and terminal display columns" matches the main change. The changeset stores complete grapheme clusters in the TUI cell grid, measures text in terminal display columns, and clears overlapping spans. The title is specific and follows the fix: prefix convention.

Full details: Description check

Explanation

The description follows the repository template. It has a Summary, Changes, Related issue ("n/a" with context), Test plan, and Checklist. It leaves the Type of change, Platforms, and Screenshots sections unticked or unfilled. The Test plan lists many commands and results, and it states that the full lint gate did not pass. Some template items are incomplete: the platform checkboxes are not ticked, the "Tested on" line is blank, and the Type of change list is not marked. The required content is present, so the check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 68885906-64e1-4bf5-85b5-ce1bd2f1e002
Commits

Reviewing files that changed from the base of the PR and between d18cea1 and 51e35a5.

Files ignored due to path filters: 1

  • Cargo.lock is excluded by !**/*.lock
Files selected for processing: 3

  • crates/perry-runtime/Cargo.toml
  • crates/perry-runtime/src/tui/cell.rs
  • crates/perry-runtime/src/tui/render.rs

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

wsafight added a commit to wsafight/perry that referenced this pull request Nov 17, 2026
@wsafight
wsafight marked this pull request as ready for review November 17, 2026 14:09
@wsafight
wsafight marked this pull request as draft November 17, 2026 14:47
@wsafight

Copy link
Copy Markdown
Contributor Author

CI triage — draft, not merge-ready

I converted this PR back to draft while distinguishing inherited failures from unresolved execution results. Local focused tests passing did not establish an all-green CI result.

The tested merge commit is b50f35b7a34b22bdb028b01d2eeaa26dd3bcdb4a; its main parent is f54495f13126658e8a7d0b9410aee1f77f196272. Comparisons below use that exact main parent, not just an older local checkout.

Check Evidence
TUI tests The Linux PR cargo-test log contains 98 passing TUI tests, 2 ignored, no TUI failures.
Overall cargo-test Both the PR and exact-parent main fail only commands::compile::compile_env::tests::codegen_env_vars_are_build_cache_inputs in the CLI binary: 1627 passed, 1 failed, 2 ignored. This is distinct from the passing TUI tests.
lint The exact-parent main job has the same 12 failing step names as the PR, including release fixtures, Windows typecheck, size/inventory and debt/custody gates. The earlier local clean-base comparison also had the same 17 failing script commands. Neither observation is a full-lint pass.
warnings Exact-parent main reports the same runtime unused import/functions in object/layout_probe.rs, object/overflow.rs, and snap_image/format.rs. The PR also reports unused items in unchanged wpfacts tests. Existing fixes include #14871.
security-audit Exact-parent main already reports zip 9.0.4, RUSTSEC-2026-0548, patched in >=9.0.5. This PR does not change any resolved package version.
WASI execution Exact-parent main and this PR both pass 9/10 probes and fail test_gap_array_methods: Array.fromAsync ends with TypeError: value is not a function. Both PR preview-1 and preview-2 execution lanes report that failure.
e2e-scoped Scope self-test fails before programs execute; it expects snap_image_literal_order twice. The self-test output is byte-identical on this branch and the pristine local 1bf3c0e8 base. Those scope sources are unchanged.
GC matrix The PR fails repsel_p4a3_numarray_barriers and gc_namespace_and_computed_dispatch_rooting; the exact-parent main rerun also fails those subjects (shard 4, shard 2).

Remaining uncertainty

Of 74 gap regressions collected so far, 67 have exactly matching case status and printed failure detail on main-parent CI runs 44091250261 / 44097228628. Seven do not have a completed same-base observation:

  • node-suite/v8/test_gap_v8_heap_snapshot_4916
  • language/error/test_gap_stack_runtime_frames
  • language/typed-array/test_gap_typed_array_number_key_plain_site
  • language/scope/test_gap_dyncode_direct_eval_patches_methods.cjs
  • node-suite/module/test_gap_native_namespace_enumeration_nskeys.cjs
  • node-suite/net/test_gap_net_constructor_class_identity
  • node-suite/zlib/test_gap_native_payload_zlib_one_shots

Main's third gap shard is stopped by a runner shutdown before reaching these cases, so absence from that log does not prove they pass or fail on main. The PR's sixth gap shard also receives a shutdown signal (exit 143), and the WASI component execution job is still pending at this audit. I am not labelling all red checks as inherited or declaring this PR safe to merge from these partial observations. No unrelated source fixes, snapshot acceptance, baseline increases, or gate suppression are included.

@wsafight

Copy link
Copy Markdown
Contributor Author

CI repair work is now submitted as separate changes, rather than stopping at identifying inherited failures:

Fix Validation
#14928 — remove duplicate e2e suite registration Scope self-test passes; the upstream e2e-scoped job is green.
#14929 — key compilation caches on PERRY_NS_PAIRS and PERRY_FACTS_NATIVE_EXTERN Actual CLI test binary: 53 cache tests passed, without a runtime symbol shim.
#14930 — require and lock patched ZIP 9.0.5 Same official RustSec DB: audit changes from one blocking ZIP advisory to exit 0; 22 updater tests pass. No advisory exemption is added.
#14932 — correct copied WASI Rust-host fixture dependency Reproduces the exact Rust host path changed: crates/perry-wasi CI assertion before the fix; 8 qualification tests and 19 release-tooling tests pass after it. Full component execution is not claimed.

Existing independent repairs such as #14871 (warnings), #14874 (release fixture), #14875 (Windows SDK), and #14878 (WASM ABI) are not duplicated here. The remaining gap/GC/debt and inventory results still need their own resolution; this PR remains draft and is not declared merge-ready.

@TheHypnoo TheHypnoo added area:stdlib Node/Perry builtin modules and APIs effort:M 1-2 days: several files or investigation impact:med A feature, module, package or platform subset type:bug Confirmed defect or regression labels Nov 18, 2026

@raliclo raliclo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this on macOS 27.3 arm64 (pinned nightly-2026-10-04), merged onto current main. Notes below; I didn't find a correctness problem in the grid/diff changes.

1. The stock TUI command now runs on macOS without a shim.
Your test plan marks RUST_TEST_THREADS=1 cargo test --locked --profile perry-dev -p perry-runtime --lib tui:: -- --test-threads=1 as blocked on macOS by the ELF snapshot symbols. Since #14931 declared those symbols on Linux only, it links and runs as-is:

build result
main 90e338d 83 passed, 0 failed, 2 ignored
main + this PR (d18cea1), merge 979fd2b1bf 98 passed, 0 failed, 2 ignored

The 15 extra tests are this PR's own; the 2 ignored are reader_lifecycle_child and registry_scaling_benchmark, the same on both sides.

2. It merges cleanly onto current main, and your CI fixes have landed.
#14928, #14929, #14930 and #14932 are all merged now, and this branch is 187 commits behind. A rebase should drop several of the inherited reds you triaged, which would make the remaining CI picture much easier to read.

3. A frame-cost number would help with the Cell change.
Cell goes from a 16-byte Copy value to roughly twice that, with Option<Arc<str>>, and flush moves from copy_from_slice to clone_from_slice. I'd expect that to be negligible at terminal sizes, but the only benchmark in the module (registry_scaling_benchmark) covers the tree registry, not diff/flush. A before/after timing for a full-screen repaint and an unchanged frame (say 200×80) would settle it.

4. FYI, no action needed: control characters keep their width.
I wondered whether the width == 0 → skip path would drop control characters. With unicode-width 0.2.2, \t, \n, \r, ESC, NUL, BEL and DEL each measure 1, as before. The only difference from the old per-char loop is that \r\n is one grapheme, so it takes 1 column instead of 2. Only true zero-width text, like ZWSP or a soft hyphen, is now skipped.

@TheHypnoo
TheHypnoo force-pushed the fix/tui-unicode-cell-width branch from d18cea1 to 51e35a5 Compare November 20, 2026 08:44
@TheHypnoo
TheHypnoo marked this pull request as ready for review November 20, 2026 08:46
@TheHypnoo
TheHypnoo merged commit 2562d4b into PerryTS:main Nov 20, 2026
138 of 163 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:stdlib Node/Perry builtin modules and APIs effort:M 1-2 days: several files or investigation impact:med A feature, module, package or platform subset type:bug Confirmed defect or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants