Skip to content

feat(ts)!: translate String as rs.String, holding std's exact capacity - #574

Merged
kixelated merged 22 commits into
mainfrom
quest/m3/ts-string-capacity
Nov 7, 2026
Merged

kixelated merged 22 commits into
mainfrom
quest/m3/ts-string-capacity

Conversation

@kixelated

@kixelated kixelated commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A translated String was a JS string, which holds no capacity. So String::capacity, and everything that takes it (into_bytes, Vec::from, Bytes::from), either guessed or was refused at translation, wherever a String's bytes might reach a read of a Bytes' capacity. The quest (ts-string-capacity) makes String a runtime type, as moq-dev/rust2#555 did for Vec.

Decision

Earlier revisions of this PR prototyped and measured two representations. The user settled it on 2026-11-06:

Which representation should rs.String take?

  1. ✅ js: a JS string, its UTF-8 length, and std's exact capacity. &str and Box<str> stay plain JS strings.
  2. utf8: an rs.ByteVec, with &str an rs.ByteSlice.
  3. Measure a hybrid first: utf8 plus a lazily cached JS string.

Why: js keeps engine string hashing and native interop (a hand-written caller still passes "name" for a &str). utf8 won on encoding (3-14x over js in the prototype), but it was 5.6-9.8x slower than main on String-keyed maps, which moq-net uses to find broadcasts, and it cost +3.1% gzip. The prototype numbers are in this PR's edit history.

This narrows the quest's Goal, which named Box<str> and &str as runtime types too. They hold no capacity, so they stay JS strings.

Approach

  • rs.String (ts/runtime/rs/str.ts) is { v, len, cap }: the text, its length in UTF-8 bytes, and std's capacity. It is an object, so a &mut String is the object, and edits (push_str, insert, truncate, retain, replace_range, ...) change it in place, checking byte offsets as std does.

  • Capacity follows std's. Every growth step goes through RawVec's amortized growth of a Vec<u8>: reserve of what is added, or push of each char where std pushes each. with_capacity, reserve, reserve_exact, shrink_to, shrink_to_fit, and their try_ forms are std's. What makes a new String gives it std's room:

    • exactly its bytes from to_owned, clone, join, and to_string of text, a char, a bool, or an unsigned integer;
    • a signed integer's most digits, plus its sign;
    • format!'s estimated_capacity;
    • std's own reserves and pushes for replace, to_lowercase, from_utf16, from_utf8_lossy, and the rest.
  • Bytes.

    • into_bytes and Vec::<u8>::from(String) encode into storage of the String's capacity.
    • String::from_utf8 and from_utf8_unchecked keep the vector's room.
    • Bytes::from(String) takes those bytes. So the translator's refusal of Bytes capacity reads (BytesMut::from(Bytes), Vec::from(Bytes), Bytes::is_unique, try_into_mut, truncate, clear) in a program that makes bytes of a String is gone, along with the crates fixture case and the reject fixture's buffers::capacity_reads and string_bytes that proved it. The stock fixture's buf::strings covers those reads against native.
  • Maps. A HashMap/HashSet keyed by String or Cow<str> files each key under its text, as a JS map key the engine hashes (rs.HASH_TEXT). A &str lookup finds it the same way. Before this, each key's text went through the runtime's hasher into a chain, at 2.3-4.0x main.

  • Cow<str>. An owned one is an rs.String. +, into_owned, clone, and clone_from follow std's capacity.

  • Panics. A hook's String payload is the String that panic_any was given, or one panic! writes its message into from empty, as std's is.

  • Boundary. A hand-written caller passes rs.stringFrom(text) for a String, with room for exactly its bytes, and reads .v for its text. &str stays a JS string.

  • Bench. The codec bench gains str/ workloads shaped on moq-net's string paths. Each runs over an ASCII path (demo/room-42/alice/camera) and one with 2-, 3-, and 4-byte chars:

    • str/<path> encode and decode: 256 size-prefixed strings through moq-net's Encoder::string and Decoder::string;
    • str/join/<path>: Path::concat and Path::strip_prefix;
    • str/keyed/<path>: 256 format!ed keys of a HashMap<String, _>, each looked up four times by &str.

    The prototype scripts under ts/bench/prototype/ are deleted.

Commits since the decision:

  • af5887420: the translation.
  • 22a87cf80: String::from(&String) clones, and the hook payload.
  • 52a202249: a dyn shim's &mut receiver, after moq-dev/rust2#593.
  • e6f549267: rs.HASH_TEXT.
  • 06272388a: str::clone_into.
  • 1d168139d: a &mut dyn BorrowMut<str> of a String lends the rs.String, not its text, after the online review below.
  • Merges of main. The latest (7dd0c5718) moves this PR's codec bench ops from 18-21 to 19-22, since main's bytes/frames (moq-dev/rust2#595) took 18.

Measured

Translated ÷ main's translator on the same bench source, one just ts bench run (seed 2495450320). Host: Ryzen 9 5900XT, shared, load about 14 on 32 threads. Measured at e6f549267; later commits don't touch these paths.

workload Node Bun Chrome Firefox
fmt/lines 1.25 1.22 0.90 1.16
str/ascii encode 1.00 1.05 1.01 0.62
str/utf8 encode 1.06 0.85 0.98 1.07
str/ascii decode 0.35 1.16 0.14 0.97
str/utf8 decode 0.46 0.97 0.22 0.70
str/join/ascii 0.40 0.86 0.09 0.71
str/join/utf8 0.32 1.08 0.08 1.15
str/keyed/ascii 1.97 1.43 2.32 2.02
str/keyed/utf8 2.79 1.47 2.63 2.65
every other workload, geomean 1.03 1.02 1.02 1.07
A/A ÷ main, geomean (noise) 0.98 1.04 0.96 1.09

Before rs.HASH_TEXT, str/keyed was 3.74/3.18 (Node), 3.46/2.93 (Bun), 4.01/3.08 (Chrome), and 2.28/2.37 (Firefox).

Most of keyed's remaining gap is format!, not the map. In Node, per key:

step this PR main
format!("{name}/{i}") 181 ns 74 ns
clone + insert 108 ns 77 ns
four gets 96 ns 79 ns

Each write into a String scans the pushed text for its UTF-8 length and grows the capacity as std does. That same cost shows in fmt/lines.

decode and join gain mostly because len is now a field. Main's rs.stringLen and rs.stringAsBytes called TextEncoder.encode on every use, and push_str concatenated.

Bundle, minified by Bun as the bench ships it (codec fixture): main 83,790 B raw / 26,444 B gzip; this PR 84,602 / 26,761 (+1.0% / +1.2%).

Impact

Consumer: moq. moq-net's paths, Encoder::string, Decoder::string, Path::concat, Path::strip_prefix, and keying by path, translate with std's capacity. The str/ workloads time them.

Public API (breaking for hand-written callers):

  • A String in a translated signature is rs.String, not string. Callers convert with rs.stringFrom(text) and read .v.
  • &str and Box<str> are unchanged.
  • rs.StrCow's owned form is rs.String.
  • rs.fmtToString and rs.fmtFormat return rs.String.
  • New runtime exports: rs.String, rs.stringFrom, and rs.HASH_TEXT.
  • rs.Hashing's hash may return a string.
  • Fewer translation refusals: the Bytes capacity reads listed above.

Quests:

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort
  • Session: 01a1b390-07e9-7402-84ca-ac570955000f, round 5
  • Head: 872f46f
  • Base: origin/main at 324a7b7
  • Score: 10/10, No in-scope defects found; the merge preserves String handling, and 32 focused cases covering generic fields, trait objects, and String stacks match native Rust.
  • Findings: none

Validation

  • just ci passes at 872f46f83.
  • A one-round bun bench/main.ts --rounds 1 --sample-ms 1 in the bench shell matched native on every route of the renumbered str/ and bytes/frames workloads. Its timings are not measurements.

Models

  • Size: L
  • Implementer: Claude Code, claude-opus-5-5, unknown effort (earlier sessions)
  • Implementer: Claude Code, claude-opus-5-5, high effort

Follow-ups

  • std's panic messages hold std's capacity: already planned, now unblocked.
  • Cheaper String writes in fmt. Each write scans its text's UTF-8 length; a literal's length could be emitted at translation. fmt/lines runs at 0.90-1.25x main, and format! of a key at 2.4x in Node.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 3 commits November 4, 2026 02:55
Ops 17 to 20 of the codec fixture's bench code a broadcast path as
moq-net's Encoder and Decoder do, join and strip one as Path::concat and
strip_prefix do, and key a HashMap by String, each over an ASCII path and
one with multi-byte chars. str-routes.py rewrites a translation into two
models of a String for just ts bench --route: a JS string with its UTF-8
length and capacity, and UTF-8 bytes in an rt.ByteVec. Prototype only,
for the decision in the PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pacity

# Conflicts:
#	ts/fixtures/codec/expected/bench.ts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title rs.String feat(ts): rs.String, measured over a JS string and over UTF-8 bytes Nov 4, 2026
kixelated and others added 8 commits November 4, 2026 09:54
…pacity

# Conflicts:
#	ts/bench/main.ts
#	ts/bench/moq.ts
#	ts/bench/workloads.ts
#	ts/fixtures/codec/expected/bench.ts
#	ts/fixtures/codec/src/bench.rs
A String is a JS string beside its UTF-8 length and capacity, edited in
place, so capacity reads, into_bytes, Vec::from, and Bytes::from give
std's. &str and Box<str> stay JS strings. Lifts the Bytes capacity
refusal and drops the utf8 prototype.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
String::from of a reference returned the source object, so mutating the
copy changed it. A panic hook's String payload is now the String panic_any
was given, or one panic! writes its message into from empty, as std's is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pacity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shim calling std's mapping, which takes what a `&mut` points at, passes
that, as a call's operands do. #593's forwarding through `&mut` impls
relies on it, so `&mut &mut String` as a `dyn Write` passed its place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pacity

# Conflicts:
#	quest/m3/ts-slice-object.md
#	ts/README.md
#	ts/fixtures/codec/expected/bench.ts
#	ts/fixtures/codec/expected/harness.ts
#	ts/fixtures/lang/arrays/expected/arrays.ts
#	ts/fixtures/lang/ascii/expected/ascii.ts
#	ts/fixtures/lang/borrows/expected/borrows.ts
#	ts/fixtures/lang/boxes/expected/boxes.ts
#	ts/fixtures/lang/capacity/expected/capacity.ts
#	ts/fixtures/lang/cells/expected/cells.ts
#	ts/fixtures/lang/chars/expected/chars.ts
#	ts/fixtures/lang/chunks/expected/chunks.ts
#	ts/fixtures/lang/clones/expected/clones.ts
#	ts/fixtures/lang/collections/expected/collections.ts
#	ts/fixtures/lang/conversions/expected/conversions.ts
#	ts/fixtures/lang/cows/expected/cows.ts
#	ts/fixtures/lang/defaults/expected/defaults.ts
#	ts/fixtures/lang/durations/expected/durations.ts
#	ts/fixtures/lang/dyns/expected/dyns.Tri.ts
#	ts/fixtures/lang/dyns/expected/dyns.ts
#	ts/fixtures/lang/dyns/expected/stds.ts
#	ts/fixtures/lang/edits/expected/edits.ts
#	ts/fixtures/lang/errors/expected/errors.ts
#	ts/fixtures/lang/fields/expected/fields.ts
#	ts/fixtures/lang/floats/expected/floats.ts
#	ts/fixtures/lang/folds/expected/folds.ts
#	ts/fixtures/lang/futures/expected/futures.ts
#	ts/fixtures/lang/hashes/expected/hashes.ts
#	ts/fixtures/lang/heap/expected/heap.ts
#	ts/fixtures/lang/ints/expected/ints.ts
#	ts/fixtures/lang/iters/expected/iters.ts
#	ts/fixtures/lang/lists/expected/lists.ts
#	ts/fixtures/lang/locks/expected/locks.ts
#	ts/fixtures/lang/nested/expected/nested.ts
#	ts/fixtures/lang/orders/expected/orders.ts
#	ts/fixtures/lang/panics/expected/located.ts
#	ts/fixtures/lang/panics/expected/panics.ts
#	ts/fixtures/lang/panics/hook/expected/hooked.Shown.ts
#	ts/fixtures/lang/panics/hook/expected/hooked.ts
#	ts/fixtures/lang/panics/hook/expected/located.ts
#	ts/fixtures/lang/panics/hook/expected/panics.ts
#	ts/fixtures/lang/refs/expected/refs.ts
#	ts/fixtures/lang/rings/expected/rings.ts
#	ts/fixtures/lang/saturates/expected/saturates.ts
#	ts/fixtures/lang/shapes/expected/fmt.Dropper.ts
#	ts/fixtures/lang/shapes/expected/fmt.Nested.ts
#	ts/fixtures/lang/shapes/expected/fmt.ts
#	ts/fixtures/lang/shapes/expected/generics.ts
#	ts/fixtures/lang/shapes/expected/shapes.ts
#	ts/fixtures/lang/shares/expected/shares.ts
#	ts/fixtures/lang/singles/expected/singles.ts
#	ts/fixtures/lang/statics/expected/statics.ts
#	ts/fixtures/lang/strs/expected/strs.ts
#	ts/fixtures/lang/utf8/expected/utf8.ts
#	ts/fixtures/lang/wrappers/expected/wrappers.ts
#	ts/fixtures/lang/wraps/expected/wraps.ts
#	ts/fixtures/lang/zeros/expected/zeros.ts
#	ts/fixtures/names/expected/lib.ts
#	ts/fixtures/stock/expected/buf.ts
#	ts/fixtures/stock/expected/harness.ts
#	ts/fixtures/stock/expected/maps.ts
#	ts/fixtures/task/expected/closures.ts
#	ts/fixtures/task/expected/drops.Key.ts
#	ts/fixtures/task/expected/drops.Wrap.ts
#	ts/fixtures/task/expected/drops.ts
#	ts/fixtures/task/expected/drops/called.ts
#	ts/fixtures/task/expected/drops/flattened.ts
#	ts/fixtures/task/expected/drops/folds.Pile.ts
#	ts/fixtures/task/expected/drops/folds.ts
#	ts/fixtures/task/expected/drops/inside.ts
#	ts/fixtures/task/expected/drops/iters.ts
#	ts/fixtures/task/expected/drops/lists.ts
#	ts/fixtures/task/expected/drops/objects.ts
#	ts/fixtures/task/expected/drops/once.ts
#	ts/fixtures/task/expected/drops/orders.Ranked.ts
#	ts/fixtures/task/expected/drops/orders.ts
#	ts/fixtures/task/expected/drops/released.ts
#	ts/fixtures/task/expected/drops/searched.ts
#	ts/fixtures/task/expected/futures.ts
#	ts/fixtures/task/expected/model.Guard.ts
#	ts/fixtures/task/expected/model.ts
kixelated and others added 3 commits November 6, 2026 14:21
A HashMap or HashSet keyed by String or Cow<str> hashed each key's text
through the runtime's hasher into a chain, 2.3-4.0x main's JS string keys
on the bench's str/keyed workloads. rs.HASH_TEXT files each key under its
text as a JS map key, which the engine hashes, so a chain holds one key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shes it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pacity

# Conflicts:
#	ts/fixtures/lang/collections/expected/collections.ts
#	ts/fixtures/lang/iters/expected/core/option.ts
#	ts/fixtures/lang/iters/expected/iters.ts
#	ts/fixtures/stock/expected/harness.ts
#	ts/fixtures/stock/src/harness.rs
@kixelated kixelated changed the title feat(ts): rs.String, measured over a JS string and over UTF-8 bytes feat(ts)!: translate String as rs.String, holding std's exact capacity Nov 6, 2026
@kixelated
kixelated marked this pull request as ready for review November 6, 2026 23:22

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head ed51944 against base b5f5545.

[P2] Preserve the mutable String place in BorrowMut trait shims — ts/src/lower/collections.rs:171–174. The new text branch returns .v for both Borrow and BorrowMut. For let mut s = String::from("abc"); let b: &mut dyn std::borrow::BorrowMut<str> = &mut s; b.borrow_mut().make_ascii_uppercase();, native Rust leaves s == "ABC". The synthesized method goes through dynamic::class_method → Body::call_def → this mapping, bypassing the direct MIR special case. It therefore returns a primitive string despite its rs.Place<string> return type, producing ill-typed TS and losing mutation access. Restrict this projection to immutable borrow, keep the String object for borrow_mut, and add a differential/type-check case through the trait object.

Verification limits: GitHub-only static review of production changes, relevant pinned Rust implementations, and targeted fixtures/snapshots in this 210-file PR. No translator, differential suite, census, benchmarks, or just ci run here; the fetched Queue runs were skipped. The finding is based on the source call path, not an executed reproducer.

(Written by OpenAI)

kixelated and others added 2 commits November 6, 2026 16:49
The text arm of the Borrow shims gave `borrow_mut` the String's JS text, so a
write through a `&mut dyn BorrowMut<str>` read `.v` of a primitive string.
Only `borrow` reads the text; `borrow_mut` lends the `rs.String`, the place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Fixed the BorrowMut finding in 1d16813. The text arm in ts/src/lower/collections.rs now answers only borrow with the String's text. borrow_mut lends the rs.String itself, the place a &mut str of it is, as the direct MIR path does.

Case 48 of the capacity fixture's strings writes through &mut dyn BorrowMut<str> and reads through &dyn Borrow<str>. Before the fix it threw TypeError on 15 differential inputs; now it matches native.

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Incremental review of head 7dd0c57 against current base b86a186, following the review of ed51944.

The previous P2 BorrowMut finding is addressed. The text projection in ts/src/lower/collections.rs now applies only to immutable borrow; borrow_mut retains the rs.String object. The generated of$String shim returns this.v for mutable borrowing and this.v.v for shared borrowing. Capacity fixture case 48 exercises the mutable trait object, reads through the shared trait object, and emits final text, length, and capacity. ASCII case conversion preserves UTF-8 byte length, so this mutation does not invalidate the cached length or capacity.

No new actionable findings in this focused follow-up. The merged benchmark dispatch consistently reserves op 18 for bytes/frames and moves string workloads to 19–22 in the Rust dispatcher, TypeScript dispatcher, and workload inputs. Direction: retain the trait-object regression coverage; no further change requested for the prior finding.

Verification limits: GitHub-only static inspection of the fix commit, current source and generated fixture, incremental comparison, and benchmark merge integration. This is not a fresh exhaustive review of all 221 changed files. I did not run the translator, TypeScript checks, differential suite, benchmarks, or just ci, and have not independently verified the execution results reported in the PR description.

(Written by OpenAI)

…pacity

# Conflicts:
#	ts/fixtures/lang/dyns/expected/dyns.Over.ts
#	ts/fixtures/lang/dyns/expected/dyns.ts
#	ts/fixtures/lang/dyns/expected/lib.ts
#	ts/fixtures/lang/iters/expected/iters.ts
@kixelated
kixelated added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 6a36cb3 Nov 7, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m3/ts-string-capacity branch November 7, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant