Core concepts · Unicode 17

Text, Bidi & IME

From bytes to grapheme clusters, bidirectional paragraphs and IME composition: zenit treats text as a system capability, not a row of glyphs laid out per code point.

12 min read · includes real window recordings

One line, four layers

Storage, edit boundaries, direction resolution and platform shaping each have exactly one authority, and together produce visual lines you can hit-test and select. No layer makes decisions on another’s behalf.

TEXT STACK

The first two layers live in src/text_core, the bidi algorithm in src/i18n, and shaping on macOS is CoreText’s job.

LayerOwnsWhere

UTF-8 document coordinatesStable byte offsets, normalized to boundaries at every edit entrysrc/text_core

UAX #29 graphemesCaret, delete and selection never split ZWJ emoji, combining marks or flagssrc/text_core/grapheme.zig

UAX #9 bidiParagraph direction, isolates, embeddings, bracket pairs, logical ↔ visual mappingsrc/i18n/bidi.zig

CoreText shapingFont fallback, ligatures, glyph runs, visual placement and caret xsrc/render/text_renderer.zig

Graphemes, not code points

A family emoji is seven Unicode scalars (four people and three ZWJs), é can be e plus a combining acute, and a flag is two regional indicators. To the user each is one character. zenit uses extended grapheme clusters as the atom for caret movement, deletion, selection and capacity truncation.

UAX #29

cursor_pos is a UTF-8 byte offset and only ever lands on a cluster boundary. Even truncation at the single-line Input’s 2048-byte capacity never leaves half a grapheme.

Color emoji go through a BGRA atlas page and the shader’s color branch; the grayscale control stays gray, and mixed runs share the normal text pipeline.

Alt+← / → jumps whole Greek and Cyrillic words, double-click selects a word, and the family emoji stays one cluster.

Bidirectional text

Text is stored in logical order and shown in visual order. UAX #9 first resolves an embedding level for every character, then rule L2 reverses runs from the highest level down. Below, abc אבג 123 sits in an LTR paragraph: the Hebrew resolves to level 1 and the digits that follow it to level 2.

UAX #9

The digits still read left to right, but the whole RTL stretch, digits included, is laid out right to left as a block.

The UAX #9 resolver is zenit’s own portable implementation in src/i18n/bidi.zig, with property tables generated from pinned Unicode 17.0.0 data. On macOS, final glyph shaping and RTL joining are still handed to CoreText a whole run at a time. The text renderer never splits an RTL run per code point.

zenit Storybook showing mixed Arabic, Hebrew and Latin text — The real Storybook: Arabic, Hebrew and latin + arabic + latin mixes laid out by one text pipeline; the virtual cursor rests on the RTL story.

Caret movement and selection in RTL and mixed-direction paragraphs.

Unicode 17 conformance

Runtime property tables are generated from Unicode 17.0.0 data pinned in the repo; the official test files are vendored unmodified under vendor/unicode/17.0.0 and pinned by SHA-256 in SHA256SUMS. “Supported” here doesn’t mean a handful of emoji unit tests. It means running the complete official suites.

490,846BidiTest type sequences

770,241paragraph-direction evaluations

91,707BidiCharacterTest sequences

766GraphemeBreakTest cases

Unicode conformance gates

python3 tools/generate_grapheme_data.py --check
python3 tools/generate_bidi_data.py --check
(cd vendor/unicode/17.0.0 && shasum -a 256 -c SHA256SUMS)
zig build test-text-core
zig build test-bidi-conformance

--check confirms the generated Zig tables match the data; test-text-core runs every GraphemeBreakTest case; test-bidi-conformance runs both complete official bidi files (through UAX #9 rule L2) and is also part of zig build test-headless.

IME as a first-class citizen

IME isn’t the final characters disguised as a keystroke. Platform events keep the phases: ime_preedit carries the composing text plus a caret offset inside it, and ime_commit carries the final text. While composing, Input and Textarea render underlined marked text, but the canonical buffer changes only on commit. There is no separate “discard” event: an empty preedit cancels the composition.

IME LIFECYCLE

Preedit and commit are two kinds of event, not snapshots of one string. After a commit the input briefly sits in commit_pending_end to swallow a duplicate of the same text the platform may send next, then returns to idle.

What first-class meansConcrete contract

Distinct eventsime_preedit / ime_commit keep the phase; an empty preedit cancels

ReconversionBoth events may carry a replacement range (UTF-8 byte span) to pull committed text back into composition or replace it in place

Visual compositionMarked-range underline and highlight, composing caret, and candidate-window position follow the visual line

Edit safetyCaret, delete, selection and capacity truncation all land on extended grapheme boundaries

TestableThe harness injects preedit / commit separately and reads back ime_phase, ime_preedit_len, buffer, cursor_pos, anchor

Real-system gatescripts/verify_ime.sh drives the full NSTextInputClient path with the system Pinyin / Japanese input sources

Input showing underlined nihongo preedit text — Preedit: underlined marked text and the I-beam are visible; the canonical buffer hasn’t changed.

Input showing committed 日本語 — Commit: 日本語 enters the canonical buffer and the preedit is cleared.

Testing IME

The E2E client exposes both phases as separate RPCs, and inputState reads an input’s real state back by test id.

ime.test.ts

import { imeCommit, imePreedit, inputState } from "./client";

// The input must already have focus (e.g. click it first).

await imePreedit("nihongo");
let s = await inputState("story.input.name");
// s.ime_phase === "composing", s.ime_preedit_len === 7, s.buffer === ""

await imeCommit("日本語");
s = await inputState("story.input.name");
// s.buffer === "日本語", s.ime_preedit_len === 0

await imePreedit("");   // empty preedit cancels an active composition

The harness focuses an Input in a real Retina window, injects preedit and commit separately, then moves on to RTL and Checkbox; the I-beam, hand cursor and click pulse are drawn into the drawable by the virtual cursor.

TIP

Injection and real IMEs test different things. Harness injection makes phase, buffer and pixel assertions repeatable; the candidate window, input-source switching and NSTextInputClient integration are still verified through the real system path in scripts/verify_ime.sh (needs a GUI session and Accessibility permission; hands off the keyboard for about 8 seconds).

See E2E harness for the full harness workflow.

← PREV

Reactivity

Styling & themes