--- title: "Text, Bidi & IME | zenit Zig UI Docs" description: "From bytes to grapheme clusters, bidirectional paragraphs and IME composition: zenit treats text as a system capability, not a row of glyphs laid out per…" url: https://ui.surf/zenit/docs/guide/text-engine language: en alternate_zh: https://ui.surf/zenit/zh/docs/guide/text-engine.md alternate_es: https://ui.surf/zenit/es/docs/guide/text-engine.md alternate_ja: https://ui.surf/zenit/ja/docs/guide/text-engine.md alternate_ko: https://ui.surf/zenit/ko/docs/guide/text-engine.md alternate_fr: https://ui.surf/zenit/fr/docs/guide/text-engine.md alternate_de: https://ui.surf/zenit/de/docs/guide/text-engine.md project: zenit v0.1.0-alpha (Zig 0.15.2, macOS) source: https://github.com/version-next/zenit --- # Text, Bidi & IME From bytes to grapheme clusters, bidirectional paragraphs and IME composition: zenit treats text as a system capability, not a row of glyphs laid out per code point. ## One line, four layers Storage, edit boundaries, direction resolution and platform shaping each have exactly one authority, and together produce visual lines you can hit-test and select. No layer makes decisions on another’s behalf. TEXT STACK The first two layers live in src/text\_core, the bidi algorithm in src/i18n, and shaping on macOS is CoreText’s job. | Layer | Owns | Where | | --- | --- | --- | | UTF-8 document coordinates | Stable byte offsets, normalized to boundaries at every edit entry | `src/text_core` | | UAX #29 graphemes | Caret, delete and selection never split ZWJ emoji, combining marks or flags | `src/text_core/grapheme.zig` | | UAX #9 bidi | Paragraph direction, isolates, embeddings, bracket pairs, logical ↔ visual mapping | `src/i18n/bidi.zig` | | CoreText shaping | Font fallback, ligatures, glyph runs, visual placement and caret x | `src/render/text_renderer.zig` | ## Graphemes, not code points A family emoji is seven Unicode scalars (four people and three ZWJs), `é` can be `e` plus a combining acute, and a flag is two regional indicators. To the user each is one character. zenit uses extended grapheme clusters as the atom for caret movement, deletion, selection and capacity truncation. UAX #29 cursor\_pos is a UTF-8 byte offset and only ever lands on a cluster boundary. Even truncation at the single-line Input’s 2048-byte capacity never leaves half a grapheme. [Video](https://ui.surf/zenit/media/stories/emoji.mp4?v=5f65f98444) Color emoji go through a BGRA atlas page and the shader’s color branch; the grayscale control stays gray, and mixed runs share the normal text pipeline. [Video](https://ui.surf/zenit/media/stories/wordnav.mp4?v=1fd0aa8709) Alt+← / → jumps whole Greek and Cyrillic words, double-click selects a word, and the family emoji stays one cluster. ## Bidirectional text Text is stored in logical order and shown in visual order. UAX #9 first resolves an embedding level for every character, then rule L2 reverses runs from the highest level down. Below, `abc אבג 123` sits in an LTR paragraph: the Hebrew resolves to level 1 and the digits that follow it to level 2. UAX #9 The digits still read left to right, but the whole RTL stretch, digits included, is laid out right to left as a block. The UAX #9 resolver is zenit’s own portable implementation in `src/i18n/bidi.zig`, with property tables generated from pinned Unicode 17.0.0 data. On macOS, final glyph shaping and RTL joining are still handed to CoreText a whole run at a time. The text renderer never splits an RTL run per code point. [![zenit Storybook showing mixed Arabic, Hebrew and Latin text](https://ui.surf/zenit/media/unicode-bidi.png?v=d1e0e89e51)](https://ui.surf/zenit/media/unicode-bidi.png?v=d1e0e89e51) The real Storybook: Arabic, Hebrew and latin + arabic + latin mixes laid out by one text pipeline; the virtual cursor rests on the RTL story. [Video](https://ui.surf/zenit/media/stories/rtl.mp4?v=bc200dee27) Caret movement and selection in RTL and mixed-direction paragraphs. ## Unicode 17 conformance Runtime property tables are generated from Unicode 17.0.0 data pinned in the repo; the official test files are vendored unmodified under `vendor/unicode/17.0.0` and pinned by SHA-256 in `SHA256SUMS`. “Supported” here doesn’t mean a handful of emoji unit tests. It means running the complete official suites. **490,846**BidiTest type sequences **770,241**paragraph-direction evaluations **91,707**BidiCharacterTest sequences **766**GraphemeBreakTest cases ```sh python3 tools/generate_grapheme_data.py --check python3 tools/generate_bidi_data.py --check (cd vendor/unicode/17.0.0 && shasum -a 256 -c SHA256SUMS) zig build test-text-core zig build test-bidi-conformance ``` `--check` confirms the generated Zig tables match the data; `test-text-core` runs every GraphemeBreakTest case; `test-bidi-conformance` runs both complete official bidi files (through UAX #9 rule L2) and is also part of `zig build test-headless`. ## IME as a first-class citizen IME isn’t the final characters disguised as a keystroke. Platform events keep the phases: `ime_preedit` carries the composing text plus a caret offset inside it, and `ime_commit` carries the final text. While composing, [Input](https://ui.surf/zenit/components/input) and Textarea render underlined marked text, but the canonical buffer changes only on commit. There is no separate “discard” event: an empty preedit cancels the composition. IME LIFECYCLE Preedit and commit are two kinds of event, not snapshots of one string. After a commit the input briefly sits in commit\_pending\_end to swallow a duplicate of the same text the platform may send next, then returns to idle. | What first-class means | Concrete contract | | --- | --- | | Distinct events | `ime_preedit` / `ime_commit` keep the phase; an empty preedit cancels | | Reconversion | Both events may carry a replacement range (UTF-8 byte span) to pull committed text back into composition or replace it in place | | Visual composition | Marked-range underline and highlight, composing caret, and candidate-window position follow the visual line | | Edit safety | Caret, delete, selection and capacity truncation all land on extended grapheme boundaries | | Testable | The harness injects preedit / commit separately and reads back `ime_phase`, `ime_preedit_len`, `buffer`, `cursor_pos`, `anchor` | | Real-system gate | `scripts/verify_ime.sh` drives the full NSTextInputClient path with the system Pinyin / Japanese input sources | [![Input showing underlined nihongo preedit text](https://ui.surf/zenit/media/ime-preedit.png?v=6c12ff351b)](https://ui.surf/zenit/media/ime-preedit.png?v=6c12ff351b) Preedit: underlined marked text and the I-beam are visible; the canonical buffer hasn’t changed. [![Input showing committed 日本語](https://ui.surf/zenit/media/ime-commit.png?v=f665179c69)](https://ui.surf/zenit/media/ime-commit.png?v=f665179c69) Commit: 日本語 enters the canonical buffer and the preedit is cleared. ## Testing IME The E2E client exposes both phases as separate RPCs, and `inputState` reads an input’s real state back by test id. `ime.test.ts` ```ts import { imeCommit, imePreedit, inputState } from "./client"; // The input must already have focus (e.g. click it first). await imePreedit("nihongo"); let s = await inputState("story.input.name"); // s.ime_phase === "composing", s.ime_preedit_len === 7, s.buffer === "" await imeCommit("日本語"); s = await inputState("story.input.name"); // s.buffer === "日本語", s.ime_preedit_len === 0 await imePreedit(""); // empty preedit cancels an active composition ``` [Video](https://ui.surf/zenit/media/e2e-text-ime.mp4?v=efa93e6b3f) The harness focuses an Input in a real Retina window, injects preedit and commit separately, then moves on to RTL and Checkbox; the I-beam, hand cursor and click pulse are drawn into the drawable by the virtual cursor. > TIP > > **Injection and real IMEs test different things.** Harness injection makes phase, buffer and pixel assertions repeatable; the candidate window, input-source switching and NSTextInputClient integration are still verified through the real system path in `scripts/verify_ime.sh` (needs a GUI session and Accessibility permission; hands off the keyboard for about 8 seconds). See [E2E harness](https://ui.surf/zenit/docs/advanced/e2e) for the full harness workflow.