Code point

The streaming decoder skeleton

decoder(machine) drives a shared Transducer and flattens its output chunks into a lazy List. UTF-8 and UTF-16 export utf8Decoder and utf16Decoder with init, update(state, unit), and end(state). The pure list driver pulls one unit at a time and invokes end at EOF, so EOF needs no sentinel in the input alphabet. Decoded output stays incremental, including the final flush chunk.

The codecs use fromStateScan to give their small per-unit operations the common lifecycle. eofFlush builds their EOF output: leftover state becomes exactly one error unit and empty state emits nothing. They differ only in how a non-empty state maps to its error unit (utf8StateToError for UTF-8, state | errorMask for an unpaired UTF-16 surrogate). Their terminal summary is null; code points are emitted items, including those emitted by end.

restart shares the first step's fresh-state dispatch with its error recovery. Each decoder dispatches a unit from the empty state in two places: the state === null arm, and recovery, where a pending state the unit refuses is flushed as an error unit and the unit is dispatched afresh. The two differ only in what goes out ahead of the unit's own output — nothing, or the flushed error — so each codec writes its classifier once and calls it as restart(fresh)([]) and restart(fresh)([error]). In UTF-16 recovery the fresh dispatch's third case, a lone low surrogate, is unreachable — the pair would have been accepted — so recovery needs no classifier of its own.

Shared code-point predicates

The code-point classification predicates — isBmpCodePoint, isHighSurrogate, isLowSurrogate, isSupplementaryPlane, and isValidCodePoint — describe fixed Unicode ranges, not per-encoding details, so they live in module.f.js alongside errorMask and decoder. They are derived from one set of boundary constants so the surrogate bounds (0xD800–0xDFFF) and the maximum code point (0x10FFFF) are spelled out exactly once. UTF-8 and UTF-16 import them instead of redefining their own range checks.

The surrogate-pair arithmetic lives beside them for the same reason: tryToSurrogatePair splits a supplementary-plane code point into its high and low surrogates and tryFromSurrogatePair combines them back. Both are derived from the same constants as isHighSurrogate, isLowSurrogate, and isSupplementaryPlane, which they invert, so UTF-16 re-spells none of 0xD800, 0xDC00, or 0x10000.

Both refuse input outside their domain with null rather than answering it (DESIGN.md §10): the arithmetic would otherwise turn 0xFFFF into a "pair" whose first half is not a high surrogate, a high surrogate and a BMP word into a plausible code point, and a fraction inside a range into the pair or code point of its truncation. The domain check is the same one a caller would have to make before calling, so UTF-16 branches on the null instead of gating first — each classification happens once.

isTextCodePoint lives here too, but answers a different question: not whether a code point is well-formed (isValidCodePoint) but whether it is text. A code point is text unless it is a control character — C0 (U+0000–U+001F), U+007F (DEL), and C1 (U+0080–U+009F) — minus the whitespace block U+0009–U+000D (TAB, LF, VT, FF, CR), which is legitimate in text. The two are deliberately distinct: a control byte such as NUL is perfectly valid UTF-8 yet not text, which is exactly what fjs/media/type needs to split text from binary. isValidCodePoint gates decoding (fromVec); isTextCodePoint gates classification.

isValidCodePoint was previously exported from utf8/module.f.mjs; it now lives solely on code_point. Importers — fjs/media/type and utf8's own fromVec — take it from code_point directly. This was a deliberate breaking change rather than a re-export, on the principle that the predicate's canonical home is the shared Unicode contract.

Emergent Testing

Idle. Press Run to start the suite.

  • ./proof.f.mjs