fjs/text · FunctionalScript

UNICODE

The shared code-point classification predicates (isBmpCodePoint, isValidCodePoint, isTextCodePoint, …) live in code_point — see code_point/README.md. So does the CodePoint type both codecs decode to and encode from, code_point/types.ts: a scalar value, an untagged surrogate or above-0x10FFFF value the UTF-8 decoder passes through unchecked (gate on isValidCodePoint for a scalar), or an errorMask-tagged error value laid out by the error tables below.

Text that says what its words are — keywords, strings, numbers — beside the text itself, for a page or a terminal to colour without reading it again, is marked: marked/README.md.

Which function to call for each encoding is on its own page: utf8, utf16 and ascii. The rest of this page is the bit-level design.

UTF-8

Requirement: no loss for UTF8 => codepoint => UTF8

utf8	utf8 code	size	codepoint
`[a]`	`0xxx_xxxx`	7 bit	`0_0000_0000_0000_0xxx_xxxx`
`[b,a]`	`110x_xxxx 10xx_xxxx`	11 bit	`0_0000_0000_0xxx_xxxx_xxxx + 0_0000_0000_1000_0000`
`[c,b,a]`	`1110_xxxx 10xx_xxxx 10xx_xxxx`	16 bit	`0_0000_xxxx_xxxx_xxxx_xxxx + 0_0000_1000_0000_0000`
`[d,c,b,a]`	`1111_0xxx 10xx_xxxx 10xx_xxxx 10xx_xxxx`	21 bit	`x_xxxx_xxxx_xxxx_xxxx_xxxx + 1_0000_0000_0000_0000`

utf8 error	utf8 code	size	codepoint	flag
`[e]`	`1111_1xxx`	3 bit		`errorByteFlag`
`[d,]`	`1111_0xxx`	3 bit		`errorByteFlag`
`[c,]`	`1110_xxxx`	4 bit		`errorByteFlag`
`[b,]`	`110x_xxxx`	5 bit		`errorByteFlag`
`[e]`	`10xx_xxxx`	6 bit	`0000_0000 1xxx_xxxx`	`errorByteFlag`
`[d,c,]`	`1111_0xxx 10xx_xxxx`	9 bit	`0000_001x xxxx_xxxx`	`errorLead4ContFlag`
`[c,b,]`	`1110_xxxx 10xx_xxxx`	10 bit	`0000_01xx xxxx_xxxx`	`errorLead3ContFlag`
`[d,c,b,]`	`1111_0xxx 10xx_xxxx 10xx_xxxx`	15 bit	`1xxx_xxxx xxxx_xxxx`	`errorLead4Cont2Flag`

Each code point is also tagged with errorMask. The flags are named once, in utf8/module.f.mjs, and both the encoder and the decoder read them from there.

Total error states:

2^6 + 2^3 + 2^5 + 2^4 + 2^10 + 2^3 + + 2^9 + 2^15
2^4 + 2^6 + 2^5 + 2^4 + 2^10 + 2^9 + 2^15
2^5 + 2^6 + 2^5 + 2^10 + 2^9 + 2^15
2^6 + 2^6 + 2^10 + 2^9 + 2^15
2^7 + 2^9 + 2^10 + 2^15
0b1000_0110_1000_000
128 + 512 + 1024 + 32_768
34_432
< 2^16

utf8/module.f.mjs

const toCodePointList: (input: List<u8|null>) => List<CodePoint>
const fromCodePointList: (input: List<CodePoint>) => List<u8>

UTF-16

Requirement: no loss for UTF16 => codepoint => UTF16

0xD800..0xDFFF
0b_1101_1000_0000_0000
0b_1101_1111_1111_1111

0b_1101_1xxx_xxxx_xxxx : 11 bits

- first : 0xD800: 0b_1101_10xx_xxxx_xxxx : 10 bit
- second: 0xDC00: 0b_1101_11xx_xxxx_xxxx : 10 bit

utf16	utf16 code	size	codepoint
`[a]`	`xxxx_xxxx_xxxx_xxxx`	16 bit	`0000_xxxx_xxxx_xxxx_xxxx`
`[b,a]`	`1101_10xx_xxxx_xxxx 1101_11xx_xxxx_xxxx`	20 bit	`xxxx_xxxx_xxxx_xxxx_xxxx + 1_0000_0000_0000_0000`

utf16 error	utf16 code	size	codepoint
`[e]`	`1101_11xx_xxxx_xxxx`	10 bit	`1101_11xx_xxxx_xxxx`
`[b,]`	`1101_10xx_xxxx_xxxx`	10 bit	`1101_10xx_xxxx_xxxx`

Total error states: 11 bit

utf16/module.f.mjs

const toCodePointList : List<u16|null>) => List<CodePoint>
const fromCodePointList: (input: List<CodePoint>) => List<u16>
const stringToList: (input: string) => List<u16>
const listToString: (input: List<u16>) => string

UTF-16 => CP => UTF-8 => CP = UTF-16

Example

UTF-16:
  1101_11xx_xxxx_xxxx
  1101_11xx_xxxx_xxxx
CP:
  1000_0000_0000_0000_1101_11xx_xxxx_xxxx
  1000_0000_0000_0000_1101_11xx_xxxx_xxxx
UTF-8:
  1111_0.101
  10.11_xxxx
  10xx_xxxx
  1111_0.101
  10.11_xxxx
  10xx_xxxx
CP:
  1000_0000_0000_0000_1101_11xx_xxxx_xxxx
  1000_0000_0000_0000_1101_11xx_xxxx_xxxx
UTF-16:
  1101_11xx_xxxx_xxxx
  1101_11xx_xxxx_xxxx