Synthetic Universal Language (SUL)

SUL is a universal encoding that bijectively maps any finite sequence of symbols to a single root symbol via a balanced tree, from which the original sequence can be uniquely recovered.

Concepts

  • A symbol is an element of a level's finite alphabet [0, n).
  • A word is a finite sequence of symbols that encodes into a single symbol of the next level.
  • A level defines a finite alphabet [0, n) and the bijection between words over that alphabet and symbols of the next level.

Word structure

A valid word has the form [s0, s1, ..., sk, t] where:

  • s0 > s1 > ... > sk — a strictly decreasing prefix of at least one symbol
  • sk <= t — a terminating symbol no less than the last prefix element

The minimum word length is 2: one prefix symbol is required to provide a value to compare against t, ensuring each word collapses to a single symbol at the next level — so the sequence strictly shrinks at every step and converges to a single root symbol in a finite number of levels.

API

The top-level encoder accepts a stream of bits and produces a single Id:

import { encode } from './module.f.mjs'
import { run } from '../types/list/module.f.js'
import type { Add } from './level/hash/types.ts'

const enc = encode(add)(storage)     // add: Add<S> — called on every merge
const root: Id = run(enc)(bits)      // consume 0n/1n bits, pad, return the root
ExportDescription
encode(add)(storage)Creates an encoder with the given initial storage
Encode<S>Shared Transducer lifecycle, plus updateWithHeight and endWithHeights, which also give each bit's height
EncodeState<S>Opaque streaming state (three literal levels + dynamic hash stack)

init is a next step holding the initial state. update(state, bit) returns a new next step; end(state) pads with 1, then 0s, and returns a done step holding the root. No items are emitted before completion. The common driver handles these steps, and a caller can save a step's state and resume it.

A bit's height is how many levels close a word on it, counting up from L1: 0 for none, 2 for L1 and L2, 5 for L1 to H2. updateWithHeight(state, bit) returns it with the next state, and endWithHeights(state) returns it for each padding bit with the root. These additional observations share the lifecycle's encoding and padding operations. The demo draws its stripes from these heights.

The Add<S> callback is invoked once per compress(left, right) merge. isSymbol is true for the terminal merge that produces a hash-level output symbol and false for Patricia trie internal merges.

Architecture

Bits flow through four stages in sequence:

bits → L1 → L2 → L3 → Hash[0] → Hash[1] → … → Id

The hash stack grows dynamically: each time a hash level emits a symbol, it is pushed into the next level (created on demand).

Sub-modules

  • level/ — level hierarchy: literal and hash levels
  • id/ — 256-bit content-addressed identifier

Demo

Cuts bits into SUL words, level by level, up to one symbol, the root: L1 words are made of bits, L2 words of L1 symbols, and so on through L3 and the hash levels H1, H2 and up. Each row switches colour where a word ends, so a stripe is the run of input bits one symbol encodes; a hatched stripe is a hash, the others hold their bits inline. After the input, a 1 and then 0s pad the bits until the root. Compare the L1 row with the L1 table in levels.md: 00, 01, 100, 101 and 11.

Input

Words, level by level:

bits
L1
L2
L3
H1
H2
H3
H4
H
01001000
e
01100101
l
01101100
l
01101100
o
01101111
,
00101100
␠
00100000
w
01110111
o
01101111
r
01110010
l
01101100
d
01100100
!
00100001
␠
00100000
S
01010011
U
01010101
L
01001100
␠
00100000
i
01101001
s
01110011
␠
00100000
a
01100001
␠
00100000
u
01110101
n
01101110
i
01101001
v
01110110
e
01100101
r
01110010
s
01110011
a
01100001
l
01101100
␠
00100000
e
01100101
n
01101110
c
01100011
o
01101111
d
01100100
i
01101001
n
01101110
g
01100111
.
00101110
end
10000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
00000

336 input bits and 229 padding bits: L1 257, L2 113, L3 48, H1 22, H2 10, H3 4, H4 1 words.

Root, a hash:

0xa9a55818da237885da693d4bf8a6bc5babfd4818ef9f4f5a1f606765eb97850a

Emergent Testing

Idle. Press Run to start the suite.

  • ./id/proof.f.mjs
  • ./level/hash/proof.f.mjs
  • ./level/literal/proof.f.mjs
  • ./proof.f.mjs