learn / 04·5 — under org · the kernel

one reader for theWHOLEgrammar

Most systems rot at the word parseable — three parsers in three languages that quietly disagree. Workbooks has exactly one. The OQL kernel is a 552-line pure-function Rust crate — string in, JSON or HTML out, no I/O — and because it's pure it compiles unchanged to wherever the string is: the CLI, the runtime, the desktop, and this page.

the kernel12 min read
A single colossal brass-and-glass reading machine in a sunlit vault, a paper tape of org text feeding into it, three bright beams of light fanning out from its core to three small workstations — a tiny engineer standing before it — 1970s sci-fi style, monumental and warm

three parsers, three TRUTHS

The parent lesson convinced you org is parseable. Fair — but parseable is exactly where most systems begin to rot. A format earns a parser on the server, then a second one in the browser so the page can render without a round trip, then a third hand-rolled one in the CLI because pulling the server in felt heavy. Three readers, three languages, three subtly different ideas of what a SCHEDULED: timestamp means or whether a tag is case-sensitive.

The rot isn't the parsers. It's that they drift. The server renders one thing, the client renders another, the CLI lints a file the engine would accept — and now your format has no canonical meaning, only a majority vote. Every bug report becomes an archaeology dig through which reader saw what. "Parseable" was supposed to be the promise; three parsers is how the promise quietly breaks.

So the only question worth asking of a format like this is the blunt one: what actually reads my file, and is it the same thing everywhere? This page is the answer, and the answer is a single program.

the DEFINITION

ker·nel /ˈkɜr·nəl/ noun

1. the one program that reads org for the whole ecosystem — a 552-line pure-function Rust crate (package oql) with two dependencies, six verbs, and zero I/O; it takes a string and returns a string, and it is the same crate everywhere the grammar is read.

There is no second copy. The whole contract is the six functions declared in its world.wit — every one of them string → string:

world oql {
  export parse-headlines: func(org: string) -> string;   ← query the tree
  export tangle-plan:     func(org: string) -> string;   ← compile a build plan
  export validate:        func(org: string) -> string;   ← diagnostics
  export check-upgrade:   func(old, new: string) -> string; ← the safety gate
  export render:          func(org: string) -> string;   ← org → HTML
  export lint:            func(org: string) -> string;   ← reserved (stub today)
}

The crate is deliberately standalone — its own Cargo.toml with its own [workspace], not a member of a parent tree. It leans on exactly two libraries: serde_json for the JSON it emits, and orgize — an MIT org-mode parser — for the heavy lifting of turning text into a syntax tree. Everything the kernel adds is opinion on top of that tree.

no host calls, no CLOCK

Every one of the six functions has the same shape: a borrowed string in, an owned string out. No function reads a file, opens a socket, asks the time, or calls back into a host. render in its entirety is Org::parse(org).to_html() — orgize parses, orgize emits, the kernel just chose the call. That austerity isn't an aesthetic. It is the deployment mechanism.

A function that touches no I/O has no opinion about where it runs. It needs no operating system, no event loop, no permissions — only a string and a place to put the answer. Which means the same source can be aimed at three different targets that share nothing else, and the feature flags decide which glue gets compiled around an otherwise identical core:

flowchart TD
  src[["lib.rs — 552 lines
six pure functions · string → string"]] src -- "default-features = false
(rlib)" --> a["native rlib
linked into the CLI"] src -- "feature: component
(cdylib → cargo component)" --> b["WIT Component
414 KB · runtime + desktop"] src -- "feature: browser
(cdylib → wasm32-unknown-unknown)" --> c["ptr/len build
537 KB · this page"] style src fill:#f2ddb0,stroke:#121316,stroke-width:2.5px style a fill:#d9dbd3,stroke:#121316 style b fill:#aee5c2,stroke:#121316 style c fill:#9fc4e8,stroke:#121316

Read that graph as one source fanning into three artifacts. The middle box is the crate itself — the six functions, written once. The left edge compiles it with no features at all, as a plain Rust library to link into a binary. The middle edge turns on the component feature and runs cargo component, producing a typed WebAssembly Component. The right edge turns on the browser feature and targets bare wasm, producing a raw module a web page can instantiate. The kernel is never ported to three surfaces. It is unchanged across them — the crate's own source says it plainly: there is no "wasm version" of the kernel logic.

six verbs, one CONTRACT

The whole surface area is six functions. Each answers one question about an org document, and every one returns either JSON or HTML — never a typed record, because JSON is the type system at this boundary, the same on all three surfaces:

verbinoutthe question it answers
parse-headlinesorgJSON rowswhat's in this tree? — states, tags, properties, timestamps
tangle-planorgJSON worldwhat does this literate document build into?
validateorgJSON findingsis the plan sound — any dangling inputs or missing languages?
check-upgradeold, newJSON findingsis replacing this world with that one safe?
renderorgHTMLwhat does this look like as a page?
lintorg[]reserved — a stub returning empty today (be honest: it does nothing yet)

The honesty matters: lint is a declared export that currently returns a hardcoded empty array. The CLI's lint verb actually maps to validate, which is the function with real diagnostics. The slot is reserved for style checks the kernel doesn't perform yet — and a page that told you otherwise would be the first drift this whole design exists to prevent.

The org the kernel reads is the full grammar from the parent lesson: TODO states, tags, :PROPERTIES: drawers, SCHEDULED and DEADLINE timestamps — parsed by a compact ~18-line routine that knows active <…> from inactive […] and reads the +, .+, ++ repeaters — and, inside source blocks, six header args: :deps :uses :in :out :persist :dir. Two of those carry deliberate weight: :persist marks a component for Motoko-style orthogonal persistence, and :dir points at a real project directory instead of inline source.

from headline to WORLD

Depth rung. render and parse-headlines are the obvious verbs; tangle-plan is the one that makes the kernel more than a parser. It reads a literate workflow — a tree of tagged headlines with source blocks — and computes a build plan shaped exactly like a WIT world: imports, exports, components, and the edges between them. None of that shape is written down by hand; it's derived.

Here is a real input — a two-component pipeline with a schedule:

* pipeline                                       :workflow:
  :PROPERTIES:
  :SCHEDULE: 0 6 * * *
  :END:
** fetch                                          :component:
#+begin_src js :uses net :out raw
export function run() { return fetch_prices() }
#+end_src
** summarize                                      :component:
#+begin_src python :in raw :out report :persist
def run(raw): return summarize(raw)
#+end_src

Run wbx tangle pipeline.org and the kernel returns one world. The rules it applied, in order: the :workflow:-tagged headline becomes the world, named pipeline. Its :component: children become components. The :SCHEDULE: property — a cron string — wins as the schedule, yielding "cron": "0 6 * * *". Every :uses becomes an import, so the union here is ["net"]. Then the edges: the kernel matches one component's :out to another's :infetch produces raw and summarize consumes it, so one edge runs fetch → summarize. Outputs nobody consumes become exports, so report is exported. And summarize carried :persist, so it's flagged persistent. The shape that falls out:

flowchart LR
  subgraph plan["pipeline.org  :workflow:  ·  SCHEDULE 0 6 * * *"]
    f["fetch  :component:
uses: net · out: raw"] s["summarize :component:
in: raw · out: report · persist"] end plan -- "kernel: tangle-plan()" --> world subgraph world["world: pipeline"] imp(["imports: net"]) f2["fetch"] -- "raw" --> s2["summarize"] exp(["exports: report"]) end style plan fill:#fbfaf6,stroke:#121316 style world fill:#fbfaf6,stroke:#121316 style imp fill:#9fc4e8,stroke:#121316 style exp fill:#13d943,stroke:#121316

Walk the right side as a story: a capability called net flows in at the top; data flows along the single raw edge from fetch to summarize; and one output, report, leaves the world unconsumed as its export. The recursion is the part that scales — a :workflow: headline nested under another becomes a sub-world, so an entire system of pipelines is one tree the kernel folds into one plan. The parent org lesson promised structure was checkable; this is the payoff — a to-do list and a build system turn out to be the same data structure, tasks with dependencies, and the kernel refuses to make you learn it twice.

the upgrade GATE

Depth rung. Once a world is live, changing it is dangerous — a redeploy can silently break the things downstream that depended on the old shape. check-upgrade is the kernel's answer: give it the old world and the new one, and it applies WIT/Candid-style subtyping rules and refuses anything breaking before it runs. The rules are short and one-directional:

what changedverdictwhy
a workflow was removederrorsomething downstream may still call it
an export was removederrora consumer of that output now breaks
a component's output type changederrorthe old contract no longer holds
a new import appearedwarna new capability is now required — surface consent
nothing breakingempty arraysafe upgrade, deploy proceeds

The verdict of that table, in one line: exports may grow but never shrink; imports may shrink but never grow; an output type may never change. Errors block the deploy outright; the new-capability warning isn't a block — it's a consent question, because asking for a new power should be visible, not silent. An empty array means the change is safe.

You can watch it refuse a bad change with nothing but stock wasmtime. Take a component Build that exported report as a string, and a new version that now requires a network-email capability and emits report as JSON instead:

$ wasmtime run --invoke 'check-upgrade("<v1>", "<v2>")' oql.wasm

[{"level":"error","scope":"Report","message":"export `report:string` removed (breaking)"},
 {"level":"warn", "scope":"Report","message":"new capability `workbook:net/email` now required"},
 {"level":"error","scope":"Report","message":"component `Build` output type changed (breaking)"}]

Three findings: two errors and one warning. Read aloud — the string export was removed, that's breaking; a new email capability is now required, that's a warning that becomes a consent prompt; and the component's output type changed, that's breaking too. Two errors mean the deploy is refused. The gate lives in the kernel; the runtime demos it today, and the upgrades lesson is where it earns its place on every deploy path. Honest scope: the rule exists and runs; wiring it in front of every redeploy is still in progress.

surface one: linked into the CLI

The first surface is the plainest: the kernel as an ordinary Rust library. The CLI crate — whose binary is named wbx — depends on it as oql = { path = "../runtime/kernel", default-features = false }. No component glue, no WASI, no wrapper. With features off, the crate is just an rlib, and it links cleanly on native targets and on the CLI's own wasm32-wasip1 target.

Three verbs map straight through: wbx query calls parse-headlines, wbx tangle calls tangle-plan, and wbx lint calls validate (the real diagnostics, not the stub). A bare - reads stdin, so org generated upstream never has to touch a temp file. The point worth holding: these verbs run with no runtime server at all. The kernel is compiled into the binary in your hand, so you can query, tangle, and validate a file on a laptop with nothing else installed.

surface two: sealed in the runtime + desktop

The second surface turns the same crate on the component feature and runs cargo component build, producing runtime/build/oql.wasm414,241 bytes, a typed WebAssembly Component, checked into git. A byte-identical copy lives at desktop/src-tauri/oql.wasm; they're verified the same bytes. One artifact, two homes.

In the BEAM runtime, a small GenServer called Workbooks.OQL embeds that wasm at compile time with File.read!, instantiates it once via Wasmex.Components, and calls the exports with call_function — no hand-rolled ptr/len ABI, no manual memory bookkeeping, because the Component model carries the types. Every render in the public plane and every published page flows through it:

sequenceDiagram
  participant K as PublishKit / PublicWeb
  participant G as Workbooks.OQL (GenServer)
  participant W as wasmtime instance (oql.wasm)
  K->>G: render(org)
  G->>W: call_function "render"
  W-->>G: HTML string
  G-->>K: HTML string
  Note over K: static page served — footer reads
"Rendered by the Workbooks OQL kernel"

Walk that exchange: a publish or public-web request asks the GenServer to render some org; the GenServer hands it to the one wasmtime instance; the instance returns HTML; the page ships. The same path drives more than rendering — the package manager calls tangle-plan to build a world, the socket dispatches parse, tangle, validate, and render over a websocket, and wb publish apply renders a workbook to self-contained HTML with no server required at publish time, because the kernel is embedded in the runtime.

The desktop does the same trick natively: oql.wasm is compiled into the Tauri shell with include_bytes! and run via wasmtime, and the app's weave(org) is just call("render", org). Its tests prove a workbook weaves with no Elixir and no Docker — the kernel is the whole rendering engine. One honest nuance lives here: the WIT world declares no imports, but the compiled artifact does import WASI 0.2, because Rust's standard library pulls it in. The host hands it a blank WASI context with no preopens — which means no filesystem, ever. render never touches a disk because there is no disk to touch.

And you don't need any Workbooks software to poke it. Stock wasmtime works: wasmtime run --invoke 'render("* Hello")' runtime/build/oql.wasm.

surface three: this very page

The third surface drops the Component model entirely and builds raw wasm for wasm32-unknown-unknown536,666 bytes, with a deliberately tiny hand-written ABI. Strings go in as a (ptr, len) pair; results come back as a single packed u64 — the high 32 bits are the pointer, the low 32 the length. All of the unsafe in the entire crate lives in the 43 lines of that ABI file; the Component path has none. This build exports only four of the six functions — parse-headlines, tangle-plan, validate, and render — and instantiates with zero imports.

The whole client is about fifteen lines of JavaScript: encode the string, ask the module for memory, write the bytes, call the function, unpack the u64 with BigInt shifts, and decode the result. That's it — no framework, no glue library:

const { instance } = await WebAssembly.instantiate(bytes);   // zero imports
const { memory, oql_alloc, oql_render } = instance.exports;
const call = (fn, s) => {
  const b = new TextEncoder().encode(s);
  const p = oql_alloc(b.length);
  new Uint8Array(memory.buffer, p, b.length).set(b);
  const packed = fn(p, b.length);                            // one u64 back
  const ptr = Number(packed >> 32n), len = Number(packed & 0xffffffffn);
  return new TextDecoder().decode(new Uint8Array(memory.buffer, ptr, len));
};

And it isn't a description — it's running right now. The textarea below holds org; the panel beside it is that org rendered to HTML by the kernel, in your browser, with no server in the loop. The exact bytes the runtime ships, built for the web, instantiated on this page. Type into it and watch the kernel work:

the kernel, in your tab — same bytes the runtime ships

Honest note on weight: that convenience costs about 537 KB of wasm (here, gzipped and inlined). The runtime's Component build is smaller at 414 KB because the Component toolchain trims harder. Neither is tiny — purity buys portability, not minimalism.

where the story BENDS

Honesty section. The "no I/O, runs anywhere" story is true at the source level, and it bends in a few real places worth naming:

  • The Component imports WASI. The WIT world declares no imports, but the compiled artifact pulls in WASI 0.2 because Rust std does. The hosts starve it instead of trusting it: a blank WASI context, no preopens, so the filesystem is unreachable in practice — but "no imports" is the source's intent, not the binary's literal truth.
  • lint is a stub. It returns []. The real diagnostics live in validate. The export is a reserved slot, not a feature.
  • The upgrade gate isn't on every deploy yet. The rules exist and run; the runtime demonstrates them. Wiring check-upgrade in front of every redeploy is ongoing work, not a finished guarantee.
  • The browser build is 537 KB. Real bytes on a real page. Inlined and gzipped here, but not free.
  • orgize does the parsing. The kernel is the opinionated lens — the tangle rules, the upgrade rules, the choice of what to emit — not a parser written from scratch. Credit where it's due.
  • A legacy fourth surface still lurks. The desktop carries an older vendored wasm-pack build from a dead monorepo, imported by one renderer path. It predates the clean browser ABI and isn't the kernel this lesson teaches — an honest aside, not a current surface.

None of these break the core claim. One crate reads org for the whole ecosystem, and the places the purity story bends are places the hosts contain, not places the meaning forks.

questions people actually ASK

Is this the org-mode parser, rewritten?

No. orgize — an MIT library — does the parsing, turning text into a syntax tree. The kernel is the opinion on top: how a tree tangles into a world, which upgrades are safe, what renders to HTML. Six functions of judgment, not a parser from scratch.

Why Rust, and not the Elixir host the runtime already has?

Because the host can't run in a browser or a standalone CLI. A pure Rust crate compiles to native, to a wasm Component, and to bare browser wasm from one source. The Elixir runtime is one consumer of the kernel — it embeds the same wasm everyone else does.

Can I call it myself, without your tooling?

Yes. Stock wasmtime invokes the Component directly — wasmtime run --invoke 'render("* Hello")' oql.wasm — and the browser build instantiates with zero imports in about fifteen lines of plain JavaScript. There's no SDK to install.

Why six string→string functions instead of typed records?

One ABI everywhere. JSON is the type system at the boundary, identical on native, Component, and browser surfaces. Strings in, strings out means the same calling convention no matter which target compiled the crate — the price of portability, paid once.

What happens if the kernel panics?

A fault traps inside wasmtime. The wasm instance is a sandbox; a panic becomes a trap that the host catches as an error, and it never reaches into the BEAM, the Tauri shell, or the page. The blast radius of a bad parse is the instance, by construction.

Does the kernel pass its own conformance check?

No — and it's a lovely fact. The runtime's conformance test asks whether a component exports run; the kernel exports render, tangle-plan, and the rest instead. It's a different kind of world, so the engine rejects it as a workbook — the reader the ecosystem is built on isn't itself one of the things it reads.

keep GOING

The kernel is the machine that reads the grammar — so the grammar is where to go next, and the things it computes each have their own lesson.