The goal
The goal is that a group keeps its shared records together, as one truth that converges. The group does not trust a server, and its members are often offline. The members know that any one of them can lose a laptop. They also know that a person can coerce any one of them.
Thus the records are in a normal SQLite file that each person carries. Access control is part of the data itself. It must stay in effect when a person copies the file, edits it offline, and merges it. The design has three properties and uses no server:
- Two or more parties each run the database and edit it offline.
- Edits merge. Two peers exchange a delta to reconcile. A subgroup converges to a checkpoint that its members agree on.
- The data does not depend on one keyholder alone. If the group loses one person, or if an attacker coerces one person, the attacker does not receive the data.
The engine is cr-sqlite, a CRDT extension to SQLite. Its merge operates on each column, and the merge is commutative, associative, and idempotent. Thus replicas collaborate on the same records and converge in any merge order. The cr-sqlite extension gives convergence. On top of it, localRbac adds all the rules about who can see and who can change each item.
Read control is a data-encryption key
A person can always read the bytes on their own disk, and you cannot prevent that. Thus the question "Can
Alice read this?" becomes "Can Alice decrypt this?" Each resource is a compartment,
for example patient:alice or notes. Each resource has its own
data-encryption key (DEK), and each DEK has a version. A record is a set of encrypted cells,
one cell for each (record, column) pair. The DEK of the resource encrypts each cell:
cell(pk = rid|resource|col, rid, resource, col, ct, dek_ver, author, sig)
A reader receives the DEK in one of two ways:
- The admin derives the DEK from one root secret when it is necessary:
DEK = HKDF(rootKey, "dek:"+resource, "v"+ver). The engine stores no resource keys. - Each other member receives a sealed DEK. For each grant, the admin seals that DEK to the
X25519 key of the member and puts it in a
keywraprow. Only that member can unseal it.
No online authority is necessary to read. listRecords decrypts each cell if you hold its key,
and it skips the other cells. A person without a grant has the same ciphertext and sees a 🔒.
Nothing leaks, because there is no plaintext to hide.
dek_ver and
seals the new key to the remaining members. The revoked user keeps the old ciphertext. That user cannot read
the data that a member writes after the revocation.
Write control is a signature that the merge checks
Encryption stops reads, but it does not stop writes. Any person can append a row to a file. Thus each row carries the Ed25519 key of its author and a signature. The engine enforces the signature at a point that a person cannot bypass: at merge. The engine replays an incoming changeset into a temporary staging database. Before the engine trusts the changeset, it makes sure that these two conditions are true:
- System rows (identities, grants, key-wraps, checkpoints) have a valid signature, and the admin is their author. These rows establish who is a writer.
- Data cells have a valid signature, and their author is a current writer for that resource.
The changeset merges only if each row passes. The merge rejects each of these items, and it gives a reason:
- A forged cell.
- A person who makes themselves the admin.
- A reader who tries to write.
Authenticity is a property of each row, not of the envelope. Thus authenticity stays intact through the transitive gossip of cr-sqlite.
Local consensus: two files, one state root
Two peers exchange changesets to reconcile. It is not necessary to send the data to know that the peers converged. Each peer calculates a state root: a SHA-256 hash of the sorted state of the cells and the grants. If the roots are equal, the states are identical. The peers compare 64 hex characters, not a database.
The engine code documents one problem. The db_version of cr-sqlite is a local clock,
and a merge assigns it again. Thus "all changes after checkpoint H" uses a local watermark, not a portable
version vector.
Group consensus: the wipe-and-rebuild ceremony
A full subgroup converges through a ceremony. A coordinator merges the diffs, records a signed checkpoint, and gives each member a rebuild slice. Then each member wipes the local database and rebuilds it from the slice. Figure 3 shows the sequence.
- Each diff holds the changes of one member after the last checkpoint.
- The coordinator can also rotate each DEK. This step is optional.
- The checkpoint is a hash of the agreed state.
- The coordinator does not send the merged database back.
- A rebuild slice holds the auth state and the checkpoint state. It also holds only the non-archived cells that the member has a right to.
This one procedure does three jobs:
- Compaction.
- Data minimization: each member receives only the data that the member has a right to.
- Redistribution from the same checkpoint.
The lock occurs in the evening, and the unlock occurs in the morning.
No single keyholder: Shamir threshold custody
One problem remains: the root secret of the admin can derive each DEK. If any one person can decrypt the full collective dataset alone, that person becomes a target for extortion. Cryptography cannot protect a person against physical force.
Thus localRbac uses Shamir's secret sharing.
It splits the custody key into n shares in the field GF(2⁸). Any k shares
reconstruct the key. Any k−1 shares reveal nothing. Then localRbac seals each share to
the key of one custodian.
sealVault encrypts a payload with a random key and then splits that key. The unlock is a
distributed operation. Each custodian seals their share again to the opener. The opener combines
k shares.
A 4-of-5 quorum continues to operate if it loses one custodian. To break it, an attacker must compromise four people at the same time. No single keyholder is a point of failure or a point of coercion.
Where the database is: demand-paged onto encrypted IndexedDB
Ordinary SQLite is demand-paged. Its pager reads and writes single pages
from the file when a query touches them. (A page is 4 KB by default.) The pager holds only a small page
cache in RAM. Thus a 50 GB database runs in a few megabytes. Only a :memory: database is
fully in RAM.
Most SQLite-in-the-browser builds discard this model. sql.js loads the whole file into
a Uint8Array on the WASM heap. localRbac keeps the demand-paged model in the browser. It injects
a VFS, the virtual-filesystem interface of SQLite. Thus the block reads and writes of the pager go
to IndexedDB, one page at a time. The whole file is never in memory.
The persistent connection opens through EncryptedIDBVFS. This is a vendored copy of
IDBBatchAtomicVFS from wa-sqlite. Each SQLite page becomes an IndexedDB block with the key
[path, offset, version]. The VFS fetches and writes each block on demand. The VFS layer
does transparent per-page AES encryption and batch-atomic commits. Figure 5 shows the path.
:memory:.
The only :memory: database is the throwaway staging connection. This connection
authorizes an incoming changeset before the changeset merges. This is the same staging step from
Write control, above. The store of record is always the encrypted IndexedDB VFS.
Two properties let localRbac run from file://. Most persistent-SQLite options cannot run there:
-
The VFS is Asyncify-based. Thus it does not use
SharedArrayBuffer, and it does not depend on theCOOP/COEPcross-origin-isolation headers. absurd-sql and threaded-OPFS builds must have those headers. -
The
crsqlite.mjsWASM is base64-inlined as adata:URI. Thus the engine opens with zero network fetches.
The result is a demand-paged, encrypted, offline database in a double-clicked HTML file. The standalone
write-up Demand-paged SQLite from file://
describes only this VFS, without the encryption or the CRDT.
The binary: built here, not installed
The engine underneath is not the published @vlcn.io/wa-sqlite.
sqlite-wasm-build compiles the engine from
pinned source. It uses the base variant (cr-sqlite + FTS5). This variant is the recipe that
reproduces the published package.
This matters here more than in other projects, because the build sets
SQLITE_OMIT_LOAD_EXTENSION. Nothing can load at runtime. Thus every capability
must be in the binary. If you install the published build, you accept the answer of another person
to one question: what extensions does this database have? The whole argument of this project is that the
rules must survive a copy of the file. Thus it is strange to leave that answer to a registry.
The variant declares itself under the same package name and exposes the same import specifiers. Thus the change to the variant moved one line and no source:
"@vlcn.io/wa-sqlite": "file:../../../sqlite-wasm-build/dist/base"
The geo variant (R*Tree, Geopoly) or the vec variant (sqlite-vec) is the same line
with a different word. This is the reason to build them as separate variants and not as one binary with
everything. One binary, every extension
describes the build, the four variants, and the six toolchain failures behind them.
sha256sum node_modules/@vlcn.io/wa-sqlite/crsqlite.wasm dist/base/crsqlite.wasm. The swap can
look correct while the registry binary still loads. @vlcn.io/crsqlite-wasm was a dependency that
nothing imported, and it pulled the published wa-sqlite in transitively. Then npm hoisted that
package above the local file: dependency, with no warning.
The storage engine — line items
packages/datalayer/src/engine/ · localRbac@fcc4079
boot-browser.tsL20–L25:openMainregistersEncryptedIDBVFSand opens/<name>.dbthrough it.openStagingis the throwaway:memory:db.boot-browser.tsL1–L5: the design note in prose. The persistent main is the encrypted IndexedDB VFS, the staging is:memory:, and the wasm is inline so that it "works onfile://".EncryptedIDBVFS.jsL2: vendored from@vlcn.io/wa-sqlite/src/examples/IDBBatchAtomicVFS.js.EncryptedIDBVFS.jsL34: each page is aFileBlockIndexedDB object with the key[path, offset, version].EncryptedIDBVFS.jsL220: the VFS is Asyncify (async I/O, noSharedArrayBuffer).sqlite.tsL111–L116: imports@vlcn.io/wa-sqlite/dist/crsqlite.mjs.openMemory()is the:memory:staging connection.
The parts of the datalayer
The engine is a headless package, and an app connects it to a UI. By design, the engine is resource-agnostic and column-agnostic. Records are a generic entity-attribute-value store. The name "notes" is a convention of the demo on top of that store. These are the parts:
The engine
packages/datalayer/src/engine/crengine.ts
This file holds the full compartmented-RBAC mechanism:
genesis(become the admin)grantandrevokeputRecordandlistRecordsimportChangeset, with the staging-authorize logicrunConsensusandrebuildSliceFor
The essential schema is eight small tables: adminroot, identity,
grantrec, dekver, keywrap, cell, archived,
and checkpoint. All of them are cr-sqlite CRRs. The exception is a local watermark.
Threshold custody
packages/datalayer/src/vault.ts · crypto/secret-sharing/shamir.ts
sealVault, openVault, and produceContribution do the k-of-n seal and
the distributed unlock. The second file holds the Shamir split and the Shamir combine in the field GF(2⁸).
tests/{shamir,vault}.test.ts has end-to-end tests for this part.
Consensus and compartments
packages/datalayer/src/consensus/ · compartment/
The consensus folder holds pure functions for the checkpoint chain: tip,
chainFrom, isDescendant, and converged. The compartment folder holds
the compartment fold. The fold does an LWW merge of the decrypted cells from all the resources that a
viewer can read. The two parts have no side effects, and you can test each part independently.
Crypto that you can replace
packages/datalayer/src/crypto/{kdf,signing,sealing,aead,hash,secret-sharing}/
Each cryptographic task is a folder. The index.ts file of the folder selects one
implementation. Thus you can replace a protocol with a change to one line. These functions are all native:
- Signatures: WebCrypto Ed25519
- Seals: X25519
- Record encryption: AES-256-GCM
- Hash and derivation: SHA-256/HKDF
Only two parts use WASM/noble. The first part is the memory-hard keystore KDF (Argon2id,
through hash-wasm). The second part is the one sync-critical page cipher. Each WASM blob is inline as
base64. Thus there are zero network fetches, and this is also true for
file://.
An identity that you hold and the app never stores
packages/datalayer/src/crypto/keystore.ts
An identity is a generated keypair. An Argon2id-encrypted keystore file wraps the keypair. The user downloads this file and keeps it. The app never writes the identity to IndexedDB.
The provision method is a public-key card exchange. You export a self-signed
{pub, name, xpub, sig} card, and the admin grants to that key. No private material goes from
one person to another person at any time.
The notes demo and the UI
packages/app/src/{notes.ts,ui.ts} · public/demo.html
This is the demo layer. notes.ts holds the full convention: a note is a record with a title
and a body in the notes resource. (It is approximately 20 lines.) demo.html shows
the built app three times in iframes: as admin, as writer, and as reader. Thus you can see role
enforcement, offline edits, and the merge on one screen.
Run the demo
The three-user demo is one HTML file, and the demo page loads it three times. The result is three machines that never communicate. Do these steps:
- Grant access to the writer.
- Deny access to the reader.
- Add a note.
- Export a delta.
- Import the delta in the adjacent frame.
The note converges, and the reader sees 🔒. All of the demo uses the real engine. The demo
simulates only the transport: it sends text between iframes as a replacement for a USB stick.
References
- rheophile10/localRbac: the source code and the description of the datalayer
- The three-user demo: one self-contained file that the demo page loads three times
- Adi Shamir, How to Share a Secret, CACM 22(11), 1979: the threshold-custody primitive
- Cargo cult: access control without enforcement is only an appearance of control
- xkcd 538, Security: the reason that no single keyholder must be a useful target for physical force
- cr-sqlite: convergent, replicated SQLite (the CRDT engine)
- wa-sqlite: async WASM SQLite with
IDBBatchAtomicVFS, the VFS that demand-pages onto IndexedDB (the storage engine, wrapped asEncryptedIDBVFS) - Demand-paged SQLite from
file://: the bare wa-sqlite and IndexedDB-VFS version, without the encryption or the CRDT - sqlite-wasm-build: the pinned-source build that this project runs on, with four variants, each a drop-in
@vlcn.io/wa-sqlite - One binary, every extension: the reason that the build exists, and the six toolchain failures on the way
- RheoServ Waumpaum · Offline HRIS: the earlier offline-CRDT work that this project continues