Skip to main content

CRDT LF SQLite

crdt_lf_sqlite_badge pub points pub likes codecov ci_badge License: MIT pub publisher

docs_badge A sqlite3 storage implementation for CRDT LF objects, providing efficient persistence for Change and Snapshot objects with document-scoped organization in a single SQLite database.

Features​

  • Compact Binary Storage: Change and Snapshot are persisted as the self-describing binary blobs produced by crdt_lf's native toBytes() / fromBytes() methods
  • Single Database, Many Documents: one database holds the changes, snapshots and peers tables
  • Document-Scoped Storage: utilities that organize data by document ID for better isolation and querying
  • Nothing suspends: sqlite3 is synchronous (FFI), and every storage method here says so in its return type. That is also what lets CRDTDocumentPersistence.openSync restore a document before it returns

Quick Start​

1. Open a database​

import 'package:crdt_lf_sqlite/crdt_lf_sqlite.dart';

void main() {
  // Open (or create) a database file...
  final storage = CRDTSqlite.open('./my_app_data.db');

  // ...or an in-memory database (useful for tests):
  final memory = CRDTSqlite.memory();

  // Your app code here...

  storage.close();
}

2. Document-scoped storage​

import 'package:crdt_lf/crdt_lf.dart';
import 'package:crdt_lf_sqlite/crdt_lf_sqlite.dart';

const documentId = 'my-document-id';

final storage = CRDTSqlite.open('./my_app_data.db');

// Open storage for a specific document
final changeStorage = storage.changeStorageForDocument(documentId);
final snapshotStorage = storage.snapshotStorageForDocument(documentId);

// Or open both at once
final documentStorage = storage.storageForDocument(documentId);

Many documents in one place​

CRDTSqlite is a CRDTStorageBackend: it lists the documents it holds, hands out the storages of each one, and deletes one whole. Code written against that interface runs on any adapter, so an app can change backend without changing anything but the line that opens it.

final backend = CRDTSqlite.open('app.db');

for (final documentId in backend.documentIds) {
  final note = backend.readDocument(documentId);
  // ...show it in a list
}

backend.deleteDocument('doc-123'); // changes, snapshots and identity
backend.close();

Keeping a whole document on disk​

Most apps do not call these methods by hand. openDocument reads the document back — its stored identity included — and follows it from there:

final note = await backend.openDocument(documentId);
final text = CRDTFugueTextHandler(note.document, 'body');

Everything written from there on is stored. The backend has the read-only half too: readDocument(id) for a preview or a list, documentAt(id, version) for the document as it was, copyDocumentTo(other, id) for a backup or a move to another adapter.

It comes from crdt_lf_persistence, which this package re-exports. See that README for the offline-first rules.

storageForDocument hands back a CRDTSqliteDocumentStorage, which backs transaction() with a real SQLite transaction: a prune drops the covered changes and rewrites the survivors, and either all of it lands or none of it does. It is built on savepoints, so a batch that opens a transaction of its own nests inside instead of failing.

close() on it does nothing on purpose. One database file holds every document, so the connection is CRDTSqlite.close()'s to release.

Document-Scoped Storage​

Data for different documents lives in the same tables and is isolated through the document_id column.

Every method answers on the spot. sqlite3 is synchronous underneath, and the return types say so: getChanges() gives a List<Change>, saveChange() gives void. The shared contract asks only for a FutureOr, so the same code still runs on drift — but here there is nothing to await.

CRDTSqliteChangeStorage​

Manages Change objects for a specific document:

final changeStorage = storage.changeStorageForDocument('doc-123');

// Save individual changes
changeStorage.saveChange(change);

// Batch save multiple changes
changeStorage.saveChanges([change1, change2, change3]);

// Load all changes for the document
final changes = changeStorage.getChanges();

// Or only part of the log, by version vector
final missing = changeStorage.getChanges(newerThan: theirVersion);
final past = changeStorage.getChanges(upTo: oldVersion);

// Delete changes
changeStorage.deleteChange(change);
changeStorage.deleteChanges([change1, change2]);

// Storage info
print('Total changes: ${changeStorage.count}');

CRDTSqliteSnapshotStorage​

Manages Snapshot objects for a specific document:

final snapshotStorage = storage.snapshotStorageForDocument('doc-123');

// Save snapshots
snapshotStorage.saveSnapshot(snapshot);
snapshotStorage.saveSnapshots([snapshot1, snapshot2]);

// Retrieve snapshots
final snapshot = snapshotStorage.getSnapshot('snapshot-id');
final allSnapshots = snapshotStorage.getSnapshots();

// Check existence
if (snapshotStorage.containsSnapshot('snapshot-id')) {
  // Snapshot exists
}

CRDTSqlitePeerIdStorage​

Keeps the PeerId the document writes under, in the peers table. Without it CRDTDocument mints a new author on every restart, and the version vector grows by one peer per session.

Read it before building the document — the id has to exist first:

final peers = storage.peerIdStorageForDocument('doc-123');

final document = CRDTDocument(
  documentId: 'doc-123',
  peerId: peers.loadOrCreate() as PeerId, // synchronous here
);

How Data Is Stored​

Both Change and Snapshot are stored as opaque binary blobs using the self-describing format provided by crdt_lf (toBytes() / fromBytes()). The schema is three tables:

CREATE TABLE changes (
  document_id TEXT    NOT NULL,
  author      TEXT    NOT NULL,
  hlc_l       INTEGER NOT NULL,
  hlc_c       INTEGER NOT NULL,
  bytes       BLOB    NOT NULL,
  PRIMARY KEY (document_id, author, hlc_l, hlc_c)
);

CREATE TABLE snapshots (
  document_id TEXT NOT NULL,
  snapshot_id TEXT NOT NULL,
  bytes       BLOB NOT NULL,
  PRIMARY KEY (document_id, snapshot_id)
);

CREATE TABLE peers (
  document_id TEXT NOT NULL,
  peer_id     TEXT NOT NULL,
  PRIMARY KEY (document_id)
);

A change is named by its author and its clock, not by one text id. An OperationId is a peer and a clock, and kept apart SQL can compare it, which is what a version vector asks. The primary key is then already the index that comparison wants. The clock takes two columns because l is 48 bits and c is 16: together they stay inside the 53 bits an integer keeps exactly in JavaScript.

Snapshots are keyed by snapshot.id, and peers holds one row per document: the PeerId this device writes it under.

Schema version​

The database carries its schema version in PRAGMA user_version, and this build writes version 2. A database written by 0.2.0 is version 1: it has no peers table and names a change with a single change_id column. The first open rebuilds it — SQLite cannot change a primary key in place — inside one savepoint, so half a rebuilt table is never left behind. The split reads the old column and not the stored bytes, so a change whose blob this build cannot decode migrates all the same. Every later open sees the version and returns without touching the database.

Examples​

A complete example is available here.

Storage Management​

// Delete all data for a specific document
backend.deleteDocument('doc-123');

// Close the database and release resources
storage.close();

Roadmap​

A roadmap is available in the project page. The roadmap provides a high-level overview of the project's goals and the current status of the project.

Apps​

Packages​

Other bricks of the crdt "system" are: