Database

Database provides transactional, CRUD-style persistence for Viper documents — backed by SQLite on disk or held entirely in memory. Documents are addressed by a typed (attachment, key, value) triple; writes are bracketed by a transaction, reads return a ValueOptional that you peel with .unwrap().

This page also covers the helpers that move a Database around: the faithful DatabaseCopier (one Database into another, blobs and all) and the DatabaseToCommitDatabaseConverter (importing a flat Database into the versioned tier).

When to use

Reach for Database when you want a plain document store with no history — the latest value for each key, nothing more. If you need an append-only mutation DAG with full version history, use CommitDatabase instead; see Commit Database. The two tiers interconvert through the converter classes below.

Use case

Class

Note

Simple CRUD, no history

Database

SQLite file or in-memory

Need version history

CommitDatabase

See Commit Database

Copy a Database faithfully

DatabaseCopier

Replicates every blob, orphans included

Import a Database into the versioned tier

DatabaseToCommitDatabaseConverter

Copies only referenced blobs

Quick Start

const {
  Database, Definitions, NameSpace, ValueUUId, Type,
} = require('@digitalsubstrate/dsviper');

// Build a schema and create or open a database
const defs = new Definitions();
const ns = new NameSpace(ValueUUId.create(), 'Demo');
const concept = defs.createConcept(ns, 'Item');
const att = defs.createAttachment(ns, 'Count', concept, Type.INT64);

const db = Database.createInMemory();        // or Database.create('data.db')
db.extendDefinitions(defs.const());          // seed/grow the schema incrementally

// Write — requires a transaction
const key = att.createKey();
db.beginTransaction();
db.set(att, key, 42);
db.commit();                                 // db.rollback() discards instead

// Read — get() returns a ValueOptional
const result = db.get(att, key);
if (!result.isNil()) {
  const count = result.unwrap();             // INT64 comes back as a bigint
  console.log(count === 42n);                // true
}

// Enumerate keys, then delete (inside a transaction)
for (const k of db.keys(att)) { /* ... */ }

db.beginTransaction();
db.delete(att, key);                         // no error if the key is absent
db.commit();

db.close();

Open an existing database read-only with Database.open(path, true), and probe a file first with Database.isCompatible(path). A failed operation (opening a missing file, writing through a read-only handle, using a closed database) throws a JS Error whose .name === 'ViperError'.

For the same API in Python, see Database.

Generated from the @digitalsubstrate/dsviper TypeScript declarations (index.d.ts) by TypeDoc.

Core Classes

Class

Description

Database

A Database is a CRUD like transactional database

DatabaseSQLite

A low-level class used to represent a CRUD like database based on SQLite3 through the Databasing interface

DatabaseRemote

Accesses a CRUD like database on a remote repository, through the Databasing interface

Databasing

Abstracts the persistence layer behind a CRUD like database

Low-Level

Class

Description

SQLite

The settings and limits of one database file and of its SQLite3 build

Reference

class Database()
A Database is a CRUD like transactional database. Use the static factory method

createInMemory(), create(…), open(…), connect(…) or connectLocal(…).

Transactional is a precondition, not a feature: reads need no transaction, and every WRITE requires one already open - set, delete and the blob operations throw without writing anything when there is none. extendDefinitions is the exception in both directions, opening an exclusive transaction of its own. A CommitDatabase does NOT work this way: its writes need no transaction at all.

One file may be open in several processes; SQLite locks it. While a transaction writes, another writer waits, and in ‘Exclusive’ mode a reader waits too - for up to 10 seconds, after which the call throws ViperError: the database is busy. Otherwise a reader sees the last committed state.

Note: Not directly instantiable.

exported from index.d

Database.[Symbol․dispose]()

Release the resource at scope exit via using (TC39 disposable); delegates to close(). Idempotent.

Database.attachmentGetting()

Return the AttachmentGetting interface.

Returns:

AttachmentGetting

Database.beginTransaction(mode)

Begin a transaction in mode ‘Deferred’ (the default), ‘Immediate’ or ‘Exclusive’.

Another spelling throws.

Transactions do not nest: calling this while inTransaction() is true throws, and the transaction already open is untouched.

Arguments:
  • mode (string | null)

Database.blob(blobId)

Return the bytes of the blob blobId names, or undefined if it is not held here.

Throws past 2 GB, where the bytes cannot be handed over in one piece. Ask blobInfo(blobId).chunked() first, and read such a blob with readBlob.

A blob databasing().createZeroBlob reserved throws until databasing().freezeBlob seals it; blobInfo answers undefined for it meanwhile.

Arguments:
  • blobId (ValueBlobId)

Returns:

ValueBlob | undefined

Database.blobGetting()

Return the BlobGetting interface.

Returns:

BlobGetting

Database.blobIds()

Return the distinct blobId of every sealed blob.

A blob databasing().createZeroBlob reserved joins it once databasing().freezeBlob seals it.

Returns:

ValueSet<ValueBlobId>

Database.blobInfo(blobId)

Return the BlobInfo for blobId, or undefined if the database does not hold it.

It gives the layout, the size and whether the blob is chunked. A blob databasing().createZeroBlob reserved answers undefined until databasing().freezeBlob seals it.

Arguments:
  • blobId (ValueBlobId)

Returns:

BlobInfo | undefined

Database.blobInfos(blobIds)

Return one BlobInfo per blobId of blobIds the database holds.

An unknown blobId is left out, so the list may be shorter than blobIds.

Arguments:
  • blobIds (ValueSet<ValueBlobId> | ValueBlobId[])

Returns:

BlobInfo[]

Database.blobStatistics()

Return the statistics for blobs.

Returns:

BlobStatistics

Database.blobStreamAppend(blobStream, blob)

Append blob to blobStream, after the bytes already written.

Requires an open transaction, and throws without writing when there is none.

A failure here closes the stream and discards what it collected, so nothing partial is left behind and there is nothing to clean up.

Arguments:
  • blobStream (BlobStream)

  • blob (ValueBlob)

Database.blobStreamClose(blobStream)

Close blobStream and return its blobId.

The blobId is computed from everything written into it. Requires an open transaction, and throws without writing when there is none.

A failure here closes the stream and discards what it collected, so nothing partial is left behind and there is nothing to clean up.

Arguments:
  • blobStream (BlobStream)

Returns:

ValueBlobId

Database.blobStreamCreate(blobLayout, size)

Return a BlobStream to fill a blob of size bytes laid out by blobLayout.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobLayout (BlobLayout)

  • size (number)

Returns:

BlobStream

Database.close()

Close the database.

Database.codecName()

Return the name of the codec.

Returns:

string

Database.commit()

Commit the transaction, so what it wrote stays.

Throws when no transaction is open.

Database.createBlob(blobLayout, blob)

Store blob under blobLayout.

Returns the blobId computed from its content and that layout.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobLayout (BlobLayout)

  • blob (ValueBlob)

Returns:

ValueBlobId

Database.createBlobFromBuffer(buffer)
Store the bytes of a Node Buffer, a typed array, an ArrayBuffer or a ValueBlob, and

return the blobId computed from them.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • buffer (ValueBlob | BlobTypedArray | ArrayBuffer | Buffer<ArrayBufferLike>)

Returns:

ValueBlobId

Database.databasing()

Return the Databasing interface.

Returns:

Databasing

Database.definitions()

Return the Definitions every value in the database is typed by.

It is a SNAPSHOT: extending the database afterwards does not show through the handle you hold, so ask again. A view obtained from Definitions.const() is live instead - same class, opposite liveness.

Returns:

DefinitionsConst

Database.definitionsHexdigest()

Return the SHA1 of the definitions, as a hex string.

Returns:

string

Database.delBlob(blobId)

Delete the blob blobId names, and return true if it was there.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobId (ValueBlobId)

Returns:

boolean

Database.delete(attachment, key)

Delete the document attachment holds at key.

Returns true if it was there.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

Returns:

boolean

Database.documentation()
Return the free text stored with the database at create(). One made without any

answers ‘not documented.’, one made in memory answers ‘In Memory’ - both are placeholders written in at create(), not substituted on read.

Returns:

string

Database.extendDefinitions(definitions)

Add every type and attachment of definitions the database lacks.

It opens an exclusive transaction of its own, so it throws when one is already open - the reverse of every other write here.

Arguments:
  • definitions (DefinitionsConst)

Returns:

DefinitionsExtendInfo

Database.get(attachment, key)

Return a ValueOptional over the document attachment holds at key.

Always a ValueOptional, never undefined: ask isNil() whether it holds anything, and unwrap() for the document itself.

An absent document answers a nil ValueOptional; a key of another concept than the one attachment files throws, so isNil() reports absence and never a mismatched key.

The document comes back as a copy; mutating it does not reach what is stored.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

Returns:

ValueOptional

Database.has(attachment, key)

Return true if attachment holds a document at key.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

Returns:

boolean

Database.inMemory()

Return true if the database is in memory.

Returns:

boolean

Database.inTransaction()

Return true if a transaction is running.

Returns:

boolean

Database.isClosed()

Return true if the database is closed.

Returns:

boolean

Database.keys(attachment)

Return every key attachment holds a document for.

The set is taken now: a later write does not reach it.

Arguments:
  • attachment (Attachment)

Returns:

ValueSet<ValueKey>

Database.path()

Return the file path the database was opened from.

A database made in memory answers ‘InMemory’ rather than a path, and that exact string is how the runtime recognises one, so no file may take it: creating or opening a file of that path throws. Over a socket the answer is whatever the server answers for its own database.

Returns:

string

Database.readBlob(blobId, size, offset)

Return size bytes of the blob blobId names, starting at offset.

The read side of blobStreamCreate, blobStreamAppend and blobStreamClose, and the only way to read a blob past 2 GB, which blob() refuses.

Arguments:
  • blobId (ValueBlobId)

  • size (number)

  • offset (number)

Returns:

ValueBlob

Database.rollback()

Roll back the transaction, so what it wrote is dropped.

Throws when no transaction is open.

Database.set(attachment, key, value)

Write value into attachment at key, replacing what was there.

The value is copied in: mutating it afterwards does not reach what was written, and writing the same object under two keys stores two independent documents.

Returns true when the document was written; a check that refuses throws instead of answering. It needs an attachment extendDefinitions has registered - an unregistered one throws whatever the transaction - and an open transaction, without which it throws and writes nothing.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

  • value (InputValue)

Returns:

boolean

Database.uuid()

Return the uuid identifying this database, generated at create().

Returns:

ValueUUId

static Database.connect(database, host, service)

Return the database named database, served by host on service.

Arguments:
  • database (string)

  • host (string)

  • service (string | null)

Returns:

Database

static Database.connectLocal(database, socketPath)

Return the database named database, served on the unix socket socketPath.

Arguments:
  • database (string)

  • socketPath (string)

Returns:

Database

static Database.create(filePath, documentation)

Create a database at filePath and return it.

documentation is stored in the file and read back by documentation().

Requires that nothing exist at filePath, and throws without creating anything when something does.

Arguments:
  • filePath (string)

  • documentation (string | null)

Returns:

Database

static Database.createInMemory()

Create and return a database in memory.

Returns:

Database

static Database.databases(host, service)

Return the names of the databases host serves on service.

Arguments:
  • host (string)

  • service (string | null)

Returns:

string[]

static Database.databasesLocal(socketPath)

Return the names of the databases served on the unix socket socketPath.

Arguments:
  • socketPath (string)

Returns:

string[]

static Database.isCompatible(filePath)

Return true if the file at filePath carries the expected SQL schema.

Arguments:
  • filePath (string)

Returns:

boolean

static Database.open(filePath, readonly)

Open the database at filePath and return it.

readonly opens it without allowing a write.

Requires a file at filePath, and throws when there is none.

Arguments:
  • filePath (string)

  • readonly (boolean | null)

Returns:

Database

class DatabaseSQLite()
A low-level class used to represent a CRUD like database based on SQLite3

through the Databasing interface.

Its own createInMemory(), create(…) and open(…) answer this low-level object, which adds sqlite(); Database.createInMemory(), Database.create(…) and Database.open(…) answer the high-level one.

Note: Not directly instantiable.

exported from index.d

DatabaseSQLite.[Symbol․dispose]()

Release the resource at scope exit via using (TC39 disposable); delegates to close(). Idempotent.

DatabaseSQLite.close()

Close the database.

DatabaseSQLite.databasing()

Return the Databasing interface.

Returns:

Databasing

DatabaseSQLite.isClosed()

Return true if the database is closed.

Returns:

boolean

DatabaseSQLite.sqlite()

Return the sqlite api.

Returns:

SQLite

static DatabaseSQLite.create(filePath, documentation)

Create a database at filePath and return it.

documentation is stored in the file and read back by documentation().

Requires that nothing exist at filePath, and throws without creating anything when something does.

Arguments:
  • filePath (string)

  • documentation (string | null)

Returns:

DatabaseSQLite

static DatabaseSQLite.createInMemory()

Create and return a database in memory.

Returns:

DatabaseSQLite

static DatabaseSQLite.isCompatible(filePath)

Return true if the file at filePath carries the expected SQL schema.

Arguments:
  • filePath (string)

Returns:

boolean

static DatabaseSQLite.open(filePath, readonly)

Open the database at filePath and return it.

readonly opens it without allowing a write.

Requires a file at filePath, and throws when there is none.

Arguments:
  • filePath (string)

  • readonly (boolean | null)

Returns:

DatabaseSQLite

class DatabaseRemote()

Accesses a CRUD like database on a remote repository, through the Databasing interface.

Note: Not directly instantiable.

exported from index.d

DatabaseRemote.[Symbol․dispose]()

Release the resource at scope exit via using (TC39 disposable); delegates to close(). Idempotent.

DatabaseRemote.close()

Close the database.

DatabaseRemote.databasing()

Return the Databasing interface.

Returns:

Databasing

DatabaseRemote.isClosed()

Return true if the database is closed.

Returns:

boolean

static DatabaseRemote.connect(database, host, service)

Return the database named database, served by host on service.

Arguments:
  • database (string)

  • host (string)

  • service (string | null)

Returns:

DatabaseRemote

static DatabaseRemote.connectLocal(database, socketPath)

Connect to socketPath and use the database.

Arguments:
  • database (string)

  • socketPath (string)

Returns:

DatabaseRemote

static DatabaseRemote.databases(host, service)

Return the names of the databases host serves on service.

Arguments:
  • host (string)

  • service (string | null)

Returns:

string[]

static DatabaseRemote.databasesLocal(socketPath)

Return the names of the databases served on the unix socket socketPath.

Arguments:
  • socketPath (string)

Returns:

string[]

class Databasing()

Abstracts the persistence layer behind a CRUD like database.

Reads need no transaction. Every WRITE requires one already open - set, delete and the blob operations throw without writing anything when there is none. extendDefinitions is the exception in both directions: it opens an exclusive transaction of its own, so it throws when one is already open.

Note: Not directly instantiable.

exported from index.d

Databasing.TRANSACTION_DEFERRED

type: readonly string

Databasing.TRANSACTION_EXCLUSIVE

type: readonly string

Databasing.TRANSACTION_IMMEDIATE

type: readonly string

Databasing.[Symbol․dispose]()

Release the resource at scope exit via using (TC39 disposable); delegates to close(). Idempotent.

Databasing.beginTransaction(mode)

Begin a transaction in mode ‘Deferred’ (the default), ‘Immediate’ or ‘Exclusive’.

Another spelling throws.

Transactions do not nest: calling this while inTransaction() is true throws, and the transaction already open is untouched.

Arguments:
  • mode (string | null)

Databasing.blob(blobId)

Return the bytes of the blob blobId names, or undefined if it is not held here.

Throws past 2 GB, where the bytes cannot be handed over in one piece. Ask blobInfo(blobId).chunked() first, and read such a blob with readBlob.

A blob reserved by createZeroBlob throws until freezeBlob seals it; blobInfo answers undefined for it meanwhile.

Arguments:
  • blobId (ValueBlobId)

Returns:

ValueBlob | undefined

Databasing.blobIds()

Return the distinct blobId of every sealed blob.

A blob createZeroBlob reserved joins it once freezeBlob seals it.

Returns:

ValueSet<ValueBlobId>

Databasing.blobInfo(blobId)

Return the BlobInfo for blobId, or undefined if this source does not hold it.

It gives the layout, the size and whether the blob is chunked. A blob createZeroBlob reserved answers undefined until freezeBlob seals it.

Arguments:
  • blobId (ValueBlobId)

Returns:

BlobInfo | undefined

Databasing.blobInfos(blobIds)

Return one BlobInfo per blobId of blobIds this source holds.

An unknown blobId is left out, so the array may be shorter than blobIds.

Arguments:
  • blobIds (ValueSet<ValueBlobId> | ValueBlobId[])

Returns:

BlobInfo[]

Databasing.blobStatistics()

Return the statistics for blobs.

Returns:

BlobStatistics

Databasing.blobStreamClose(streamId, blobId)

Close the stream streamId names.

What it collected is stored under blobId. Requires an open transaction, and throws without writing when there is none.

Arguments:
  • streamId (ValueUUId)

  • blobId (ValueBlobId)

Databasing.blobStreamCreate(blobLayout, size)

Open a stream for a blob of size bytes laid out by blobLayout.

Returns the uuid naming it. Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobLayout (BlobLayout)

  • size (number)

Returns:

ValueUUId

Databasing.blobStreamDelete(streamId)

Drop the stream streamId names, discarding what it collected.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • streamId (ValueUUId)

Databasing.blobStreamWrite(streamId, blob, offset)

Write blob into the stream streamId names, at offset.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • streamId (ValueUUId)

  • blob (ValueBlob)

  • offset (number)

Databasing.close()

Close the database.

Databasing.codecName()

Return the name of the codec.

Returns:

string

Databasing.commit()

Commit the transaction, so what it wrote stays.

Throws when no transaction is open.

Databasing.createBlob(blobId, blobLayout, blob)

Store blob under blobId and blobLayout.

Returns true if it was not already there. Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobId (ValueBlobId)

  • blobLayout (BlobLayout)

  • blob (ValueBlob)

Returns:

boolean

Databasing.createZeroBlob(blobId, blobLayout, size)

Reserve size zeroed bytes under blobId and blobLayout.

The blob is filled later. Returns true if it was created. Requires an open transaction, and throws without writing when there is none.

Until freezeBlob seals it, blobIds leaves it out, blobInfo answers undefined, and blob and readBlob throw.

Arguments:
  • blobId (ValueBlobId)

  • blobLayout (BlobLayout)

  • size (number)

Returns:

boolean

Databasing.dataVersion()

Return the counter another connection’s commit bumps.

An unchanged value means nothing was committed elsewhere since the last read.

Returns:

number

Databasing.definitions()

Return the Definitions every value in this source is typed by.

It is a SNAPSHOT: extending the source afterwards does not show through the handle you hold, so ask again. A view obtained from Definitions.const() is live instead - same class, opposite liveness.

Returns:

DefinitionsConst

Databasing.definitionsHexdigest()

Return the SHA1 of the definitions, as a hex string.

Returns:

string

Databasing.delBlob(blobId)

Delete the blob blobId names, and return true if it was there.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobId (ValueBlobId)

Returns:

boolean

Databasing.delete(attachment, key)

Delete the document attachment holds at key.

Returns true if it was there. Requires an open transaction, and throws without writing when there is none.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

Returns:

boolean

Databasing.documentation()
Return the free text stored with the database at create(). One made without any

answers ‘not documented.’, one made in memory answers ‘In Memory’ - both are placeholders written in at create(), not substituted on read.

Returns:

string

Databasing.extendDefinitions(other)

Add every type and attachment of other this source does not hold yet.

It opens an exclusive transaction of its own, so it throws when one is already open - the reverse of every other write here.

Arguments:
  • other (DefinitionsConst)

Returns:

DefinitionsExtendInfo

Databasing.freezeBlob(blobId)

Seal the blob blobId names against further writes.

Returns true if it was still open. Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobId (ValueBlobId)

Returns:

boolean

Databasing.get(attachment, key)

Return a ValueOptional over the document attachment holds at key.

Always a ValueOptional, never undefined: ask isNil() whether it holds anything, and unwrap() for the document itself.

An absent document answers a nil ValueOptional; a key of another concept than the one attachment files throws, so isNil() reports absence and never a mismatched key.

The document comes back as a copy; mutating it does not reach what is stored.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

Returns:

ValueOptional

Databasing.has(attachment, key)

Return true if attachment holds a document at key.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

Returns:

boolean

Databasing.inTransaction()

Return true if a transaction is running.

Returns:

boolean

Databasing.isClosed()

Return true if the database is closed.

Returns:

boolean

Databasing.keys(attachment)

Return every key attachment holds a document for.

The set is taken now: a later write does not reach it.

Arguments:
  • attachment (Attachment)

Returns:

ValueSet<ValueKey>

Databasing.path()

Return the file path behind this source.

A database made in memory answers ‘InMemory’ rather than a path, and that exact string is how the runtime recognises one, so no file may take it: creating or opening a file of that path throws. Over a socket the answer is whatever the server answers for its own database.

Returns:

string

Databasing.readBlob(blobId, size, offset)

Return size bytes of the blob blobId names, starting at offset.

The read side of blobStreamCreate, blobStreamWrite and blobStreamClose, and the only way to read a blob past 2 GB, which blob() refuses.

Arguments:
  • blobId (ValueBlobId)

  • size (number)

  • offset (number)

Returns:

ValueBlob

Databasing.rollback()

Roll back the transaction, so what it wrote is dropped.

Throws when no transaction is open.

Databasing.set(attachment, key, value)

Write value into attachment at key, replacing what was there.

The value is copied in: mutating it afterwards does not reach what was written, and writing the same object under two keys stores two independent documents.

Returns true when the document was written; a check that refuses throws instead of answering. It needs an attachment extendDefinitions has registered - an unregistered one throws whatever the transaction - and an open transaction, without which it throws and writes nothing.

Arguments:
  • attachment (Attachment)

  • key (ValueKey)

  • value (InputValue)

Returns:

boolean

Databasing.uuid()

Return the uuid identifying the database behind this source.

Returns:

ValueUUId

Databasing.writeBlob(blobId, blob, offset)

Write the bytes of blob into the stored blob blobId names, at offset.

Requires an open transaction, and throws without writing when there is none.

Arguments:
  • blobId (ValueBlobId)

  • blob (ValueBlob)

  • offset (number)

class SQLite()

The settings and limits of one database file and of its SQLite3 build.

The pragmas, page size and maximum size belong to the file; the compile options and maxLength() to the library it runs on, the same for every file.

Note: Not directly instantiable.

exported from index.d

SQLite.compileOptions()

Return the SQLite compile-time options, as [name, value] pairs.

An option set without a value, such as a feature flag, has the value ‘on’.

Returns:

[string, string][]

SQLite.getPragma(key)

Return the value of the SQLite pragma key, as an array of strings.

Arguments:
  • key (string)

Returns:

string[]

SQLite.maxDatabaseSize()
Return the largest the database file may grow to, in bytes: pageSize() times

maxPageCount().

Returns:

bigint

SQLite.maxLength()

Return the largest string or blob this SQLite3 build will handle, in bytes.

Returns:

bigint

SQLite.maxPageCount()

Return how many pages the database file may grow to.

Returns:

bigint

SQLite.pageSize()

Return the size of a database page in bytes.

Returns:

bigint

SQLite.pragmas()

Return the list of pragmas.

Returns:

string[]

SQLite.setPragma(key, value)

Set the SQLite pragma key to value.

Arguments:
  • key (string)

  • value (string)

static SQLite.isSqlite3(filePath)

Return true if the file at filePath is a SQLite3 database.

Arguments:
  • filePath (string)

Returns:

boolean