Binary Data (Blobs)¶

Blobs store raw binary payloads — images, meshes, buffers — under a typed BlobLayout that records the element data type (float, int, uchar, …) and its component count. A blob’s bytes live in a ValueBlob; storing one in a database yields a content-addressable ValueBlobId (the SHA-1 of layout + bytes), so identical payloads deduplicate automatically.

When to use: reach for blobs when a value is best expressed as a packed byte buffer rather than a structured Viper value. BlobLayout describes the element shape; BlobEncoder packs typed elements into a ValueBlob; BlobArray and BlobView reinterpret those bytes as a sequence of typed elements; and the database blob API (createBlob, blob, blobInfo) persists and retrieves them by id.

Quick Start¶

const {
  BlobLayout, BlobEncoder, BlobArray, BlobArrayBuilder, ValueBlob, ValueBlobId,
  Database,
} = require('@digitalsubstrate/dsviper');

// A layout: 3-component float elements (e.g. vec3 positions).
const layout = new BlobLayout('float', 3);
layout.representation();   // 'float-3'
layout.components();       // 3
layout.byteCount();        // 12  (3 * 4 bytes)

// Layouts also parse from their string form.
BlobLayout.parse('uchar-1').representation();   // 'uchar-1'

// Pack typed elements into a blob with an encoder, element by element. The Node
// BlobArray is a read-only view, so the bytes are built first, then reinterpreted.
const enc = new BlobEncoder(layout);
enc.write([1.0, 2.0, 3.0]);   // one 3-component element
enc.write([4.0, 5.0, 6.0]);
const blob = enc.endEncoding();   // -> ValueBlob

// Reinterpret the bytes as a typed array; iterate to read elements back.
const array = new BlobArray(layout, blob);
const elems = [...array];     // ValueVec rows of 3 components
elems[0].at(0);               // 1.0

// Or write the bytes whole, on a builder that owns them until build() seals
// them: layout and count are fixed at construction, so there is no resize.
const builder = new BlobArrayBuilder(layout, 2);            // 2 * 3 floats = 24 bytes
builder.copy(Buffer.from(new Float32Array([1, 2, 3, 4, 5, 6]).buffer));
const sealed = builder.build();   // -> ValueBlob; the builder is spent
builder.isBuilt();                // true

// Raw bytes round-trip through ValueBlob directly.
const raw = new ValueBlob(Buffer.from([1, 2, 3, 4]));
raw.size();                   // 4
raw.sha1();                   // hex digest of the bytes

// Persist to a database: createBlob returns a content-addressable id.
const db = Database.createInMemory();
db.beginTransaction();
const blobId = db.createBlob(layout, blob);
db.commit();

const info = db.blobInfo(blobId);
Number(info.size());                  // byte count (BlobInfo.size() is a BigInt)
info.blobLayout().representation();   // 'float-3'
db.blob(blobId).equals(blob);         // true
db.close();

Content-addressable storage: createBlob hashes layout plus bytes, so writing the same payload twice yields the same ValueBlobId and is stored once. A different layout over the same bytes is a different id.

Choosing the right class¶

Need	Class	Example
Raw byte payload	`ValueBlob`	`new ValueBlob(Buffer.from([...]))`
Element shape (data type + components)	`BlobLayout`	`new BlobLayout('float', 3)`
Pack typed elements into bytes	`BlobEncoder`	`enc.write(...)`, `enc.endEncoding()`
Fill the bytes of a blob, before it is a value	`BlobArrayBuilder` / `BlobPackBuilder`	`new BlobArrayBuilder(layout, count)`, `build()`
Read bytes back as typed elements	`BlobArray` / `BlobView`	`new BlobArray(layout, blob)`
Database reference (content-addressed)	`ValueBlobId`	`new ValueBlobId(layout, blob)`
Stored-blob metadata and stats	`BlobInfo` / `BlobStatistics`	`db.blobInfo(id)`
Stream a large blob in chunks	`BlobStream`	`db.blobStreamCreate(layout, size)`

{tip} The full database blob surface — createBlob, createBlobFromBuffer, blobIds, blob, readBlob, blobInfo, blobInfos, blobStatistics, and the blobStreamCreate/blobStreamAppend/blobStreamClose streaming API — is documented on Database. The Python counterpart of this page is Binary Data (Blobs). {/tip}

Generated from the @digitalsubstrate/dsviper TypeScript declarations (index.d.ts) by TypeDoc.

Core Classes¶

Class	Description
`BlobLayout`	Describes the layout of one element in a blob: a data type and a component count
`BlobArray`	Reads blob as a flat array of blobLayout.dataType
`BlobArrayBuilder`	Fills the bytes of one typed array, then seals them into a ValueBlob
`BlobPack`	Several memory regions packed into one blob, described by a BlobPackDescriptor
`BlobPackBuilder`	Fills the regions a BlobPackDescriptor describes, then seals them into a ValueBlob a BlobPack reads
`BlobPackDescriptor`	Describes the binary regions a BlobPack will hold
`BlobPackRegion`	One region of a BlobPack - its name, the layout of its elements, how many it holds, and where it sits in the pack’s blob

Blob I/O¶

Class	Description
`BlobStream`	The handle on a blob being written to a database past 2 GB, in chunks
`BlobEncoder`	Builds a blob element by element: write(value) appends, endEncoding() returns the finished blob
`BlobEncoderLayout`	A BlobLayout together with the Viper types it reads as
`BlobView`	Interprets the bytes of a blob through its layout
`BlobData`	The bytes of one blob, with the blobId and layout that go with them

Database Integration¶

Class	Description
`BlobGetting`	Retrieves blobs from a persistence layer
`BlobInfo`	What a store knows about one blob without reading it: its id, layout, size, SQLite rowid, and whether it is chunked (over 2 GB)
`BlobStatistics`	How many blobs a store holds and how big they are: count, total, smallest, largest, in bytes

Reference¶

class BlobLayout(dataType, components)¶

Describes the layout of one element in a blob: a data type and a component count.

Stored with a blob, the layout labels its bytes and is part of its id. Nothing checks their count against it until a BlobView or a BlobArray reads them, which refuses a size that is not a whole number of elements.

exported from index.d

Arguments:

dataType (string | null)
components (number | null)

BlobLayout.byteCount()¶

Return the size of an element in bytes.

Returns:: number

BlobLayout.compare(other)¶

Return -1, 0 or 1. Throws if other is not a BlobLayout.

Arguments:

other (BlobLayout)

Returns:

number

BlobLayout.components()¶

Return the number of components packed in one element.

Returns:: number

BlobLayout.dataType()¶

Return the code of the data type of one component, 0 to 10.

dataTypeRepresentation() renders it as uchar, ushort, uint, ulong, char, short, int, long, half, float or double, which is the spelling the constructor takes.

Returns:: number

BlobLayout.dataTypeByteCount()¶

Return the size of the data type in bytes.

Returns:: number

BlobLayout.dataTypeRepresentation()¶

Return the name of the data type - uchar, ushort, uint, ulong, char, short, int,: long, half, float or double. This is what the constructor takes, where dataType() answers a code.

Returns:: string

BlobLayout.equals(other)¶

Return true if both are equal. other of another type answers false.

Arguments:

other (unknown)

Returns:

boolean

BlobLayout.glAttribParams()¶

Return the gl.vertexAttribPointer parameter bundle for this layout. Throws for the: 64-bit integer dataTypes (‘ulong’/’long’ - no GL vertex-attribute type).

Returns:: GlAttribParams

BlobLayout.representation()¶

Return a string representation.

Returns:: string

static BlobLayout.parse(representation)¶

Return a BlobLayout by parsing representation.

representation is ‘<dataType>-<components>’ - ‘uchar-1’, ‘float-3’, ‘double-16’. Throw if it is not two parts separated by a dash, if the data type is unknown, or if the component count is zero.

Arguments:

representation (string)

Returns:

BlobLayout

class BlobArray(blobLayout, blob)¶

Reads blob as a flat array of blobLayout.dataType.

The blob must hold a whole number of elements - its size a multiple of blobLayout.byteCount() - or the constructor throws.

The binary reading of a blob: an index answers one datum, as a native scalar, and toTypedArray() hands out the whole run at once. Use BlobView to read the same bytes element by element.

Iteration walks the data as well, so it yields blobLayout.components() times what BlobView.count() reports over the same bytes.

A BlobArrayBuilder writes the bytes a blob is sealed with, and fromTypedArray reads client geometry instead.

exported from index.d

Arguments:

blobLayout (BlobLayout)
blob (ValueBlob)

BlobArray.[Symbol․iterator]()¶

Iterate the data lazily, as native scalars. blobView() iterates the elements.

Returns:: Iterator<number | bigint>

BlobArray.at(index)¶

Return the datum at the given index (out of range throws).

One datum, not one element: an element packs blobLayout.components() of them, so the index runs to the data count. A number, or a bigint for the 64-bit types, whose every bit is significant. Use blobView() to read elements as Values.

A negative index counts from the end, as Array.prototype.at does: at(-1) is the last datum. Out of range still throws.

Arguments:

index (number)

Returns:

number | bigint

BlobArray.blob()¶

Return the ValueBlob the array reads. The same bytes, not a copy - unlike: toTypedArray().

Returns:: ValueBlob

BlobArray.blobLayout()¶

Return the BlobLayout of one element: its data type and how many components it: packs.

Returns:: BlobLayout

BlobArray.blobView()¶

Return a BlobView over the same bytes, indexed by element rather than by data.

Returns:: BlobView

BlobArray.dataCount()¶

Return how many data the array holds: an element packs blobLayout.components(): data, and an index addresses one datum.

Returns:: number

BlobArray.toTypedArray()¶

Project the whole blob as a flat TypedArray of its scalar components (copy-first,: GPU-ready). The kind follows the layout dataType. Requires a little-endian host.

Returns:: BlobTypedArray

static BlobArray.fromTypedArray(blobLayout, typedArray)¶

Build a BlobArray from a TypedArray (client geometry -> Viper). The TypedArray kind: must match the layout dataType, and its byte length must be a whole number of elements (multiple of the layout stride). Requires a little-endian host.

Arguments:

blobLayout (BlobLayout)
typedArray (BlobTypedArray)

Returns:

BlobArray

class BlobArrayBuilder(blobLayout, count)¶

Fills the bytes of one typed array, then seals them into a ValueBlob.

Write with copy(); build() hands the bytes over once. A blob’s bytes never change afterwards.

It is constructed in ELEMENTS and measured in DATA, which are the same number only when an element packs one datum: dataCount() is count() times blobLayout.components(), and byteCount() is dataCount() times the data type’s width. copy() takes the whole array at once, so the count a caller must get right is the TypedArray’s length - dataCount(), not count().

exported from index.d

Arguments:

blobLayout (BlobLayout)
count (number)

BlobArrayBuilder.blobLayout()¶

Return the BlobLayout of one element: its data type and how many components it: packs.

Returns:: BlobLayout

BlobArrayBuilder.build()¶

Return the ValueBlob of the bytes written so far, and empty the builder.

The bytes are moved, not copied, so a value never shares them with a writer: every later write is refused.

Returns:: ValueBlob

BlobArrayBuilder.byteCount()¶

Return how many bytes the array occupies: dataCount() times: blobLayout.dataTypeByteCount().

Returns:: number

BlobArrayBuilder.copy(buffer)¶

Write buffer over the whole array: byteCount() bytes, which is a TypedArray of: dataCount() items of the layout data type, not count().

Arguments:

buffer (BlobTypedArray | ArrayBuffer | Buffer<ArrayBufferLike>)

BlobArrayBuilder.count()¶

Return how many elements the array holds, fixed at construction.

This is NOT how many values the array stores: an element packs blobLayout.components() data. dataCount() is that total.

Returns:: number

BlobArrayBuilder.dataCount()¶

Return how many data the array holds: count() times blobLayout.components().

The length the TypedArray given to copy() must have.

Returns:: number

BlobArrayBuilder.isBuilt()¶

Return whether build() has handed the bytes over to a blob.

Returns:: boolean

class BlobPack(descriptor)¶

Several memory regions packed into one blob, described by a BlobPackDescriptor.

Write with a BlobPackBuilder, whose build() answers the blob; fromBlob() reads that blob back. query() or check() then answers a BlobPackRegion, which says where its bytes sit - see its blob().

new BlobPack(descriptor) builds a pack whose regions are all zeros: the pack a BlobPackBuilder over descriptor answers before anything is written.

exported from index.d

Arguments:

descriptor (BlobPackDescriptor)

BlobPack.[Symbol․iterator]()¶

Iterate the region names, in registration order.

Returns:: Iterator<string>

BlobPack.blob()¶

Return the blob used to encode the blob pack.

Returns:: ValueBlob

BlobPack.check(name)¶

Return the BlobPackRegion registered under name, or throw if the pack has no such: region.

Arguments:

name (string)

Returns:

BlobPackRegion

BlobPack.query(name)¶

Return the BlobPackRegion registered under name, or undefined if the pack has no such: region.

Arguments:

name (string)

Returns:

BlobPackRegion | undefined

BlobPack.regions()¶

Return the BlobPackRegion list, in the order the descriptor registered them.

Returns:: BlobPackRegion[]

static BlobPack.fromBlob(blob)¶

Return a BlobPack from a blob or throw.

Arguments:

blob (ValueBlob)

Returns:

BlobPack

class BlobPackBuilder(descriptor)¶

Fills the regions a BlobPackDescriptor describes, then seals them into a

ValueBlob a BlobPack reads.

copy(name, buffer) writes one region whole; build() hands the bytes over once, and a blob’s bytes never change afterwards.

exported from index.d

Arguments:

descriptor (BlobPackDescriptor)

BlobPackBuilder.[Symbol․iterator]()¶

Iterate the region names, in registration order.

Returns:: Iterator<string>

BlobPackBuilder.build()¶

Return the ValueBlob of the regions written so far, and empty the builder.

Read it back with BlobPack.fromBlob. The bytes are moved, not copied, so a value never shares them with a writer: every later write is refused.

Returns:: ValueBlob

BlobPackBuilder.byteCount(name)¶

Return how many bytes the region name occupies, or the whole pack when name is: undefined.

Arguments:

name (string | null)

Returns:

number

BlobPackBuilder.copy(name, buffer)¶

Write buffer over the region name; its size must match that region.

Arguments:

name (string)
buffer (BlobTypedArray | ArrayBuffer | Buffer<ArrayBufferLike>)

BlobPackBuilder.isBuilt()¶

Return whether build() has handed the bytes over to a blob.

Returns:: boolean

class BlobPackDescriptor()¶

Describes the binary regions a BlobPack will hold.

exported from index.d

BlobPackDescriptor.addRegion(name, blobLayout, count)¶

Throw if name is empty, longer than 31 bytes or already registered, or if count is zero.

Arguments:

name (string)
blobLayout (BlobLayout)
count (number)

class BlobPackRegion()¶

One region of a BlobPack - its name, the layout of its elements, how many it

holds, and where it sits in the pack’s blob.

It describes a region; it does not write one. The bytes are filled before the pack is a value, on a BlobPackBuilder.

Note: Not directly instantiable.

exported from index.d

BlobPackRegion.at(index)¶

Return one datum of the region, as BlobArray.at does: a number, or a bigint for the: 64-bit types. The index runs to dataCount(); a negative one counts from the end. Out of range throws.

Arguments:

index (number)

Returns:

number | bigint

BlobPackRegion.blob()¶

Return the whole blob of the BlobPack this region belongs to.

The region is the slice of it that starts at offset() and runs byteCount() bytes; a BlobArray or a BlobView built on a ValueBlob of that slice reads it through blobLayout().

Returns:: ValueBlob

BlobPackRegion.blobLayout()¶

Return the BlobLayout of one element of the region.

Returns:: BlobLayout

BlobPackRegion.byteCount()¶

Return the size of the region in bytes.

Returns:: number

BlobPackRegion.count()¶

Return the number of elements the region holds.

Returns:: number

BlobPackRegion.dataCount()¶

Return the number of data the region holds - count() elements of as many: components as the layout packs.

Returns:: number

BlobPackRegion.name()¶

Return the name the region was registered under, at most 31 bytes.

Returns:: string

BlobPackRegion.offset()¶

Return where the region starts in the pack blob, in bytes.

Returns:: number

class BlobStream()¶

The handle on a blob being written to a database past 2 GB, in chunks.

It reports where the write stands - offset, remaining, isClosed - and its blobId once the last byte is in. It moves nothing itself. The three CommitDatabase methods blobStreamCreate, blobStreamAppend and blobStreamClose do the writing.

Note: Not directly instantiable.

exported from index.d

BlobStream.blobId()¶

Return the ValueBlobId of the written blob.

It is computed from the blob’s layout and every byte appended. Throw while the stream is open: the hash is only complete once every byte has been written.

Returns:: ValueBlobId

BlobStream.isClosed()¶

Return true if the stream is closed.

Returns:: boolean

BlobStream.offset()¶

Return the current offset.

Returns:: number

BlobStream.remaining()¶

Return the remaining bytes count to fill the blob.

Returns:: number

class BlobEncoder(blobLayout)¶

Builds a blob element by element: write(value) appends, endEncoding(): returns the finished blob.

exported from index.d

Arguments:

blobLayout (BlobLayout)

BlobEncoder.blobLayout()¶

Return the layout of the blob.

Returns:: BlobLayout

BlobEncoder.encoderLayout()¶

Return the BlobEncoderLayout: the blob layout together with the Viper types write(): accepts for one element and for one component.

Returns:: BlobEncoderLayout

BlobEncoder.endEncoding()¶

Return the encoded blob and end the encoding.

A write after this throws.

Returns:: ValueBlob

BlobEncoder.write(value)¶

Write one element at the end of the blob.

value is a number when the layout packs one component, and an array of blobLayout.components() of them otherwise - or a value of encoderLayout().type(). A value of another type, or an array of another length, throws.

Arguments:

value (InputValue)

class BlobEncoderLayout()¶

A BlobLayout together with the Viper types it reads as.

type() is the Viper type of one element - vec<float, 3> for a float-3 layout - and elementType() the type of one component, float.

Note: Not directly instantiable.

exported from index.d

BlobEncoderLayout.blobLayout()¶

Return the layout.

Returns:: BlobLayout

BlobEncoderLayout.elementType()¶

Return the type of one component.

Returns:: NumericType

BlobEncoderLayout.type()¶

Return the type of one element.

Returns:: TypeVec | NumericType

class BlobView(blobLayout, blob)¶

Interprets the bytes of a blob through its layout.

at() and iteration walk ELEMENTS: one element is a native number when the layout packs one component, a ValueVec of blobLayout.components() values otherwise. BlobArray reads the same bytes one datum at a time.

The blob must hold a whole number of elements - its size a multiple of blobLayout.byteCount() - or the constructor throws.

exported from index.d

Arguments:

blobLayout (BlobLayout)
blob (ValueBlob)

BlobView.[Symbol․iterator]()¶

Iterate the decoded elements lazily.

Returns:: Iterator<OutputValue>

BlobView.at(index)¶

Return the element at the given index (out of range throws).

A negative index counts from the end, as Array.prototype.at does: at(-1) is the last element. Out of range still throws.

Arguments:

index (number)

Returns:

OutputValue

BlobView.blob()¶

Return the ValueBlob the view reads. The bytes are shared, not copied.

Returns:: ValueBlob

BlobView.blobLayout()¶

Return the BlobLayout of one element: its data type and how many components it: packs.

Returns:: BlobLayout

BlobView.count()¶

Return the number of elements in the blob.

Returns:: number

BlobView.dataCount()¶

Return the number of data in the blob - count() elements of as many components as: the layout packs.

Returns:: number

BlobView.encoderLayout()¶

Return the BlobEncoderLayout: the blob layout together with the Viper types an: element and a component decode to.

Returns:: BlobEncoderLayout

BlobView.toTypedArray()¶

Project the whole blob as a flat TypedArray of its scalar components (copy-first:: a fresh, aligned buffer - safe to keep and upload to the GPU). The kind follows the layout dataType (float -> Float32Array, half -> Uint16Array, …). Requires a little-endian host.

Returns:: BlobTypedArray

class BlobData()¶

The bytes of one blob, with the blobId and layout that go with them.

What one database hands another when they synchronize.

Note: Not directly instantiable.

exported from index.d

BlobData.blob()¶

Return the bytes of the blob.

Returns:: ValueBlob

BlobData.blobId()¶

Return the ValueBlobId, computed from the immutable content and the layout.

Returns:: ValueBlobId

BlobData.blobLayout()¶

Return the layout of the blob.

Returns:: BlobLayout

BlobData.size()¶

Return the size of the blob.

Returns:: number

class BlobGetting()¶

Retrieves blobs from a persistence layer.

Note: Not directly instantiable.

exported from index.d

BlobGetting.blob(blobId)¶

Return the bytes of the blob blobId names, or undefined if it is not held here.

Throws past 2 GB, where the bytes cannot be handed over in one piece. Ask blobInfo(blobId).chunked() first, and read such a blob with readBlob.

A blob reserved and not yet sealed - createZeroBlob, then freezeBlob, on Databasing or CommitDatabasing - throws; blobInfo answers undefined for it meanwhile.

Arguments:

blobId (ValueBlobId)

Returns:

ValueBlob | undefined

BlobGetting.blobIds()¶

Return the distinct blobId of every sealed blob.

A blob reserved and not yet sealed - createZeroBlob, then freezeBlob, on Databasing or CommitDatabasing - is left out.

Returns:: ValueSet<ValueBlobId>

BlobGetting.blobInfo(blobId)¶

Return the BlobInfo for blobId, or undefined if there is no such blob.

It gives the layout, the size, whether the blob is chunked, and the rowid that reaches it. A blob reserved and not yet sealed answers undefined - createZeroBlob, then freezeBlob, on Databasing or CommitDatabasing.

Arguments:

blobId (ValueBlobId)

Returns:

BlobInfo | undefined

BlobGetting.blobInfos(blobIds)¶

Return one BlobInfo per blobId of blobIds this source holds.

An unknown blobId is left out, so the list may be shorter than blobIds.

Arguments:

blobIds (ValueSet<ValueBlobId> | ValueBlobId[])

Returns:

BlobInfo[]

BlobGetting.blobStatistics()¶

Return the statistics for blobs.

Returns:: BlobStatistics

BlobGetting.readBlob(blobId, size, offset)¶

Return size bytes of the blob blobId names, starting at offset.

The read side of the blob stream that wrote it, and the only way to read a blob past 2 GB, which blob() refuses.

Arguments:

blobId (ValueBlobId)
size (number | bigint)
offset (number | bigint)

Returns:

ValueBlob

class BlobInfo()¶

What a store knows about one blob without reading it: its id, layout,

size, SQLite rowid, and whether it is chunked (over 2 GB).

Note: Not directly instantiable.

exported from index.d

BlobInfo.blobId()¶

Return the identifier.

Returns:: ValueBlobId

BlobInfo.blobLayout()¶

Return the layout.

Returns:: BlobLayout

BlobInfo.chunked()¶

Return true if the blob is stored in several chunks, which the database does above: 2 GB. Such a blob is written with blobStreamCreate, blobStreamAppend and blobStreamClose, and read with readBlob; blob() refuses it.

Returns:: boolean

BlobInfo.rowId()¶

Return the rowid that reaches the blob in its store.

Returns:: number

BlobInfo.size()¶

Return the size of the blob in bytes.

Returns:: number

class BlobStatistics()¶

How many blobs a store holds and how big they are: count, total, smallest,

largest, in bytes.

Note: Not directly instantiable.

exported from index.d

BlobStatistics.count()¶

Return the number of blobs.

Returns:: number

BlobStatistics.maxSize()¶

Return the size of the biggest blob in bytes.

Returns:: number

BlobStatistics.minSize()¶

Return the size of the smallest blob in bytes.

Returns:: number

BlobStatistics.totalSize()¶

Return the total size in bytes.

Returns:: number