Binary data (blobs)¶

Blobs store raw binary payloads — images, meshes, buffers — under a typed BlobLayout that records the element data type (float, int, uchar, …) and its component count. A blob’s bytes live in a ValueBlob; storing one in a database yields a content-addressable ValueBlobId (the SHA-1 of layout + bytes), so identical payloads deduplicate automatically.

When to use: reach for blobs when a value is best expressed as a packed byte buffer rather than a structured Viper value. BlobLayout describes the element shape; BlobEncoder packs typed elements into a ValueBlob; BlobArray and BlobView reinterpret those bytes as a sequence of typed elements; and the database blob API (createBlob, blob, blobInfo) persists and retrieves them by id.

Quick Start¶

const {
  BlobLayout, BlobEncoder, BlobArray, ValueBlob, ValueBlobId, Database,
} = require('@digitalsubstrate/dsviper');

// A layout: 3-component float elements (e.g. vec3 positions).
const layout = new BlobLayout('float', 3);
layout.representation();   // 'float-3'
layout.components();       // 3
layout.byteCount();        // 12  (3 * 4 bytes)

// Layouts also parse from their string form.
BlobLayout.parse('uchar-1').representation();   // 'uchar-1'

// Pack typed elements into a blob with an encoder. The Node BlobArray is a
// read-only view, so build data through BlobEncoder, then reinterpret it.
const enc = new BlobEncoder(layout);
enc.write([1.0, 2.0, 3.0]);   // one 3-component element
enc.write([4.0, 5.0, 6.0]);
const blob = enc.endEncoding();   // -> ValueBlob

// Reinterpret the bytes as a typed array; iterate to read elements back.
const array = BlobArray.fromBlob(layout, blob);
const elems = [...array];     // ValueVec rows of 3 components
elems[0].at(0);               // 1.0

// Raw bytes round-trip through ValueBlob directly.
const raw = new ValueBlob(Buffer.from([1, 2, 3, 4]));
raw.size();                   // 4
raw.sha1();                   // hex digest of the bytes

// Persist to a database: createBlob returns a content-addressable id.
const db = Database.createInMemory();
db.beginTransaction();
const blobId = db.createBlob(layout, blob);
db.commit();

const info = db.blobInfo(blobId);
Number(info.size());                  // byte count (BlobInfo.size() is a BigInt)
info.blobLayout().representation();   // 'float-3'
db.blob(blobId).equals(blob);         // true
db.close();

Content-addressable storage: createBlob hashes layout plus bytes, so writing the same payload twice yields the same ValueBlobId and is stored once. A different layout over the same bytes is a different id.

Choosing the right class¶

Need	Class	Example
Raw byte payload	`ValueBlob`	`new ValueBlob(Buffer.from([...]))`
Element shape (data type + components)	`BlobLayout`	`new BlobLayout('float', 3)`
Pack typed elements into bytes	`BlobEncoder`	`enc.write(...)`, `enc.endEncoding()`
Read bytes back as typed elements	`BlobArray` / `BlobView`	`BlobArray.fromBlob(layout, blob)`
Database reference (content-addressed)	`ValueBlobId`	`new ValueBlobId(layout, blob)`
Stored-blob metadata and stats	`BlobInfo` / `BlobStatistics`	`db.blobInfo(id)`
Stream a large blob in chunks	`BlobStream`	`db.blobStreamCreate(layout, size)`

{tip} The full database blob surface — createBlob, createBlobFromBuffer, blobIds, blob, readBlob, blobInfo, blobInfos, blobStatistics, and the blobStreamCreate/blobStreamAppend/blobStreamClose streaming API — is documented on Database. The Python counterpart of this page is Binary Data (Blobs). {/tip}

Generated from the @digitalsubstrate/dsviper TypeScript declarations (index.d.ts) by TypeDoc.

Summary¶

Class	Description
`BlobArray`	A class used to reinterpret the blob of the internal BlobView as an array<blob_layout.data_type>
`BlobData`	A class used to represent the data associated with a blob during the synchronization of two databases
`BlobEncoder`	A class used to encode a blob
`BlobEncoderLayout`	A class used to represent the layout of one element
`BlobGetting`	An interface used to retrieve blobs from a persistence layer
`BlobInfo`	A class used to represent various information of a blob in the persistence layer
`BlobLayout`	A class used to describe the layout of the element in a blob
`BlobPack`	A class used to implements memory regions in one blob
`BlobPackDescriptor`	A class used to describe binary regions
`BlobPackRegion`	A class used for a region that implement the buffer protocol
`BlobStatistics`	A class used to represent the statistics about blobs
`BlobStream`	A class used to copy a huge blob (> 2GB) to a database
`BlobView`	A class used to interpret the bytes of a blob from the layout

Reference¶

class BlobArray(blobLayout, size)¶

A class used to reinterpret the blob of the internal BlobView as: an array<blob_layout.data_type>.

exported from index.d

Arguments:

blobLayout (BlobLayout)
size (number)

BlobArray.[Symbol․iterator]()¶

Iterate the decoded elements lazily.

Returns:: Iterator<OutputValue>

BlobArray.at(index)¶

Return the element at the given index (out of range throws).

Arguments:

index (number)

Returns:

OutputValue

BlobArray.blob()¶

Return the blob.

Returns:: ValueBlob

BlobArray.blobLayout()¶

Return the blob layout.

Returns:: BlobLayout

BlobArray.blobView()¶

Return the BlobView.

Returns:: BlobView

static BlobArray.fromBlob(blobLayout, blob)¶

Return a BlobBuffer from a layout and a blob.

Arguments:

blobLayout (BlobLayout)
blob (ValueBlob)

Returns:

BlobArray

class BlobData()¶

A class used to represent the data associated with a blob during the

synchronization of two databases.

Note: Not directly instantiable.

exported from index.d

BlobData.blob()¶

Return the blob.

Returns:: ValueBlob

BlobData.blobId()¶

Return the identifier the blob.

Returns:: ValueBlobId

BlobData.blobLayout()¶

Return the layout of the blob.

Returns:: BlobLayout

BlobData.size()¶

Return the size of the blob.

Returns:: number

class BlobEncoder(blobLayout)¶

A class used to encode a blob.

exported from index.d

Arguments:

blobLayout (BlobLayout)

BlobEncoder.blobLayout()¶

Return the layout of the blob.

Returns:: BlobLayout

BlobEncoder.encoderLayout()¶

Return the encoder layout.

Returns:: BlobEncoderLayout

BlobEncoder.endEncoding()¶

Return the encoded blob.

Returns:: ValueBlob

BlobEncoder.write(value)¶

Write a value at the end of the blob.: The parameter value is an int, a real or a tuple compatible with the layout.

Arguments:

value (InputValue)

class BlobEncoderLayout()¶

A class used to represent the layout of one element. The blob is interpreted as

an immutable vector<blob_layout>.

Note: Not directly instantiable.

exported from index.d

BlobEncoderLayout.blobLayout()¶

Return the layout.

Returns:: BlobLayout

BlobEncoderLayout.elementType()¶

Return the type of the element.

Returns:: NumericType

BlobEncoderLayout.type()¶

Return the type of the value for an element.

Returns:: TypeVec | NumericType

class BlobGetting()¶

An interface used to retrieve blobs from a persistence layer.

Note: Not directly instantiable.

exported from index.d

BlobGetting.blob(blobId)¶

Return a blob or null.

Arguments:

blobId (ValueBlobId)

Returns:

ValueBlob

BlobGetting.blobIds()¶

Return the set of blob_id.

Returns:: ValueBlobId[]

BlobGetting.blobInfo(blobId)¶

Return the info for the blob.

Arguments:

blobId (ValueBlobId)

Returns:

BlobInfo

BlobGetting.blobInfos(blobIds)¶

Return a list of BlobInfo.

Arguments:

blobIds (ValueBlobId[])

Returns:

BlobInfo[]

BlobGetting.blobStatistics()¶

Return the statistics for blobs.

Returns:: BlobStatistics

class BlobInfo()¶

A class used to represent various information of a blob in the persistence

layer.

Note: Not directly instantiable.

exported from index.d

BlobInfo.blobId()¶

Return the identifier.

Returns:: ValueBlobId

BlobInfo.blobLayout()¶

Return the layout.

Returns:: BlobLayout

BlobInfo.chunked()¶

Return true if the blob is greater than 2GB and requires the BlobIO API.

Returns:: boolean

BlobInfo.rowId()¶

Return the SQLite3 rowid.

Returns:: number

BlobInfo.size()¶

Return the size.

Returns:: number

class BlobLayout(dataType, components)¶

A class used to describe the layout of the element in a blob.: data_type is ‘uchar’, ‘ushort’, ‘uint’, ‘ulong’, ‘char’, ‘short’, ‘int’, ‘long’, ‘half’, ‘float’ or ‘double’, and components is limited to 255. The default layout is ‘uchar-1’ (aka a BLOB).

exported from index.d

Arguments:

dataType (string)
components (number)

BlobLayout.byteCount()¶

Return the size of an element in bytes.

Returns:: number

BlobLayout.components()¶

Return the number of packed values in an element.

Returns:: number

BlobLayout.dataType()¶

Return the data type.

Returns:: number

BlobLayout.dataTypeByteCount()¶

Return the size of the data type in bytes.

Returns:: number

BlobLayout.dataTypeRepresentation()¶

Return the data_type representation.

Returns:: string

BlobLayout.representation()¶

Return a string representation.

Returns:: string

static BlobLayout.parse(representation)¶

Return a BlobLayout by parsing the string ‘<data_type>-<components>’: ex: ‘uchar-1’, ‘float-2’, ‘float-3’, ‘half-4’, ‘double-16’.

Arguments:

representation (string)

Returns:

BlobLayout

class BlobPack(descriptor)¶

A class used to implements memory regions in one blob.

exported from index.d

Arguments:

descriptor (BlobPackDescriptor)

BlobPack.[Symbol․iterator]()¶

Returns:: Iterator<string>

BlobPack.blob()¶

Return the blob used to encode the blob pack.

Returns:: ValueBlob

BlobPack.check(name)¶

Return the BlobPackItem or throw.

Arguments:

name (string)

Returns:

BlobPackRegion

BlobPack.query(name)¶

Return the BlobPackItem or null.

Arguments:

name (string)

Returns:

BlobPackRegion | undefined

BlobPack.regions()¶

Return a list of BlobPackItem.

Returns:: BlobPackRegion[]

static BlobPack.fromBlob(blob)¶

Return a BlobPack from a blob or throw.

Arguments:

blob (ValueBlob)

Returns:

BlobPack

class BlobPackDescriptor()¶

A class used to describe binary regions.

exported from index.d

BlobPackDescriptor.addRegion(name, blobLayout, count)¶

Add and named region, where count is the number of elements defined by: blob_layout type.

Arguments:

name (string)
blobLayout (BlobLayout)
count (number)

class BlobPackRegion()¶

A class used for a region that implement the buffer protocol.

Note: Not directly instantiable.

exported from index.d

BlobPackRegion.blob()¶

Return the blob used by the BlobPack.

Returns:: ValueBlob

BlobPackRegion.blobLayout()¶

Return the blob layout.

Returns:: BlobLayout

BlobPackRegion.byteCount()¶

Return the number of byte.

Returns:: number

BlobPackRegion.copy(buffer)¶

Copy the content of the buffer to the region.

Arguments:

buffer (Buffer)

BlobPackRegion.count()¶

Return the number of element.

Returns:: number

BlobPackRegion.dataCount()¶

Return the number of data.

Returns:: number

BlobPackRegion.name()¶

Return the name.

Returns:: string

BlobPackRegion.offset()¶

Return the offset in the blob.

Returns:: number

class BlobStatistics()¶

A class used to represent the statistics about blobs.

Note: Not directly instantiable.

exported from index.d

BlobStatistics.count()¶

Return the number of blobs.

Returns:: number

BlobStatistics.maxSize()¶

Return the size of the biggest blob in bytes.

Returns:: number

BlobStatistics.minSize()¶

Return the size of the smallest blob in bytes.

Returns:: number

BlobStatistics.totalSize()¶

Return the total size in bytes.

Returns:: number

class BlobStream()¶

A class used to copy a huge blob (> 2GB) to a database. Use the method

database.blobStreamCreate(blob_layout, size) to create a BlobStream, then the method database.blobStreamAppend(…) to incrementally fill the content and finally the method database.blobStreamClose(…) to get the blob_id computed from the immutable content of the blob and its layout.

Note: Not directly instantiable.

exported from index.d

BlobStream.blobId()¶

Return the identifier of the blob.

Returns:: ValueBlobId

BlobStream.isClosed()¶

Return true if the stream is closed.

Returns:: boolean

BlobStream.offset()¶

Return the current offset.

Returns:: number

BlobStream.remaining()¶

Return the remaining bytes count to fill the blob.

Returns:: number

class BlobView(blobLayout, blob)¶

A class used to interpret the bytes of a blob from the layout.

exported from index.d

Arguments:

blobLayout (BlobLayout)
blob (ValueBlob)

BlobView.[Symbol․iterator]()¶

Iterate the decoded elements lazily.

Returns:: Iterator<OutputValue>

BlobView.at(index)¶

Return the element at the given index (out of range throws).

Arguments:

index (number)

Returns:

OutputValue

BlobView.blob()¶

Return the blob.

Returns:: ValueBlob

BlobView.blobLayout()¶

Return the blob_layout.

Returns:: BlobLayout

BlobView.count()¶

Return the number of elements in the blob.

Returns:: number

BlobView.dataCount()¶

Return the number of data in the blob.

Returns:: number

BlobView.encoderLayout()¶

Return the encoder layout.

Returns:: BlobEncoderLayout