Templated Features

A project does not pick templates; it picks features. The pack’s features.json is the map between the two: per target, each feature names the templates it renders and the features it requires. Templates stay flat in one directory per target (cpp/, python/, typescript/), and the selection is expressed one level up.

A project names its features in its kibo.toml, and kibo-project does the rest: it resolves the closure, renders each template, writes the embedded definitions and copies the runtime, as features.json declares. Calling the jar by hand is covered in Command-line usage.

The catalogue

C++

The requires of the C++ features was measured, not decided: it is the #include graph of the generated C++, interface and build unioned, since asking for a feature means compiling it.

Feature

What it gives

Requires

Templates

Base

The type system and everything that cannot be pulled apart from it. Data, Codec, Model and AnyConcept include one another in both directions; there is no cut.

—

data, codec, model, any_concept (.hpp and .cpp each)

Fields

The name of every field, as a constant (fields::S::f, a std::string_view): code that handles structures through the dynamic API gets completion instead of a string it can misspell. Selectable alone.

—

fields

Paths

The path of every field, built once from its name (paths::S::f()). CommitAttachments addresses a field through it.

Fields

paths (.hpp and .cpp)

DatabaseAttachments

The attachments a unit declares, in a Viper::Database: keys, has, get, set, del, documents read, written and deleted whole, a write inside a transaction. Nothing of Commit.

Base

database_attachments

CommitAttachments

The attachments a unit declares, in a Commit database: keys, has, get over a state; set, diff and the field operations over a Viper::CommitAttachmentMutating, stored when the state is committed.

Base, Paths

commit_attachments

Reads

The attachments a unit declares, reading only: keys, has, get over a Viper::AttachmentGetting, a Viper::Database or a Commit state, for code that reads documents without knowing which database holds them. Neither usage requires it.

Base

reads

Pool

A function pool, host side: the functions the application implements, and the pool that registers them, for its embedded Python or a service.

Base

pool

PoolRemote

A function pool, client side: Remote, which calls the pool through a service. Selectable without Pool, so a client does not link the functions only a host implements.

Base

remote

AttachmentPool

An attachment function pool, host side: the functions over a Commit state the application implements, and the pool that registers them.

CommitAttachments

attachment_pool

AttachmentPoolRemote

An attachment function pool, client side: Remote, which calls the pool through a service, over a Commit state.

CommitAttachments

attachment_remote

Every C++ file lands in the target’s output directory, flat. A file rendered once per DSM namespace is named <infrastructure>_<unit>_<template>, a file rendered once for the model <infrastructure>_<template>, a file rendered once per pool <infrastructure>_<pool>_<template>. For a model named features with one namespace Demo, one function pool Tools and one attachment function pool PlayerModel:

Feature

Files

Base

features_demo_data, features_demo_model, features_demo_codec, features_codec, features_any_concept (.hpp, .cpp), and features_resources.hpp

Fields

features_demo_fields

Paths

features_demo_paths (.hpp, .cpp)

DatabaseAttachments

features_demo_database_attachments

CommitAttachments

features_demo_commit_attachments

Reads

features_demo_reads

Pool

features_tools_pool

PoolRemote

features_tools_remote

AttachmentPool

features_player_model_attachment_pool

AttachmentPoolRemote

features_player_model_attachment_remote

Python

Feature

What it gives

Requires

Templates

Base

The package: its entry point, the types each namespace declares, and the container bindings the model mentions.

—

__init__.py, data.py, containers.py

Fields

The name of every field, as a constant (fields.S.f).

—

fields.py

Paths

The path of every field, built once from its name (paths.S.f).

Fields

paths.py

DatabaseAttachments

The attachments a unit declares, in a dsviper.Database: keys, has, get, enumerate, set, delete. Imported by its path, import <package>.<unit>.database_attachments.

Base

database_attachments.py

CommitAttachments

The attachments a unit declares, in a Commit database: keys, has, get, enumerate, diff_keys over a state; set, diff and the field operations over a CommitAttachmentMutating. Imported as <package>.<unit>.commit_attachments.

Base, Paths

commit_attachments.py

Reads

The attachments a unit declares, reading only: keys, has, get over an AttachmentGetting, a Database’s attachment_getting() or a Commit state’s. Imported as <package>.<unit>.reads.

Base

reads.py

Pool

A function pool in the host’s embedded Python: Pool, over the dsviper.FunctionPool the C++ host gives. Most projects have none: a pool is a service surface, not a consequence of having a model.

Base

pool/__init__.py, pool/embedded.py

PoolRemote

A function pool, client side: Remote, which calls the pool through a service.

Base

pool/__init__.py, pool/remote.py

AttachmentPool

An attachment function pool in the host’s embedded Python: Pool, over the dsviper.AttachmentFunctionPool the C++ host gives.

CommitAttachments

attachment_pool/__init__.py, attachment_pool/embedded.py

AttachmentPoolRemote

An attachment function pool, client side: Remote, over a Commit state.

CommitAttachments

attachment_pool/__init__.py, attachment_pool/remote.py

Wheel

What makes the package installable, and therefore typed.

Base

pyproject.toml, py.typed

The package is the directory <output>/<infrastructure>/. Base renders __init__.py at its root (definitions(), the units) and in each unit, <unit>/data.py and containers.py; Fields, Paths and the three attachment usages render <unit>/fields.py, <unit>/paths.py, <unit>/database_attachments.py, <unit>/commit_attachments.py and <unit>/reads.py, which a unit’s __init__.py does not import. A pool is a subpackage, <pool>/__init__.py, which re-exports nothing, with a module per side: <pool>/embedded.py (Pool, the Pool and AttachmentPool features) and <pool>/remote.py (Remote, PoolRemote and AttachmentPoolRemote). Wheel is the one feature whose two files do not land beside the modules: py.typed goes inside the package, pyproject.toml one level up, at the target’s output. Without py.typed a type checker reports Stub file not found and every annotation of the package is invisible; without pyproject.toml, pip install refuses the directory. See Python Wheels.

TypeScript

Feature

What it gives

Requires

Templates

Base

The package: its entry point, the types each namespace declares, and the container bindings the model mentions.

—

index.ts, definitions.ts, data.ts, containers.ts

Fields

The name of every field, as a constant.

—

fields.ts

Paths

The path of every field, built once from its name.

Fields

paths.ts

DatabaseAttachments

The attachments a unit declares, in a dsviper.Database: keys, has, get, enumerate, set, del. Imported by its path, from "<package>/<unit>/database_attachments".

Base

database_attachments.ts

CommitAttachments

The attachments a unit declares, in a Commit database: keys, has, get, enumerate, diffKeys over a state; set, diff and the field operations over a CommitAttachmentMutating. Imported as "<package>/<unit>/commit_attachments".

Base, Paths

commit_attachments.ts

PoolRemote

A function pool, client side: Remote, which calls the pool through a service; every function pool of the model from <package>/pools. Most projects have none: a pool is a service surface, not a consequence of having a model.

Base

pool/index.ts, pools.ts, pool/remote.ts

AttachmentPoolRemote

An attachment function pool, client side: Remote, over a Commit state; every attachment pool of the model from <package>/attachment_pools.

CommitAttachments

attachment_pool/index.ts, attachment_pools.ts, attachment_pool/remote.ts

Package

What makes the package buildable and installable: package.json and tsconfig.json, at the package root, the sources staying in src/.

Base

package.json, tsconfig.json

The sources are the directory <output>/src/. Base renders index.ts at its root and in each unit, definitions.ts, <unit>/data.ts and containers.ts; Fields, Paths and the two attachment usages render <unit>/fields.ts, <unit>/paths.ts, <unit>/database_attachments.ts and <unit>/commit_attachments.ts, which a unit’s index.ts does not export. No host embeds Node, so TypeScript has the client side only, and no Reads: PoolRemote renders pools.ts (every function pool, one namespace each) and <pool>/index.ts, <pool>/remote.ts (Remote); AttachmentPoolRemote the same with attachment_pools.ts. Package renders package.json and tsconfig.json at the target’s output, the package root. See npm Packages.

Resolving a selection

resolve.py, beside features.json, walks requires depth first and returns the closure, dependencies first, then the templates of that closure, deduplicated, in order. A name the target does not declare is refused with the list of those it does; a cycle is refused.

$ ./resolve.py cpp CommitAttachments
features : Base Fields Paths CommitAttachments
  data.cpp.stg
  data.hpp.stg
  codec.cpp.stg
  codec.hpp.stg
  model.cpp.stg
  model.hpp.stg
  any_concept.cpp.stg
  any_concept.hpp.stg
  fields.hpp.stg
  paths.cpp.stg
  paths.hpp.stg
  commit_attachments.cpp.stg
  commit_attachments.hpp.stg

From Python, resolve.closure("cpp", ["CommitAttachments"]) gives the feature names and resolve.templates("cpp", ["CommitAttachments"]) the template paths. kibo-project does the same walk, and kibo_project.py plan shows it, with where each template lands — sources for the directory the templates render into, root for the target’s output itself:

[python] python -n features -> /…/python
    features: Base, DatabaseAttachments, Wheel
    python/__init__.py.stg                   sources
    python/data.py.stg                       sources
    python/containers.py.stg                 sources
    python/database_attachments.py.stg       sources
    python/pyproject.toml.stg                root
    python/py.typed.stg                      sources
[typescript] typescript -n features -> /…/typescript
    features: Base, Fields, Paths, CommitAttachments, Package
    typescript/index.ts.stg                  sources
    typescript/definitions.ts.stg            sources
    typescript/data.ts.stg                   sources
    typescript/containers.ts.stg             sources
    typescript/fields.ts.stg                 sources
    typescript/paths.ts.stg                  sources
    typescript/commit_attachments.ts.stg     sources
    typescript/package.json.stg              root
    typescript/tsconfig.json.stg             root

What sits beside the templates

Besides the features, features.json declares what a tool driving kibo needs to know of the pack, so that the tool carries no knowledge of its own:

  • generator.kibo — the oldest kibo exposing the Template Model the templates consume, >=2.1.0. A floor: a later kibo exposing the same Template Model renders them too.

  • layout, per target — where the templates render, below the target’s output (sources), which templates render at the output itself (root), the runtime copied beside the sources, and the embedded definitions (resources) with their encoding.

  • reserved, per target and per family of names — the names the pack’s own code takes: the members every generated class inherits from the runtime’s proxy, which a field would mask (wrap_value, copy, type in Python; wrapValue, equals, toJSON in TypeScript), and the modules at the package’s root, which a namespace or a pool would replace (containers, definitions, resources, pools, and in TypeScript attachment_pools and index). A DSM name meeting one stops the generation, saying how to spell it otherwise for that target — kibo-project.

  • validation, per target — the checks run once a target is written. C++ needs none: its compiler refuses invalid code and says why. Python and TypeScript can run code a name has broken without a word, so Python is imported, every structure built and the package checked with mypy --strict; TypeScript is checked with tsc --noEmit. A check whose tool is not found fails.

Target

sources

root

runtime

resources

cpp

the output

—

—

<infrastructure>_resources.hpp, a byte array <infrastructure>_resources_definitions

python

<infrastructure>

pyproject.toml

python/runtime copied as _codegen

resources.py, base64 of the zlib-compressed definitions, B64_DEFINITIONS

typescript

src

package.json, tsconfig.json

typescript/runtime copied as _codegen

resources.ts, base64 of the definitions, B64_DEFINITIONS

The runtime and the embedded definitions go with a feature: "with": "Base" for both, in every target. They are written beside the templates of the feature that reads them, so a target that does not render Base — the client side of a pool, rendered with with_requirements = false — gets neither.

The runtime belongs to the pack, not to dsviper: the proxy base, the registry that wraps and unwraps values, the container views (Sequence, Mapping, Ordered, Optional, Variant) and AttachmentBase, the base every generated attachment derives from, holding its runtime id and descriptor, written once and copied into every generated package (_codegen/ in Python: proxy.py, container.py, attachment.py; src/_codegen/ in TypeScript: proxy.ts, container.ts, attachment.ts, registry.ts, value.ts). It versions with the templates that call it. Python names the optional’s view Option: an annotation Optional[D] reads as typing.Optional, a value or None, where it is a box opened with unwrap(). The declared classes keep their name in both languages (Optional_of_…). The embedded definitions are decoded once by the package’s definitions(), or by codec::definitions() in C++.

A project’s own features

A project can add its features to the pack’s selection: a manifest of the same shape as features.json, whose templates sit beside it (<manifest dir>/<language>/<template>), and whose features may require the pack’s. A feature name the pack already declares is refused.

{
  "cpp": {
    "Report": {
      "doc": "A report of every structure the model declares.",
      "templates": ["report.md.stg"],
      "requires": ["Fields"]
    }
  }
}

resolve.py takes it with --with (repeatable), and from Python with extra=[...]:

$ ./resolve.py cpp Report --with my-templates/features.json
features : Fields Report
  fields.cpp.stg
  fields.hpp.stg
  report.md.stg

In a project, kibo-project reads the manifests listed in [generator] manifests, paths relative to the project file:

[generator]
templates = "2"
manifests = ["my-templates/features.json"]

[target.cpp]
features = ["Report"]
output = "report"

A third-party template writes its own header: its licence and the runtime it needs are its author’s to state. The templates such a manifest names read the Template Model, as the pack’s do.

Picking a feature set

A selection is a target of the project’s kibo.toml; name the features you want and the closure follows. This project renders the C++ with the Commit attachments and a function pool on both sides, the Python package as an installable wheel over a Database, and the TypeScript package with its manifest and a pool’s client:

[project]
definitions = "definitions"
infrastructure = "features"

[generator]
templates = "2"

[target.cpp]
features = ["CommitAttachments", "Pool", "PoolRemote"]
output = "cpp/generated"

[target.python]
features = ["DatabaseAttachments", "Wheel"]
output = "python"

[target.typescript]
features = ["Base", "PoolRemote", "Package"]
output = "typescript"

Common selections:

Use

Target

Features

Types and the codec only

cpp

Base

Typed documents in a Database

cpp

DatabaseAttachments (pulls in Base)

Typed documents in a Commit database

cpp

CommitAttachments (pulls in Base, Paths and Fields)

A library that reads documents from either database

cpp

Reads

Field names for code that uses the dynamic API

cpp

Fields

A service: server and client in one binary

cpp

CommitAttachments, Pool, PoolRemote (and AttachmentPool, AttachmentPoolRemote)

A client of a service, in another binary

cpp

PoolRemote, with with_requirements = false beside a target that renders Base

Python inside a host application that embeds it

python

Base and a usage (add Pool, AttachmentPool if the host exposes pools)

An installable Python package

python

a usage, Wheel (and PoolRemote)

A buildable TypeScript package

typescript

a usage, Package (and PoolRemote)

Several targets of one language — a server and its client, say — name each target for what it produces and state its language; see the project file.