Skip to content

About

tad: an open, transparent, easily accessible database - text files in git, tables declared as data, every row checked

Topics

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

tad

tad is an open, transparent, easily accessible database. Open: plain text in git, under a free licence, readable and changeable by anyone with an editor, nothing hidden in a binary store. Transparent: every row is checked against tables declared as data, every change is a reviewable one-line diff. Easily accessible: no server to run, no schema migration to learn - one small program, and the data opens in any tool.

tad keeps a small database as plain text in git, with three ways into the same data: files you can read and diff (one JSON object per line, one file per table, so a change to one row is a one-line diff), text you can reshape with jq and Miller, and SQL with joins (DuckDB), every change brought back to the same canonical files. That is tad's whole concern: what is done with the data - pages, the observatory, anything else - belongs to the tools that use it (rendering still lives here until it moves to a tool of its own: 2026-11-12-tad-is-text-as-data). Why, and what was weighed: the research why-tad-exists in this repository's data. Each repository declares its tables as data in tables.toml, and tad checks every row against it, keeps the files in one canonical form, edits a row only when the result still checks out, and renders the data into documents.

tad init DIR                   # start a bare database: tables.toml, data/, tad pinned to this release, the check CI runs
tad check                      # every row, and every rule across tables
tad canon                      # rewrite data/ in canonical form (only when it checks out)
tad set TABLE --key ID --set field=value ...   # add or update one row, checked; nothing kept that breaks a rule
tad render [--format md,rst]   # data/ -> docs/: one page per table, Markdown and/or reStructuredText
tad sql "SELECT ..." [--file F]   # ask the data in SQL (DuckDB): a query changes nothing; a change is kept only if every rule holds
tad query NAME --PARAM V ...    # (tad q) run queries/NAME.sql: a named query; it reads, and changes nothing
tad record NAME --PARAM V ...   # (tad r) run records/NAME.sql: a named record; kept only if every rule holds
tad statements [queries|records]   # the named statements this place keeps, with their parameters
tad completions bash|zsh|fish|nushell; tad man   # derived from the command line's contract (argenv)
tad engine                     # install the SQL engine tad uses: DuckDB's pinned binary, checked by fingerprint
tad declare --from-sql schema.sql   # translate a DuckDB schema into a declaration (refusing what it does not know)

Starting a database

tad init DIR (. for here) starts a bare database and nothing of a domain: tables.toml with no table yet, data/, bin/install-tad.sh pinning the release of tad that started it (the download checked against the release's checksums), .github/workflows/tad.yml - the check CI runs: every row and rule hold, the files are canonical and the pages what the data renders to - and the pages of the empty declaration, so that check holds from the first commit. Where any of these files is there already it writes nothing. A knowledge base - standing tables for decisions, research and work, the about link, the named queries an agent answers by - is grown on top of it by ideal-knowledge-base, which pins tad as tad's own consumers do. This took the place of text-as-data-template's scaffold (decision 2026-11-17-tad-init).

Named statements

What a place asks of its data, and how it writes it, is kept as SQL files beside tables.toml: queries/NAME.sql read and change nothing (one that would change a table is refused); records/NAME.sql write, and what they write is kept only if every row and rule still checks out, as with tad sql. The leading comment is the statement's header: its first line says what it does, and each line beginning --NAME declares a parameter as it is given on the command line, with its type (the declaration's types), ? when it may be left out (null), or = DEFAULT (today for a date):

-- A decision, with what was rejected.
-- --project   string        the place it is of
-- --decision  string
-- --rejected  string?
-- --date      date = today
INSERT INTO decisions (id, project, date, decision, rejected)
VALUES ($date || '-' || $project, $project, $date, $decision, $rejected);

tad r decision --project lab-tools --decision "..." runs it. In the body $NAME ($run_at for --run-at) is written by tad as a literal of its type - a value that is not of its type is refused, and a string can never close its quotes - and never inside quoted text or a comment. A new statement is a file in the same change as the work that needs it: tad is not released for it. The shell completions ask tad for the names as they complete them (tad statements queries), so they are always the place's own.

The declaration

tables.toml declares each table (after Frictionless Data's Table Schema): its key, and its fields - each with a name, a type (string, integer, number, boolean, date, datetime or list), and as needed required, enum, pattern, starts_with, minimum and maximum. A table may also say which pairs of fields must differ (distinct) and which field a given field requires (requires). A table may allow fields it does not name (extra_fields = true, for a shared minimum that projects extend) and name the field a new row is dated by (dated, filled with today by tad set). Rules across tables: unique_ids (ids unique across tables), references (a field names a row of other tables), appears_in (every row is cited somewhere) and no_mirror (a symmetric relation is not recorded both ways). A row is named by the table's key when that is one field, else by id.

Rendering

One page per table, one section per row; shapes are found from the data, not from names: a relation table (from_id, to_id, relation) folds into the pages it connects; a sources table (title, url) and two-column citation tables become "Sources" and "Cited by"; a table with a status and two <x>_id columns is drawn as a grid; a table named about heads the index. Output is deterministic, so CI can require the documents to be exactly what the data renders to.

Presentation, declared as data

With nothing declared, every table renders as above. A repository may declare in tables.toml how its pages read, so it never needs a renderer of its own: [site] gives the index a title, an intro, an index order (with count = { field, open_unless } for "N open of M"), and relation_words (each relation kind in words, from either end: serves = ["Serves", "Served by"]); [pages.<table>] gives a page an intro, a group_by field with ordered groups (and skip_empty_groups), an order_by, and what each row shows, in order - a field (or several fields, joined) as a plain paragraph or under a label, its relations, its sources - or a list layout instead of sections. The software-engineering canon renders this way.

Using it

A repository installs tad from a pinned release, checking it against the release's checksums (see bin/install-tad.sh in any of the repositories below), and its CI runs make verify. Releases are built by the package conveyor for x86_64 and aarch64: tarballs, .deb and .rpm, with checksums and signed provenance.

A pinned version is kept current by bin/update-pins.sh, run weekly on the lab machine: for each repository listed in consumers it moves the pin, re-renders, and lands the change only if that repository's CI passes.

In use by: linux-phone-oco/docs, software-engineering-canon, ideal-project-playbook and git-rhizoid.github.io.

History

tad began as a Python engine on DuckDB, vendored into each repository. On 2026-11-11 it was rewritten in Rust and proven equal before any repository switched: the same verdicts on all four repositories and on sixteen deliberately broken copies, the same canonical bytes, the same edits, and the same 74 documents in both formats, byte for byte. The vendored copies had drifted (three of four were older); the switch ended that. The Python engine's last version is in this repository's history at commit 02da2bc (https://github.com/creation-guidelines/tad-engine/tree/02da2bc/engine).

About

tad: an open, transparent, easily accessible database - text files in git, tables declared as data, every row checked

Topics

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages