Skip to content

development.md

Latest commit

 

History

History
262 lines (216 loc) · 13.2 KB

development.md

File metadata and controls

262 lines (216 loc) · 13.2 KB
Raw

comshu — internals and development

What the tool is, and how to install and run it, is in ../README.md. This file is everything else.

Unified GPU-communication profiler merging SigmaTrace, Snoopie, snoopie-sanitized, and (later) ucTrace. Design docs live two levels up: UNIFICATION.md (what/why) and ARCHITECTURE.md (layers, interfaces, patterns).

Piece Name
Core library (L0–L2) libcomshu
The command (launch, read, export, query, serve) comshu
Namespace comshu:: (core, rec, hook, attr, gpu, provider)
Env prefix COMSHU_*

BUGS-FOUND.md — defects in other software that comshu found, with the evidence that identified each and the explanations that were excluded. The claim a tool makes is what it found, not what it can in principle see.

Layout

libcomshu/        L0–L2 core — vendor-free (no cuda*/hip*/ibv_*/Sanitizer_* types allowed)
                 checked by tools/check_vendor_free.py (ctest `vendor_free`)
  include/comshu/
    core/        record model (Event/Entity/Artifact/Edge), time, ids, neutral vocabulary
    rec/         Recorder, ThreadLocalBuffer, InternTable, sinks (the hot path)
    hook/        HookTrait / CompositeHookManager        [ported from SigmaTrace/Hook]
    attr/        Collector/Filter/Attributor pipeline    [from snoopie-sanitized design]
providers/       L3 — each builds one .so (vendor types live HERE, not in core)
  gpumem/        GPU memory-access provider (Snoopie engine)
  ibverbs/       proof-of-concept network provider (from SigmaTrace)
  ucx/           UCX as the application asked for it (public UCP symbols)
  ucx_deep/      UCX as it reached the wire (the uct_iface_ops_t table)
gpu-backends/    vendor capture engines feeding the gpumem provider
  nvbit/         phase-1 default (confirmed working on du04: CUDA 13.0, driver 580)
  sanitizer/     Compute-Sanitizer (stub; phase 2)
comshu/           the binary: verb dispatch (main.cpp) + the launcher (run.cpp)
comshu-read/      the reading half — read/dump/export/link/sql/types, linked into comshu
comshu-ui/        the serving half — `comshu ui`, linked in when built with DuckDB
ui/              the Svelte frontend, built to ui/dist and embedded in the binary
tests/           core unit/smoke tests (no GPU required)

The vendor-free rule bans types and headers, not names. Core may name a vendor symbol, because interposition and dlsym work by name and need no header: core/mem_attr.hpp declares libcuda's signature itself rather than including cuda.h, core/dmabuf_registry.hpp copies CUresult's ABI under the name CuResult, and src/engine.cpp defines five extern "C" doors whose signatures are void* and size_t throughout. The property being protected is that core compiles and runs on a host with no CUDA and no verbs — which is MN5's own core partition. tools/check_vendor_free.py enforces exactly that, and carries the allowlist of symbols core interposes on purpose.

Usage

Pick what to capture by short name; comshu assembles the run and reads it back.

comshu                                  # the verbs
comshu list                             # what can I capture, and how?
comshu run -p ib      -- ./app args     # InfiniBand wire traffic (inter-node)
comshu run -p p2p     -- ./app args     # GPU-initiated NVLink peer r/w (intra-node, Snoopie)
comshu run -p ib,p2p  -- ./app args     # both — see "replay" below
comshu read comshu-out                   # what is in the capture
comshu export comshu-out --csv out.csv   # tables for anything that reads CSV
comshu ui comshu-out                     # the panels, at 127.0.0.1:8777
agent/run.sh comshu-out                 # ... and the chat pane, if a model is up

agent/ is an optional sidecar that lets you ask the capture questions in English and get prose, the SQL behind it, and a chart. It answers with a query rather than with data, so every number it gives you is one click from being re-run in the Ask panel — see ../agent/README.md. The panels work exactly the same without it.

One binary with verbs, not three tools: comshu-read and comshu-ui are gone as executables (they are still the directories the code lives in). It also removes an argv[1] that used to mean a capture in some invocations and an export in others — read/dump/export take a capture, link/sql take an export.

Providers are chosen with -p name[,name...] (each accepts short aliases, e.g. ib/net, p2p/nvlink/snoopie). Add --dry-run to print the plan.

Replay. Some engines must own the whole runtime — NVBit (p2p) takes over the CUDA runtime and dlsym, so it can't share a process with anything else. When a selection mixes such an exclusive provider with others, one process can't host them, so comshu replays: it runs the workload once per exclusive provider and drops each pass under one -o directory (out/ib/, out/gpumem/, …). The passes are wall-clock-aligned, and comshu read stitches them back into one per-rank picture (each provider owns a disjoint record-type range, so a rank's network and NVLink traffic merge cleanly). Hook-based providers (all the network ones) share a single pass — no replay needed among them.

Phase 1 scope (decided 2026-07-15)

  1. New four-archetype record model with interning, from day one.
  2. gpumem provider with the NVBit backend first (Sanitizer stubbed).
  3. ibverbs as the single network proof provider; the rest of SigmaTrace's providers follow in phase 2. ucTrace deferred. (Superseded 2026-08-27: ucx and then ucx_deep landed in phase 1. The ids 800-899 reserved for "ucTrace" are now the UCT-layer provider, which reaches UCX's data plane through the uct_iface_ops_t dispatch table rather than by reimplementing UCT internals — see plans/PLAN-20260827-1449-ucx-deep-uct-ops-table-done.md.)
  4. Test bed: ssh a100 (du04, 4x A100). Multinode validation deferred until supercomputer access.

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build

Core + tests build with no CUDA/verbs installed; providers are enabled individually (-DCOMSHU_PROVIDER_GPUMEM=ON, -DCOMSHU_PROVIDER_IBVERBS=ON) and require their vendor SDKs.

Build the engine with gcc or clang, never NVHPC's nvc++. nvc++ leaves __builtin_ia32_rdtsc as an undefined symbol instead of inlining it, so every hook dies at load and the capture is empty (nm -D --undefined-only libcomshu_engine.so shows it). This is why a gpumem-only tree, which configures under nvidia-hpc-sdk, installs only the provider (COMSHU_INSTALL_CORE defaults OFF there). The fix is __rdtsc() from <x86intrin.h> in the clock helpers, but that is on every hook's hot path, so it needs its own MN5 run.

Install

cmake --install lays down a self-locating tree — the launcher finds the engine through $ORIGIN/../lib/comshu and its providers through the matching search root, so an installed comshu needs no COMSHU_PROVIDER_PATH, COMSHU_ENGINE_PATH or LD_LIBRARY_PATH:

<prefix>/bin/comshu
<prefix>/lib/comshu/{libcomshu_engine,libibverbs_trace,libgpumem_trace}.so
<prefix>/include/comshu/...

COMSHU_INSTALL_RPATH_EXTRA bakes absolute directories into the installed RUNPATH, which is what lets the result run in a bare login shell with no module load. On MareNostrum 5 (installed at ~/.local, already on PATH):

# core + ibverbs, from the gcc-13.2 tree
cmake -B build -DCOMSHU_PROVIDER_IBVERBS=ON \
      -DCMAKE_INSTALL_PREFIX=$HOME/.local \
      -DCOMSHU_INSTALL_RPATH_EXTRA=/apps/ACC/GCC/13.2.0/lib64
cmake --build build -j && cmake --install build

# the CUDA provider, from its own NVHPC tree (see run_on_mn5.sh for the flags)
cmake --install build-gpumem              # installs gpumem only, by design

That second command installs only the provider: COMSHU_INSTALL_CORE defaults OFF for a gpumem-only tree, because nvc++ miscompiles the engine (see things-to-circle-back-to.md). Don't override it.

New dtags are used throughout, so the baked RUNPATH is searched after LD_LIBRARY_PATH — module-loading a different toolchain still wins. The one runtime dependency an install cannot supply is nvdisasm on PATH, which NVBit needs when -p p2p actually runs on a GPU node.

Formatting

.clang-format is the house style, derived from the tree rather than picked: 4-space indent, char* p, unindented namespaces, 100 columns. Two settings are load-bearing and shouldn't be relaxed without reading the comments on them — ReflowComments: false (the comments here carry tables and aligned prose that reflowing destroys) and SortIncludes: false (includes are grouped on purpose, several with explanatory trailing comments).

tools/format.sh              # format the staged files
tools/format.sh --all        # format every tracked first-party source
tools/format.sh --check --all

third_party/, the vendored NVBit, and libcomshu/schema/generated/ each carry a DisableFormat: true marker, so neither the hook nor an editor-on-save will touch them.

The pre-commit hook enforces formatting only once the whole tree is already formatted — until then it stays inert, so a one-file commit never drags a repo-wide reformat behind it. Run tools/format.sh --all once, commit that on its own, and enforcement switches on by itself.

Gates

There is no hosted CI (the repo is mirrored between machines, not on a forge), so the gates are local and named:

  • Every commit: tools/hooks/pre-commit (install once with tools/install_hooks.sh) builds the core configuration and runs its tests, including tools/schema_check.py. Skippable with --no-verify; the MN5 gate is what catches you.
  • Anything touching capture semantics (hook engine, records, providers, stream format): tests/hpc/run_on_mn5.sh — sync, build both providers on MareNostrum 5, run the 8-case suite, compare SQL goldens. This is the merge gate; a change is not done until this board is green. It syncs by pushing HEAD to a bare repo on MN5 which the remote build tree clones, so every board is attributable to a commit; a dirty tree is refused (--rsync to override while iterating).

Adding a record type

One entry per layer; the build refuses a missing one:

  1. COMSHU_RECORD in core/records.hpp (or the provider's *_records.hpp), type_id inside the provider's range from core/ids.hpp.
  2. A struct in libcomshu/schema/comshu.fbs with the // type N, kind comment, and kSchemaHash in rec/stream_header.hpp updated (schema_check.py prints the new value).
  3. A case in libcomshu/tests/schema_conformance.cpp.
  4. A Table<> specialization + TableRecords entry in comshu-read/main.cpp (the compiler demands both or neither).
  5. A row in tools/gen_figures.py's FAMILIES and a box in the family figure it belongs to (ctest figures_current).

tools/schema_check.py runs at build time and cross-checks 1-3; it also holds tools/read_pcs.py's enum-name tables equal to the reader's.

If the record carries an enum class field, three more, each of which fails loudly rather than silently — that is why they are listed rather than left to be discovered:

  • the enum in comshu.fbs too, with identical member names and values (schema_check.py compares the bodies);
  • a COMSHU_SCHEMA_ENUM line per enumerator in schema_conformance.cpp, plus its _MAX in the count assertion — a reordered enumerator changes no size and no offset, so nothing else can see it;
  • rows in link_sql.hpp's vocabulary table, and the enum's name in tools/gen_raw_columns.py's ENUMS set. That set is closed by construction: an unknown type is a generator error, never a dropped column.

One thing the checklist cannot tell you, and it decides how the record is read: export --csv may print an enum as a name while --parquet always writes the number. gpu_access.op does (Store vs 1), and the consequence is that no single SQL assertion can name that column — tests/hpc/suite/sql_check.sh runs assertions against Parquet when there is one and against CSV+sqlite when there is not. A new record should print the number in both (cuda_memcpy does) and leave the names to the vocabulary table.

Adding a provider

providers/<name>/ building one .so: implement provider::IProvider, COMSHU_EXPORT_PROVIDER, claim a record-id range in core/ids.hpp, declare it at startup via comshu_engine_declare_provider (the engine refuses overlapping ranges), and add a registry row in comshu/main.cpp.

If it emits EntityClose for anything, add that entity to the opens CTE in tests/hpc/suite/asserts/entity_close_accounted.sql in the same change, and a quiet/fires pair to asserts_fire.sh. This has been forgotten three times — the ib family, UCX's four entities, then ucx_deep's two — and the cost is not a missing check: every close the new provider emits is reported as "names an entity no open record created", so the assertion becomes noise for every other provider in the same capture, on exactly the runs where it is most needed. Hook definitions follow providers/ibverbs/ib_hooks.hpp; the manager contract is documented in comshu/hook/CompositeHookManager.hpp.