From 28731894179e5dbd7950dbf95f4c93c4d4623cac Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 13:46:09 +0000 Subject: [PATCH 01/11] Adversarial audit of PythonTap, October 2026 docs/AUDIT-2026-10.md: a second full audit after the plan's phases landed, read against what a user's class or Max can do to each promise. Confirmed with probes on the core: worker::stop() blocks forever on a process() that never returns (Max hangs when DSP toggles or the object goes); NaN samples escape when process() raises mid-vector; and os._exit() ends the host, so the ReadMe's "nothing your code does can take Max down" overclaims. By reading: the reserved-name list misses names Max calls with C arguments (dspstate, fileusage, patchlineupdate), user print() and numpy warnings post from the audio thread, the worker thread runs on the platform's default stack, helper modules never hot-reload, and a handful of low items. Records what was attacked and held (the reload/audio race, the worker ring, GIL lifetime, loading, supply chain, CI) and the tests each finding calls for. Release, ASan/UBSan and TSan batteries: 85/85 each. assemble-package.py leaves the audits out of the shipped package, as it does the plan. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- docs/AUDIT-2026-10.md | 322 ++++++++++++++++++++++++++++++++++++ scripts/assemble-package.py | 4 +- 2 files changed, 324 insertions(+), 2 deletions(-) create mode 100644 docs/AUDIT-2026-10.md diff --git a/docs/AUDIT-2026-10.md b/docs/AUDIT-2026-10.md new file mode 100644 index 0000000..0726493 --- /dev/null +++ b/docs/AUDIT-2026-10.md @@ -0,0 +1,322 @@ +# tap.python~ — adversarial audit, October 2026 + +A second full audit of PythonTap, after the first (September 2026) became `PRODUCTION-PLAN.md` and +its phases landed. Same four dimensions — threading and real-time safety, CPython C-API use, +build/CI/supply chain, tests and documentation — read adversarially: for each promise the ReadMe or +a header makes, what a user's class (or Max) can do to break it. Anchors are to commit `41c1ae0` +(0.10.0 merged into `main`). + +What was run here (Linux, CPython 3.13.14, numpy 2.5.3, attrs 26.1.0): + +| Battery | Result | +|---|---| +| `core/tests`, Release | 85/85 | +| `core/tests`, ASan + UBSan (`halt_on_error=1`) | 85/85, no sanitizer report | +| `core/tests`, TSan (`halt_on_error=1`) | 85/85, no race report | +| Max glue against min's mock kernel (`build-linux`) | 1/1 | +| Three probes written for this audit (below) | two contract breaks and one overclaim confirmed | + +Max itself was not available, so everything about Max's dispatch is from min-api's and the Max +SDK's sources, as the headers cite them; those items say so. + +## Summary + +The design discipline the plan set out holds up: the GIL rules (one thread state per thread; build +the new binding completely, swap with no Python call in between; take your own references before +anything that can yield), the SPSC worker ring, the double-checked block-buffer swap, the +`SystemExit` handling, the identifier check that closes path traversal, the isolated interpreter +configuration, hash-pinned downloads, least-privilege CI with SHA-pinned actions. None of the +attacks on those found a hole, and the sanitizer batteries agree. + +What did not hold are mostly at the edges of the contract rather than in its core: + +1. **Worker mode can hang Max** on a `process()` that never returns — `worker::stop()` joins the + thread, and `stop()` runs when DSP toggles, the chain recompiles, or the object is deleted. + Confirmed with a probe. (The plan's bar 1 says "no hang".) +2. **NaN leaks when `process()` raises mid-vector** — the samples computed before the exception are + never sanitized. Confirmed with a probe; the ReadMe promises they are replaced with 0.0. +3. **The reserved-name list misses names Max calls with C arguments** (`dspstate`, `fileusage`, + `patchlineupdate`, …): the same crash mechanism 6.1 found for `filechanged`, reachable by naming a + Python method `dspstate`. By reading min and the SDK; not run in Max. +4. **"Nothing your code does can take Max down" is an overclaim** — `os._exit()` ends the process + (confirmed), a crashing C extension or an infinite loop is outside the guard. +5. **User code still prints and allocates on the audio thread** through `print()` and, more + insidiously, numpy's `RuntimeWarning`s, which format through `linecache` (disk I/O on the first + one) and post to Max's console from the audio thread. The core's own reports are deferred; the + user's are not. +6. **The worker thread has the platform's default stack** (512 KB on macOS for a `std::thread`, + where CPython gives its own threads 16 MB), so deep recursion or a heavy extension on the worker + can overflow where it would not in Max's own threads. +7. Smaller items: helper modules never hot-reload (and keep the `.pyc` staleness hazard 3.6a fixed + for class files); a few type-hint shapes map surprisingly (`float | int` becomes a symbol + attribute); Windows non-ASCII package paths break the file watcher; a zip-slip gap through + symlink entries in `assemble-package.py --merge`; plan D2 describes a loader that was replaced. + +Each item below has what to do about it and how to pin it. + +## Findings + +Severity: **high** — a crash, hang or process exit a user can reach; **medium** — a documented +contract broken, or a real-time hazard; **low** — a surprise, a gap in hardening, or a doc that +has drifted. + +### A1 (high) — worker mode: `stop()` blocks forever on a `process()` that never returns + +`worker::stop()` (`core/include/tap/python/worker.h:129-145`) sets `stopping`, bumps `signal` and +`join()`s the thread (`:142`). The thread is inside `processor::process()` inside the user's Python, +and checks `stopping` only between vectors. A `process()` that loops forever — or blocks in a C +call — never comes back, and `stop()` never returns. `stop()` runs on Max's main thread from +`dspsetup` (`set_up_worker`, every time the chain compiles: DSP on/off, a cord change, `@mode` or +`@latency` set, a save that changes the ports), from `start()`, and from the destructor (object +deleted, patch closed, Max quitting). In each case Max hangs. + +Probe (`scripts` folder with `never_returns.py`: `while True: pass` in `process()`; a worker started +with latency 2; one vector pushed; `stop()` called with a 3 s watchdog): + +``` +worker::stop() still blocked after 3 s on a process() that never returns -> host would hang +``` + +Direct mode has the same stall but no join: the audio thread is simply gone, which Max's own +watchdog treats as audio dropping out; the main thread stays usable to fix the file. Worker mode +turns a bad class into a frozen Max, and the mode exists to protect against bad Python. + +*Do:* bound the join. After `stopping` is set and the thread has not finished within, say, 250 ms, +inject an exception into it with `PyThreadState_SetAsyncExc(thread_id, exc)` (the worker records +its `PyThreadState_GetID()` or `threading.get_ident()` as it starts; `exc` through a builtins +lookup, per the no-data-symbols rule), which raises at the next bytecode in a pure-Python loop; +`process()` then returns raising, the processor records it and unbinds as it does for any +exception, and the join completes. If it still has not finished (a blocking C call, `time.sleep`), +detach the thread and leak the ring rather than hang the host, and say so once in the console. +Pin with a core test on `never_returns.py`: `stop()` returns within a bound, the exception is +reported through `flush_reports()`, a `load()` re-arms audio. Document the remainder (a blocking C +call cannot be interrupted) in the ReadMe's errors paragraph. + +### A2 (medium) — NaN escapes when `process()` raises mid-vector + +`process_samples()` sanitizes the written channels after the loop (`processor.h:889-891`), but the +two early returns — the input conversion failing (`:855-860`) and the call raising (`:865-870`) — +silence the rest of the vector and return without sanitizing the samples already written. Probe +(`nan_then_raise.py`: returns `nan` until an input ≥ 0.75, then raises; input +`0.1 0.1 0.1 0.9 0.1 0.1`): + +``` +nan nan nan 0 0 0 +NaN samples escaped: 3 +``` + +The ReadMe says non-finite output "is replaced with 0.0". It is a one-vector leak and only when +the class also raises, but a NaN into the rest of a signal chain propagates. The block path is +not affected (its result is copied and sanitized or the vector is silenced whole). + +*Do:* sanitize `[0, i)` of each written channel before both returns. Pin with the probe as a test +(`test_realtime.cpp`, beside "Non-finite output is replaced with 0.0"). + +### A3 (high, by reading; not run in Max) — reserved names miss what Max calls with C arguments + +The crash 6.1 found — min registered `filechanged` with its `A_GIMME` wrapper, Max called it with +C arguments, every save crashed Max — has a general form: `object_addmethod` with an `A_GIMME` +trampoline (`tap.python_tilde_message.h`) for any name Max sends with a C signature. The object's +`reserved_messages()` (`tap.python_tilde.h:374-379`) lists nineteen names; min's own list of +messages it registers as `A_CANT` (`source/min-api/include/c74_min_message.h:46-54` and the +`MIN_WRAPPER_ADDMETHOD` table in `c74_min_object_wrapper.h:575-614`) also has `dspstate`, +`fileusage`, `patchlineupdate`, `edclose`, `okclose`, `oksize`, `paint`, the mouse and focus +messages, `key`, `mousewheel`, `getplaystate`, `jitclass_setup`, `maxclass_setup`, `maxob_setup`, +`mop_setup`, `setup`. For an MSP object without a UI the ones Max actually sends are `dspstate` +(a `long`, on every DSP on/off, to any MSP object that has the method), `fileusage` (a handle, to +every object when a collective or standalone is built) and `patchlineupdate` (when a cord +connects); `inputchanged` and `multichanneloutputs` are the mc-era additions. A class with +`def dspstate(self, on: int)` therefore registers an object method Max finds and calls as +`(x, long)`, and `python_mess_gimme` reads a `long` as a `t_symbol*`. + +*Do:* reserve min's `cant` list plus `dspstate`, `inputchanged`, `multichanneloutputs`, and, as a +general guard, any name the Max class already answers (`class_method(c, gensym(name))` or +`object_getmethod(maxobj(), …)` non-null). Keep the data in one place the core test +(`test_safety.cpp`, "Methods named like reserved host messages") iterates over. In Max, add a step +to the `faults` runtime test: a fixture with `dspstate`, `fileusage` and `patchlineupdate` methods, +DSP toggled and a cord connected. + +### A4 (medium) — the honest-limits text overclaims + +ReadMe line 17: "nothing your code does, `sys.exit()` included, can take Max down"; line 101: "it +never takes Max down". The plan's bar 1 adds "no hang". What the guards cover is every *Python +exception*, `SystemExit` included; what they cannot cover is a process exit or a fault below +Python. Probe (`hard_exit.py`: `os._exit(3)` in a message): + +``` +exit 3 +``` + +Also outside the guard: `os.abort()`, `signal.raise_signal`, a segfault in a C extension (ctypes, +a broken wheel), and an infinite loop — in a message handler it freezes Max's main thread, in +`process()` the audio thread (and, A1, the main thread too in worker mode). + +*Do:* rewrite the two ReadMe sentences and bar 1 to say exactly that: no exception, `sys.exit()` +included, takes Max down or stops the object; what happens below Python (a C extension crashing, +`os._exit()`) or never returns to it (an endless loop) is not caught, and worker mode confines the +latter to the worker thread (after A1). Add the limit to CLAUDE.md's "Honest limits are pinned, +not hidden" so it is not rediscovered. + +### A5 (medium) — user code posts to the console from the audio thread + +The core's own audio-thread problems are recorded and flushed from the main thread (2.1). The +user's output is not: `print()` in `process()`, and any `warnings.warn` — numpy raises a +`RuntimeWarning` for `1/x` at zero or `log` of a negative in a block `process()`, one keystroke +away — goes through `sys.stderr` → `_maxconsole.write` → `console_post()` +(`runtime.h:355-367`), which takes a `std::mutex` (`:357`) and calls the sink while holding it, +and the object's sink `console_line()` (`tap.python_tilde.h:412-420`) calls `object_post` / +`object_error` directly, on whatever thread it is. For a warning, Python first formats it: +`warnings.formatwarning` → `linecache.getline`, which reads the source file from disk the first +time. So the first warning costs a file read plus a console post on the audio thread; every later +one a mutex and a post. In worker mode the same happens on the worker thread, which is the +real-time thread the mode set up (2.5). + +*Do:* make the Max sink real-time safe: when `!systhread_ismainthread()`, push the line into a +bounded lock-free queue drained by a qelem (the object's `m_reports` already exists) instead of +posting; drop with a count when full. Replace `console_post`'s mutex-held sink call with +copy-then-release (or a per-thread buffer). Document that `print()` in `process()` is deferred +and that a flood is coalesced. Pin: a core test that `print()` from the audio thread calls the +sink on the thread the host nominates (today it asserts nothing about which thread). + +### A6 (medium, by reading) — the worker thread runs on the default stack + +`worker::start()` creates the thread with `std::thread` (`worker.h:110`), which takes the +platform's default stack: 8 MB on Linux, 1 MB on Windows, **512 KB on macOS** for a secondary +thread. CPython gives the threads it creates 16 MB on macOS (`THREAD_STACK_SIZE` in +`Python/thread_pthread.h`) because 512 KB is known to be too little for Python code that recurses +or calls into extensions. Max's audio thread is what it is (direct mode shares the hazard, and it +is Max's), but the worker is ours, and the mode is sold as the one for heavy Python. A class that +recurses a few hundred frames through a C boundary (sorting with a key, `__getattr__`, a numpy +callback) or calls a deep extension will segfault on the worker where it would not on a `threading` +thread. + +*Do:* create the worker with `pthread_create` and `pthread_attr_setstacksize` (16 MB, matching +CPython), or on Windows `CreateThread` with a stack size; the thread body stays as it is. Pin with +a fixture that recurses to the default `sys.getrecursionlimit()` through a C call on the worker. +(Verify the macOS default on the runtime-test Mac first: `ulimit -s` is the main thread's; the +secondary-thread default is in `pthread_attr_getstacksize` of a fresh attr.) + +### A7 (low) — helper modules never hot-reload, and keep the `.pyc` hazard + +The ReadMe (line 89) says other files in `python/` can be imported as helpers. They are imported +through the normal machinery into `sys.modules`, so a save of a helper is neither watched nor +re-executed — not even by `filechanged`, because an unchanged class file hits the source cache and +the file is not re-run, and even when it is, `import helper` returns the cached module. They also +keep the `.pyc` staleness check (mtime at 1 s + size) that 3.6a removed for class files. The +first user to split a class into modules will save the helper and wonder why nothing changes. + +*Do:* say so in the ReadMe ("helpers are loaded once per Max session; restart Max, or put what +you are live-coding in the class file"), or, in the loader, when a class file *is* re-executed, +drop from `sys.modules` every module whose `__file__` is under `scripts_directory()` first, and +set `sys.dont_write_bytecode` at start-up so no `.pyc` is written for anything under `python/`. +Pin either way with a core test (a helper edited between two loads). + +### A8 (low) — type-hint shapes that map surprisingly + +By running the support module's `hint_kind`/`return_shape` on plain CPython (`runtime.h:243-330`): + +| Hint | Maps to | Effect | +|---|---|---| +| `float \| int` | `'str'` → symbol attribute | `@x 0.5` stores `""` (symbol coercion of a number) | +| `Annotated[float, …]`, `Final[float]` | `'Annotated'`, `'Final'` → symbol | same | +| `-> tuple[float, float] \| None` | `(1, 'tuple')` | bound as one output; every sample then reported non-numeric | +| `np.float64` | `'float64'` → symbol | a numpy-typed field becomes a symbol attribute | + +`Optional[X]`, `X | None`, string hints and `ClassVar` are handled well. + +*Do:* unwrap `Annotated` and `Final` to their first argument; for a `Union` of several non-None +members prefer `float`, then `int`, then `bool`, else symbol; treat `Optional[tuple[...]]` as the +tuple in `return_shape`; map numpy scalar types (`np.floating`, `np.integer` subclasses) to float +and int. Extend `test_types.cpp` with a `typed_more.py`. + +### A9 (low, by reading) — Windows: non-ASCII package paths break the watcher + +`watched_file.string()` (`tap.python_tilde.h:230`) and `path.string()` in the core's diagnostics +convert a `std::filesystem::path` to the *ANSI code page* on MSVC, while Max's path API takes +UTF-8. For a user whose `Documents` folder has a non-ASCII character, `locatefile_extended` +(`:233`) fails and the console says "Unable to watch …": hot reload is silently off. The Python +side is right (`path_to_unicode` uses `wstring()`), so loading works and only the watcher (and +the text of "No file …") is affected. + +*Do:* convert with `u8string()` (then `reinterpret_cast` to `char*`) wherever a path meets a Max +or console API. No Windows machine here to confirm; the runbook's Windows step (6.4) can. + +### A10 (low) — zip-slip through symlink entries in `assemble-package.py --merge` + +`extract()` (`scripts/assemble-package.py:248-270`) refuses `..` in entry *names*, but re-creates a +symlink entry with whatever target the zip says (`:263`), and later entries written through that +link land wherever it points. The inputs are CI's own artifacts from the same workflow run, so +this needs a compromised runner to matter; cheap to close. + +*Do:* reject a link target that is absolute or resolves outside `destination`, and check every +`target` with `resolve().is_relative_to(destination.resolve())` before writing. + +### A11 (low) — pins by tag where the policy is by SHA + +`style.yml:27` uses `tap/taphouse/.github/workflows/drift-check.yml@v5` (own organization, so a +moved tag is a self-inflicted risk); `core/tests/CMakeLists.txt` fetches Catch2 by tag `v3.8.0` +(tests only). Pin the taphouse workflow by SHA like the third-party actions, or note the exception +in the workflow comment that states the policy. + +### A12 (low) — plan D2 describes a loader that was replaced + +`PRODUCTION-PLAN.md:27` says `importlib.util.spec_from_file_location` under +`_tap_python_user.`; 3.5 and the loader compile and `exec` the source as +`_tap_python_`. Also worth a line somewhere: the module has `__file__` but no `__spec__` or +`__package__`, so relative imports and `importlib.resources` on the class module do not work +(flat folder by design); and two externals embedding CPython in one Max (another Python package +loading its own `libpython`) is untested and unlikely to work — Phase 7 mentions sharing an +interpreter with plugins; the limit belongs in the ReadMe now. + +## What was attacked and held + +Recorded so the next audit need not repeat it. + +- **The reload/audio race, every variant.** A reload between the audio thread's load of + `m_process_function` and `m_instance` (both read under the GIL after a lock-free check, swapped + together with no Python call between); a reload during `ensure_block_buffers`'s own numpy calls + (the double check at `processor.h:1069` catches the other thread's set, and the audio thread + swaps in its own set at its own size with no yield before use); `prepare()` swapping the block + arrays while a vector holds them (own references, `:935-936`); `set_attribute`/`call` reading + `m_attributes`/`m_messages` while `load()` exchanges them (both under the GIL, copies and + references taken before the first call that can yield). TSan agrees across the battery. +- **The worker ring.** The write slot `k % slot_count` and the read slot `(k − L) % slot_count` + never coincide because `slot_count ≥ L + 16`; a dropped vector (ring full) is passed over by the + worker (`in_seq ≠ next`) and read as silence without a second "late" count; the worker cannot be + writing a slot the audio thread reads because it only processes `next < written`; the + signal/`wait` ordering loses no wake-up (signal loaded before `written`); `stop()`'s borrow of + the ring waits at most one vector. The one-report-per-load CAS (`record()`) does not carry a + count across a flush into the next load. +- **GIL lifetime.** `thread_state_keeper` pairs `PyGILState_Ensure`/`PyEval_SaveThread` once per + thread and undoes them at thread exit (a Core Audio thread torn down on a device change exits + cleanly); the interpreter's own thread is excluded; `start()`/`stop()`/destructors are called + without the GIL by the wrapper, as the headers require. +- **Reporting.** `record_exception()` keeps one exception per flush and never prints; the worker's + `record()` never allocates; `SystemExit` in every entry point is displayed, not honored + (`test_safety.cpp`), including through `PyErr_DisplayException` for a traceback whose frames + reference the instance. +- **Loading.** `is_identifier()` refuses `../x`, `two words`, `gain.py`; the path is built from + `scripts_directory()` alone; a class file named `json.py` neither shadows nor is shadowed by the + standard library; the 2 s once-per-save rule reports a broken save once and a later object again; + `__name__` is `_tap_python_`, so `if __name__ == "__main__"` blocks do not run. +- **Supply chain and CI.** Both installers verify the archive SHA256 from a committed lock and + install wheels with `--require-hashes --only-binary :all:`; a failed install restores the previous + runtime; `update-locks.py` is the only writer of the locks; workflows run with `contents: read` + (write only on the release job), pin third-party actions by SHA, cancel superseded runs, and check + the link shape (weak dylib, `@loader_path` rpath only, delay-load, no CPython data imports) on the + real binaries. The release's `all platforms` job validates package names with a regex before + merging. +- **Max glue.** `filechanged` is a min `message<>` with default thread safety, so a `filechanged` + sent from the scheduler thread (a `metro` in Overdrive) is deferred to the main thread by min + (`c74_min_message.h:313-321`); `load()` therefore stays main-thread only as the header requires. + The C trampolines are `guarded()`; the perform routine catches everything; the file watcher owns a + nobox object with the SDK's `A_CANT` signature. + +## Coverage gaps the findings expose + +Tests that would have caught the confirmed items, to add with the fixes: an exception raised +mid-vector after NaNs (A2); a `process()` that never returns against `worker::stop()` (A1, after +the fix); a fixture named for each reserved name, iterated from the one list the object passes to +the core (A3); `print()` and a numpy warning from the audio thread, asserting which thread the sink +sees (A5); deep recursion on the worker (A6); a helper module edited between loads (A7); the +hint table in A8. diff --git a/scripts/assemble-package.py b/scripts/assemble-package.py index bc6a84d..b0a670a 100644 --- a/scripts/assemble-package.py +++ b/scripts/assemble-package.py @@ -56,9 +56,9 @@ # Never shipped from the copied folders. # maxtest_*.py: the runtime tests' fixtures, copied into python/ while runtime-tests/run.py runs -# PRODUCTION-PLAN.md: the development roadmap in docs/, beside the reference page Max reads +# PRODUCTION-PLAN.md, AUDIT-*.md: the development roadmap and audits in docs/, beside the reference page Max reads IGNORED = shutil.ignore_patterns("__pycache__", "*.pyc", ".ipynb_checkpoints", ".DS_Store", "maxtest_*", - "PRODUCTION-PLAN.md") + "PRODUCTION-PLAN.md", "AUDIT-*.md") # Each platform's folder for its runtime in a package for every platform (plan 4.8); the external From 0abac79f0bfbbdd735914ae6f0123ed77da88656 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 15:47:38 +0000 Subject: [PATCH 02/11] Plan: Phase 8 closes the October 2026 audit's findings docs/PRODUCTION-PLAN.md gains Phase 8, one PR per item in severity order, each fix against a test that fails first: 8.1 the documents say what is true (the honest limits, helpers, D2) plus the zip-slip and tag-pin hardening; 8.2 NaN sanitized before process()'s early returns and the reserved names Max calls with C arguments; 8.3 a bounded worker stop (async exception, then detach) and a 16 MiB worker stack; 8.4 user console output deferred off the audio thread; 8.5 helper modules reloaded with the class file; 8.6 the type-hint mapping; 8.7 UTF-8 paths on Windows; 8.8 the Mac session that runs the new runtime tests. Sequenced before 1.0; the audit points at it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- docs/AUDIT-2026-10.md | 3 +- docs/PRODUCTION-PLAN.md | 102 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 1 deletion(-) diff --git a/docs/AUDIT-2026-10.md b/docs/AUDIT-2026-10.md index 0726493..18bbdc0 100644 --- a/docs/AUDIT-2026-10.md +++ b/docs/AUDIT-2026-10.md @@ -52,7 +52,8 @@ What did not hold are mostly at the edges of the contract rather than in its cor attribute); Windows non-ASCII package paths break the file watcher; a zip-slip gap through symlink entries in `assemble-package.py --merge`; plan D2 describes a loader that was replaced. -Each item below has what to do about it and how to pin it. +Each item below has what to do about it and how to pin it; the plan to close them is Phase 8 of +`PRODUCTION-PLAN.md`. ## Findings diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index 9092f97..af450d1 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -543,6 +543,105 @@ passed; a third ran the soak (6.2) and measured performance (6.3). To continue: (two processors; another broken save; a later object) and the runtime test `announce-once` (a broken save, five objects in Max, one report — five before the change). +## Phase 8 — the October 2026 audit's findings (before 1.0) + +`docs/AUDIT-2026-10.md` (findings A1–A12) read the shipped 0.10.0 adversarially: the design +discipline held, the edges of the contract did not. This phase closes them, one PR each as below, +highest severity first, each fix landing against a test that fails before it (the house rule), and +each contract change recorded in `CHANGELOG.md` (D5). The honest limits that remain are written +down rather than discovered again. + +- [ ] **8.1 Say what is true (A4, A7a, A12 — docs and small hardening, no behavior change).** + ReadMe lines 17 and 101, and bar 1 above: *no Python exception*, `sys.exit()` included, takes + Max down or stops the object — what happens below Python (`os._exit()`, a C extension crashing) + or never returns to it (an endless loop in a message freezes Max's main thread; in `process()` + the audio thread, and until 8.3 the main thread too in worker mode) is not caught. Helper modules + in `python/` are imported once per Max session and are not reloaded by a save (until 8.5). + Another external that embeds its own CPython in the same Max is untested and unsupported. Add + these to CLAUDE.md's honest-limits rule. Correct D2 (the loader compiles and `exec`s the source + as `_tap_python_`; `__file__` only, no `__spec__`, so no relative imports). Harden + `assemble-package.py`'s `extract()` against symlink zip-slip (A10: refuse a link target that is + absolute or resolves outside the destination, and check every written path), and pin the + taphouse drift workflow by SHA or note the exception beside the policy comment (A11). *Done + when:* the three documents agree with each other and with the code; `--merge` of a crafted zip + with a `../` symlink entry fails. +- [ ] **8.2 Crash and contract fixes in the core and the glue (A2, A3).** + *A2:* `process_samples()` sanitizes the samples already written before each of its two early + returns (input conversion failing; the call raising). Test first, in `test_realtime.cpp`: the + audit's `nan_then_raise.py` (NaN until an input ≥ 0.75, then an exception) renders + `0 0 0 0 0 0`, not `nan nan nan 0 0 0`. + *A3:* one list of reserved names, in the object (`reserved_messages()`), extended with every + message min registers as `A_CANT` (`c74_min_message.h`'s `message_type::cant` names and the + `MIN_WRAPPER_ADDMETHOD` table: `dspstate`, `fileusage`, `patchlineupdate`, `edclose`, `okclose`, + `oksize`, `paint`, the mouse, focus and key messages, `mousewheel`, `getplaystate`, the `*_setup` + names, `setup`, `dictionary`) plus Max's mc-era `inputchanged` and `multichanneloutputs`; and a + general guard — a name the Max class already answers (`class_method(c, gensym(name))` non-null) + is reserved too, so min's own registrations never need listing by hand. Tests: the core's + reserved-name scenario stays (it takes any list); a glue test instantiates + `[tap.python~ maxtest_mock_reserved]` whose class defines `dspstate`, `fileusage`, + `patchlineupdate` and `inputchanged`, and checks none is registered and each is announced once; + the `faults` runtime test gains a step that toggles DSP and connects a cord with such a class + loaded (the crash 6.1 found, in its general form). CHANGELOG: the new reserved names. +- [ ] **8.3 Worker mode never hangs Max (A1, A6 — `worker.h`).** + *A1:* `stop()` bounds its join. The worker records its thread identifier + (`PyThread_get_thread_ident()`) as it starts; if the thread has not finished within 100 ms of + `stopping` being set, `stop()` injects `PyThreadState_SetAsyncExc(ident, WorkerStopped)` — a + `RuntimeError` subclass defined in the support module, looked up by name (no CPython data + symbols) — which raises at the next bytecode of a pure-Python loop; `process()` returns raising, + the processor records it and unbinds as for any exception, and the join completes. If it still + has not finished after another 250 ms (a blocking C call, `time.sleep`), `stop()` detaches the + thread, releases the ring and the processor to it (leaked on purpose: the thread may yet wake and + use them), and logs once that the class stalled and was abandoned; the object then runs a fresh + processor. Tests (`test_worker.cpp`): `never_returns.py` (`while True: pass`) — `stop()` returns + within 1 s, `flush_reports()` prints `WorkerStopped`, a `load()` re-arms audio; `time.sleep(5)` + in `process()` — `stop()` returns within 1 s having detached, and the test process still exits + cleanly. Runtime test (`worker`): the same two classes in Max, DSP toggled, patch closed. + *A6:* the worker thread is created with an explicit stack of 16 MiB (CPython's own + `THREAD_STACK_SIZE` on macOS) through `pthread_attr_setstacksize` / `_beginthreadex`, in a small + `native_thread` helper that keeps `std::thread`'s join/detach shape. Measure first on the Mac: + the secondary-thread default from a fresh `pthread_attr_t`, and the depth at which a fixture + recursing through a C boundary (`sorted(key=…)` calling itself) crashes on a `std::thread` + against a `threading.Thread`; then pin that fixture running to `sys.getrecursionlimit()` on the + worker. CHANGELOG: a stalled class in worker mode is interrupted or abandoned, never waited for. +- [ ] **8.4 Nothing of the user's prints on the audio thread either (A5).** `runtime_options` + gains `is_main_thread` (the host's predicate: `systhread_ismainthread` in Max) and + `console_ready` (real-time safe; the object sets its `m_reports` qelem). `console_post()` on a + thread the host does not call main appends the line to a bounded lock-free queue (MPSC, or one + SPSC ring per thread state through the keeper) and calls `console_ready`; it never blocks on the + console mutex (`try_lock`; a full queue or a contended lock drops the line and counts it). The + host's `flush_console()`, from its main thread, drains the queue to the sink in order and + appends "… and N lines dropped" when it had to. Lines from the main thread keep posting at once, + so the core battery's synchronous `console()` checks stand; the audio-thread tests call + `flush_console()`. In Max, `print()` in `process()` and numpy's `RuntimeWarning`s then reach the + console from the main thread; what remains on the audio thread is Python's own formatting of a + warning (`linecache` reads the file once), which the ReadMe names. Tests: `print()` from an audio + thread reaches the sink only on the nominated thread; a flood of 10,000 lines drops with a count + and never blocks the producer (timed). CHANGELOG: console output from `process()` is deferred and + coalesced. +- [ ] **8.5 Helper modules follow the class file (A7b).** When a load *executes* the class file + (the loader says so), it first drops from `sys.modules` every module whose `__file__` is under + `scripts_directory()`, so the fresh execution imports the helpers afresh; the runtime sets + `sys.dont_write_bytecode = True` at start-up, so no `.pyc` is written for anything under + `python/` and 3.6a's hazard cannot return for helpers. A helper saved on its own is still not + watched — decide then whether the watcher should cover the folder (Max's file watcher watches a + file; a folder watch is a `t_filewatcher` per file or a poll) or whether "save the class file to + pick up a helper" is the rule the ReadMe states. Test: a class importing `helper.py`, the helper + edited, the class file saved → the new helper runs; edited alone → it does not, and the ReadMe + says so. +- [ ] **8.6 Type hints that map as a reader expects (A8).** In `hint_kind`: unwrap `Annotated` + and `Final` to their first argument; for a `Union` of several members other than `None`, prefer + `float`, then `int`, then `bool`, else symbol; map numpy scalar types (`np.floating` → `float`, + `np.integer` → `int`, `np.bool_` → `bool`). In `return_shape`: see through `Optional[tuple[…]]` + to the tuple. Test: `typed_more.py` in `test_types.cpp` with the audit's table. CHANGELOG: the + mapping changes (a `float | int` field was a symbol attribute). +- [ ] **8.7 Windows: paths meet Max as UTF-8 (A9).** Every `path.string()` handed to a Max call or + a console line (`locatefile_extended` in the constructor, the core's "No file …" and + "Failed to load …") goes through `u8string()`. Verify in the runbook's Windows step with the + package under a folder named with a non-ASCII character: the watcher starts, a save reloads. +- [ ] **8.8 The Mac session for this phase.** Runtime tests for 8.2 (`faults`) and 8.3 (`worker`); + 8.3's stack measurement; the macOS half of 8.7 is not needed (POSIX paths are UTF-8). Then + repeat the runbook's step 4 on the release packages, and tag. + ## Phase 7 — plugin front ends (optional, post-1.0) - [ ] **7.1** A CLAP (MIT, simpler) or VST3 wrapper over the core. Needs its own answer to fixed @@ -561,6 +660,9 @@ passed; a third ran the soak (6.2) and measured performance (6.3). To continue: 8. Phase 2.4–2.6 — multichannel, worker mode, reload stalls (features; may follow 1.0). 9. Phase 6 — in-Max validation before tagging 1.0. 10. Phase 4.7–4.8 — uv for the tooling, then one package for every platform on each tag. +11. Phase 8 — the audit's findings, in the order above: 8.1 (docs and hardening, first, so the + ReadMe stops overclaiming while the fixes land), 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, then the Mac + session 8.8 — all before 1.0. ## External prerequisites From 66d1493b9c6bf3aa23cbd4edb8d0f5cb52ede9d9 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 15:53:27 +0000 Subject: [PATCH 03/11] Plan: Phase 8 revised after an audit of the plan itself 8.3: WorkerStopped derives from BaseException so a class's except Exception cannot swallow it; the processor recognizes it and keeps the binding (a slow vector loses one vector, not audio); the detached thread shares the ring and the object leaks its processor only at destruction while such a thread lives. 8.4: lines assembled per thread, so the existing cross-thread interleave goes with the mutex; ordering caveat stated. 8.5: the class modules and compiled modules are excluded from the sys.modules drop; two-version and unchanged-file limits stated. 8.6: mixed unions map to any, not to a preferred member. 8.2: the class_method guard is pinned not to take int/float/symbol/bang/list away; fileusage is a by-hand step. 1.0 gate stated. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- docs/PRODUCTION-PLAN.md | 143 +++++++++++++++++++++++++--------------- 1 file changed, 91 insertions(+), 52 deletions(-) diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index af450d1..6a717e3 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -549,7 +549,8 @@ passed; a third ran the soak (6.2) and measured performance (6.3). To continue: discipline held, the edges of the contract did not. This phase closes them, one PR each as below, highest severity first, each fix landing against a test that fails before it (the house rule), and each contract change recorded in `CHANGELOG.md` (D5). The honest limits that remain are written -down rather than discovered again. +down rather than discovered again. *This plan was itself audited before being adopted +(2026-10-01); what that changed is marked "revised".* - [ ] **8.1 Say what is true (A4, A7a, A12 — docs and small hardening, no behavior change).** ReadMe lines 17 and 101, and bar 1 above: *no Python exception*, `sys.exit()` included, takes @@ -575,27 +576,46 @@ down rather than discovered again. `MIN_WRAPPER_ADDMETHOD` table: `dspstate`, `fileusage`, `patchlineupdate`, `edclose`, `okclose`, `oksize`, `paint`, the mouse, focus and key messages, `mousewheel`, `getplaystate`, the `*_setup` names, `setup`, `dictionary`) plus Max's mc-era `inputchanged` and `multichanneloutputs`; and a - general guard — a name the Max class already answers (`class_method(c, gensym(name))` non-null) - is reserved too, so min's own registrations never need listing by hand. Tests: the core's - reserved-name scenario stays (it takes any list); a glue test instantiates - `[tap.python~ maxtest_mock_reserved]` whose class defines `dspstate`, `fileusage`, - `patchlineupdate` and `inputchanged`, and checks none is registered and each is announced once; - the `faults` runtime test gains a step that toggles DSP and connects a cord with such a class - loaded (the crash 6.1 found, in its general form). CHANGELOG: the new reserved names. + general guard — a name the Max class already answers (`class_method(object_class(maxobj()), + gensym(name))` non-null) is reserved too, so min's own registrations never need listing by hand. + *Revised:* the guard must not take away what is promised: a glue test asserts `int`, `float`, + `symbol`, `bang`, `list` and an ordinary name stay exposed with the guard on (min registers + `int`/`float`/`bang` only for classes that declare them; this one does not — the test is the + proof, since Max's own class setup is not ours to read). Tests: the core's reserved-name scenario + stays (it takes any list); a glue test instantiates `[tap.python~ maxtest_mock_reserved]` whose + class defines `dspstate`, `fileusage`, `patchlineupdate` and `inputchanged`, and checks none is + registered and each is announced once; the `faults` runtime test gains a step that toggles DSP + and connects a cord with such a class loaded (the crash 6.1 found, in its general form). + `fileusage` is sent only by Build Collective/Application, so it is a runbook step by hand, not a + runtime test. CHANGELOG: the new reserved names. - [ ] **8.3 Worker mode never hangs Max (A1, A6 — `worker.h`).** - *A1:* `stop()` bounds its join. The worker records its thread identifier - (`PyThread_get_thread_ident()`) as it starts; if the thread has not finished within 100 ms of - `stopping` being set, `stop()` injects `PyThreadState_SetAsyncExc(ident, WorkerStopped)` — a - `RuntimeError` subclass defined in the support module, looked up by name (no CPython data - symbols) — which raises at the next bytecode of a pure-Python loop; `process()` returns raising, - the processor records it and unbinds as for any exception, and the join completes. If it still - has not finished after another 250 ms (a blocking C call, `time.sleep`), `stop()` detaches the - thread, releases the ring and the processor to it (leaked on purpose: the thread may yet wake and - use them), and logs once that the class stalled and was abandoned; the object then runs a fresh - processor. Tests (`test_worker.cpp`): `never_returns.py` (`while True: pass`) — `stop()` returns - within 1 s, `flush_reports()` prints `WorkerStopped`, a `load()` re-arms audio; `time.sleep(5)` - in `process()` — `stop()` returns within 1 s having detached, and the test process still exits - cleanly. Runtime test (`worker`): the same two classes in Max, DSP toggled, patch closed. + *A1:* `stop()` bounds its join. The worker records its OS thread identifier + (`PyThread_get_thread_ident()`, no GIL needed) as it starts; if the thread has not finished + within 100 ms of `stopping` being set, `stop()` takes the GIL briefly (the hung thread yields it + every 0.5 ms) and injects `PyThreadState_SetAsyncExc(ident, WorkerStopped)`, which raises at the + next bytecode of a pure-Python loop. *Revised, three ways.* (i) `WorkerStopped` derives from + **`BaseException`**, like `KeyboardInterrupt`, so a class's `except Exception:` does not swallow + it and keep looping; it is defined in the support module and looked up by name (no CPython data + symbols). (ii) The processor **recognizes `WorkerStopped` and does not unbind**: it is our + interruption, not the class's fault — a `process()` that was merely slow (a 200 ms numpy call + when the chain recompiled) loses that vector and carries on; it is reported once as "process() + was interrupted while the worker stopped". (iii) If the thread still has not finished after + another 250 ms (a blocking C call; `time.sleep`, which delivers the exception only when it + returns), `stop()` **detaches** it. The ring is then shared (`shared_ptr`, the thread holding one + reference), so a thread that wakes later frees it; the processor it runs stays owned by the + object — attributes, messages and a fixing save keep working, and the next `dspsetup` starts a + fresh worker — and the object's destructor hands the processor to a process-wide leak list + while a detached thread is still alive (an atomic count the thread decrements as it exits), + rather than destroy what the thread may still touch. The console says once that the class + stalled and was abandoned, and that it keeps costing a core until Max restarts (it still holds + the GIL in 0.5 ms slices). `stop()` thus holds Max's main thread for at most ~350 ms. Direct + mode is unchanged (8.1 documents it). Tests (`test_worker.cpp`): `never_returns.py` + (`while True: pass`, inside `try: … except Exception: pass`) — `stop()` returns within 1 s, + `flush_reports()` prints the interruption, audio is still bound and the next vector runs; + `time.sleep(5)` in `process()` — `stop()` returns within 1 s having detached, a new `start()` + works, and the test process exits cleanly (leaks are off under ASan already); a `process()` that + takes 50 ms — `stop()` waits for it and injects nothing. Runtime test (`worker`): the same + classes in Max, DSP toggled, patch closed. *A6:* the worker thread is created with an explicit stack of 16 MiB (CPython's own `THREAD_STACK_SIZE` on macOS) through `pthread_attr_setstacksize` / `_beginthreadex`, in a small `native_thread` helper that keeps `std::thread`'s join/detach shape. Measure first on the Mac: @@ -604,43 +624,62 @@ down rather than discovered again. against a `threading.Thread`; then pin that fixture running to `sys.getrecursionlimit()` on the worker. CHANGELOG: a stalled class in worker mode is interrupted or abandoned, never waited for. - [ ] **8.4 Nothing of the user's prints on the audio thread either (A5).** `runtime_options` - gains `is_main_thread` (the host's predicate: `systhread_ismainthread` in Max) and - `console_ready` (real-time safe; the object sets its `m_reports` qelem). `console_post()` on a - thread the host does not call main appends the line to a bounded lock-free queue (MPSC, or one - SPSC ring per thread state through the keeper) and calls `console_ready`; it never blocks on the - console mutex (`try_lock`; a full queue or a contended lock drops the line and counts it). The - host's `flush_console()`, from its main thread, drains the queue to the sink in order and - appends "… and N lines dropped" when it had to. Lines from the main thread keep posting at once, - so the core battery's synchronous `console()` checks stand; the audio-thread tests call - `flush_console()`. In Max, `print()` in `process()` and numpy's `RuntimeWarning`s then reach the - console from the main thread; what remains on the audio thread is Python's own formatting of a - warning (`linecache` reads the file once), which the ReadMe names. Tests: `print()` from an audio - thread reaches the sink only on the nominated thread; a flood of 10,000 lines drops with a count - and never blocks the producer (timed). CHANGELOG: console output from `process()` is deferred and - coalesced. + gains `is_main_thread` (the host's predicate — `systhread_ismainthread` in Max; the default, + with no host predicate, is "always", which keeps the core battery's synchronous `console()` + checks as they are) and `console_ready` (real-time safe; the object sets its `m_reports` qelem). + *Revised:* lines are assembled per thread in a `thread_local` buffer — today the two per-stream + buffers are shared under one mutex, so partial writes from two threads already interleave — and + only a *complete line* is dispatched: on a main thread straight to the sink; on any other thread + into a bounded lock-free MPSC queue (dropping and counting when full, never blocking), with + `console_ready` called. The host's `flush_console()`, from its main thread, drains the queue to + the sink in order and appends "… and N lines dropped" when it had to. The audio-thread tests call + `flush_console()`. Consequences to document: deferred lines arrive after main-thread lines + printed later (ordering across threads is not kept); the scheduler thread's prints are deferred + too (harmless); what still happens on the audio thread is Python's own formatting of a warning + (`linecache` reads the file once). Tests: `print()` from an audio thread reaches the sink only on + the nominated thread; two threads printing half-lines never produce a mixed line; a flood of + 10,000 lines drops with a count and never blocks the producer (timed). CHANGELOG: console output + from `process()` is deferred and coalesced. - [ ] **8.5 Helper modules follow the class file (A7b).** When a load *executes* the class file - (the loader says so), it first drops from `sys.modules` every module whose `__file__` is under - `scripts_directory()`, so the fresh execution imports the helpers afresh; the runtime sets - `sys.dont_write_bytecode = True` at start-up, so no `.pyc` is written for anything under - `python/` and 3.6a's hazard cannot return for helpers. A helper saved on its own is still not - watched — decide then whether the watcher should cover the folder (Max's file watcher watches a - file; a folder watch is a `t_filewatcher` per file or a poll) or whether "save the class file to - pick up a helper" is the rule the ReadMe states. Test: a class importing `helper.py`, the helper - edited, the class file saved → the new helper runs; edited alone → it does not, and the ReadMe - says so. + (the loader says so), it first drops from `sys.modules` every module whose `__file__` is a `.py` + under `scripts_directory()` — *revised:* **except the class modules themselves** + (`_tap_python_*`, which `typing.get_type_hints` and the loader's cache depend on) and anything + that is not a source module (a compiled extension cannot be re-imported) — so the fresh + execution imports the helpers afresh; the runtime sets `sys.dont_write_bytecode = True` at + start-up, so no `.pyc` is written for anything under `python/` and 3.6a's hazard cannot return + for helpers (the shipped runtime and installed wheels carry their `.pyc` already, so nothing + slows). Two honest limits to state: a helper shared by two class files runs in two versions + until each class file has been re-executed (module-level state is not shared between them + meanwhile); and only a save that *changes* the class file re-executes it — an unchanged file hits + the source cache — so "edit the helper, then save the class file" works only if the class file + changed too. Decide in the PR whether `filechanged` sent by a patcher (as against the watcher's) + should force re-execution — the watcher would then send a distinct, undocumented message — or + whether the folder should be watched. Test: a class importing `helper.py`, the helper edited, + the class file changed and loaded → the new helper runs; edited alone → it does not, and the + ReadMe says so. CHANGELOG: helpers reload with the class file. - [ ] **8.6 Type hints that map as a reader expects (A8).** In `hint_kind`: unwrap `Annotated` - and `Final` to their first argument; for a `Union` of several members other than `None`, prefer - `float`, then `int`, then `bool`, else symbol; map numpy scalar types (`np.floating` → `float`, - `np.integer` → `int`, `np.bool_` → `bool`). In `return_shape`: see through `Optional[tuple[…]]` - to the tuple. Test: `typed_more.py` in `test_types.cpp` with the audit's table. CHANGELOG: the - mapping changes (a `float | int` field was a symbol attribute). + and `Final` to their first argument; *revised:* a `Union` of several members other than `None` + maps to **`any`** — the atom passes as it is, as for an unannotated parameter — rather than to + one preferred member, since preferring `float` for `str | float` would silently lose strings as + preferring `str` loses numbers today (on the Max side an `any` attribute is a symbol attribute, + as now); a hint that is a *subclass* of `float`, `int`, `bool` or `str` maps to the base + (`np.float64` subclasses `float`), and the numpy scalar families that do not (`np.int64`, + `np.bool_`) are recognized by their `__mro__` names without the support module importing numpy. + In `return_shape`: see through `Optional[tuple[…]]` to the tuple. Test: `typed_more.py` in + `test_types.cpp` with the audit's table. CHANGELOG: the mapping changes (a `float | int` field + was a symbol attribute). - [ ] **8.7 Windows: paths meet Max as UTF-8 (A9).** Every `path.string()` handed to a Max call or a console line (`locatefile_extended` in the constructor, the core's "No file …" and - "Failed to load …") goes through `u8string()`. Verify in the runbook's Windows step with the - package under a folder named with a non-ASCII character: the watcher starts, a save reloads. + "Failed to load …") goes through `u8string()` (C++20: `reinterpret_cast` its `c_str()`). Verify + in the runbook's Windows step with the package under a folder named with a non-ASCII character: + the watcher starts, a save reloads. A bug fix with no contract change: the one item here that + may follow 1.0. - [ ] **8.8 The Mac session for this phase.** Runtime tests for 8.2 (`faults`) and 8.3 (`worker`); 8.3's stack measurement; the macOS half of 8.7 is not needed (POSIX paths are UTF-8). Then - repeat the runbook's step 4 on the release packages, and tag. + repeat the runbook's step 4 on the release packages, and tag `v0.11.0`. + +*The 1.0 gate (revised):* 8.1–8.6 change what the ReadMe promises or what a class can rely on, so +they land before 1.0, where D5 still allows them; 8.7 may follow it. ## Phase 7 — plugin front ends (optional, post-1.0) From 8aedb16e36e3f1c81bb2729dcba1b12860671d39 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 15:58:24 +0000 Subject: [PATCH 04/11] 8.1: say what is true, and two small hardenings The ReadMe's claim that nothing a class does can take Max down becomes what the guards cover: no Python exception, sys.exit() included. Its errors paragraph names what never reaches Python's exception machinery and so is not caught (os._exit(), a crash in a C extension, code that never returns and which thread it freezes), that helper modules load once per Max session, and that a second embedded CPython in the same Max is unsupported. The plan's bar 1 says the same; D2 now describes the loader as built (compile and exec as _tap_python_, no __spec__). CLAUDE.md's honest-limits rule carries the list. assemble-package.py's extract() refuses any entry whose path would leave the destination, through `..`, an absolute path or a symlink an earlier entry made, and any symlink whose target is absolute or resolves outside; checked with crafted zips. style.yml says why the family's own drift check is the one reference pinned by release rather than SHA. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- .github/workflows/style.yml | 4 +++- CLAUDE.md | 6 +++++- ReadMe.md | 6 +++--- docs/PRODUCTION-PLAN.md | 19 +++++++++++++++---- scripts/assemble-package.py | 23 +++++++++++++++++++---- 5 files changed, 45 insertions(+), 13 deletions(-) diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml index 213e38a..5cd4584 100644 --- a/.github/workflows/style.yml +++ b/.github/workflows/style.yml @@ -14,7 +14,9 @@ name: Tap House Style on: [push, pull_request] # Least privilege: the jobs only read the repository. Third-party actions are pinned by -# commit SHA (the tag each SHA was cut from is noted beside it). +# commit SHA (the tag each SHA was cut from is noted beside it). The one reference by tag is the +# family's own drift check (tap/taphouse, below): its `ref` input must name the same release of +# the canonical files, and the family moves that tag together, so it is pinned by release, not SHA. permissions: contents: read diff --git a/CLAUDE.md b/CLAUDE.md index dadcd0c..f7fe930 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,7 +145,11 @@ attaches all the zips + SHA256s to a release — a pre-release for 0.x, a draft - **Honest limits are pinned, not hidden.** A known bug or limit gets a test that states it (named for the promise, with the plan item that will change it); fixes land against a test that reproduces - the bug first. The core battery is where that happens. + the bug first. The core battery is where that happens. Limits no test can pin are written in the + ReadMe's errors paragraph and stay there: the guards catch every Python *exception*, not a + process exit below Python (`os._exit()`), a crash in a C extension, or code that never returns; + helper modules load once per session; a second embedded CPython in the same Max is unsupported. + Never let a document claim more than that. - **Style:** `STYLE.md`, `.clang-format`, `.clang-tidy`, `.pre-commit-config.yaml`, `scripts/tidy.sh` and the SessionStart hook are canonical TapHouse copies — never hand-edit them (CI drift-checks). New files use the STYLE.md §3 SPDX banner. clang-tidy compiles with a clang front end; treat it as a diff --git a/ReadMe.md b/ReadMe.md index b6926ea..28c6425 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -14,7 +14,7 @@ Write Max objects in Python. - The class's **type-annotated attributes** become Max attributes (`@gain 0.5` in the object box, `gain 0.5`, `getgain`, attrui — it all works). - The class's **public methods** become Max messages, called according to their signatures, with arguments converted according to their type hints. - The source file is **watched and hot-reloaded** every time you save it, keeping attribute values, so you can live-code DSP with Max running. -- Python's `print()` output and tracebacks land in the **Max console** — and nothing your code does, `sys.exit()` included, can take Max down. +- Python's `print()` output and tracebacks land in the **Max console** — and no exception your code raises, `sys.exit()` included, can take Max down. ```python from attrs import define, field @@ -86,7 +86,7 @@ Without a runtime the object still loads, and says in the Max console what is mi ## Writing a class -Python sources live in the package's `python` folder. `[tap.python~ name]` loads `python/name.py` and instantiates the class `name` defined in it (the file, class, and argument must share the same name, which must be a valid Python identifier; with no argument, `default` is loaded). The file is loaded by its path, never through `import`, so a file named like a standard-library module (`random.py`, `json.py`) works and does not shadow that module for anyone else. Other files in the `python` folder can be imported by your class as helper modules (the folder is on `sys.path`, after the standard library). Text is UTF-8: the interpreter runs in Python's UTF-8 mode, so `open()` reads and writes UTF-8 unless you pass another `encoding`, whatever the machine's locale. +Python sources live in the package's `python` folder. `[tap.python~ name]` loads `python/name.py` and instantiates the class `name` defined in it (the file, class, and argument must share the same name, which must be a valid Python identifier; with no argument, `default` is loaded). The file is loaded by its path, never through `import`, so a file named like a standard-library module (`random.py`, `json.py`) works and does not shadow that module for anyone else. Other files in the `python` folder can be imported by your class as helper modules (the folder is on `sys.path`, after the standard library) — they are imported once per Max session, as any module is, so a save of a helper is not picked up until Max restarts; what you are live-coding belongs in the class file. Text is UTF-8: the interpreter runs in Python's UTF-8 mode, so `open()` reads and writes UTF-8 unless you pass another `encoding`, whatever the machine's locale. - **Attributes** — class-level annotated fields (e.g. via `attrs`) become Max attributes. `int` maps to a Max `long`, `float` to `float64`, `bool` to an on/off `long` (your field receives a real `bool`), anything else to a symbol; `Optional[X]` and `X | None` count as `X`. Names starting with `_` are private, and `ClassVar`s are not fields. If a hint cannot be resolved (say a name imported only under `TYPE_CHECKING`), the console says so and the annotation is read as written. - **Messages** — public methods (including classmethods and staticmethods) become Max messages, called according to their signature: parameters with defaults are optional, `*args` takes any number of arguments, and a keyword-only parameter must have a default (a method with a required keyword-only parameter is not exposed). Argument hints (`int`, `float`, `bool`, `str`) drive the conversion from Max atoms; an unannotated parameter receives the atom as it is (an `int`, `float` or `str`). Methods named `int`, `float`, `symbol`, and `bang` map to those standard Max messages. Names Max or the object handle themselves (`filechanged`, `dsp64`, `notify`, `assist`, `loadbang`, `dblclick`, `anything`, …, and the object's own attributes `mode`, `latency` and `latencysamples`) are not exposed — as methods, or as fields; the console names any skipped this way so you can rename it. @@ -98,7 +98,7 @@ Python sources live in the package's `python` folder. `[tap.python~ name]` loads - **Worker mode** — by default `process()` runs on Max's audio thread, so whatever else holds Python's interpreter — a reload, a message, another instance — holds up the audio while it does. With `@mode worker` it runs on a thread of its own instead, `@latency` milliseconds behind the audio (30 by default, rounded up to whole signal vectors): the audio thread then only copies vectors to and from that thread, and never waits for Python. Max computes a whole I/O vector's worth of signal vectors at once, so `@latency` must be longer than the I/O vector (Options > Audio Status: 512 samples is 11.6 ms at 44.1 kHz) — what is left over is the time Python has to keep up; raise it with a larger I/O vector. If Python falls further behind, the vectors it is late for are output as silence and the console says so, once per load; the delay stays the same. The read-only `@latencysamples` gives the delay in samples, for aligning other signal paths (a `delay~`, say); in direct mode it is 0. The worker thread has the real-time scheduling of an audio thread. `@mode` and `@latency` take effect as soon as they are set, by rebuilding the signal chain. - **Audio settings** — an optional method `prepare(self, sample_rate: float, vector_size: int) -> None` is called with Max's sample rate and vector size before the object processes any audio, again whenever they change, and on every reload before the new code runs. `python/allpass.py` uses it to size its delay line. - **Hot reload** — saving the `.py` file reloads it in place, as a fresh module (names you deleted from the file are gone): the attributes and messages follow the new class, and audio resumes with the new code. Until the new code is ready the object keeps running the old one, so a successful reload swaps in without a gap in the audio. Attribute values carry over — set from the patcher or by your own code — for every attribute the new class still has with the same type; an attribute whose type changed starts from its new default, and one you removed disappears from the object. If the file has an error, the object prints the traceback to the Max console and outputs silence until the next successful reload (and the attribute values come back with it); a save that breaks a file shared by many objects is reported once, not by each, while an object created later with the file still broken says so again. Every object using the same file shares one execution of it per save, and the console says so once — `Loaded name.py: process() bound, one call per sample` (or per vector), from whichever object ran the file — along with anything true of the class, such as a method skipped for its name; a save that changes nothing reloads silently. Errors that belong to one object, such as an exception in its constructor, are still reported by each. Max's file watcher notices a save within a couple of seconds, but it coalesces saves made in quick succession — a script saving once a second, say — so the object then reloads for only some of them, and can lag behind until they stop; sending the object `filechanged` reloads it at once. -- **Errors** — an exception raised by your code (in `process()`, a message, an attribute setter, the constructor or at import) prints its traceback to the Max console; it never takes Max down. That includes `sys.exit()`, which is reported like any other exception rather than quitting Max. +- **Errors** — an exception raised by your code (in `process()`, a message, an attribute setter, the constructor or at import) prints its traceback to the Max console and never takes Max down. That includes `sys.exit()`, which is reported like any other exception rather than quitting Max. What the object cannot catch is what never reaches Python's exception machinery: a process exit below it (`os._exit()`, `os.abort()`), a crash in a C extension (a broken wheel, `ctypes`), or code that never returns — an endless loop in a message or the constructor freezes Max's main thread, and in `process()` it stalls the audio thread (in worker mode, the worker thread; stopping that worker then waits on it). Another Max external that embeds its own CPython alongside this one is untested and unsupported: two interpreters in one Max share process-wide state neither expects to. ### A note on performance diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index 6a717e3..a1253aa 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -10,7 +10,11 @@ note the PR that closed them. ## The bar 1. **User Python cannot take Max down.** No crash, no process exit, no hang from anything a user - script does (`sys.exit()`, exceptions, non-numeric returns, NaN, pathological names). + script does *that reaches Python's exception machinery* (`sys.exit()`, exceptions, non-numeric + returns, NaN, pathological names). What never reaches it is outside the guard and said so in the + ReadMe (8.1): a process exit below Python (`os._exit()`), a crash in a C extension, and code that + never returns — which freezes the thread it runs on (main for a message, audio for `process()`; + worker mode confines the latter to the worker thread and, since 8.3, stops it or abandons it). 2. **The audio thread is protected.** No crash, no allocation or console posting on the hot path, and a defined, documented behavior (silence, not a stall) when Python misbehaves. 3. **Every documented behavior is pinned by a test** — unit (mock kernel) where possible, in-Max @@ -24,7 +28,7 @@ note the PR that closed them. |---|---|---| | D1 | Where `process()` runs | **Direct now, worker later.** A block path called once per vector on the audio thread (zero latency); the per-sample path stays as a documented slow path; an opt-in worker-thread mode (`@mode worker`, lock-free FIFO, ≥1 vector latency, audio thread never takes the GIL) follows. | | D1a | Opting into the block path | **By type hint.** `process(self, x: np.ndarray) -> np.ndarray` is called per vector; `process(self, x: float) -> float` per sample. Detected at bind time. | -| D2 | How `[tap.python~ name]` loads code | **File-based.** `python/.py` is loaded by path (`importlib.util.spec_from_file_location`) under a private module name (`_tap_python_user.`); `name` must be an identifier; `python/` is *appended* to `sys.path` so sibling-helper imports keep working without shadowing the stdlib. | +| D2 | How `[tap.python~ name]` loads code | **File-based.** `python/.py` is loaded by path — its source compiled and executed by the loader in the support module (3.5), never imported, so no `.pyc` is read or written for it (3.6a) — as a fresh module registered in `sys.modules` as `_tap_python_` (with `__file__` only: no `__spec__` or `__package__`, so no relative imports; the folder is flat by design); `name` must be an identifier; `python/` is *appended* to `sys.path` so sibling-helper imports keep working without shadowing the stdlib. *(Revised with 8.1: the first draft named `importlib.util.spec_from_file_location` and `_tap_python_user.`.)* | | D3 | How users get the runtime | **Bundled in the release.** CI assembles a complete per-platform package (externals, `support/` with CPython + attrs + numpy, `python/`, help, docs, licenses). `scripts/install-runtime.*` stays for source builds. Signing/notarization steps are wired but **skip cleanly until credentials exist** (none yet). | | D4 | Python version policy | **Pin 3.13; upgrade deliberately.** One CPython minor per release; a move (e.g. to 3.14's deferred annotations) is its own PR with tests. No free-threaded or subinterpreter builds until numpy supports them. | | D5 | Compatibility before 1.0 | **Breaking changes to the class contract are allowed** where they buy correctness (reserved names, file-based loading, signature dispatch). Each is recorded in a `CHANGELOG.md`; the shipped examples are updated in the same PR. | @@ -552,7 +556,7 @@ each contract change recorded in `CHANGELOG.md` (D5). The honest limits that rem down rather than discovered again. *This plan was itself audited before being adopted (2026-10-01); what that changed is marked "revised".* -- [ ] **8.1 Say what is true (A4, A7a, A12 — docs and small hardening, no behavior change).** +- [x] **8.1 Say what is true (A4, A7a, A12 — docs and small hardening, no behavior change).** ReadMe lines 17 and 101, and bar 1 above: *no Python exception*, `sys.exit()` included, takes Max down or stops the object — what happens below Python (`os._exit()`, a C extension crashing) or never returns to it (an endless loop in a message freezes Max's main thread; in `process()` @@ -565,7 +569,14 @@ down rather than discovered again. *This plan was itself audited before being ad absolute or resolves outside the destination, and check every written path), and pin the taphouse drift workflow by SHA or note the exception beside the policy comment (A11). *Done when:* the three documents agree with each other and with the code; `--merge` of a crafted zip - with a `../` symlink entry fails. + with a `../` symlink entry fails. *Done:* the ReadMe's two sentences and bar 1 now name what the + guards cover and what they cannot; helpers and a second interpreter are stated; D2 corrected; + CLAUDE.md carries the rule. `extract()` refuses an entry whose path leaves the destination + (through `..`, an absolute path, or a symlink an earlier entry made) and a symlink whose target + is absolute or resolves outside — checked with four crafted zips (a `..` name, an absolute link, + an escaping link, a write through an escaping link), all refused, and an honest zip whose + `bin/python3` link survives. The taphouse drift check stays pinned by release, with the reason + beside the policy comment. - [ ] **8.2 Crash and contract fixes in the core and the glue (A2, A3).** *A2:* `process_samples()` sanitizes the samples already written before each of its two early returns (input conversion failing; the call raising). Test first, in `test_realtime.cpp`: the diff --git a/scripts/assemble-package.py b/scripts/assemble-package.py index b0a670a..13eae98 100644 --- a/scripts/assemble-package.py +++ b/scripts/assemble-package.py @@ -248,19 +248,34 @@ def same_content(a: Path, b: Path) -> bool: def extract(archive: Path, destination: Path) -> None: """Unzip a single-platform release zip as the platform that made it would: entries named with backslashes (as some Windows zip writers make them) as folders, and the macOS runtime's symlinks - and executable bits as they were — which Python's zipfile leaves out.""" + and executable bits as they were — which Python's zipfile leaves out. + + Nothing is written outside `destination`: an entry whose path leaves it (through `..`, or + through a symlink an earlier entry made) is refused, as is a symlink whose target is absolute + or resolves outside it — a crafted zip must fail, not write elsewhere.""" + root = destination.resolve() + + def inside(path: Path) -> bool: + return path.resolve().is_relative_to(root) # resolve() follows the symlinks made so far + with zipfile.ZipFile(archive) as zip_file: for info in zip_file.infolist(): name = info.filename.replace("\\", "/") - if name.startswith("__MACOSX/") or ".." in Path(name).parts: - continue # resource forks; and never a path out of the destination + if name.startswith("__MACOSX/"): + continue # resource forks target = destination / name + if ".." in Path(name).parts or Path(name).is_absolute() or not inside(target): + raise AssemblyError(f"{archive.name}: entry {info.filename!r} would land outside {destination}") mode = info.external_attr >> 16 if info.create_system == 3 else 0 # Unix if name.endswith("/"): target.mkdir(parents=True, exist_ok=True) elif stat.S_ISLNK(mode): + link = zip_file.read(info).decode("utf-8") + if Path(link).is_absolute() or not inside(target.parent / link): + raise AssemblyError(f"{archive.name}: symlink {info.filename!r} -> {link!r} points outside " + f"{destination}") target.parent.mkdir(parents=True, exist_ok=True) - os.symlink(zip_file.read(info).decode("utf-8"), target) + os.symlink(link, target) else: target.parent.mkdir(parents=True, exist_ok=True) target.write_bytes(zip_file.read(info)) From 5512c86a84d35707a87ccccd48c959310744820c Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 16:06:56 +0000 Subject: [PATCH 05/11] 8.2: sanitize before process() raises; reserve what Max calls with C arguments A2: every path out of process_samples() goes through one sanitize(io, written, frame_count), the two early returns included, so a vector that held NaNs before the exception no longer lets them through; the block path shares the helper. Pinned by a core scenario on nan_then_raise.py, which rendered three NaNs before the change. A3: the object reserves every message min treats as A_CANT plus Max's dspstate, inputchanged and multichanneloutputs, and the processor takes a host_answers predicate beside the list: answered_by_max() asks object_getmethod(), excluding the messages the object itself added for the previous incarnation's methods, which are still registered while load() runs. A Python method of such a name was registered with the A_GIMME trampoline, which Max then called with a long or a pointer: the filechanged crash of 6.1 in its general form. Pinned by a core scenario for the predicate, a glue scenario that the four C-argument names are absent and int/float/symbol/bang/list/greet present, and new steps in the faults runtime test (DSP toggled, a cord connected by thispatcher, with those methods in the class), for the Mac session. CHANGELOG entries for both and for 8.1. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- CHANGELOG.md | 25 + core/include/tap/python/processor.h | 58 +- core/tests/python/nan_then_raise.py | 9 + core/tests/test_realtime.cpp | 19 + core/tests/test_safety.cpp | 17 + docs/PRODUCTION-PLAN.md | 17 +- runtime-tests/make_patchers.py | 14 +- .../tap.python~.faults.maxtest.maxpat | 1479 ++++++++++++----- runtime-tests/python/maxtest_faults.py | 14 + .../tap.python_tilde/tap.python_tilde.h | 95 +- .../tap.python_tilde_test.cpp | 34 + 11 files changed, 1334 insertions(+), 447 deletions(-) create mode 100644 core/tests/python/nan_then_raise.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d1d8c4c..9b5177e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,31 @@ Changes to the Python class contract and to the object's behavior, newest first. breaking changes to the contract are allowed where they buy correctness (D5 in `docs/PRODUCTION-PLAN.md`); each is recorded here. +## Unreleased + +### Changed — more names are reserved + +- **A method or field named like a message Max sends with C arguments is not exposed**, with the + console saying so: `dspstate`, `fileusage`, `patchlineupdate`, `inputchanged`, + `multichanneloutputs`, `edclose`, `okclose`, `oksize`, `paint`, `dictionary`, `getplaystate`, + `key`, the mouse and focus messages, `mousewheel`, and the `*_setup` names — every message min + treats as `A_CANT`, plus Max's own — and, as a guard, any name the Max object already answers + itself. A Python method of such a name used to be registered as an ordinary message, which Max + then called with a `long` or a pointer: the crash that every save once caused through + `filechanged`, in its general form. (Plan 8.2, audit A3.) + +### Fixed + +- **Samples computed before `process()` raised are sanitized.** A `process()` that returned NaN for + part of a vector and then raised let those NaNs through; the contract says non-finite output is + replaced with 0.0, and now it is on that path too. (Plan 8.2, audit A2.) +- **The documents say what the guards cover.** No Python exception, `sys.exit()` included, takes + Max down; what never reaches Python's exception machinery — `os._exit()`, a crash in a C + extension, code that never returns — is not caught, and the ReadMe now says so, as it says that + helper modules load once per Max session and that a second embedded CPython in the same Max is + unsupported. `assemble-package.py --merge` refuses a zip whose entries or symlinks would land + outside the package. (Plan 8.1.) + ## 0.10.0 — 2026-10-01 ### Added — one package for every platform diff --git a/core/include/tap/python/processor.h b/core/include/tap/python/processor.h index b06efac..efc11c9 100644 --- a/core/include/tap/python/processor.h +++ b/core/include/tap/python/processor.h @@ -82,12 +82,18 @@ namespace tap::python { /// @param report_ready called on the audio thread when something needs reporting; the /// host must then call flush_reports() from its main thread. Must be /// real-time safe (no locks, no allocation). + /// @param host_answers true for a name the host object already answers (in Max, a method + /// its class registered), which a Python method or field must not + /// replace either: reserved like the list, with the same diagnostic. + /// Called on the main thread during load(), with the GIL held. explicit processor(std::string source_name, log_function log = {}, - std::vector reserved_messages = {}, std::function report_ready = {}) + std::vector reserved_messages = {}, std::function report_ready = {}, + std::function host_answers = {}) : m_source_name{std::move(source_name)} , m_log{std::move(log)} , m_reserved_messages{std::move(reserved_messages)} - , m_report_ready{std::move(report_ready)} {} + , m_report_ready{std::move(report_ready)} + , m_host_answers{std::move(host_answers)} {} ~processor() { if (!Py_IsInitialized()) { @@ -562,18 +568,20 @@ namespace tap::python { bool m_prepared{}; // Reports recorded on the audio thread for flush_reports(), and whether each kind has // already been recorded since the last load(). - std::function m_report_ready; - std::atomic m_pending_exception{}; // strong - std::atomic m_pending_non_numeric{}; // strong: the returned object's type - std::atomic m_pending_non_finite{}; - std::atomic m_pending_bad_length{k_no_length}; - std::atomic m_bad_length_expected{}; - std::atomic m_warned_non_numeric{}; - std::atomic m_warned_non_finite{}; - std::atomic m_warned_bad_length{}; - std::atomic m_pending_bad_count{k_no_length}; - std::atomic m_bad_count_expected{}; - std::atomic m_warned_bad_count{}; + std::function m_report_ready; + // Names the host object answers itself (main thread; see the constructor). + std::function m_host_answers; + std::atomic m_pending_exception{}; // strong + std::atomic m_pending_non_numeric{}; // strong: the returned object's type + std::atomic m_pending_non_finite{}; + std::atomic m_pending_bad_length{k_no_length}; + std::atomic m_bad_length_expected{}; + std::atomic m_warned_non_numeric{}; + std::atomic m_warned_non_finite{}; + std::atomic m_warned_bad_length{}; + std::atomic m_pending_bad_count{k_no_length}; + std::atomic m_bad_count_expected{}; + std::atomic m_warned_bad_count{}; void log(const log_level level, const std::string& text) const { if (m_log) { @@ -833,6 +841,15 @@ namespace tap::python { std::size_t outputs; }; + /// sanitize() the first `written` of the host's output channels: every path out of a vector, + /// including the early returns when process() raises, goes through here, so no sample the + /// class wrote reaches the host unchecked. + void sanitize(const channels& io, const std::size_t written, const std::size_t frame_count) { + for (std::size_t c = 0; c < written; ++c) { + sanitize(io.out[c], frame_count); + } + } + /// Zero channels [first, end) of `out`, from `first_frame` to `frame_count`. static void silence(double* const* out, const std::size_t first, const std::size_t end, const std::size_t first_frame, const std::size_t frame_count) { @@ -856,6 +873,7 @@ namespace tap::python { release_arguments(call_args, c); record_exception(); silence(io.out, 0, written, i, frame_count); + sanitize(io, written, frame_count); // the samples before this one are still the class's return; } } @@ -866,6 +884,7 @@ namespace tap::python { record_exception(); unbind_process(function); // load() re-arms it silence(io.out, 0, written, i, frame_count); + sanitize(io, written, frame_count); // the samples before this one are still the class's return; } if (io.outputs == 1) { @@ -886,9 +905,7 @@ namespace tap::python { } Py_DECREF(result); } - for (std::size_t c = 0; c < written; ++c) { - sanitize(io.out[c], frame_count); - } + sanitize(io, written, frame_count); } /// Release the `count` input values after `call_args[0]`. @@ -959,9 +976,7 @@ namespace tap::python { silence(io.out, 0, written, 0, frame_count); } Py_DECREF(result); - for (std::size_t c = 0; c < written; ++c) { - sanitize(io.out[c], frame_count); - } + sanitize(io, written, frame_count); } static bool is_native_double(const Py_buffer& view) { @@ -1116,7 +1131,8 @@ namespace tap::python { } bool is_reserved(const std::string& name) const { - return std::find(m_reserved_messages.begin(), m_reserved_messages.end(), name) != m_reserved_messages.end(); + return std::find(m_reserved_messages.begin(), m_reserved_messages.end(), name) != m_reserved_messages.end() + || (m_host_answers && m_host_answers(name)); } /// A support-module call with one argument; a new reference, or nullptr with an error set. diff --git a/core/tests/python/nan_then_raise.py b/core/tests/python/nan_then_raise.py new file mode 100644 index 0000000..46c83c0 --- /dev/null +++ b/core/tests/python/nan_then_raise.py @@ -0,0 +1,9 @@ +# Test fixture: NaN output until an input of 0.75 or more, which raises (plan 8.2, audit A2): +# the samples written before the exception must still be sanitized. + + +class nan_then_raise: + def process(self, x: float) -> float: + if x >= 0.75: + raise RuntimeError("nan_then_raise: raised on purpose") + return float("nan") diff --git a/core/tests/test_realtime.cpp b/core/tests/test_realtime.cpp index a56fb26..976033f 100644 --- a/core/tests/test_realtime.cpp +++ b/core/tests/test_realtime.cpp @@ -119,6 +119,25 @@ SCENARIO("Non-finite output is replaced with 0.0 and reported once per load") { CHECK(h.log.lines().empty()); } +SCENARIO("Samples written before process() raises are still sanitized (plan 8.2)") { + ensure_runtime(); + harness h{"nan_then_raise"}; + std::vector in{0.1, 0.1, 0.1, 0.9, 0.1, 0.1}; // NaN, NaN, NaN, then an exception + std::vector out(6, 12345.0); + REQUIRE(h.p.load()); + + h.p.process(in.data(), out.data(), out.size()); + THEN("the NaNs before the exception are 0.0, as the contract says, and the rest is silence") { + CHECK(out == std::vector{0.0, 0.0, 0.0, 0.0, 0.0, 0.0}); + } + THEN("both the exception and the non-finite output are reported") { + CHECK_FALSE(h.p.has_process()); + h.p.flush_reports(); + CHECK(h.log.contains("process() raised an exception", log_level::error)); + CHECK(h.log.contains("non-finite sample (NaN or infinity)", log_level::error)); + } +} + // 2.2 — the numpy block path SCENARIO("A process() hinted np.ndarray is called once per vector") { diff --git a/core/tests/test_safety.cpp b/core/tests/test_safety.cpp index 67ab0cb..9db05aa 100644 --- a/core/tests/test_safety.cpp +++ b/core/tests/test_safety.cpp @@ -152,6 +152,23 @@ SCENARIO("Methods named like reserved host messages are not exposed") { CHECK(p.has_process()); } +SCENARIO("A name the host object already answers is reserved too (plan 8.2)") { + ensure_runtime(); + log_capture log; + // the host says it answers `allowed` itself (in Max: a method its class registered) + processor p{"reserved", log.sink(), {"filechanged"}, {}, [](const std::string& name) { return name == "allowed"; }}; + REQUIRE(p.load()); + + std::vector names; + for (const auto& m : p.messages()) { + names.push_back(m.name); + } + CHECK(names == std::vector{"dsp64"}); // not in the list, not answered by the host + CHECK(log.contains("allowed() is reserved", log_level::error)); + CHECK(log.contains("filechanged() is reserved", log_level::error)); + CHECK(p.has_process()); // process() is never asked of the host +} + SCENARIO("Fields named like reserved host attributes are not exposed (plan 2.5)") { ensure_runtime(); log_capture log; diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index a1253aa..2809904 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -577,7 +577,7 @@ down rather than discovered again. *This plan was itself audited before being ad an escaping link, a write through an escaping link), all refused, and an honest zip whose `bin/python3` link survives. The taphouse drift check stays pinned by release, with the reason beside the policy comment. -- [ ] **8.2 Crash and contract fixes in the core and the glue (A2, A3).** +- [x] **8.2 Crash and contract fixes in the core and the glue (A2, A3).** *A2:* `process_samples()` sanitizes the samples already written before each of its two early returns (input conversion failing; the call raising). Test first, in `test_realtime.cpp`: the audit's `nan_then_raise.py` (NaN until an input ≥ 0.75, then an exception) renders @@ -598,7 +598,20 @@ down rather than discovered again. *This plan was itself audited before being ad registered and each is announced once; the `faults` runtime test gains a step that toggles DSP and connects a cord with such a class loaded (the crash 6.1 found, in its general form). `fileusage` is sent only by Build Collective/Application, so it is a runbook step by hand, not a - runtime test. CHANGELOG: the new reserved names. + runtime test. CHANGELOG: the new reserved names. *Done:* every path out of `process_samples()` goes through one + `sanitize(io, written, frame_count)`; the fixture renders `0 0 0 0 0 0` where it rendered three + NaNs, and reports both the exception and the non-finite output. The processor takes a + `host_answers` predicate beside the list; the object's `reserved_messages()` carries min's + `A_CANT` names plus `dspstate`, `inputchanged` and `multichanneloutputs`, and `answered_by_max()` + asks `object_getmethod()` — excluding the messages the object itself added for the previous + incarnation's methods, which are still registered while `load()` runs and must not reserve the + class's own names on a reload. Pinned by a core scenario for the predicate, the glue scenario + with `maxtest_mock_reserved` (the four C-argument names absent, `int`/`float`/`symbol`/`bang`/ + `list`/`greet` present, audio bound), and the `faults` runtime test's new steps (DSP toggled, a + cord connected by `thispatcher`, with those methods in the class) — to run in the Mac session + (8.8), as is Build Collective for `fileusage`. In the mock kernel `object_getmethod()` always + answers null, so the guard's own effect is seen only in Max: the existing + `attributes-and-messages` runtime test is what would show it over-reserving. - [ ] **8.3 Worker mode never hangs Max (A1, A6 — `worker.h`).** *A1:* `stop()` bounds its join. The worker records its OS thread identifier (`PyThread_get_thread_ident()`, no GIL needed) as it starts; if the thread has not finished diff --git a/runtime-tests/make_patchers.py b/runtime-tests/make_patchers.py index b0882bd..9bbbcb9 100644 --- a/runtime-tests/make_patchers.py +++ b/runtime-tests/make_patchers.py @@ -680,10 +680,15 @@ def faults() -> Test: t = Test("tap.python~.faults.maxtest.maxpat", "sys.exit() in a message is reported, not obeyed; NaN from process() is output as 0 " "and reported once; an exception in process() is reported once, from the main " - "thread, and silences the object — ignoring messages — until the file is reloaded.") + "thread, and silences the object — ignoring messages — until the file is reloaded. Its " + "class also has methods named dspstate, patchlineupdate, inputchanged and fileusage, " + "which Max calls with C arguments (plan 8.2): they must not be exposed, so toggling DSP " + "and connecting a cord with it loaded must pass (fileusage is Build Collective's: by hand).") source = t.signal(1.0) - py = t.python("maxtest_faults") + py = t.patcher.box("tap.python~ maxtest_faults", 1, 1, column=2, outlettype=["signal"], varname="faults") t.patcher.connect(source, 0, py) + t.patcher.box("sig~ 1.", 1, 1, column=2, outlettype=["signal"], varname="second_source") + scripting = t.obj("thispatcher") for pattern in ("SystemExit", "non-finite", "audio.disabled"): t.count_errors(pattern) t.step(t.sample_equals("passes-signal", py, 1.0)) @@ -702,6 +707,11 @@ def faults() -> Test: t.step(t.sample_equals("messages-ignored-until-reload", py, 0.0)) t.step(t.send("filechanged", py)) t.step(t.sample_equals("reload-restores-audio", py, 1.0)) + t.step(t.dsp(False)) + t.step(t.dsp(True), wait=100) + t.step(t.sample_equals("survives-dsp-toggle-with-c-argument-names", py, 1.0), wait=1000) + t.step(t.send("script connect second_source 0 faults 0", scripting)) + t.step(t.sample_equals("survives-a-new-cord-with-c-argument-names", py, 2.0)) # both sources summed return t diff --git a/runtime-tests/patchers/tap.python~.faults.maxtest.maxpat b/runtime-tests/patchers/tap.python~.faults.maxtest.maxpat index fbcc2d3..9e87d43 100644 --- a/runtime-tests/patchers/tap.python~.faults.maxtest.maxpat +++ b/runtime-tests/patchers/tap.python~.faults.maxtest.maxpat @@ -35,7 +35,7 @@ 480, 80.0 ], - "text": "tap.python~.faults\n\nsys.exit() in a message is reported, not obeyed; NaN from process() is output as 0 and reported once; an exception in process() is reported once, from the main thread, and silences the object \u2014 ignoring messages \u2014 until the file is reloaded.", + "text": "tap.python~.faults\n\nsys.exit() in a message is reported, not obeyed; NaN from process() is output as 0 and reported once; an exception in process() is reported once, from the main thread, and silences the object \u2014 ignoring messages \u2014 until the file is reloaded. Its class also has methods named dspstate, patchlineupdate, inputchanged and fileusage, which Max calls with C arguments (plan 8.2): they must not be exposed, so toggling DSP and connecting a cord with it loaded must pass (fileusage is Build Collective's: by hand).", "linecount": 4 } }, @@ -72,7 +72,8 @@ 198.0, 22.0 ], - "text": "tap.python~ maxtest_faults" + "text": "tap.python~ maxtest_faults", + "varname": "faults" } }, { @@ -81,6 +82,43 @@ "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, + "outlettype": [ + "signal" + ], + "patching_rect": [ + 520.0, + 80.0, + 65.0, + 22.0 + ], + "text": "sig~ 1.", + "varname": "second_source" + } + }, + { + "box": { + "id": "obj-5", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 520.0, + 110.0, + 93.0, + 22.0 + ], + "text": "thispatcher" + } + }, + { + "box": { + "id": "obj-6", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, "outlettype": [ "" ], @@ -95,7 +133,7 @@ }, { "box": { - "id": "obj-5", + "id": "obj-7", "maxclass": "newobj", "numinlets": 1, "numoutlets": 5, @@ -117,7 +155,7 @@ }, { "box": { - "id": "obj-6", + "id": "obj-8", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -135,7 +173,7 @@ }, { "box": { - "id": "obj-7", + "id": "obj-9", "maxclass": "newobj", "numinlets": 3, "numoutlets": 4, @@ -156,7 +194,7 @@ }, { "box": { - "id": "obj-8", + "id": "obj-10", "maxclass": "newobj", "numinlets": 1, "numoutlets": 5, @@ -178,7 +216,7 @@ }, { "box": { - "id": "obj-9", + "id": "obj-11", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -196,7 +234,7 @@ }, { "box": { - "id": "obj-10", + "id": "obj-12", "maxclass": "newobj", "numinlets": 3, "numoutlets": 4, @@ -217,7 +255,7 @@ }, { "box": { - "id": "obj-11", + "id": "obj-13", "maxclass": "newobj", "numinlets": 1, "numoutlets": 5, @@ -239,7 +277,7 @@ }, { "box": { - "id": "obj-12", + "id": "obj-14", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -257,7 +295,7 @@ }, { "box": { - "id": "obj-13", + "id": "obj-15", "maxclass": "newobj", "numinlets": 3, "numoutlets": 4, @@ -278,7 +316,7 @@ }, { "box": { - "id": "obj-14", + "id": "obj-16", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -296,7 +334,7 @@ }, { "box": { - "id": "obj-15", + "id": "obj-17", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -314,7 +352,7 @@ }, { "box": { - "id": "obj-16", + "id": "obj-18", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -332,7 +370,7 @@ }, { "box": { - "id": "obj-17", + "id": "obj-19", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -350,7 +388,7 @@ }, { "box": { - "id": "obj-18", + "id": "obj-20", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -368,7 +406,7 @@ }, { "box": { - "id": "obj-19", + "id": "obj-21", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -386,7 +424,7 @@ }, { "box": { - "id": "obj-20", + "id": "obj-22", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -404,7 +442,7 @@ }, { "box": { - "id": "obj-21", + "id": "obj-23", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -422,7 +460,7 @@ }, { "box": { - "id": "obj-22", + "id": "obj-24", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -440,7 +478,7 @@ }, { "box": { - "id": "obj-23", + "id": "obj-25", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -458,7 +496,7 @@ }, { "box": { - "id": "obj-24", + "id": "obj-26", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -476,7 +514,7 @@ }, { "box": { - "id": "obj-25", + "id": "obj-27", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -494,7 +532,7 @@ }, { "box": { - "id": "obj-26", + "id": "obj-28", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -512,7 +550,7 @@ }, { "box": { - "id": "obj-27", + "id": "obj-29", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -530,7 +568,7 @@ }, { "box": { - "id": "obj-28", + "id": "obj-30", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -548,7 +586,7 @@ }, { "box": { - "id": "obj-29", + "id": "obj-31", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -566,7 +604,7 @@ }, { "box": { - "id": "obj-30", + "id": "obj-32", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -584,7 +622,7 @@ }, { "box": { - "id": "obj-31", + "id": "obj-33", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -602,7 +640,7 @@ }, { "box": { - "id": "obj-32", + "id": "obj-34", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -620,7 +658,7 @@ }, { "box": { - "id": "obj-33", + "id": "obj-35", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -638,7 +676,7 @@ }, { "box": { - "id": "obj-34", + "id": "obj-36", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -656,7 +694,7 @@ }, { "box": { - "id": "obj-35", + "id": "obj-37", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -674,7 +712,7 @@ }, { "box": { - "id": "obj-36", + "id": "obj-38", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -692,7 +730,7 @@ }, { "box": { - "id": "obj-37", + "id": "obj-39", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -710,7 +748,7 @@ }, { "box": { - "id": "obj-38", + "id": "obj-40", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -728,7 +766,7 @@ }, { "box": { - "id": "obj-39", + "id": "obj-41", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -746,7 +784,7 @@ }, { "box": { - "id": "obj-40", + "id": "obj-42", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -764,7 +802,7 @@ }, { "box": { - "id": "obj-41", + "id": "obj-43", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -782,7 +820,7 @@ }, { "box": { - "id": "obj-42", + "id": "obj-44", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -800,7 +838,7 @@ }, { "box": { - "id": "obj-43", + "id": "obj-45", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -818,7 +856,7 @@ }, { "box": { - "id": "obj-44", + "id": "obj-46", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -836,7 +874,7 @@ }, { "box": { - "id": "obj-45", + "id": "obj-47", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -854,7 +892,7 @@ }, { "box": { - "id": "obj-46", + "id": "obj-48", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -872,7 +910,7 @@ }, { "box": { - "id": "obj-47", + "id": "obj-49", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -890,7 +928,7 @@ }, { "box": { - "id": "obj-48", + "id": "obj-50", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -908,7 +946,7 @@ }, { "box": { - "id": "obj-49", + "id": "obj-51", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -926,139 +964,133 @@ }, { "box": { - "id": "obj-50", - "maxclass": "newobj", - "numinlets": 1, - "numoutlets": 0, - "outlettype": [], + "id": "obj-52", + "maxclass": "message", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], "patching_rect": [ - 20.0, - 108.0, - 114.0, + 270.0, + 200.0, + 86.0, 22.0 ], - "text": "test.terminate" + "text": "; dsp stop" } }, { "box": { - "id": "obj-51", - "maxclass": "newobj", - "numinlets": 1, + "id": "obj-53", + "maxclass": "message", + "numinlets": 2, "numoutlets": 1, "outlettype": [ "" ], "patching_rect": [ - 1020.0, - 500.0, - 72.0, + 270.0, + 230.0, + 93.0, 22.0 ], - "text": "tosymbol" + "text": "; dsp start" } }, { "box": { - "id": "obj-52", + "id": "obj-54", "maxclass": "newobj", "numinlets": 1, - "numoutlets": 5, + "numoutlets": 1, "outlettype": [ - "", - "", - "", - "", "" ], "patching_rect": [ - 1020.0, - 530.0, + 770.0, + 740.0, 177.0, 22.0 ], - "text": "regexp \\\" @substitute '" + "text": "test.sample~ @autorun 0" } }, { "box": { - "id": "obj-53", + "id": "obj-55", "maxclass": "newobj", - "numinlets": 1, + "numinlets": 2, "numoutlets": 1, "outlettype": [ "" ], "patching_rect": [ - 1020.0, - 560.0, - 86.0, + 770.0, + 770.0, + 121.0, 22.0 ], - "text": "fromsymbol" + "text": "test.equals 1.0" } }, { "box": { - "id": "obj-54", + "id": "obj-56", "maxclass": "newobj", "numinlets": 1, - "numoutlets": 2, + "numoutlets": 1, "outlettype": [ - "", "" ], "patching_rect": [ - 1020.0, - 590.0, - 51.0, + 770.0, + 800.0, + 387.0, 22.0 ], - "text": "t l b" + "text": "test.assert survives-dsp-toggle-with-c-argument-names" } }, { "box": { - "id": "obj-55", - "maxclass": "newobj", - "numinlets": 3, - "numoutlets": 4, + "id": "obj-57", + "maxclass": "message", + "numinlets": 2, + "numoutlets": 1, "outlettype": [ - "", - "", - "", "" ], "patching_rect": [ - 1020.0, - 620.0, - 135.0, + 270.0, + 260.0, + 289.0, 22.0 ], - "text": "counter 1 1000000" + "text": "script connect second_source 0 faults 0" } }, { "box": { - "id": "obj-56", + "id": "obj-58", "maxclass": "newobj", - "numinlets": 2, + "numinlets": 1, "numoutlets": 1, "outlettype": [ "" ], "patching_rect": [ - 1020.0, - 650.0, - 51.0, + 770.0, + 830.0, + 177.0, 22.0 ], - "text": "<= 50" + "text": "test.sample~ @autorun 0" } }, { "box": { - "id": "obj-57", + "id": "obj-59", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1066,51 +1098,219 @@ "" ], "patching_rect": [ - 1020.0, - 680.0, - 72.0, + 770.0, + 860.0, + 121.0, 22.0 ], - "text": "gate 1 1" + "text": "test.equals 2.0" } }, { "box": { - "id": "obj-58", + "id": "obj-60", "maxclass": "newobj", "numinlets": 1, - "numoutlets": 0, - "outlettype": [], + "numoutlets": 1, + "outlettype": [ + "" + ], "patching_rect": [ - 1020.0, - 710.0, - 170.0, + 770.0, + 890.0, + 387.0, 22.0 ], - "text": "test.log console-error" + "text": "test.assert survives-a-new-cord-with-c-argument-names" } }, { "box": { - "id": "obj-59", + "id": "obj-61", "maxclass": "newobj", "numinlets": 1, - "numoutlets": 1, - "outlettype": [ - "" - ], + "numoutlets": 0, + "outlettype": [], "patching_rect": [ 20.0, - 138.0, - 72.0, + 108.0, + 114.0, 22.0 ], - "text": "loadbang" + "text": "test.terminate" } }, { "box": { - "id": "obj-60", + "id": "obj-62", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 500.0, + 72.0, + 22.0 + ], + "text": "tosymbol" + } + }, + { + "box": { + "id": "obj-63", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 5, + "outlettype": [ + "", + "", + "", + "", + "" + ], + "patching_rect": [ + 1020.0, + 530.0, + 177.0, + 22.0 + ], + "text": "regexp \\\" @substitute '" + } + }, + { + "box": { + "id": "obj-64", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 560.0, + 86.0, + 22.0 + ], + "text": "fromsymbol" + } + }, + { + "box": { + "id": "obj-65", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 1020.0, + 590.0, + 51.0, + 22.0 + ], + "text": "t l b" + } + }, + { + "box": { + "id": "obj-66", + "maxclass": "newobj", + "numinlets": 3, + "numoutlets": 4, + "outlettype": [ + "", + "", + "", + "" + ], + "patching_rect": [ + 1020.0, + 620.0, + 135.0, + 22.0 + ], + "text": "counter 1 1000000" + } + }, + { + "box": { + "id": "obj-67", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 650.0, + 51.0, + 22.0 + ], + "text": "<= 50" + } + }, + { + "box": { + "id": "obj-68", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 680.0, + 72.0, + 22.0 + ], + "text": "gate 1 1" + } + }, + { + "box": { + "id": "obj-69", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 0, + "outlettype": [], + "patching_rect": [ + 1020.0, + 710.0, + 170.0, + 22.0 + ], + "text": "test.log console-error" + } + }, + { + "box": { + "id": "obj-70", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 138.0, + 72.0, + 22.0 + ], + "text": "loadbang" + } + }, + { + "box": { + "id": "obj-71", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1129,7 +1329,7 @@ }, { "box": { - "id": "obj-61", + "id": "obj-72", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -1147,7 +1347,7 @@ }, { "box": { - "id": "obj-62", + "id": "obj-73", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1165,7 +1365,7 @@ }, { "box": { - "id": "obj-63", + "id": "obj-74", "maxclass": "newobj", "numinlets": 1, "numoutlets": 4, @@ -1186,7 +1386,7 @@ }, { "box": { - "id": "obj-64", + "id": "obj-75", "maxclass": "newobj", "numinlets": 2, "numoutlets": 2, @@ -1205,7 +1405,7 @@ }, { "box": { - "id": "obj-65", + "id": "obj-76", "maxclass": "newobj", "numinlets": 2, "numoutlets": 2, @@ -1224,7 +1424,7 @@ }, { "box": { - "id": "obj-66", + "id": "obj-77", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -1242,7 +1442,7 @@ }, { "box": { - "id": "obj-67", + "id": "obj-78", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -1251,7 +1451,7 @@ ], "patching_rect": [ 770.0, - 740.0, + 920.0, 191.0, 22.0 ], @@ -1260,7 +1460,7 @@ }, { "box": { - "id": "obj-68", + "id": "obj-79", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1278,7 +1478,7 @@ }, { "box": { - "id": "obj-69", + "id": "obj-80", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1297,7 +1497,7 @@ }, { "box": { - "id": "obj-70", + "id": "obj-81", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1315,7 +1515,7 @@ }, { "box": { - "id": "obj-71", + "id": "obj-82", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1334,7 +1534,7 @@ }, { "box": { - "id": "obj-72", + "id": "obj-83", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1352,7 +1552,7 @@ }, { "box": { - "id": "obj-73", + "id": "obj-84", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -1372,7 +1572,7 @@ }, { "box": { - "id": "obj-74", + "id": "obj-85", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1390,7 +1590,7 @@ }, { "box": { - "id": "obj-75", + "id": "obj-86", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1409,7 +1609,7 @@ }, { "box": { - "id": "obj-76", + "id": "obj-87", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1427,7 +1627,7 @@ }, { "box": { - "id": "obj-77", + "id": "obj-88", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -1447,7 +1647,7 @@ }, { "box": { - "id": "obj-78", + "id": "obj-89", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1465,7 +1665,7 @@ }, { "box": { - "id": "obj-79", + "id": "obj-90", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1484,7 +1684,7 @@ }, { "box": { - "id": "obj-80", + "id": "obj-91", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1502,7 +1702,7 @@ }, { "box": { - "id": "obj-81", + "id": "obj-92", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1521,7 +1721,7 @@ }, { "box": { - "id": "obj-82", + "id": "obj-93", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1539,7 +1739,7 @@ }, { "box": { - "id": "obj-83", + "id": "obj-94", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1558,7 +1758,7 @@ }, { "box": { - "id": "obj-84", + "id": "obj-95", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1576,7 +1776,7 @@ }, { "box": { - "id": "obj-85", + "id": "obj-96", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -1596,7 +1796,7 @@ }, { "box": { - "id": "obj-86", + "id": "obj-97", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1614,7 +1814,7 @@ }, { "box": { - "id": "obj-87", + "id": "obj-98", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1633,7 +1833,7 @@ }, { "box": { - "id": "obj-88", + "id": "obj-99", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1651,7 +1851,7 @@ }, { "box": { - "id": "obj-89", + "id": "obj-100", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1670,7 +1870,7 @@ }, { "box": { - "id": "obj-90", + "id": "obj-101", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1688,7 +1888,7 @@ }, { "box": { - "id": "obj-91", + "id": "obj-102", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1707,7 +1907,7 @@ }, { "box": { - "id": "obj-92", + "id": "obj-103", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1725,7 +1925,7 @@ }, { "box": { - "id": "obj-93", + "id": "obj-104", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1744,7 +1944,7 @@ }, { "box": { - "id": "obj-94", + "id": "obj-105", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1757,12 +1957,12 @@ 79.0, 22.0 ], - "text": "delay 500" + "text": "delay 150" } }, { "box": { - "id": "obj-95", + "id": "obj-106", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1778,29 +1978,394 @@ ], "text": "t b b" } - } - ], - "lines": [ + }, { - "patchline": { - "source": [ - "obj-2", - 0 + "box": { + "id": "obj-107", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" ], - "destination": [ - "obj-3", - 0 - ] + "patching_rect": [ + 20.0, + 1218.0, + 79.0, + 22.0 + ], + "text": "delay 100" } }, { - "patchline": { + "box": { + "id": "obj-108", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 20.0, + 1248.0, + 51.0, + 22.0 + ], + "text": "t b b" + } + }, + { + "box": { + "id": "obj-109", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 1278.0, + 86.0, + 22.0 + ], + "text": "delay 1000" + } + }, + { + "box": { + "id": "obj-110", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 20.0, + 1308.0, + 51.0, + 22.0 + ], + "text": "t b b" + } + }, + { + "box": { + "id": "obj-111", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 1338.0, + 79.0, + 22.0 + ], + "text": "delay 150" + } + }, + { + "box": { + "id": "obj-112", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 20.0, + 1368.0, + 51.0, + 22.0 + ], + "text": "t b b" + } + }, + { + "box": { + "id": "obj-113", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 1398.0, + 79.0, + 22.0 + ], + "text": "delay 150" + } + }, + { + "box": { + "id": "obj-114", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 20.0, + 1428.0, + 51.0, + 22.0 + ], + "text": "t b b" + } + }, + { + "box": { + "id": "obj-115", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 1458.0, + 79.0, + 22.0 + ], + "text": "delay 500" + } + }, + { + "box": { + "id": "obj-116", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 20.0, + 1488.0, + 51.0, + 22.0 + ], + "text": "t b b" + } + } + ], + "lines": [ + { + "patchline": { "source": [ - "obj-4", + "obj-2", + 0 + ], + "destination": [ + "obj-3", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-6", + 0 + ], + "destination": [ + "obj-7", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-7", + 2 + ], + "destination": [ + "obj-8", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-8", + 0 + ], + "destination": [ + "obj-9", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-6", + 0 + ], + "destination": [ + "obj-10", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-10", + 2 + ], + "destination": [ + "obj-11", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-11", + 0 + ], + "destination": [ + "obj-12", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-6", + 0 + ], + "destination": [ + "obj-13", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-13", + 2 + ], + "destination": [ + "obj-14", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-14", + 0 + ], + "destination": [ + "obj-15", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-3", + 0 + ], + "destination": [ + "obj-16", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-16", + 0 + ], + "destination": [ + "obj-17", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-17", + 0 + ], + "destination": [ + "obj-18", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-19", + 0 + ], + "destination": [ + "obj-3", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-9", + 0 + ], + "destination": [ + "obj-20", + 1 + ] + } + }, + { + "patchline": { + "source": [ + "obj-20", + 0 + ], + "destination": [ + "obj-21", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-21", 0 ], "destination": [ - "obj-5", + "obj-22", 0 ] } @@ -1808,11 +2373,11 @@ { "patchline": { "source": [ - "obj-5", - 2 + "obj-3", + 0 ], "destination": [ - "obj-6", + "obj-23", 0 ] } @@ -1820,11 +2385,11 @@ { "patchline": { "source": [ - "obj-6", + "obj-23", 0 ], "destination": [ - "obj-7", + "obj-24", 0 ] } @@ -1832,11 +2397,11 @@ { "patchline": { "source": [ - "obj-4", + "obj-24", 0 ], "destination": [ - "obj-8", + "obj-25", 0 ] } @@ -1844,11 +2409,11 @@ { "patchline": { "source": [ - "obj-8", - 2 + "obj-26", + 0 ], "destination": [ - "obj-9", + "obj-3", 0 ] } @@ -1856,11 +2421,11 @@ { "patchline": { "source": [ - "obj-9", + "obj-3", 0 ], "destination": [ - "obj-10", + "obj-27", 0 ] } @@ -1868,11 +2433,11 @@ { "patchline": { "source": [ - "obj-4", + "obj-27", 0 ], "destination": [ - "obj-11", + "obj-28", 0 ] } @@ -1880,11 +2445,11 @@ { "patchline": { "source": [ - "obj-11", - 2 + "obj-28", + 0 ], "destination": [ - "obj-12", + "obj-29", 0 ] } @@ -1896,7 +2461,43 @@ 0 ], "destination": [ - "obj-13", + "obj-30", + 1 + ] + } + }, + { + "patchline": { + "source": [ + "obj-30", + 0 + ], + "destination": [ + "obj-31", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-31", + 0 + ], + "destination": [ + "obj-32", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-33", + 0 + ], + "destination": [ + "obj-3", 0 ] } @@ -1908,7 +2509,7 @@ 0 ], "destination": [ - "obj-14", + "obj-34", 0 ] } @@ -1916,11 +2517,11 @@ { "patchline": { "source": [ - "obj-14", + "obj-34", 0 ], "destination": [ - "obj-15", + "obj-35", 0 ] } @@ -1928,11 +2529,11 @@ { "patchline": { "source": [ - "obj-15", + "obj-35", 0 ], "destination": [ - "obj-16", + "obj-36", 0 ] } @@ -1940,7 +2541,7 @@ { "patchline": { "source": [ - "obj-17", + "obj-37", 0 ], "destination": [ @@ -1952,23 +2553,23 @@ { "patchline": { "source": [ - "obj-7", + "obj-3", 0 ], "destination": [ - "obj-18", - 1 + "obj-38", + 0 ] } }, { "patchline": { "source": [ - "obj-18", + "obj-38", 0 ], "destination": [ - "obj-19", + "obj-39", 0 ] } @@ -1976,11 +2577,11 @@ { "patchline": { "source": [ - "obj-19", + "obj-39", 0 ], "destination": [ - "obj-20", + "obj-40", 0 ] } @@ -1988,23 +2589,23 @@ { "patchline": { "source": [ - "obj-3", + "obj-15", 0 ], "destination": [ - "obj-21", - 0 + "obj-41", + 1 ] } }, { "patchline": { "source": [ - "obj-21", + "obj-41", 0 ], "destination": [ - "obj-22", + "obj-42", 0 ] } @@ -2012,11 +2613,11 @@ { "patchline": { "source": [ - "obj-22", + "obj-42", 0 ], "destination": [ - "obj-23", + "obj-43", 0 ] } @@ -2024,7 +2625,7 @@ { "patchline": { "source": [ - "obj-24", + "obj-44", 0 ], "destination": [ @@ -2040,7 +2641,7 @@ 0 ], "destination": [ - "obj-25", + "obj-45", 0 ] } @@ -2048,11 +2649,11 @@ { "patchline": { "source": [ - "obj-25", + "obj-45", 0 ], "destination": [ - "obj-26", + "obj-46", 0 ] } @@ -2060,11 +2661,11 @@ { "patchline": { "source": [ - "obj-26", + "obj-46", 0 ], "destination": [ - "obj-27", + "obj-47", 0 ] } @@ -2072,23 +2673,23 @@ { "patchline": { "source": [ - "obj-10", + "obj-48", 0 ], "destination": [ - "obj-28", - 1 + "obj-3", + 0 ] } }, { "patchline": { "source": [ - "obj-28", + "obj-3", 0 ], "destination": [ - "obj-29", + "obj-49", 0 ] } @@ -2096,11 +2697,11 @@ { "patchline": { "source": [ - "obj-29", + "obj-49", 0 ], "destination": [ - "obj-30", + "obj-50", 0 ] } @@ -2108,11 +2709,11 @@ { "patchline": { "source": [ - "obj-31", + "obj-50", 0 ], "destination": [ - "obj-3", + "obj-51", 0 ] } @@ -2124,7 +2725,7 @@ 0 ], "destination": [ - "obj-32", + "obj-54", 0 ] } @@ -2132,11 +2733,11 @@ { "patchline": { "source": [ - "obj-32", + "obj-54", 0 ], "destination": [ - "obj-33", + "obj-55", 0 ] } @@ -2144,11 +2745,11 @@ { "patchline": { "source": [ - "obj-33", + "obj-55", 0 ], "destination": [ - "obj-34", + "obj-56", 0 ] } @@ -2156,11 +2757,11 @@ { "patchline": { "source": [ - "obj-35", + "obj-57", 0 ], "destination": [ - "obj-3", + "obj-5", 0 ] } @@ -2172,7 +2773,7 @@ 0 ], "destination": [ - "obj-36", + "obj-58", 0 ] } @@ -2180,11 +2781,11 @@ { "patchline": { "source": [ - "obj-36", + "obj-58", 0 ], "destination": [ - "obj-37", + "obj-59", 0 ] } @@ -2192,11 +2793,11 @@ { "patchline": { "source": [ - "obj-37", + "obj-59", 0 ], "destination": [ - "obj-38", + "obj-60", 0 ] } @@ -2204,23 +2805,23 @@ { "patchline": { "source": [ - "obj-13", + "obj-6", 0 ], "destination": [ - "obj-39", - 1 + "obj-62", + 0 ] } }, { "patchline": { "source": [ - "obj-39", + "obj-62", 0 ], "destination": [ - "obj-40", + "obj-63", 0 ] } @@ -2228,11 +2829,11 @@ { "patchline": { "source": [ - "obj-40", + "obj-63", 0 ], "destination": [ - "obj-41", + "obj-64", 0 ] } @@ -2240,11 +2841,23 @@ { "patchline": { "source": [ - "obj-42", + "obj-63", + 3 + ], + "destination": [ + "obj-64", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-64", 0 ], "destination": [ - "obj-3", + "obj-65", 0 ] } @@ -2252,11 +2865,23 @@ { "patchline": { "source": [ - "obj-3", + "obj-65", + 1 + ], + "destination": [ + "obj-66", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-66", 0 ], "destination": [ - "obj-43", + "obj-67", 0 ] } @@ -2264,11 +2889,11 @@ { "patchline": { "source": [ - "obj-43", + "obj-67", 0 ], "destination": [ - "obj-44", + "obj-68", 0 ] } @@ -2276,11 +2901,47 @@ { "patchline": { "source": [ - "obj-44", + "obj-65", 0 ], "destination": [ - "obj-45", + "obj-68", + 1 + ] + } + }, + { + "patchline": { + "source": [ + "obj-68", + 0 + ], + "destination": [ + "obj-69", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-70", + 0 + ], + "destination": [ + "obj-71", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-71", + 1 + ], + "destination": [ + "obj-73", 0 ] } @@ -2288,11 +2949,11 @@ { "patchline": { "source": [ - "obj-46", + "obj-71", 0 ], "destination": [ - "obj-3", + "obj-72", 0 ] } @@ -2300,11 +2961,11 @@ { "patchline": { "source": [ - "obj-3", + "obj-73", 0 ], "destination": [ - "obj-47", + "obj-61", 0 ] } @@ -2312,11 +2973,11 @@ { "patchline": { "source": [ - "obj-47", + "obj-74", 0 ], "destination": [ - "obj-48", + "obj-75", 0 ] } @@ -2324,11 +2985,11 @@ { "patchline": { "source": [ - "obj-48", + "obj-75", 0 ], "destination": [ - "obj-49", + "obj-76", 0 ] } @@ -2336,11 +2997,11 @@ { "patchline": { "source": [ - "obj-4", + "obj-76", 0 ], "destination": [ - "obj-51", + "obj-77", 0 ] } @@ -2348,11 +3009,11 @@ { "patchline": { "source": [ - "obj-51", + "obj-77", 0 ], "destination": [ - "obj-52", + "obj-78", 0 ] } @@ -2360,11 +3021,11 @@ { "patchline": { "source": [ - "obj-52", + "obj-76", 0 ], "destination": [ - "obj-53", + "obj-79", 0 ] } @@ -2372,11 +3033,11 @@ { "patchline": { "source": [ - "obj-52", - 3 + "obj-79", + 0 ], "destination": [ - "obj-53", + "obj-80", 0 ] } @@ -2384,11 +3045,11 @@ { "patchline": { "source": [ - "obj-53", - 0 + "obj-80", + 1 ], "destination": [ - "obj-54", + "obj-16", 0 ] } @@ -2396,11 +3057,11 @@ { "patchline": { "source": [ - "obj-54", - 1 + "obj-80", + 0 ], "destination": [ - "obj-55", + "obj-81", 0 ] } @@ -2408,11 +3069,11 @@ { "patchline": { "source": [ - "obj-55", + "obj-81", 0 ], "destination": [ - "obj-56", + "obj-82", 0 ] } @@ -2420,11 +3081,11 @@ { "patchline": { "source": [ - "obj-56", - 0 + "obj-82", + 1 ], "destination": [ - "obj-57", + "obj-19", 0 ] } @@ -2432,23 +3093,23 @@ { "patchline": { "source": [ - "obj-54", + "obj-82", 0 ], "destination": [ - "obj-57", - 1 + "obj-83", + 0 ] } }, { "patchline": { "source": [ - "obj-57", + "obj-83", 0 ], "destination": [ - "obj-58", + "obj-84", 0 ] } @@ -2456,11 +3117,11 @@ { "patchline": { "source": [ - "obj-59", - 0 + "obj-84", + 2 ], "destination": [ - "obj-60", + "obj-20", 0 ] } @@ -2468,11 +3129,11 @@ { "patchline": { "source": [ - "obj-60", + "obj-84", 1 ], "destination": [ - "obj-62", + "obj-23", 0 ] } @@ -2480,11 +3141,11 @@ { "patchline": { "source": [ - "obj-60", + "obj-84", 0 ], "destination": [ - "obj-61", + "obj-85", 0 ] } @@ -2492,11 +3153,11 @@ { "patchline": { "source": [ - "obj-62", + "obj-85", 0 ], "destination": [ - "obj-50", + "obj-86", 0 ] } @@ -2504,11 +3165,11 @@ { "patchline": { "source": [ - "obj-63", - 0 + "obj-86", + 1 ], "destination": [ - "obj-64", + "obj-26", 0 ] } @@ -2516,11 +3177,11 @@ { "patchline": { "source": [ - "obj-64", + "obj-86", 0 ], "destination": [ - "obj-65", + "obj-87", 0 ] } @@ -2528,11 +3189,11 @@ { "patchline": { "source": [ - "obj-65", + "obj-87", 0 ], "destination": [ - "obj-66", + "obj-88", 0 ] } @@ -2540,11 +3201,11 @@ { "patchline": { "source": [ - "obj-66", - 0 + "obj-88", + 2 ], "destination": [ - "obj-67", + "obj-27", 0 ] } @@ -2552,11 +3213,11 @@ { "patchline": { "source": [ - "obj-65", - 0 + "obj-88", + 1 ], "destination": [ - "obj-68", + "obj-30", 0 ] } @@ -2564,11 +3225,11 @@ { "patchline": { "source": [ - "obj-68", + "obj-88", 0 ], "destination": [ - "obj-69", + "obj-89", 0 ] } @@ -2576,11 +3237,11 @@ { "patchline": { "source": [ - "obj-69", - 1 + "obj-89", + 0 ], "destination": [ - "obj-14", + "obj-90", 0 ] } @@ -2588,11 +3249,11 @@ { "patchline": { "source": [ - "obj-69", - 0 + "obj-90", + 1 ], "destination": [ - "obj-70", + "obj-33", 0 ] } @@ -2600,11 +3261,11 @@ { "patchline": { "source": [ - "obj-70", + "obj-90", 0 ], "destination": [ - "obj-71", + "obj-91", 0 ] } @@ -2612,11 +3273,11 @@ { "patchline": { "source": [ - "obj-71", - 1 + "obj-91", + 0 ], "destination": [ - "obj-17", + "obj-92", 0 ] } @@ -2624,11 +3285,11 @@ { "patchline": { "source": [ - "obj-71", - 0 + "obj-92", + 1 ], "destination": [ - "obj-72", + "obj-34", 0 ] } @@ -2636,11 +3297,11 @@ { "patchline": { "source": [ - "obj-72", + "obj-92", 0 ], "destination": [ - "obj-73", + "obj-93", 0 ] } @@ -2648,11 +3309,11 @@ { "patchline": { "source": [ - "obj-73", - 2 + "obj-93", + 0 ], "destination": [ - "obj-18", + "obj-94", 0 ] } @@ -2660,11 +3321,11 @@ { "patchline": { "source": [ - "obj-73", + "obj-94", 1 ], "destination": [ - "obj-21", + "obj-37", 0 ] } @@ -2672,11 +3333,11 @@ { "patchline": { "source": [ - "obj-73", + "obj-94", 0 ], "destination": [ - "obj-74", + "obj-95", 0 ] } @@ -2684,11 +3345,11 @@ { "patchline": { "source": [ - "obj-74", + "obj-95", 0 ], "destination": [ - "obj-75", + "obj-96", 0 ] } @@ -2696,11 +3357,11 @@ { "patchline": { "source": [ - "obj-75", - 1 + "obj-96", + 2 ], "destination": [ - "obj-24", + "obj-38", 0 ] } @@ -2708,11 +3369,11 @@ { "patchline": { "source": [ - "obj-75", - 0 + "obj-96", + 1 ], "destination": [ - "obj-76", + "obj-41", 0 ] } @@ -2720,11 +3381,11 @@ { "patchline": { "source": [ - "obj-76", + "obj-96", 0 ], "destination": [ - "obj-77", + "obj-97", 0 ] } @@ -2732,11 +3393,11 @@ { "patchline": { "source": [ - "obj-77", - 2 + "obj-97", + 0 ], "destination": [ - "obj-25", + "obj-98", 0 ] } @@ -2744,11 +3405,11 @@ { "patchline": { "source": [ - "obj-77", + "obj-98", 1 ], "destination": [ - "obj-28", + "obj-44", 0 ] } @@ -2756,11 +3417,11 @@ { "patchline": { "source": [ - "obj-77", + "obj-98", 0 ], "destination": [ - "obj-78", + "obj-99", 0 ] } @@ -2768,11 +3429,11 @@ { "patchline": { "source": [ - "obj-78", + "obj-99", 0 ], "destination": [ - "obj-79", + "obj-100", 0 ] } @@ -2780,11 +3441,11 @@ { "patchline": { "source": [ - "obj-79", + "obj-100", 1 ], "destination": [ - "obj-31", + "obj-45", 0 ] } @@ -2792,11 +3453,11 @@ { "patchline": { "source": [ - "obj-79", + "obj-100", 0 ], "destination": [ - "obj-80", + "obj-101", 0 ] } @@ -2804,11 +3465,11 @@ { "patchline": { "source": [ - "obj-80", + "obj-101", 0 ], "destination": [ - "obj-81", + "obj-102", 0 ] } @@ -2816,11 +3477,11 @@ { "patchline": { "source": [ - "obj-81", + "obj-102", 1 ], "destination": [ - "obj-32", + "obj-48", 0 ] } @@ -2828,11 +3489,11 @@ { "patchline": { "source": [ - "obj-81", + "obj-102", 0 ], "destination": [ - "obj-82", + "obj-103", 0 ] } @@ -2840,11 +3501,11 @@ { "patchline": { "source": [ - "obj-82", + "obj-103", 0 ], "destination": [ - "obj-83", + "obj-104", 0 ] } @@ -2852,11 +3513,11 @@ { "patchline": { "source": [ - "obj-83", + "obj-104", 1 ], "destination": [ - "obj-35", + "obj-49", 0 ] } @@ -2864,11 +3525,11 @@ { "patchline": { "source": [ - "obj-83", + "obj-104", 0 ], "destination": [ - "obj-84", + "obj-105", 0 ] } @@ -2876,23 +3537,11 @@ { "patchline": { "source": [ - "obj-84", - 0 - ], - "destination": [ - "obj-85", + "obj-105", 0 - ] - } - }, - { - "patchline": { - "source": [ - "obj-85", - 2 ], "destination": [ - "obj-36", + "obj-106", 0 ] } @@ -2900,11 +3549,11 @@ { "patchline": { "source": [ - "obj-85", + "obj-106", 1 ], "destination": [ - "obj-39", + "obj-52", 0 ] } @@ -2912,11 +3561,11 @@ { "patchline": { "source": [ - "obj-85", + "obj-106", 0 ], "destination": [ - "obj-86", + "obj-107", 0 ] } @@ -2924,11 +3573,11 @@ { "patchline": { "source": [ - "obj-86", + "obj-107", 0 ], "destination": [ - "obj-87", + "obj-108", 0 ] } @@ -2936,11 +3585,11 @@ { "patchline": { "source": [ - "obj-87", + "obj-108", 1 ], "destination": [ - "obj-42", + "obj-53", 0 ] } @@ -2948,11 +3597,11 @@ { "patchline": { "source": [ - "obj-87", + "obj-108", 0 ], "destination": [ - "obj-88", + "obj-109", 0 ] } @@ -2960,11 +3609,11 @@ { "patchline": { "source": [ - "obj-88", + "obj-109", 0 ], "destination": [ - "obj-89", + "obj-110", 0 ] } @@ -2972,11 +3621,11 @@ { "patchline": { "source": [ - "obj-89", + "obj-110", 1 ], "destination": [ - "obj-43", + "obj-54", 0 ] } @@ -2984,11 +3633,11 @@ { "patchline": { "source": [ - "obj-89", + "obj-110", 0 ], "destination": [ - "obj-90", + "obj-111", 0 ] } @@ -2996,11 +3645,11 @@ { "patchline": { "source": [ - "obj-90", + "obj-111", 0 ], "destination": [ - "obj-91", + "obj-112", 0 ] } @@ -3008,11 +3657,11 @@ { "patchline": { "source": [ - "obj-91", + "obj-112", 1 ], "destination": [ - "obj-46", + "obj-57", 0 ] } @@ -3020,11 +3669,11 @@ { "patchline": { "source": [ - "obj-91", + "obj-112", 0 ], "destination": [ - "obj-92", + "obj-113", 0 ] } @@ -3032,11 +3681,11 @@ { "patchline": { "source": [ - "obj-92", + "obj-113", 0 ], "destination": [ - "obj-93", + "obj-114", 0 ] } @@ -3044,11 +3693,11 @@ { "patchline": { "source": [ - "obj-93", + "obj-114", 1 ], "destination": [ - "obj-47", + "obj-58", 0 ] } @@ -3056,11 +3705,11 @@ { "patchline": { "source": [ - "obj-93", + "obj-114", 0 ], "destination": [ - "obj-94", + "obj-115", 0 ] } @@ -3068,11 +3717,11 @@ { "patchline": { "source": [ - "obj-94", + "obj-115", 0 ], "destination": [ - "obj-95", + "obj-116", 0 ] } @@ -3080,11 +3729,11 @@ { "patchline": { "source": [ - "obj-95", + "obj-116", 1 ], "destination": [ - "obj-50", + "obj-61", 0 ] } diff --git a/runtime-tests/python/maxtest_faults.py b/runtime-tests/python/maxtest_faults.py index 0eb3d04..094972e 100644 --- a/runtime-tests/python/maxtest_faults.py +++ b/runtime-tests/python/maxtest_faults.py @@ -18,6 +18,20 @@ def fail(self) -> None: def heal(self) -> None: self._fault = "" + # Named like messages Max sends with C arguments (plan 8.2): the object must not expose them, + # or Max's call would crash it when DSP toggles, a cord connects, or a collective is built. + def dspstate(self, on: int) -> None: + self._fault = "dspstate was called" + + def patchlineupdate(self) -> None: + self._fault = "patchlineupdate was called" + + def inputchanged(self) -> None: + self._fault = "inputchanged was called" + + def fileusage(self) -> None: + self._fault = "fileusage was called" + def process(self, x: float) -> float: if self._fault == "raise": raise RuntimeError("maxtest: process() raised on purpose") diff --git a/source/projects/tap.python_tilde/tap.python_tilde.h b/source/projects/tap.python_tilde/tap.python_tilde.h index d7c413a..a221107 100644 --- a/source/projects/tap.python_tilde/tap.python_tilde.h +++ b/source/projects/tap.python_tilde/tap.python_tilde.h @@ -212,9 +212,10 @@ class python : public object, public vector_operator<> { cout << std::string{text} << endl; } }; - m_processor = std::make_unique(m_python_source, object_log, reserved_messages(), - [this] { m_reports.set(); }); - m_worker = std::make_unique( + m_processor = std::make_unique( + m_python_source, object_log, reserved_messages(), [this] { m_reports.set(); }, + [this](const std::string& name) { return answered_by_max(name); }); + m_worker = std::make_unique( *m_processor, object_log, [this] { m_reports.set(); }, audio_thread_scheduling); update_source(); @@ -353,6 +354,17 @@ class python : public object, public vector_operator<> { } } + /// The names of the Max messages made for the class's methods (for the tests). + std::vector python_message_names() const { + std::vector names; + names.reserve(m_python_messages.size()); + for (const auto& element : m_python_messages) { + names.push_back(element.first); + } + std::sort(names.begin(), names.end()); + return names; + } + private: string m_python_source{}; std::filesystem::path m_scripts_dir{}; @@ -371,11 +383,80 @@ class python : public object, public vector_operator<> { /// never replace: min's own class methods, the messages Max sends every object, this object's /// file watcher (a Python method named filechanged would silently disable hot reload), and its /// own attributes (worker mode's, plan 2.5). + /// + /// Above all, every message Max sends with C arguments (plan 8.2, audit A3): a Python method + /// of such a name would be registered with the A_GIMME trampoline, and Max calling it with a + /// long or a pointer reads them as a symbol and an atom list — the crash 6.1 found for + /// filechanged, in its general form. The names are those min treats as A_CANT + /// (c74_min_message.h, message_type::cant; the MIN_WRAPPER_ADDMETHOD table in + /// c74_min_object_wrapper.h) plus Max's own dspstate, inputchanged and multichanneloutputs. + /// Anything the Max class already answers is reserved as well: answered_by_max(). static std::vector reserved_messages() { - return {"anything", "appendtodictionary", "assist", "dblclick", "dsp", - "dsp64", "dspsetup", "filechanged", "getvalueof", "inletinfo", - "latency", "latencysamples", "loadbang", "mode", "notify", - "preset", "savestate", "setvalueof", "signal"}; + return {"anything", + "appendtodictionary", + "assist", + "dblclick", + "dictionary", + "dsp", + "dsp64", + "dspsetup", + "dspstate", + "edclose", + "filechanged", + "fileusage", + "focusgained", + "focuslost", + "getplaystate", + "getvalueof", + "inletinfo", + "inputchanged", + "jitclass_setup", + "key", + "latency", + "latencysamples", + "loadbang", + "maxclass_setup", + "maxob_setup", + "mode", + "mop_setup", + "mousedoubleclick", + "mousedown", + "mousedrag", + "mousedragdelta", + "mouseenter", + "mouseleave", + "mousemove", + "mouseup", + "mousewheel", + "mt_mousedown", + "mt_mousedrag", + "mt_mouseenter", + "mt_mouseleave", + "mt_mousemove", + "mt_mouseup", + "multichanneloutputs", + "notify", + "okclose", + "oksize", + "paint", + "patchlineupdate", + "preset", + "savestate", + "setup", + "setvalueof", + "signal"}; + } + + /// Whether the Max object already answers `name` itself — a method its class registered (min's + /// dsp64, assist, the ones above), which Max would call before anything added for a Python + /// method, or with C arguments. The messages this object added for the previous incarnation's + /// Python methods are its own, not Max's: still registered while load() runs, they must not + /// make the class's methods reserved on a reload. Main thread. + bool answered_by_max(const std::string& name) { + if (m_python_messages.find(name) != m_python_messages.end()) { + return false; + } + return c74::max::object_getmethod(maxobj(), c74::max::gensym(name.c_str())) != nullptr; } /// The worker thread's scheduling (plan 2.5): the real-time class an audio thread has, so that a diff --git a/source/projects/tap.python_tilde/tap.python_tilde_test.cpp b/source/projects/tap.python_tilde/tap.python_tilde_test.cpp index e93b800..12cf7ea 100644 --- a/source/projects/tap.python_tilde/tap.python_tilde_test.cpp +++ b/source/projects/tap.python_tilde/tap.python_tilde_test.cpp @@ -414,6 +414,40 @@ SCENARIO("A save that changes process()'s inputs or outputs changes the object's std::filesystem::remove(file); } +SCENARIO("Methods named like messages Max sends with C arguments are not exposed, and the promised ones are " + "(plan 8.2)") { + ext_main(nullptr); + const auto file = tap::python::package_root() / "python" / "maxtest_mock_reserved.py"; + write_file(file, "class maxtest_mock_reserved:\n" + // what Max calls with C arguments: a Python method of the name would crash Max + " def dspstate(self, on: int) -> None:\n pass\n" + " def fileusage(self) -> None:\n pass\n" + " def patchlineupdate(self) -> None:\n pass\n" + " def inputchanged(self) -> None:\n pass\n" + // what the ReadMe promises stays a message, whatever the guard answers + " def int(self, n: int) -> None:\n pass\n" + " def float(self, x: float) -> None:\n pass\n" + " def symbol(self, s: str) -> None:\n pass\n" + " def bang(self) -> None:\n pass\n" + " def list(self, *args: float) -> None:\n pass\n" + " def greet(self, name: str) -> None:\n pass\n" + " def process(self, x: float) -> float:\n return x\n"); + const auto argument = symbol_atom("maxtest_mock_reserved"); + auto* wrapped = c74::min::wrapper_new(c74::min::symbol("dummy"), 1, &argument); + REQUIRE(wrapped); + python& my_object = wrapped->m_min_object; + + THEN("the reserved names are not registered, and every promised name is") { + CHECK(my_object.python_message_names() + == std::vector{"bang", "float", "greet", "int", "list", "symbol"}); + } + THEN("audio is bound regardless") { + CHECK(all_equal(render(my_object, 0.5), 0.5)); + } + c74::max::object_free(wrapped); + std::filesystem::remove(file); +} + SCENARIO("With @mode worker, process() runs on a thread of its own, @latency milliseconds behind (plan 2.5)") { ext_main(nullptr); test_wrapper an_instance; From ab45f3261c52f4302ba8fad3b9a774f04e1006c3 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 16:17:04 +0000 Subject: [PATCH 06/11] 8.3: a worker that never hangs Max; a 16 MiB worker stack worker::stop() no longer waits forever for a process() that does not return. It waits 100 ms, then raises WorkerStopped into the thread with PyThreadState_SetAsyncExc: a BaseException defined in the support module, so a class's except Exception cannot swallow it; the processor recognizes it (take_interruption), silences that vector, reports it once per load and keeps the class's audio bound. If the thread still has not returned 250 ms later it is blocked in a call Python cannot interrupt, and stop() detaches it: the ring is shared with the thread, has_abandoned_thread() says one is still running, and the Max object's destructor then leaks its worker and processor rather than free what the thread may yet touch, saying so. The worker thread is made by detail::native_thread (pthread_attr_setstacksize / _beginthreadex) with a 16 MiB stack, what CPython gives its own threads on macOS, where a std::thread's 512 KiB is too little for Python recursing through a C boundary. Pinned by four core scenarios (a process() that never returns; one blocked in time.sleep; a merely slow one, waited for; recursion to the limit through sorted's key on the worker), clean under ASan/UBSan and TSan, and new steps in the worker runtime test for the Mac session. ReadMe and CHANGELOG say what stopping the worker now does. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- CHANGELOG.md | 14 + ReadMe.md | 4 +- core/include/tap/python/processor.h | 47 +- core/include/tap/python/runtime.h | 8 + core/include/tap/python/worker.h | 226 +++- core/tests/python/deep_recursion.py | 17 + core/tests/python/hangs.py | 14 + core/tests/python/sleeps.py | 12 + core/tests/test_worker.cpp | 93 ++ docs/PRODUCTION-PLAN.md | 17 +- runtime-tests/make_patchers.py | 15 +- .../tap.python~.worker.maxtest.maxpat | 962 +++++++++++++----- runtime-tests/python/maxtest_stall.py | 6 + .../tap.python_tilde/tap.python_tilde.h | 16 +- 14 files changed, 1145 insertions(+), 306 deletions(-) create mode 100644 core/tests/python/deep_recursion.py create mode 100644 core/tests/python/hangs.py create mode 100644 core/tests/python/sleeps.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b5177e..61ba343 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,20 @@ breaking changes to the contract are allowed where they buy correctness (D5 in ## Unreleased +### Changed — worker mode never waits forever + +- **A `process()` that does not return when the worker stops is interrupted, then abandoned.** + Stopping the worker — DSP off, a chain rebuild, the object deleted — waits up to 100 ms, then + raises `WorkerStopped` (a `BaseException`) into the thread: a pure-Python loop ends, that vector + is silence, the console says so once per load, and the class's audio stays bound. If it still has + not returned 250 ms later it is blocked in a call Python cannot interrupt, and the thread is + abandoned: audio continues on a new worker, the console says the thread keeps costing a core + until Max quits, and the object keeps its Python state alive for it. Before, stopping the worker + waited for `process()` to return, so such a class froze Max when DSP toggled or the patch closed. + The worker thread also runs on a 16 MiB stack, what CPython gives its own threads on macOS, where + the default 512 KiB was too little for Python that recurses through a C boundary. (Plan 8.3, + audit A1 and A6.) + ### Changed — more names are reserved - **A method or field named like a message Max sends with C arguments is not exposed**, with the diff --git a/ReadMe.md b/ReadMe.md index 28c6425..b91aed3 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -95,10 +95,10 @@ Python sources live in the package's `python` folder. `[tap.python~ name]` loads - `process(self, x: float) -> float` is called **once per sample** — simplest for sketching; the call is cheap, but every line of Python in it runs once per sample (see [the performance note](#a-note-on-performance)). Its parameters are the object's signal inlets and its return hint its outlets: `process(self, left: np.ndarray, right: np.ndarray) -> tuple[np.ndarray, np.ndarray]` makes an object with two of each (see `python/stereo_width.py`), and `process(self) -> float`, with no inputs, a generator — the first inlet is always there, for messages. The inputs are all `np.ndarray` or all per sample, and a tuple return must say how many values it has (`tuple[float, float]`). A save that changes how many changes the object's inlets and outlets to match, keeping the patch cords of those that stay. Wrap the object in `mc.` to run one instance per channel of a multichannel signal. Output that is not a number, or not finite (NaN, infinity), is replaced with 0.0 and reported once in the Max console, as is a return with the wrong number of values. -- **Worker mode** — by default `process()` runs on Max's audio thread, so whatever else holds Python's interpreter — a reload, a message, another instance — holds up the audio while it does. With `@mode worker` it runs on a thread of its own instead, `@latency` milliseconds behind the audio (30 by default, rounded up to whole signal vectors): the audio thread then only copies vectors to and from that thread, and never waits for Python. Max computes a whole I/O vector's worth of signal vectors at once, so `@latency` must be longer than the I/O vector (Options > Audio Status: 512 samples is 11.6 ms at 44.1 kHz) — what is left over is the time Python has to keep up; raise it with a larger I/O vector. If Python falls further behind, the vectors it is late for are output as silence and the console says so, once per load; the delay stays the same. The read-only `@latencysamples` gives the delay in samples, for aligning other signal paths (a `delay~`, say); in direct mode it is 0. The worker thread has the real-time scheduling of an audio thread. `@mode` and `@latency` take effect as soon as they are set, by rebuilding the signal chain. +- **Worker mode** — by default `process()` runs on Max's audio thread, so whatever else holds Python's interpreter — a reload, a message, another instance — holds up the audio while it does. With `@mode worker` it runs on a thread of its own instead, `@latency` milliseconds behind the audio (30 by default, rounded up to whole signal vectors): the audio thread then only copies vectors to and from that thread, and never waits for Python. Max computes a whole I/O vector's worth of signal vectors at once, so `@latency` must be longer than the I/O vector (Options > Audio Status: 512 samples is 11.6 ms at 44.1 kHz) — what is left over is the time Python has to keep up; raise it with a larger I/O vector. If Python falls further behind, the vectors it is late for are output as silence and the console says so, once per load; the delay stays the same. The read-only `@latencysamples` gives the delay in samples, for aligning other signal paths (a `delay~`, say); in direct mode it is 0. The worker thread has the real-time scheduling of an audio thread. `@mode` and `@latency` take effect as soon as they are set, by rebuilding the signal chain. Stopping the worker (DSP off, a chain rebuild, the object deleted) waits up to 100 ms for the vector in progress; a `process()` that has not returned by then is interrupted with a `WorkerStopped` exception (a `BaseException`, so `except Exception:` does not swallow it), that vector is silence, the console says so once, and the class stays bound; one that still has not returned after a further 250 ms is blocked in a call Python cannot interrupt (`time.sleep`, a long C call) and is abandoned: audio continues on a new worker, while the abandoned thread keeps costing a core until Max quits. - **Audio settings** — an optional method `prepare(self, sample_rate: float, vector_size: int) -> None` is called with Max's sample rate and vector size before the object processes any audio, again whenever they change, and on every reload before the new code runs. `python/allpass.py` uses it to size its delay line. - **Hot reload** — saving the `.py` file reloads it in place, as a fresh module (names you deleted from the file are gone): the attributes and messages follow the new class, and audio resumes with the new code. Until the new code is ready the object keeps running the old one, so a successful reload swaps in without a gap in the audio. Attribute values carry over — set from the patcher or by your own code — for every attribute the new class still has with the same type; an attribute whose type changed starts from its new default, and one you removed disappears from the object. If the file has an error, the object prints the traceback to the Max console and outputs silence until the next successful reload (and the attribute values come back with it); a save that breaks a file shared by many objects is reported once, not by each, while an object created later with the file still broken says so again. Every object using the same file shares one execution of it per save, and the console says so once — `Loaded name.py: process() bound, one call per sample` (or per vector), from whichever object ran the file — along with anything true of the class, such as a method skipped for its name; a save that changes nothing reloads silently. Errors that belong to one object, such as an exception in its constructor, are still reported by each. Max's file watcher notices a save within a couple of seconds, but it coalesces saves made in quick succession — a script saving once a second, say — so the object then reloads for only some of them, and can lag behind until they stop; sending the object `filechanged` reloads it at once. -- **Errors** — an exception raised by your code (in `process()`, a message, an attribute setter, the constructor or at import) prints its traceback to the Max console and never takes Max down. That includes `sys.exit()`, which is reported like any other exception rather than quitting Max. What the object cannot catch is what never reaches Python's exception machinery: a process exit below it (`os._exit()`, `os.abort()`), a crash in a C extension (a broken wheel, `ctypes`), or code that never returns — an endless loop in a message or the constructor freezes Max's main thread, and in `process()` it stalls the audio thread (in worker mode, the worker thread; stopping that worker then waits on it). Another Max external that embeds its own CPython alongside this one is untested and unsupported: two interpreters in one Max share process-wide state neither expects to. +- **Errors** — an exception raised by your code (in `process()`, a message, an attribute setter, the constructor or at import) prints its traceback to the Max console and never takes Max down. That includes `sys.exit()`, which is reported like any other exception rather than quitting Max. What the object cannot catch is what never reaches Python's exception machinery: a process exit below it (`os._exit()`, `os.abort()`), a crash in a C extension (a broken wheel, `ctypes`), or code that never returns — an endless loop in a message or the constructor freezes Max's main thread, and in `process()` it stalls the audio thread (in worker mode, only the worker thread, which is interrupted or abandoned when the worker stops — see above). Another Max external that embeds its own CPython alongside this one is untested and unsupported: two interpreters in one Max share process-wide state neither expects to. ### A note on performance diff --git a/core/include/tap/python/processor.h b/core/include/tap/python/processor.h index efc11c9..247bb23 100644 --- a/core/include/tap/python/processor.h +++ b/core/include/tap/python/processor.h @@ -298,6 +298,10 @@ namespace tap::python { + ", not a number — output as 0.0 (reported once per load)"); Py_DECREF(type); } + if (m_pending_interrupted.exchange(false)) { + log(log_level::error, "process() had not returned when the worker stopped and was interrupted — " + "that vector is silence; audio stays bound (reported once per load)"); + } if (m_pending_non_finite.exchange(false)) { log(log_level::error, "process() produced a non-finite sample (NaN or infinity) — replaced with 0.0 (reported once per " @@ -574,6 +578,7 @@ namespace tap::python { std::atomic m_pending_exception{}; // strong std::atomic m_pending_non_numeric{}; // strong: the returned object's type std::atomic m_pending_non_finite{}; + std::atomic m_pending_interrupted{}; // WorkerStopped raised into process() (8.3) std::atomic m_pending_bad_length{k_no_length}; std::atomic m_bad_length_expected{}; std::atomic m_warned_non_numeric{}; @@ -770,10 +775,11 @@ namespace tap::python { } void reset_warnings() { - m_warned_non_numeric = false; - m_warned_non_finite = false; - m_warned_bad_length = false; - m_warned_bad_count = false; + m_pending_interrupted = false; + m_warned_non_numeric = false; + m_warned_non_finite = false; + m_warned_bad_length = false; + m_warned_bad_count = false; } void notify_report() const { @@ -782,6 +788,27 @@ namespace tap::python { } } + /// If the pending exception is the worker's own interruption — WorkerStopped, raised into a + /// process() that had not returned when the worker stopped (plan 8.3) — clear it and record + /// that, and say so: it is not the class's fault, so the caller keeps audio bound. Otherwise + /// leave the exception pending and return false. Caller holds the GIL. + bool take_interruption() { + PyObject* exception = PyErr_GetRaisedException(); + if (!exception) { + return false; + } + PyObject* interruption = detail::support("WorkerStopped"); // borrowed + if (interruption && PyErr_GivenExceptionMatches(exception, interruption)) { + Py_DECREF(exception); + if (!m_pending_interrupted.exchange(true)) { + notify_report(); + } + return true; + } + PyErr_SetRaisedException(exception); // steals the reference + return false; + } + /// Record the pending exception (audio thread; caller holds the GIL). Only the first since /// the last flush is kept — the rest of that vector is silenced, and process() unbound. void record_exception() { @@ -881,8 +908,10 @@ namespace tap::python { release_arguments(call_args, io.inputs); if (!result) { - record_exception(); - unbind_process(function); // load() re-arms it + if (!take_interruption()) { // the worker's interruption keeps audio bound (8.3) + record_exception(); + unbind_process(function); // load() re-arms it + } silence(io.out, 0, written, i, frame_count); sanitize(io, written, frame_count); // the samples before this one are still the class's return; @@ -956,8 +985,10 @@ namespace tap::python { release_arguments(call_args, io.inputs); if (!result) { - record_exception(); - unbind_process(function); + if (!take_interruption()) { // the worker's interruption keeps audio bound (8.3) + record_exception(); + unbind_process(function); + } silence(io.out, 0, written, 0, frame_count); return; } diff --git a/core/include/tap/python/runtime.h b/core/include/tap/python/runtime.h index 6cb924c..7574534 100644 --- a/core/include/tap/python/runtime.h +++ b/core/include/tap/python/runtime.h @@ -196,9 +196,17 @@ namespace tap::python { // return_count is how many values a return hint declares: 1, or n for tuple[a, b, ...] (-1 // when a tuple's length is not said: tuple, tuple[float, ...]); return_element_kind is the // hint kind of the value, or of a tuple's first element. + // + // WorkerStopped: raised into a process() that has not returned when its worker is stopped + // (plan 8.3; worker.h). A BaseException, like KeyboardInterrupt, so that a class's + // `except Exception:` cannot swallow it and keep looping; the processor recognizes it and + // keeps the class's audio bound — it is the host's interruption, not the class's fault. inline constexpr const char* k_support_source = R"( import inspect, re, sys, time, types, typing +class WorkerStopped(BaseException): + """tap.python~ stopped the worker thread while process() had not returned.""" + _cache = {} _failed = {} _REPORT_WINDOW = 2.0 diff --git a/core/include/tap/python/worker.h b/core/include/tap/python/worker.h index dc562ca..32a504d 100644 --- a/core/include/tap/python/worker.h +++ b/core/include/tap/python/worker.h @@ -26,11 +26,25 @@ // chain while it prepares the next). The audio thread borrows the ring for each vector by taking // an atomic pointer, so start() and stop() wait at most one vector for it, and a vector that // arrives while they hold it is silence. +// +// A process() that does not return cannot be waited for (plan 8.3): stop() gives it a grace +// period, then raises WorkerStopped into its thread (PyThreadState_SetAsyncExc, delivered at the +// next bytecode of a pure-Python loop; the processor keeps the class's audio bound and reports the +// interruption), and if it still has not returned — blocked in a call Python cannot interrupt — +// detaches the thread and abandons it, so the host never hangs on it. The abandoned thread keeps +// its ring (shared), but still refers to this worker and its processor: while +// has_abandoned_thread() is true, a host must leak both rather than destroy them (the Max object +// does), and the thread goes on costing a core until the process exits. +// +// The worker thread runs on a 16 MiB stack (detail::native_thread), what CPython gives the threads +// it creates on macOS, where a plain std::thread gets 512 KiB — too little for Python that recurses +// through a C boundary or calls into a deep extension. #pragma once #include #include +#include #include #include #include @@ -38,14 +52,133 @@ #include #include #include +#include #include #include #include +#ifdef _WIN32 +#include +#include +#else +#include +#endif + #include "tap/python/processor.h" namespace tap::python { + namespace detail { + + /// A thread with the stack size given to it (std::thread offers no way to choose one): the + /// join/detach shape of std::thread, nothing more. Plan 8.3. + class native_thread { + public: + /// What CPython gives its own threads on macOS (THREAD_STACK_SIZE in thread_pthread.h). + static constexpr std::size_t k_stack_bytes = std::size_t{16} << 20; + + native_thread() = default; + + /// Start `body` on a new thread with a stack of `stack_bytes`; throws std::system_error if + /// the system refuses. + explicit native_thread(std::function body, const std::size_t stack_bytes = k_stack_bytes) { + auto start = std::make_unique>(std::move(body)); +#ifdef _WIN32 + const auto handle = _beginthreadex(nullptr, static_cast(stack_bytes), &native_thread::run, + start.get(), 0, nullptr); + if (handle == 0) { + throw std::system_error{errno, std::generic_category(), "_beginthreadex"}; + } + m_handle = reinterpret_cast(handle); +#else + pthread_attr_t attributes; + pthread_attr_init(&attributes); + pthread_attr_setstacksize(&attributes, stack_bytes); + const int error = pthread_create(&m_thread, &attributes, &native_thread::run, start.get()); + pthread_attr_destroy(&attributes); + if (error != 0) { + throw std::system_error{error, std::generic_category(), "pthread_create"}; + } +#endif + m_joinable = true; + start.release(); // the thread owns it now + } + + ~native_thread() { + if (m_joinable) { + std::terminate(); // as std::thread: a running thread must be joined or detached + } + } + + native_thread(const native_thread&) = delete; + native_thread& operator=(const native_thread&) = delete; + + native_thread(native_thread&& other) noexcept { swap(other); } + native_thread& operator=(native_thread&& other) noexcept { + if (m_joinable) { + std::terminate(); + } + swap(other); + return *this; + } + + bool joinable() const noexcept { return m_joinable; } + + void join() { + if (!m_joinable) { + return; + } +#ifdef _WIN32 + WaitForSingleObject(m_handle, INFINITE); + CloseHandle(m_handle); +#else + pthread_join(m_thread, nullptr); +#endif + m_joinable = false; + } + + void detach() { + if (!m_joinable) { + return; + } +#ifdef _WIN32 + CloseHandle(m_handle); +#else + pthread_detach(m_thread); +#endif + m_joinable = false; + } + + private: + void swap(native_thread& other) noexcept { +#ifdef _WIN32 + std::swap(m_handle, other.m_handle); +#else + std::swap(m_thread, other.m_thread); +#endif + std::swap(m_joinable, other.m_joinable); + } + +#ifdef _WIN32 + static unsigned __stdcall run(void* start) { + std::unique_ptr> body{static_cast*>(start)}; + (*body)(); + return 0; + } + HANDLE m_handle{}; +#else + static void* run(void* start) { + std::unique_ptr> body{static_cast*>(start)}; + (*body)(); + return nullptr; + } + pthread_t m_thread{}; +#endif + bool m_joinable{}; + }; + + } // namespace detail + class worker { public: /// The latency, in milliseconds, unless the host says otherwise (plan 2.5). A host that computes @@ -87,6 +220,12 @@ namespace tap::python { worker(const worker&) = delete; worker& operator=(const worker&) = delete; + /// How long stop() waits for a vector in progress before interrupting it, and then before + /// abandoning the thread (plan 8.3). A vector normally takes a fraction of its period; a + /// class whose vector takes longer is already late. + static constexpr std::chrono::milliseconds k_grace{100}; + static constexpr std::chrono::milliseconds k_interrupt_grace{250}; + /// Start the worker, or restart it with new settings: a ring for `inputs` and `outputs` host /// channels of `vector_size` frames, `latency` vectors deep (at least 1), with room for the /// worker to fall a quarter of a second behind at `sample_rate` (at least 16 vectors). @@ -103,20 +242,24 @@ namespace tap::python { ? static_cast(std::ceil(k_backlog_seconds * sample_rate / static_cast(frames))) : std::size_t{0}; auto next = - std::make_unique(inputs, outputs, frames, vectors, vectors + std::max(k_min_backlog, backlog)); + std::make_shared(inputs, outputs, frames, vectors, vectors + std::max(k_min_backlog, backlog)); ring* r = next.get(); const auto period = std::isfinite(sample_rate) && sample_rate > 0.0 ? static_cast(frames) / sample_rate : 0.0; - r->thread = std::thread{[this, r, period] { + // the thread keeps the ring alive itself, so one that stop() abandons outlives the worker's hold on it + r->thread = detail::native_thread{[this, keep = next, period] { + ring& own = *keep; if (m_thread_setup) { m_thread_setup(period); } + own.ident.store(PyThread_get_thread_ident(), std::memory_order_release); if (Py_IsInitialized()) { gil_lock lock; // the thread's Python thread state, made now rather than on its first vector } - r->ready.store(true, std::memory_order_release); - r->ready.notify_one(); - run(*r); + own.ready.store(true, std::memory_order_release); + own.ready.notify_one(); + run(own); + own.finished.store(true, std::memory_order_release); }}; r->ready.wait(false, std::memory_order_acquire); // ready before the audio thread can give it work m_owned = std::move(next); @@ -124,8 +267,10 @@ namespace tap::python { m_ring.store(r, std::memory_order_release); } - /// Stop the worker and join its thread; process() outputs silence until the next start(). - /// Main thread. + /// Stop the worker; process() outputs silence until the next start(). Joins its thread when + /// it returns within the grace periods, interrupting a process() that has not (see above), + /// and otherwise abandons it: returns within about k_grace + k_interrupt_grace in every + /// case. Main thread, never holding the GIL. void stop() { if (!m_owned) { return; @@ -139,6 +284,20 @@ namespace tap::python { r->stopping.store(true, std::memory_order_release); r->signal.fetch_add(1, std::memory_order_release); r->signal.notify_one(); + if (!finishes_within(*r, k_grace)) { + interrupt(*r); + if (!finishes_within(*r, k_interrupt_grace)) { + r->thread.detach(); + m_abandoned.push_back(m_owned); // the thread holds its own; this one answers has_abandoned_thread() + m_owned.reset(); + m_latency = 0; + log(log_level::error, + "process() did not return when the worker stopped, even when interrupted: the class is " + "blocked in a call Python cannot interrupt. Its thread is abandoned and keeps costing a " + "core until Max restarts; audio continues on a new worker at the next compile"); + return; + } + } r->thread.join(); m_owned.reset(); m_latency = 0; @@ -147,6 +306,18 @@ namespace tap::python { /// True between start() and stop(). Main thread. bool running() const noexcept { return m_owned != nullptr; } + /// True while a thread stop() abandoned is still running (plan 8.3). It still refers to this + /// worker and to the processor it runs: a host must then leak both rather than destroy them. + /// Main thread. + bool has_abandoned_thread() { + m_abandoned.erase(std::remove_if(m_abandoned.begin(), m_abandoned.end(), + [](const std::shared_ptr& r) { + return r->finished.load(std::memory_order_acquire); + }), + m_abandoned.end()); + return !m_abandoned.empty(); + } + /// The latency in samples: the vectors it was started with times the vector size; 0 when /// stopped. Main thread. std::size_t latency_samples() const noexcept { return m_latency; } @@ -276,8 +447,10 @@ namespace tap::python { std::atomic done{}; // vectors the worker has finished (or passed over) std::atomic signal{}; // bumped to wake the worker std::atomic stopping{}; - std::atomic ready{}; // the thread has started and has its Python thread state - std::thread thread; + std::atomic ready{}; // the thread has started and has its Python thread state + std::atomic finished{}; // the thread is about to exit (plan 8.3) + std::atomic ident{}; // the thread's identifier, for PyThreadState_SetAsyncExc + detail::native_thread thread; }; processor& m_target; @@ -285,15 +458,44 @@ namespace tap::python { std::function m_report_ready; std::function m_thread_setup; - std::atomic m_ring{}; // the running ring, while the audio thread is not using it - std::unique_ptr m_owned; // the running ring (main thread) - std::size_t m_latency{}; // in samples (main thread) + std::atomic m_ring{}; // the running ring, while the audio thread is not using it + std::shared_ptr m_owned; // the running ring (main thread) + std::vector> m_abandoned; // rings whose threads stop() gave up on (main thread) + std::size_t m_latency{}; // in samples (main thread) std::atomic m_late{}; // vectors output as silence since the last flush std::atomic m_dropped{}; // vectors whose inputs were dropped std::atomic m_late_load{k_none}; // the load for which late vectors were last reported std::atomic m_dropped_load{k_none}; // the same, for dropped ones + /// Whether the thread of `r` finishes within `limit`, polled; the thread sets `finished` as + /// its last act, so a join after a true answer returns at once. + static bool finishes_within(const ring& r, const std::chrono::milliseconds limit) { + const auto deadline = std::chrono::steady_clock::now() + limit; + while (!r.finished.load(std::memory_order_acquire)) { + if (std::chrono::steady_clock::now() >= deadline) { + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds{1}); + } + return true; + } + + /// Raise WorkerStopped into the thread of `r` (plan 8.3): delivered at its next bytecode, so + /// a pure-Python loop raises and process() returns; a blocking C call delivers it only when + /// it returns. Main thread; takes the GIL for the call (the thread yields it every switch + /// interval) and releases it before anything waits on the thread. + static void interrupt(const ring& r) { + if (!Py_IsInitialized()) { + return; + } + gil_lock lock; + PyObject* interruption = detail::support("WorkerStopped"); // borrowed + if (interruption) { + PyThreadState_SetAsyncExc(r.ident.load(std::memory_order_acquire), interruption); + } + } + /// The worker thread: process each queued vector in order, sleeping while there is none. void run(ring& r) { std::vector in(r.inputs); diff --git a/core/tests/python/deep_recursion.py b/core/tests/python/deep_recursion.py new file mode 100644 index 0000000..fd2e040 --- /dev/null +++ b/core/tests/python/deep_recursion.py @@ -0,0 +1,17 @@ +# Test fixture: recursion through a C boundary (sorted's key) to the recursion limit, on whatever +# thread calls process() — the worker's stack must take it (plan 8.3, audit A6). Returns its input. +import sys + + +class deep_recursion: + def process(self, x: float) -> float: + found = [] + + def down(n: int) -> int: + if n <= 0: + found.append(x) + return 0 + return sorted([n], key=lambda v: down(v - 1))[0] # two Python frames and a C call per level + + down((sys.getrecursionlimit() - 100) // 2) + return found[0] diff --git a/core/tests/python/hangs.py b/core/tests/python/hangs.py new file mode 100644 index 0000000..9cb2c41 --- /dev/null +++ b/core/tests/python/hangs.py @@ -0,0 +1,14 @@ +# Test fixture: a process() that never returns while `forever` is set (plan 8.3). The loop swallows +# every Exception, so only a BaseException — the worker's WorkerStopped — can end it from outside. + + +class hangs: + forever: bool = True + + def process(self, x: float) -> float: + while self.forever: + try: + x = x * 1.0 + except Exception: + pass + return x diff --git a/core/tests/python/sleeps.py b/core/tests/python/sleeps.py new file mode 100644 index 0000000..b9ecdc7 --- /dev/null +++ b/core/tests/python/sleeps.py @@ -0,0 +1,12 @@ +# Test fixture: a process() blocked in a call Python cannot interrupt — time.sleep delivers an +# asynchronous exception only when it returns (plan 8.3). Identity once `seconds` is 0. +import time + + +class sleeps: + seconds: float = 3.0 + + def process(self, x: float) -> float: + if self.seconds: + time.sleep(self.seconds) + return x diff --git a/core/tests/test_worker.cpp b/core/tests/test_worker.cpp index 9fe1a8a..42a280e 100644 --- a/core/tests/test_worker.cpp +++ b/core/tests/test_worker.cpp @@ -295,6 +295,99 @@ SCENARIO("A worker can be restarted with new settings, and stopped (plan 2.5)") } } +// 8.3 — a worker never hangs the host + +SCENARIO("A process() that never returns is interrupted when the worker stops, and audio stays bound (plan 8.3)") { + ensure_runtime(); + harness h{"hangs"}; + REQUIRE(h.p.load()); + h.w.start(1, 1, k_frames, 2, 48000.0); + h.push({1.0}, 1); // the worker enters process() and never leaves it + std::this_thread::sleep_for(std::chrono::milliseconds{50}); + REQUIRE(h.w.processed() == 0); + + const auto before = std::chrono::steady_clock::now(); + h.w.stop(); + const auto took = std::chrono::steady_clock::now() - before; + THEN("stop() returns within a second, having joined the thread") { + CHECK(took < std::chrono::seconds{1}); + CHECK_FALSE(h.w.running()); + CHECK_FALSE(h.w.has_abandoned_thread()); + } + THEN("the interruption is reported, from the main thread, and the class's audio is still bound") { + CHECK(h.notified.load() == 1); + h.p.flush_reports(); + CHECK(h.log.contains("interrupted", log_level::error)); + CHECK_FALSE(h.log.contains("audio disabled", log_level::error)); + CHECK(h.p.has_process()); + } + THEN("the same class runs again once it returns") { + REQUIRE(h.p.set_attribute("forever", std::int64_t{0})); + h.w.start(1, 1, k_frames, 1, 48000.0); + h.push({2.0}, 1); + h.wait_for(1); + CHECK(all_equal(h.push({3.0}, 1)[0], 2.0)); + } +} + +SCENARIO("A process() blocked in a call Python cannot interrupt is abandoned, not waited for (plan 8.3)") { + ensure_runtime(); + // the abandoned thread may wake and touch the processor and worker later: leaked, as a host must + auto* h = new harness{"sleeps"}; // NOLINT(cppcoreguidelines-owning-memory) + REQUIRE(h->p.load()); + h->w.start(1, 1, k_frames, 2, 48000.0); + h->push({1.0}, 1); // the worker sleeps 3 s inside process() + std::this_thread::sleep_for(std::chrono::milliseconds{50}); + + const auto before = std::chrono::steady_clock::now(); + h->w.stop(); + const auto took = std::chrono::steady_clock::now() - before; + THEN("stop() returns within a second, having abandoned the thread, and says so") { + CHECK(took < std::chrono::seconds{1}); + CHECK_FALSE(h->w.running()); + CHECK(h->w.has_abandoned_thread()); + CHECK(h->log.contains("abandoned", log_level::error)); + } + THEN("a new worker runs the same class once it no longer blocks") { + REQUIRE(h->p.set_attribute("seconds", 0.0)); + h->w.start(1, 1, k_frames, 1, 48000.0); + h->push({2.0}, 1); + h->wait_for(1); + CHECK(all_equal(h->push({3.0}, 1)[0], 2.0)); + h->w.stop(); + } + if (!h->w.has_abandoned_thread()) { + delete h; // NOLINT(cppcoreguidelines-owning-memory) + } +} + +SCENARIO("A vector that is merely slow is waited for, not interrupted (plan 8.3)") { + ensure_runtime(); + harness h{"stalls"}; + REQUIRE(h.p.load()); + h.w.start(1, 1, k_frames, 2, 48000.0); + REQUIRE(h.p.set_attribute("stall", value{0.05})); + h.push({1.0}, 1); + std::this_thread::sleep_for(std::chrono::milliseconds{10}); // into the 50 ms stall + h.w.stop(); + h.p.flush_reports(); + CHECK(h.notified.load() == 0); + CHECK_FALSE(h.log.contains("interrupted", log_level::error)); + CHECK_FALSE(h.w.has_abandoned_thread()); +} + +SCENARIO("The worker's stack takes recursion through a C boundary to the recursion limit (plan 8.3)") { + ensure_runtime(); + harness h{"deep_recursion"}; + REQUIRE(h.p.load()); + h.w.start(1, 1, 4, 1, 48000.0); + h.push({0.5}, 1, 4); + h.wait_for(1); + const auto outputs = h.push({0.5}, 1, 4); + CHECK(all_equal(outputs[0], 0.5)); // every sample made it down and back + CHECK(h.notified.load() == 0); // no exception: the recursion did not hit a limit +} + SCENARIO("A reload while the worker runs takes effect without a pause on the audio thread (plan 2.5)") { ensure_runtime(); write_script("worker_reload", diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index 2809904..de36f04 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -612,7 +612,7 @@ down rather than discovered again. *This plan was itself audited before being ad (8.8), as is Build Collective for `fileusage`. In the mock kernel `object_getmethod()` always answers null, so the guard's own effect is seen only in Max: the existing `attributes-and-messages` runtime test is what would show it over-reserving. -- [ ] **8.3 Worker mode never hangs Max (A1, A6 — `worker.h`).** +- [x] **8.3 Worker mode never hangs Max (A1, A6 — `worker.h`).** *A1:* `stop()` bounds its join. The worker records its OS thread identifier (`PyThread_get_thread_ident()`, no GIL needed) as it starts; if the thread has not finished within 100 ms of `stopping` being set, `stop()` takes the GIL briefly (the hung thread yields it @@ -646,7 +646,20 @@ down rather than discovered again. *This plan was itself audited before being ad the secondary-thread default from a fresh `pthread_attr_t`, and the depth at which a fixture recursing through a C boundary (`sorted(key=…)` calling itself) crashes on a `std::thread` against a `threading.Thread`; then pin that fixture running to `sys.getrecursionlimit()` on the - worker. CHANGELOG: a stalled class in worker mode is interrupted or abandoned, never waited for. + worker. CHANGELOG: a stalled class in worker mode is interrupted or abandoned, never waited for. *Done:* as designed — `WorkerStopped(BaseException)` in the support module; + `processor::take_interruption()` recognizes it, records "interrupted" (once per load) and keeps + the binding; `worker::stop()` polls `finished` for 100 ms, injects with + `PyThreadState_SetAsyncExc` under a brief `gil_lock`, polls 250 ms more, then detaches; the ring + is a `shared_ptr` the thread also holds, `has_abandoned_thread()` prunes the finished ones, and the + Max object's destructor leaks the worker and processor while one lives. `detail::native_thread` + (pthread / `_beginthreadex`, 16 MiB) replaces `std::thread`. Four core scenarios: `hangs.py` + (`while self.forever` inside `except Exception`) — stop() in 0.5 s, interruption reported, audio + bound, the class runs on; `sleeps.py` (`time.sleep(3)`) — stop() in under a second, abandoned + and said so, a new worker runs the class (the harness is leaked, as a host must); `stalls.py` at + 50 ms — waited for, nothing reported; `deep_recursion.py` (sorted's key recursing to the limit) — + passes on the worker. Release, ASan/UBSan and TSan clean. The `worker` runtime test gains `hang 1` + → `mode direct` → `hang 0`, expecting one "interrupted" line and two errors in all — step timing + to confirm in the Mac session (8.8), with the stack measurement the item asks for. - [ ] **8.4 Nothing of the user's prints on the audio thread either (A5).** `runtime_options` gains `is_main_thread` (the host's predicate — `systhread_ismainthread` in Max; the default, with no host predicate, is "always", which keeps the core battery's synchronous `console()` diff --git a/runtime-tests/make_patchers.py b/runtime-tests/make_patchers.py index 9bbbcb9..2f01329 100644 --- a/runtime-tests/make_patchers.py +++ b/runtime-tests/make_patchers.py @@ -613,7 +613,9 @@ def worker(latency_ms: float | None = None, quiet_windows: int = 0) -> Test: "Worker mode (plan 2.5), at its default latency: with @mode worker the output is the input " "delayed by @latencysamples, sample for sample what delay~ gives, and stays so through a " "reload; a class that stalls for 0.2 s is late, reported once, and comes back at the same " - "latency; @mode direct takes the delay away.") + "latency; @mode direct takes the delay away. A process() that never returns is interrupted " + "when the worker stops for the switch to direct (plan 8.3), reported once, and the class " + "runs on once it returns.") ramp = t.obj("phasor~ 50", signal=True) # a new value every sample, so any delay shows py = t.python("maxtest_stall @mode worker" + (f" @latency {latency_ms}" if latency_ms is not None else "")) delay = t.obj("delay~ 48000 0", inlets=2, signal=True) @@ -627,6 +629,7 @@ def worker(latency_ms: float | None = None, quiet_windows: int = 0) -> Test: t.patcher.connect(delay_by_latency, 1, py) t.patcher.connect(delay_by_latency, 0, delay, 1) t.count_errors("late.for") + t.count_errors("interrupted") watch = t.no_change("delayed-by-latencysamples", late, 0.0) after = t.no_change("same-latency-after-stall", late, 0.0) @@ -644,10 +647,16 @@ def worker(latency_ms: float | None = None, quiet_windows: int = 0) -> Test: t.step(after[1], wait=1000) t.step(t.send("filechanged", py)) t.step(t.sample_equals("delayed-after-reload", late, 0.0), wait=500) - t.step(t.send("mode direct", py)) + # the worker enters a process() that never returns; switching to direct stops it, which must + # interrupt the vector (within 350 ms) rather than hang Max; direct mode then runs the same + # class, which loops until hang is cleared a step later (plan 8.3) + t.step(t.send("hang 1", py)) + t.step(t.send("mode direct", py), wait=500) + t.step(t.send("hang 0", py), wait=500) t.step(t.attribute_equals("direct-reports-no-latency", py, "latencysamples", 0), t.sample_equals("direct-has-no-delay", now, 0.0), - t.errors_are("console-only-the-stall", "== 1"), wait=500) + t.errors_are("hang-interrupted-once", "== 1", "interrupted"), + t.errors_are("console-only-the-stall-and-the-interruption", "== 2"), wait=1000) return t diff --git a/runtime-tests/patchers/tap.python~.worker.maxtest.maxpat b/runtime-tests/patchers/tap.python~.worker.maxtest.maxpat index 7a8ce54..9863076 100644 --- a/runtime-tests/patchers/tap.python~.worker.maxtest.maxpat +++ b/runtime-tests/patchers/tap.python~.worker.maxtest.maxpat @@ -35,7 +35,7 @@ 480, 80.0 ], - "text": "tap.python~.worker\n\nWorker mode (plan 2.5), at its default latency: with @mode worker the output is the input delayed by @latencysamples, sample for sample what delay~ gives, and stays so through a reload; a class that stalls for 0.2 s is late, reported once, and comes back at the same latency; @mode direct takes the delay away.", + "text": "tap.python~.worker\n\nWorker mode (plan 2.5), at its default latency: with @mode worker the output is the input delayed by @latencysamples, sample for sample what delay~ gives, and stays so through a reload; a class that stalls for 0.2 s is late, reported once, and comes back at the same latency; @mode direct takes the delay away. A process() that never returns is interrupted when the worker stops for the switch to direct (plan 8.3), reported once, and the class runs on once it returns.", "linecount": 4 } }, @@ -232,6 +232,67 @@ "box": { "id": "obj-12", "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 5, + "outlettype": [ + "", + "", + "", + "", + "" + ], + "patching_rect": [ + 1020.0, + 140.0, + 142.0, + 22.0 + ], + "text": "regexp interrupted" + } + }, + { + "box": { + "id": "obj-13", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 170.0, + 40.0, + 22.0 + ], + "text": "t b" + } + }, + { + "box": { + "id": "obj-14", + "maxclass": "newobj", + "numinlets": 3, + "numoutlets": 4, + "outlettype": [ + "", + "", + "", + "" + ], + "patching_rect": [ + 1020.0, + 200.0, + 135.0, + 22.0 + ], + "text": "counter 1 1000000" + } + }, + { + "box": { + "id": "obj-15", + "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, "outlettype": [ @@ -248,7 +309,7 @@ }, { "box": { - "id": "obj-13", + "id": "obj-16", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -266,7 +327,7 @@ }, { "box": { - "id": "obj-14", + "id": "obj-17", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -285,7 +346,7 @@ }, { "box": { - "id": "obj-15", + "id": "obj-18", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -303,7 +364,7 @@ }, { "box": { - "id": "obj-16", + "id": "obj-19", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -321,7 +382,7 @@ }, { "box": { - "id": "obj-17", + "id": "obj-20", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -339,7 +400,7 @@ }, { "box": { - "id": "obj-18", + "id": "obj-21", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -357,7 +418,7 @@ }, { "box": { - "id": "obj-19", + "id": "obj-22", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -375,7 +436,7 @@ }, { "box": { - "id": "obj-20", + "id": "obj-23", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -393,7 +454,7 @@ }, { "box": { - "id": "obj-21", + "id": "obj-24", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -411,7 +472,7 @@ }, { "box": { - "id": "obj-22", + "id": "obj-25", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -430,7 +491,7 @@ }, { "box": { - "id": "obj-23", + "id": "obj-26", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -448,7 +509,7 @@ }, { "box": { - "id": "obj-24", + "id": "obj-27", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -466,7 +527,7 @@ }, { "box": { - "id": "obj-25", + "id": "obj-28", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -484,7 +545,7 @@ }, { "box": { - "id": "obj-26", + "id": "obj-29", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -502,7 +563,7 @@ }, { "box": { - "id": "obj-27", + "id": "obj-30", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -520,7 +581,7 @@ }, { "box": { - "id": "obj-28", + "id": "obj-31", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -540,7 +601,7 @@ }, { "box": { - "id": "obj-29", + "id": "obj-32", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -558,7 +619,7 @@ }, { "box": { - "id": "obj-30", + "id": "obj-33", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -576,7 +637,7 @@ }, { "box": { - "id": "obj-31", + "id": "obj-34", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -587,7 +648,7 @@ ], "patching_rect": [ 1020.0, - 140.0, + 230.0, 240.0, 22.0 ], @@ -596,7 +657,7 @@ }, { "box": { - "id": "obj-32", + "id": "obj-35", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -605,7 +666,7 @@ ], "patching_rect": [ 1020.0, - 170.0, + 260.0, 170.0, 22.0 ], @@ -614,14 +675,14 @@ }, { "box": { - "id": "obj-33", + "id": "obj-36", "maxclass": "newobj", "numinlets": 1, "numoutlets": 0, "outlettype": [], "patching_rect": [ 1020.0, - 200.0, + 290.0, 128.0, 22.0 ], @@ -630,7 +691,7 @@ }, { "box": { - "id": "obj-34", + "id": "obj-37", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -648,7 +709,7 @@ }, { "box": { - "id": "obj-35", + "id": "obj-38", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -657,7 +718,7 @@ ], "patching_rect": [ 1020.0, - 230.0, + 320.0, 40.0, 22.0 ], @@ -666,7 +727,7 @@ }, { "box": { - "id": "obj-36", + "id": "obj-39", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -675,7 +736,7 @@ ], "patching_rect": [ 1020.0, - 260.0, + 350.0, 44.0, 22.0 ], @@ -684,7 +745,7 @@ }, { "box": { - "id": "obj-37", + "id": "obj-40", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -702,7 +763,7 @@ }, { "box": { - "id": "obj-38", + "id": "obj-41", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -720,7 +781,7 @@ }, { "box": { - "id": "obj-39", + "id": "obj-42", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -738,7 +799,7 @@ }, { "box": { - "id": "obj-40", + "id": "obj-43", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -756,7 +817,7 @@ }, { "box": { - "id": "obj-41", + "id": "obj-44", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -774,7 +835,7 @@ }, { "box": { - "id": "obj-42", + "id": "obj-45", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -784,6 +845,24 @@ "patching_rect": [ 270.0, 80.0, + 58.0, + 22.0 + ], + "text": "hang 1" + } + }, + { + "box": { + "id": "obj-46", + "maxclass": "message", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 270.0, + 110.0, 93.0, 22.0 ], @@ -792,7 +871,25 @@ }, { "box": { - "id": "obj-43", + "id": "obj-47", + "maxclass": "message", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 270.0, + 140.0, + 58.0, + 22.0 + ], + "text": "hang 0" + } + }, + { + "box": { + "id": "obj-48", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -812,7 +909,7 @@ }, { "box": { - "id": "obj-44", + "id": "obj-49", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -830,7 +927,7 @@ }, { "box": { - "id": "obj-45", + "id": "obj-50", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -848,7 +945,7 @@ }, { "box": { - "id": "obj-46", + "id": "obj-51", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -866,7 +963,7 @@ }, { "box": { - "id": "obj-47", + "id": "obj-52", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -884,7 +981,7 @@ }, { "box": { - "id": "obj-48", + "id": "obj-53", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -902,7 +999,61 @@ }, { "box": { - "id": "obj-49", + "id": "obj-54", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 380.0, + 40.0, + 22.0 + ], + "text": "i" + } + }, + { + "box": { + "id": "obj-55", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 1020.0, + 410.0, + 44.0, + 22.0 + ], + "text": "== 1" + } + }, + { + "box": { + "id": "obj-56", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 770.0, + 920.0, + 247.0, + 22.0 + ], + "text": "test.assert hang-interrupted-once" + } + }, + { + "box": { + "id": "obj-57", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -911,7 +1062,7 @@ ], "patching_rect": [ 1020.0, - 290.0, + 440.0, 40.0, 22.0 ], @@ -920,7 +1071,7 @@ }, { "box": { - "id": "obj-50", + "id": "obj-58", "maxclass": "newobj", "numinlets": 3, "numoutlets": 4, @@ -932,7 +1083,7 @@ ], "patching_rect": [ 1020.0, - 320.0, + 470.0, 135.0, 22.0 ], @@ -941,7 +1092,7 @@ }, { "box": { - "id": "obj-51", + "id": "obj-59", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -950,7 +1101,7 @@ ], "patching_rect": [ 1020.0, - 350.0, + 500.0, 40.0, 22.0 ], @@ -959,7 +1110,7 @@ }, { "box": { - "id": "obj-52", + "id": "obj-60", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -968,16 +1119,16 @@ ], "patching_rect": [ 1020.0, - 380.0, + 530.0, 44.0, 22.0 ], - "text": "== 1" + "text": "== 2" } }, { "box": { - "id": "obj-53", + "id": "obj-61", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -986,16 +1137,16 @@ ], "patching_rect": [ 770.0, - 920.0, - 254.0, + 950.0, + 401.0, 22.0 ], - "text": "test.assert console-only-the-stall" + "text": "test.assert console-only-the-stall-and-the-interruption" } }, { "box": { - "id": "obj-54", + "id": "obj-62", "maxclass": "newobj", "numinlets": 1, "numoutlets": 0, @@ -1011,7 +1162,7 @@ }, { "box": { - "id": "obj-55", + "id": "obj-63", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -1020,7 +1171,7 @@ ], "patching_rect": [ 1020.0, - 410.0, + 560.0, 72.0, 22.0 ], @@ -1029,7 +1180,7 @@ }, { "box": { - "id": "obj-56", + "id": "obj-64", "maxclass": "newobj", "numinlets": 1, "numoutlets": 5, @@ -1042,7 +1193,7 @@ ], "patching_rect": [ 1020.0, - 440.0, + 590.0, 177.0, 22.0 ], @@ -1051,7 +1202,7 @@ }, { "box": { - "id": "obj-57", + "id": "obj-65", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -1060,7 +1211,7 @@ ], "patching_rect": [ 1020.0, - 470.0, + 620.0, 86.0, 22.0 ], @@ -1069,7 +1220,7 @@ }, { "box": { - "id": "obj-58", + "id": "obj-66", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1079,7 +1230,7 @@ ], "patching_rect": [ 1020.0, - 500.0, + 650.0, 51.0, 22.0 ], @@ -1088,7 +1239,7 @@ }, { "box": { - "id": "obj-59", + "id": "obj-67", "maxclass": "newobj", "numinlets": 3, "numoutlets": 4, @@ -1100,7 +1251,7 @@ ], "patching_rect": [ 1020.0, - 530.0, + 680.0, 135.0, 22.0 ], @@ -1109,7 +1260,7 @@ }, { "box": { - "id": "obj-60", + "id": "obj-68", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1118,7 +1269,7 @@ ], "patching_rect": [ 1020.0, - 560.0, + 710.0, 51.0, 22.0 ], @@ -1127,7 +1278,7 @@ }, { "box": { - "id": "obj-61", + "id": "obj-69", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1136,7 +1287,7 @@ ], "patching_rect": [ 1020.0, - 590.0, + 740.0, 72.0, 22.0 ], @@ -1145,14 +1296,14 @@ }, { "box": { - "id": "obj-62", + "id": "obj-70", "maxclass": "newobj", "numinlets": 1, "numoutlets": 0, "outlettype": [], "patching_rect": [ 1020.0, - 620.0, + 770.0, 170.0, 22.0 ], @@ -1161,7 +1312,7 @@ }, { "box": { - "id": "obj-63", + "id": "obj-71", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -1179,7 +1330,7 @@ }, { "box": { - "id": "obj-64", + "id": "obj-72", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1198,7 +1349,7 @@ }, { "box": { - "id": "obj-65", + "id": "obj-73", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -1216,7 +1367,7 @@ }, { "box": { - "id": "obj-66", + "id": "obj-74", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1234,7 +1385,7 @@ }, { "box": { - "id": "obj-67", + "id": "obj-75", "maxclass": "newobj", "numinlets": 1, "numoutlets": 4, @@ -1255,7 +1406,7 @@ }, { "box": { - "id": "obj-68", + "id": "obj-76", "maxclass": "newobj", "numinlets": 2, "numoutlets": 2, @@ -1274,7 +1425,7 @@ }, { "box": { - "id": "obj-69", + "id": "obj-77", "maxclass": "newobj", "numinlets": 2, "numoutlets": 2, @@ -1293,7 +1444,7 @@ }, { "box": { - "id": "obj-70", + "id": "obj-78", "maxclass": "message", "numinlets": 2, "numoutlets": 1, @@ -1311,7 +1462,7 @@ }, { "box": { - "id": "obj-71", + "id": "obj-79", "maxclass": "newobj", "numinlets": 1, "numoutlets": 1, @@ -1320,7 +1471,7 @@ ], "patching_rect": [ 770.0, - 950.0, + 980.0, 191.0, 22.0 ], @@ -1329,7 +1480,7 @@ }, { "box": { - "id": "obj-72", + "id": "obj-80", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1347,7 +1498,7 @@ }, { "box": { - "id": "obj-73", + "id": "obj-81", "maxclass": "newobj", "numinlets": 1, "numoutlets": 4, @@ -1368,7 +1519,7 @@ }, { "box": { - "id": "obj-74", + "id": "obj-82", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1386,7 +1537,7 @@ }, { "box": { - "id": "obj-75", + "id": "obj-83", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1405,7 +1556,7 @@ }, { "box": { - "id": "obj-76", + "id": "obj-84", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1423,7 +1574,7 @@ }, { "box": { - "id": "obj-77", + "id": "obj-85", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1442,7 +1593,7 @@ }, { "box": { - "id": "obj-78", + "id": "obj-86", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1460,7 +1611,7 @@ }, { "box": { - "id": "obj-79", + "id": "obj-87", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1479,7 +1630,7 @@ }, { "box": { - "id": "obj-80", + "id": "obj-88", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1497,7 +1648,7 @@ }, { "box": { - "id": "obj-81", + "id": "obj-89", "maxclass": "newobj", "numinlets": 1, "numoutlets": 3, @@ -1517,7 +1668,7 @@ }, { "box": { - "id": "obj-82", + "id": "obj-90", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1535,7 +1686,7 @@ }, { "box": { - "id": "obj-83", + "id": "obj-91", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1554,7 +1705,7 @@ }, { "box": { - "id": "obj-84", + "id": "obj-92", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1572,7 +1723,7 @@ }, { "box": { - "id": "obj-85", + "id": "obj-93", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1591,7 +1742,7 @@ }, { "box": { - "id": "obj-86", + "id": "obj-94", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1609,7 +1760,7 @@ }, { "box": { - "id": "obj-87", + "id": "obj-95", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1628,7 +1779,7 @@ }, { "box": { - "id": "obj-88", + "id": "obj-96", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1646,7 +1797,7 @@ }, { "box": { - "id": "obj-89", + "id": "obj-97", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1665,7 +1816,7 @@ }, { "box": { - "id": "obj-90", + "id": "obj-98", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1683,28 +1834,26 @@ }, { "box": { - "id": "obj-91", + "id": "obj-99", "maxclass": "newobj", "numinlets": 1, - "numoutlets": 4, + "numoutlets": 2, "outlettype": [ - "", - "", "", "" ], "patching_rect": [ 20.0, 948.0, - 79.0, + 51.0, 22.0 ], - "text": "t b b b b" + "text": "t b b" } }, { "box": { - "id": "obj-92", + "id": "obj-100", "maxclass": "newobj", "numinlets": 2, "numoutlets": 1, @@ -1722,7 +1871,7 @@ }, { "box": { - "id": "obj-93", + "id": "obj-101", "maxclass": "newobj", "numinlets": 1, "numoutlets": 2, @@ -1738,6 +1887,83 @@ ], "text": "t b b" } + }, + { + "box": { + "id": "obj-102", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 1038.0, + 86.0, + 22.0 + ], + "text": "delay 1000" + } + }, + { + "box": { + "id": "obj-103", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 5, + "outlettype": [ + "", + "", + "", + "", + "" + ], + "patching_rect": [ + 20.0, + 1068.0, + 93.0, + 22.0 + ], + "text": "t b b b b b" + } + }, + { + "box": { + "id": "obj-104", + "maxclass": "newobj", + "numinlets": 2, + "numoutlets": 1, + "outlettype": [ + "" + ], + "patching_rect": [ + 20.0, + 1098.0, + 79.0, + 22.0 + ], + "text": "delay 500" + } + }, + { + "box": { + "id": "obj-105", + "maxclass": "newobj", + "numinlets": 1, + "numoutlets": 2, + "outlettype": [ + "", + "" + ], + "patching_rect": [ + 20.0, + 1128.0, + 51.0, + 22.0 + ], + "text": "t b b" + } } ], "lines": [ @@ -1876,7 +2102,7 @@ { "patchline": { "source": [ - "obj-5", + "obj-8", 0 ], "destination": [ @@ -1889,7 +2115,7 @@ "patchline": { "source": [ "obj-12", - 0 + 2 ], "destination": [ "obj-13", @@ -1900,11 +2126,11 @@ { "patchline": { "source": [ - "obj-14", - 1 + "obj-13", + 0 ], "destination": [ - "obj-13", + "obj-14", 0 ] } @@ -1912,11 +2138,11 @@ { "patchline": { "source": [ - "obj-14", + "obj-5", 0 ], "destination": [ - "obj-17", + "obj-15", 0 ] } @@ -1924,7 +2150,7 @@ { "patchline": { "source": [ - "obj-17", + "obj-15", 0 ], "destination": [ @@ -1936,11 +2162,11 @@ { "patchline": { "source": [ - "obj-15", - 0 + "obj-17", + 1 ], "destination": [ - "obj-13", + "obj-16", 0 ] } @@ -1948,23 +2174,23 @@ { "patchline": { "source": [ - "obj-13", + "obj-17", 0 ], "destination": [ - "obj-16", - 1 + "obj-20", + 0 ] } }, { "patchline": { "source": [ - "obj-16", + "obj-20", 0 ], "destination": [ - "obj-18", + "obj-19", 0 ] } @@ -1976,7 +2202,7 @@ 0 ], "destination": [ - "obj-19", + "obj-16", 0 ] } @@ -1984,19 +2210,19 @@ { "patchline": { "source": [ - "obj-5", + "obj-16", 0 ], "destination": [ - "obj-20", - 0 + "obj-19", + 1 ] } }, { "patchline": { "source": [ - "obj-20", + "obj-19", 0 ], "destination": [ @@ -2008,11 +2234,11 @@ { "patchline": { "source": [ - "obj-22", - 1 + "obj-21", + 0 ], "destination": [ - "obj-21", + "obj-22", 0 ] } @@ -2020,11 +2246,11 @@ { "patchline": { "source": [ - "obj-22", + "obj-5", 0 ], "destination": [ - "obj-25", + "obj-23", 0 ] } @@ -2032,7 +2258,7 @@ { "patchline": { "source": [ - "obj-25", + "obj-23", 0 ], "destination": [ @@ -2044,11 +2270,11 @@ { "patchline": { "source": [ - "obj-23", - 0 + "obj-25", + 1 ], "destination": [ - "obj-21", + "obj-24", 0 ] } @@ -2056,23 +2282,23 @@ { "patchline": { "source": [ - "obj-21", + "obj-25", 0 ], "destination": [ - "obj-24", - 1 + "obj-28", + 0 ] } }, { "patchline": { "source": [ - "obj-24", + "obj-28", 0 ], "destination": [ - "obj-26", + "obj-27", 0 ] } @@ -2084,7 +2310,7 @@ 0 ], "destination": [ - "obj-27", + "obj-24", 0 ] } @@ -2092,19 +2318,19 @@ { "patchline": { "source": [ - "obj-28", - 1 + "obj-24", + 0 ], "destination": [ - "obj-3", - 0 + "obj-27", + 1 ] } }, { "patchline": { "source": [ - "obj-28", + "obj-27", 0 ], "destination": [ @@ -2165,7 +2391,7 @@ "patchline": { "source": [ "obj-34", - 0 + 1 ], "destination": [ "obj-3", @@ -2176,12 +2402,12 @@ { "patchline": { "source": [ - "obj-11", + "obj-34", 0 ], "destination": [ "obj-35", - 1 + 0 ] } }, @@ -2200,11 +2426,11 @@ { "patchline": { "source": [ - "obj-36", + "obj-37", 0 ], "destination": [ - "obj-37", + "obj-3", 0 ] } @@ -2212,19 +2438,19 @@ { "patchline": { "source": [ - "obj-38", + "obj-11", 0 ], "destination": [ - "obj-3", - 0 + "obj-38", + 1 ] } }, { "patchline": { "source": [ - "obj-5", + "obj-38", 0 ], "destination": [ @@ -2248,11 +2474,11 @@ { "patchline": { "source": [ - "obj-40", + "obj-41", 0 ], "destination": [ - "obj-41", + "obj-3", 0 ] } @@ -2260,11 +2486,11 @@ { "patchline": { "source": [ - "obj-42", + "obj-5", 0 ], "destination": [ - "obj-3", + "obj-42", 0 ] } @@ -2272,11 +2498,11 @@ { "patchline": { "source": [ - "obj-43", - 1 + "obj-42", + 0 ], "destination": [ - "obj-3", + "obj-43", 0 ] } @@ -2296,11 +2522,11 @@ { "patchline": { "source": [ - "obj-44", + "obj-45", 0 ], "destination": [ - "obj-45", + "obj-3", 0 ] } @@ -2308,11 +2534,11 @@ { "patchline": { "source": [ - "obj-6", + "obj-46", 0 ], "destination": [ - "obj-46", + "obj-3", 0 ] } @@ -2320,11 +2546,11 @@ { "patchline": { "source": [ - "obj-46", + "obj-47", 0 ], "destination": [ - "obj-47", + "obj-3", 0 ] } @@ -2332,11 +2558,11 @@ { "patchline": { "source": [ - "obj-47", - 0 + "obj-48", + 1 ], "destination": [ - "obj-48", + "obj-3", 0 ] } @@ -2344,7 +2570,7 @@ { "patchline": { "source": [ - "obj-8", + "obj-48", 0 ], "destination": [ @@ -2368,12 +2594,12 @@ { "patchline": { "source": [ - "obj-50", + "obj-6", 0 ], "destination": [ "obj-51", - 1 + 0 ] } }, @@ -2404,23 +2630,23 @@ { "patchline": { "source": [ - "obj-8", + "obj-14", 0 ], "destination": [ - "obj-55", - 0 + "obj-54", + 1 ] } }, { "patchline": { "source": [ - "obj-55", + "obj-54", 0 ], "destination": [ - "obj-56", + "obj-55", 0 ] } @@ -2428,11 +2654,11 @@ { "patchline": { "source": [ - "obj-56", + "obj-55", 0 ], "destination": [ - "obj-57", + "obj-56", 0 ] } @@ -2440,8 +2666,8 @@ { "patchline": { "source": [ - "obj-56", - 3 + "obj-8", + 0 ], "destination": [ "obj-57", @@ -2465,11 +2691,11 @@ "patchline": { "source": [ "obj-58", - 1 + 0 ], "destination": [ "obj-59", - 0 + 1 ] } }, @@ -2500,23 +2726,23 @@ { "patchline": { "source": [ - "obj-58", + "obj-8", 0 ], "destination": [ - "obj-61", - 1 + "obj-63", + 0 ] } }, { "patchline": { "source": [ - "obj-61", + "obj-63", 0 ], "destination": [ - "obj-62", + "obj-64", 0 ] } @@ -2524,11 +2750,11 @@ { "patchline": { "source": [ - "obj-63", + "obj-64", 0 ], "destination": [ - "obj-64", + "obj-65", 0 ] } @@ -2537,10 +2763,10 @@ "patchline": { "source": [ "obj-64", - 1 + 3 ], "destination": [ - "obj-66", + "obj-65", 0 ] } @@ -2548,11 +2774,11 @@ { "patchline": { "source": [ - "obj-64", + "obj-65", 0 ], "destination": [ - "obj-65", + "obj-66", 0 ] } @@ -2561,10 +2787,10 @@ "patchline": { "source": [ "obj-66", - 0 + 1 ], "destination": [ - "obj-54", + "obj-67", 0 ] } @@ -2593,6 +2819,18 @@ ] } }, + { + "patchline": { + "source": [ + "obj-66", + 0 + ], + "destination": [ + "obj-69", + 1 + ] + } + }, { "patchline": { "source": [ @@ -2608,11 +2846,11 @@ { "patchline": { "source": [ - "obj-70", + "obj-71", 0 ], "destination": [ - "obj-71", + "obj-72", 0 ] } @@ -2620,11 +2858,11 @@ { "patchline": { "source": [ - "obj-69", - 0 + "obj-72", + 1 ], "destination": [ - "obj-72", + "obj-74", 0 ] } @@ -2644,11 +2882,95 @@ { "patchline": { "source": [ - "obj-73", + "obj-74", + 0 + ], + "destination": [ + "obj-62", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-75", + 0 + ], + "destination": [ + "obj-76", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-76", + 0 + ], + "destination": [ + "obj-77", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-77", + 0 + ], + "destination": [ + "obj-78", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-78", + 0 + ], + "destination": [ + "obj-79", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-77", + 0 + ], + "destination": [ + "obj-80", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-80", + 0 + ], + "destination": [ + "obj-81", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-81", 3 ], "destination": [ - "obj-28", + "obj-31", 0 ] } @@ -2656,7 +2978,7 @@ { "patchline": { "source": [ - "obj-73", + "obj-81", 2 ], "destination": [ @@ -2668,11 +2990,11 @@ { "patchline": { "source": [ - "obj-73", + "obj-81", 1 ], "destination": [ - "obj-31", + "obj-34", 0 ] } @@ -2680,11 +3002,11 @@ { "patchline": { "source": [ - "obj-73", + "obj-81", 0 ], "destination": [ - "obj-74", + "obj-82", 0 ] } @@ -2692,11 +3014,11 @@ { "patchline": { "source": [ - "obj-74", + "obj-82", 0 ], "destination": [ - "obj-75", + "obj-83", 0 ] } @@ -2704,11 +3026,11 @@ { "patchline": { "source": [ - "obj-75", + "obj-83", 1 ], "destination": [ - "obj-14", + "obj-17", 0 ] } @@ -2716,11 +3038,11 @@ { "patchline": { "source": [ - "obj-75", + "obj-83", 0 ], "destination": [ - "obj-76", + "obj-84", 0 ] } @@ -2728,11 +3050,11 @@ { "patchline": { "source": [ - "obj-76", + "obj-84", 0 ], "destination": [ - "obj-77", + "obj-85", 0 ] } @@ -2740,11 +3062,11 @@ { "patchline": { "source": [ - "obj-77", + "obj-85", 1 ], "destination": [ - "obj-15", + "obj-18", 0 ] } @@ -2752,11 +3074,11 @@ { "patchline": { "source": [ - "obj-77", + "obj-85", 0 ], "destination": [ - "obj-78", + "obj-86", 0 ] } @@ -2764,11 +3086,11 @@ { "patchline": { "source": [ - "obj-78", + "obj-86", 0 ], "destination": [ - "obj-79", + "obj-87", 0 ] } @@ -2776,11 +3098,11 @@ { "patchline": { "source": [ - "obj-79", + "obj-87", 1 ], "destination": [ - "obj-34", + "obj-37", 0 ] } @@ -2788,11 +3110,11 @@ { "patchline": { "source": [ - "obj-79", + "obj-87", 0 ], "destination": [ - "obj-80", + "obj-88", 0 ] } @@ -2800,11 +3122,11 @@ { "patchline": { "source": [ - "obj-80", + "obj-88", 0 ], "destination": [ - "obj-81", + "obj-89", 0 ] } @@ -2812,11 +3134,11 @@ { "patchline": { "source": [ - "obj-81", + "obj-89", 2 ], "destination": [ - "obj-35", + "obj-38", 0 ] } @@ -2824,11 +3146,11 @@ { "patchline": { "source": [ - "obj-81", + "obj-89", 1 ], "destination": [ - "obj-22", + "obj-25", 0 ] } @@ -2836,11 +3158,11 @@ { "patchline": { "source": [ - "obj-81", + "obj-89", 0 ], "destination": [ - "obj-82", + "obj-90", 0 ] } @@ -2848,11 +3170,11 @@ { "patchline": { "source": [ - "obj-82", + "obj-90", 0 ], "destination": [ - "obj-83", + "obj-91", 0 ] } @@ -2860,11 +3182,11 @@ { "patchline": { "source": [ - "obj-83", + "obj-91", 1 ], "destination": [ - "obj-23", + "obj-26", 0 ] } @@ -2872,11 +3194,11 @@ { "patchline": { "source": [ - "obj-83", + "obj-91", 0 ], "destination": [ - "obj-84", + "obj-92", 0 ] } @@ -2884,11 +3206,11 @@ { "patchline": { "source": [ - "obj-84", + "obj-92", 0 ], "destination": [ - "obj-85", + "obj-93", 0 ] } @@ -2896,11 +3218,11 @@ { "patchline": { "source": [ - "obj-85", + "obj-93", 1 ], "destination": [ - "obj-38", + "obj-41", 0 ] } @@ -2908,11 +3230,11 @@ { "patchline": { "source": [ - "obj-85", + "obj-93", 0 ], "destination": [ - "obj-86", + "obj-94", 0 ] } @@ -2920,11 +3242,11 @@ { "patchline": { "source": [ - "obj-86", + "obj-94", 0 ], "destination": [ - "obj-87", + "obj-95", 0 ] } @@ -2932,11 +3254,11 @@ { "patchline": { "source": [ - "obj-87", + "obj-95", 1 ], "destination": [ - "obj-39", + "obj-42", 0 ] } @@ -2944,11 +3266,11 @@ { "patchline": { "source": [ - "obj-87", + "obj-95", 0 ], "destination": [ - "obj-88", + "obj-96", 0 ] } @@ -2956,11 +3278,11 @@ { "patchline": { "source": [ - "obj-88", + "obj-96", 0 ], "destination": [ - "obj-89", + "obj-97", 0 ] } @@ -2968,11 +3290,11 @@ { "patchline": { "source": [ - "obj-89", + "obj-97", 1 ], "destination": [ - "obj-42", + "obj-45", 0 ] } @@ -2980,11 +3302,11 @@ { "patchline": { "source": [ - "obj-89", + "obj-97", 0 ], "destination": [ - "obj-90", + "obj-98", 0 ] } @@ -2992,11 +3314,11 @@ { "patchline": { "source": [ - "obj-90", + "obj-98", 0 ], "destination": [ - "obj-91", + "obj-99", 0 ] } @@ -3004,11 +3326,95 @@ { "patchline": { "source": [ - "obj-91", + "obj-99", + 1 + ], + "destination": [ + "obj-46", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-99", + 0 + ], + "destination": [ + "obj-100", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-100", + 0 + ], + "destination": [ + "obj-101", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-101", + 1 + ], + "destination": [ + "obj-47", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-101", + 0 + ], + "destination": [ + "obj-102", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-102", + 0 + ], + "destination": [ + "obj-103", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-103", + 4 + ], + "destination": [ + "obj-48", + 0 + ] + } + }, + { + "patchline": { + "source": [ + "obj-103", 3 ], "destination": [ - "obj-43", + "obj-51", 0 ] } @@ -3016,11 +3422,11 @@ { "patchline": { "source": [ - "obj-91", + "obj-103", 2 ], "destination": [ - "obj-46", + "obj-54", 0 ] } @@ -3028,11 +3434,11 @@ { "patchline": { "source": [ - "obj-91", + "obj-103", 1 ], "destination": [ - "obj-51", + "obj-59", 0 ] } @@ -3040,11 +3446,11 @@ { "patchline": { "source": [ - "obj-91", + "obj-103", 0 ], "destination": [ - "obj-92", + "obj-104", 0 ] } @@ -3052,11 +3458,11 @@ { "patchline": { "source": [ - "obj-92", + "obj-104", 0 ], "destination": [ - "obj-93", + "obj-105", 0 ] } @@ -3064,11 +3470,11 @@ { "patchline": { "source": [ - "obj-93", + "obj-105", 1 ], "destination": [ - "obj-54", + "obj-62", 0 ] } diff --git a/runtime-tests/python/maxtest_stall.py b/runtime-tests/python/maxtest_stall.py index cff5aae..98e8f3c 100644 --- a/runtime-tests/python/maxtest_stall.py +++ b/runtime-tests/python/maxtest_stall.py @@ -5,9 +5,15 @@ class maxtest_stall: stall: float = 0.0 # seconds to sleep at the next sample, once + hang: int = 0 # while set, process() never returns (plan 8.3): the worker must interrupt it def process(self, x: float) -> float: if self.stall: seconds, self.stall = self.stall, 0.0 time.sleep(seconds) + while self.hang: + try: + pass + except Exception: # which cannot catch the worker's WorkerStopped + pass return x diff --git a/source/projects/tap.python_tilde/tap.python_tilde.h b/source/projects/tap.python_tilde/tap.python_tilde.h index a221107..d4528bc 100644 --- a/source/projects/tap.python_tilde/tap.python_tilde.h +++ b/source/projects/tap.python_tilde/tap.python_tilde.h @@ -243,7 +243,21 @@ class python : public object, public vector_operator<> { ~python() { m_file_watch.reset(); m_use_worker = false; - m_worker.reset(); // joins the worker thread, before the processor it runs goes + if (m_worker) { + m_worker->stop(); // joins the worker thread, before the processor it runs goes + if (m_worker->has_abandoned_thread()) { + // a process() that never returned still runs on the thread stop() gave up on, through + // this worker and this processor: leak both rather than free what it may yet touch + // (plan 8.3); the process exits with the thread + cerr << "a stalled process() is still running on an abandoned thread; this object's Python " + "state is kept alive for it until Max quits" + << endl; + m_worker.release(); + m_processor.release(); + return; + } + } + m_worker.reset(); m_processor.reset(); // releases the Python objects under the GIL } From dd389b923fc8e8eaca9e198149f750e6a1bdbf3a Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 16:25:25 +0000 Subject: [PATCH 07/11] 8.4: nothing a class prints is posted from the audio thread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Console lines are assembled per thread (thread_local buffers, so two threads printing pieces of a line never mix them) and dispatched whole: at once on the host's main thread, or on any thread when the host gives no predicate, and otherwise into detail::console_queue, Vyukov's bounded lock-free queue (256 slots of 480 bytes; a longer line is cut with an ellipsis; a full queue drops and counts), with the host's console_ready called. flush_console() drains it to the sink in order from the main thread and adds one line for what was dropped. runtime_options gained is_main_thread and console_ready; set_console_threading() sets them later. The Max object nominates systhread_ismainthread and one process-wide qelem that flushes, so a print() or a numpy RuntimeWarning in process() no longer takes a lock and posts from the audio thread (or the worker's). Pinned by test_console.cpp: a line from another thread reaches the sink only through the flush, a flush=True partial line queues whole, 80 lines printed in pieces by two threads at a 1 µs switch interval come out unmixed, a flood of 10,000 keeps 256 and counts 9,744 dropped, a 2,000-character line is cut. The battery's other console checks stand unchanged. Release, ASan/UBSan and TSan clean; ReadMe and CHANGELOG say what is deferred. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- CHANGELOG.md | 10 + ReadMe.md | 4 +- core/include/tap/python/runtime.h | 204 ++++++++++++++++-- core/tests/CMakeLists.txt | 1 + core/tests/test_console.cpp | 130 +++++++++++ docs/PRODUCTION-PLAN.md | 15 +- .../tap.python_tilde/tap.python_tilde.h | 20 +- 7 files changed, 359 insertions(+), 25 deletions(-) create mode 100644 core/tests/test_console.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 61ba343..35eaf51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,16 @@ breaking changes to the contract are allowed where they buy correctness (D5 in ## Unreleased +### Changed — nothing of yours posts from the audio thread + +- **What `process()` prints, or warns, is posted from Max's main thread.** A `print()` or a numpy + `RuntimeWarning` in `process()` used to post to the console from the audio thread (or the worker + thread), taking a lock on the way. Now a complete line printed on any thread but Max's main one + is queued, lock-free and without allocating, and posted from the main thread a moment later; + lines are assembled per thread, so two threads printing pieces of a line no longer mix them. A + flood is coalesced: 256 lines queue, the rest are dropped and counted in one line. Lines printed + on the main thread post at once, as before. (Plan 8.4, audit A5.) + ### Changed — worker mode never waits forever - **A `process()` that does not return when the worker stops is interrupted, then abandoned.** diff --git a/ReadMe.md b/ReadMe.md index b91aed3..184170a 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -14,7 +14,7 @@ Write Max objects in Python. - The class's **type-annotated attributes** become Max attributes (`@gain 0.5` in the object box, `gain 0.5`, `getgain`, attrui — it all works). - The class's **public methods** become Max messages, called according to their signatures, with arguments converted according to their type hints. - The source file is **watched and hot-reloaded** every time you save it, keeping attribute values, so you can live-code DSP with Max running. -- Python's `print()` output and tracebacks land in the **Max console** — and no exception your code raises, `sys.exit()` included, can take Max down. +- Python's `print()` output and tracebacks land in the **Max console**, posted from Max's main thread whatever thread printed — and no exception your code raises, `sys.exit()` included, can take Max down. ```python from attrs import define, field @@ -102,7 +102,7 @@ Python sources live in the package's `python` folder. `[tap.python~ name]` loads ### A note on performance -`process()` runs on the audio thread, holding Python's global interpreter lock. The per-sample form (`x: float`) makes one Python call per sample. The call itself is cheap — a fraction of a percent of a core, in the measurements below — but every line of Python in it runs 48,000 or 96,000 times a second, so the cost is your code's: `allpass.py`, a few lines of indexing and arithmetic, costs more than twenty times the call — and `numpy_allpass.py`, the same filter written a vector at a time, a small fraction of it. It is fantastic for sketching and live-coding an algorithm. The numpy form (`x: np.ndarray`) makes one call per signal vector, and numpy then works on the whole vector at C speed: it is the one to use when the algorithm has real work in it and has to keep up. Either way, in the default direct mode, the audio thread waits while other Python code holds the interpreter — a reload, a message, or another instance: all `tap.python~` instances share one interpreter, so heavy Python work in one can steal time from the others. [Worker mode](#writing-a-class) (`@mode worker`) takes Python off the audio thread, at the cost of a fixed delay. The object has Python hand the interpreter to a waiting thread after half a millisecond — a tenth of CPython's default — so a reload, which holds it for a few milliseconds, delays the audio a little at a time instead of all at once: measured with `core/bench`'s reload benchmark (96 kHz, 512-sample buffers, a save every 100 ms), no buffer was late at 0.5 ms, where CPython's default left some late in most runs. Errors in `process()` are printed from Max's main thread, never from the audio thread. +`process()` runs on the audio thread, holding Python's global interpreter lock. The per-sample form (`x: float`) makes one Python call per sample. The call itself is cheap — a fraction of a percent of a core, in the measurements below — but every line of Python in it runs 48,000 or 96,000 times a second, so the cost is your code's: `allpass.py`, a few lines of indexing and arithmetic, costs more than twenty times the call — and `numpy_allpass.py`, the same filter written a vector at a time, a small fraction of it. It is fantastic for sketching and live-coding an algorithm. The numpy form (`x: np.ndarray`) makes one call per signal vector, and numpy then works on the whole vector at C speed: it is the one to use when the algorithm has real work in it and has to keep up. Either way, in the default direct mode, the audio thread waits while other Python code holds the interpreter — a reload, a message, or another instance: all `tap.python~` instances share one interpreter, so heavy Python work in one can steal time from the others. [Worker mode](#writing-a-class) (`@mode worker`) takes Python off the audio thread, at the cost of a fixed delay. The object has Python hand the interpreter to a waiting thread after half a millisecond — a tenth of CPython's default — so a reload, which holds it for a few milliseconds, delays the audio a little at a time instead of all at once: measured with `core/bench`'s reload benchmark (96 kHz, 512-sample buffers, a save every 100 ms), no buffer was late at 0.5 ms, where CPython's default left some late in most runs. Errors in `process()` are printed from Max's main thread, never from the audio thread — and so is anything `process()` prints, or a warning it raises (numpy's `RuntimeWarning` for a division by zero, say): on any thread but Max's main one, a complete line is queued, without locking or allocating, and posted from the main thread a moment later, so it may appear after lines printed meanwhile from the main thread. A flood is coalesced: the queue holds 256 lines, and what does not fit is dropped and counted in one line. What still happens on the audio thread is Python's own work of formatting a warning, which reads the source file once. What `process()` costs, measured by `core/bench` — which calls it exactly as Max's audio thread does, one vector at a time — as the share of one CPU core it needs: diff --git a/core/include/tap/python/runtime.h b/core/include/tap/python/runtime.h index 7574534..aa1337a 100644 --- a/core/include/tap/python/runtime.h +++ b/core/include/tap/python/runtime.h @@ -13,6 +13,12 @@ #include // CPython must precede the standard headers // standard library +#include +#include +#include +#include +#include +#include #include #include #include @@ -26,8 +32,10 @@ namespace tap::python { /// Severity of a line of output, whether from Python's sys.stdout/sys.stderr or from the core. enum class log_level { info, error }; - /// Receives complete lines (without the trailing newline). May be called from any thread, - /// including the audio thread, so an implementation must be thread-safe. + /// Receives complete lines (without the trailing newline). The console's sink is called on the + /// thread that printed when the host gave no is_main_thread predicate (runtime_options), and + /// only on the host's main thread when it did (plan 8.4); a processor's or worker's log is + /// called on the main thread, except where their constructors say otherwise. using log_function = std::function; /// What initialize() needs from the host. @@ -40,6 +48,15 @@ namespace tap::python { /// thread waiting past an I/O buffer's deadline; 0.5 ms does not (plan 2.6: measured by /// core/bench's tap_python_reload_bench) and is under one 64-sample vector at 96 kHz. double switch_interval = 0.0005; + /// Whether the calling thread may post to the console at once (the host's main thread). On + /// any other thread a complete line is queued instead — lock-free, allocation-free, dropping + /// and counting when the queue is full — and console_ready is called; the host then calls + /// flush_console() from its main thread (plan 8.4). Without a predicate every thread posts + /// at once (the core battery's default). Both may be replaced by set_console_threading(). + std::function is_main_thread{}; + /// Called on the printing thread when a line was queued: must be real-time safe (in Max, + /// qelem_set); idempotent, as one call may stand for several lines. + std::function console_ready{}; }; /// The outcome of initialize(): ok, or the reason the interpreter could not start. @@ -135,13 +152,109 @@ namespace tap::python { // _maxconsole: a tiny built-in module that sys.stdout/sys.stderr are rebound to, so that // print() and tracebacks reach the host's console instead of disappearing. Output is - // buffered per stream and forwarded a line at a time. + // assembled into lines per thread and per stream (two threads writing half-lines never + // mix), and each complete line is posted at once on the host's main thread or queued from + // any other (plan 8.4). + + /// A bounded, lock-free, multi-producer single-consumer queue of console lines (Vyukov's + /// bounded queue): a producer never blocks or allocates, and a line that finds it full is + /// dropped and counted. Lines longer than a slot are cut, with an ellipsis. + class console_queue { + public: + static constexpr std::size_t k_slots = 256; // a power of two + static constexpr std::size_t k_line_bytes = 480; + + console_queue() { + for (std::size_t i = 0; i < k_slots; ++i) { + m_slots[i].sequence.store(i, std::memory_order_relaxed); + } + } + + /// Queue `text` as one line; false, and counted, if the queue is full. Any thread. + bool push(const log_level level, const std::string_view text) noexcept { + auto position = m_enqueue.load(std::memory_order_relaxed); + slot* cell = nullptr; + for (;;) { + cell = &m_slots[position & (k_slots - 1)]; + const auto sequence = cell->sequence.load(std::memory_order_acquire); + const auto gap = static_cast(sequence) - static_cast(position); + if (gap == 0) { + if (m_enqueue.compare_exchange_weak(position, position + 1, std::memory_order_relaxed)) { + break; + } + } + else if (gap < 0) { + m_dropped.fetch_add(1, std::memory_order_relaxed); + return false; + } + else { + position = m_enqueue.load(std::memory_order_relaxed); + } + } + cell->level = level; + if (text.size() <= k_line_bytes) { + cell->length = static_cast(text.size()); + std::memcpy(cell->text.data(), text.data(), text.size()); + } + else { + constexpr std::string_view k_ellipsis = "\u2026"; + const auto kept = k_line_bytes - k_ellipsis.size(); + std::memcpy(cell->text.data(), text.data(), kept); + std::memcpy(cell->text.data() + kept, k_ellipsis.data(), k_ellipsis.size()); + cell->length = static_cast(k_line_bytes); + } + cell->sequence.store(position + 1, std::memory_order_release); + return true; + } + + /// Take the oldest line, if any. One consumer thread. + bool pop(log_level& level, std::string& text) { + auto position = m_dequeue.load(std::memory_order_relaxed); + slot* cell = nullptr; + for (;;) { + cell = &m_slots[position & (k_slots - 1)]; + const auto sequence = cell->sequence.load(std::memory_order_acquire); + const auto gap = static_cast(sequence) - static_cast(position + 1); + if (gap == 0) { + if (m_dequeue.compare_exchange_weak(position, position + 1, std::memory_order_relaxed)) { + break; + } + } + else if (gap < 0) { + return false; + } + else { + position = m_dequeue.load(std::memory_order_relaxed); + } + } + level = cell->level; + text.assign(cell->text.data(), cell->length); + cell->sequence.store(position + k_slots, std::memory_order_release); + return true; + } + + /// Lines dropped since the last call. + std::uint64_t take_dropped() noexcept { return m_dropped.exchange(0, std::memory_order_relaxed); } + + private: + struct slot { + std::atomic sequence{}; + log_level level{}; + std::uint16_t length{}; + std::array text{}; + }; + std::array m_slots; + std::atomic m_enqueue{0}; + std::atomic m_dequeue{0}; + std::atomic m_dropped{0}; + }; struct console_state { - std::mutex mutex; - log_function sink; - std::string buffer_out; - std::string buffer_err; + std::mutex mutex; // guards sink, and the main thread's posting + log_function sink; + std::function is_main_thread; // set before any other thread prints + std::function ready; + console_queue queue; }; inline console_state& console() { @@ -149,6 +262,31 @@ namespace tap::python { return s_console; } + /// The calling thread's partial line for one stream (plan 8.4: per thread, so two threads + /// writing pieces never share a buffer). + inline std::string& console_buffer(const log_level level) { + thread_local std::string s_out; + thread_local std::string s_err; + return level == log_level::error ? s_err : s_out; + } + + /// Post one complete line: at once on the host's main thread (or on any thread, without a + /// predicate), queued for flush_console() from any other. + inline void console_dispatch(const log_level level, const std::string_view line) { + auto& state = console(); + if (!state.is_main_thread || state.is_main_thread()) { + std::lock_guard lock{state.mutex}; + if (state.sink) { + state.sink(level, line); + } + return; + } + state.queue.push(level, line); // a full queue drops and counts; never blocks + if (state.ready) { + state.ready(); + } + } + inline std::filesystem::path& scripts_directory() { static std::filesystem::path s_scripts_directory; return s_scripts_directory; @@ -361,28 +499,21 @@ def return_shape(hint): } inline void console_post(const std::string_view text, const log_level level) { - auto& state = console(); - std::lock_guard lock{state.mutex}; - auto& buffer = (level == log_level::error) ? state.buffer_err : state.buffer_out; + auto& buffer = console_buffer(level); buffer.append(text); size_t pos; while ((pos = buffer.find('\n')) != std::string::npos) { - if (state.sink) { - state.sink(level, std::string_view{buffer}.substr(0, pos)); - } + console_dispatch(level, std::string_view{buffer}.substr(0, pos)); buffer.erase(0, pos + 1); } } - /// Forward a pending partial line (text without its newline yet) as a line of its own. + /// Forward this thread's pending partial line (text without its newline yet) as a line of + /// its own. inline void console_flush(const log_level level) { - auto& state = console(); - std::lock_guard lock{state.mutex}; - auto& buffer = (level == log_level::error) ? state.buffer_err : state.buffer_out; + auto& buffer = console_buffer(level); if (!buffer.empty()) { - if (state.sink) { - state.sink(level, buffer); - } + console_dispatch(level, buffer); buffer.clear(); } } @@ -475,6 +606,38 @@ def return_shape(hint): state.sink = std::move(sink); } + /// Replace the console's main-thread predicate and the callback for queued lines (see + /// runtime_options). Main thread, before any other thread prints: the printing threads read + /// both without a lock. + inline void set_console_threading(std::function is_main_thread, std::function console_ready) { + auto& state = detail::console(); + state.is_main_thread = std::move(is_main_thread); + state.ready = std::move(console_ready); + } + + /// Post the lines other threads queued since the last flush, in order, to the sink — and, if + /// the queue overflowed meanwhile, one more saying how many were dropped. The host's main + /// thread, when console_ready has been called (idempotent: cheap when there is nothing). + inline void flush_console() { + auto& state = detail::console(); + log_level level{}; + std::string text; + while (state.queue.pop(level, text)) { + std::lock_guard lock{state.mutex}; + if (state.sink) { + state.sink(level, text); + } + } + if (const auto dropped = state.queue.take_dropped(); dropped != 0) { + std::lock_guard lock{state.mutex}; + if (state.sink) { + state.sink(log_level::error, "\u2026 and " + std::to_string(dropped) + + " console line(s) were dropped: printed faster than the main " + "thread could post them"); + } + } + } + /// Load `/.py` by path as the module `_tap_python_` (see /// detail::k_support_source). Returns a new reference to the module, or nullptr with a Python /// error set; `executed` tells whether the source was (re)executed or the cached module reused. @@ -522,6 +685,7 @@ def return_shape(hint): std::call_once(s_once, [&] { set_console(options.console); + set_console_threading(options.is_main_thread, options.console_ready); detail::scripts_directory() = options.scripts_dir; detail::init_thread() = std::this_thread::get_id(); PyImport_AppendInittab("_maxconsole", detail::console_module_init); diff --git a/core/tests/CMakeLists.txt b/core/tests/CMakeLists.txt index 380a962..0bfa08d 100644 --- a/core/tests/CMakeLists.txt +++ b/core/tests/CMakeLists.txt @@ -36,6 +36,7 @@ add_executable(tap_python_core_tests test_types.cpp test_channels.cpp test_worker.cpp + test_console.cpp ) target_link_libraries(tap_python_core_tests PRIVATE tap::python Python3::Python Catch2::Catch2WithMain) diff --git a/core/tests/test_console.cpp b/core/tests/test_console.cpp new file mode 100644 index 0000000..139700e --- /dev/null +++ b/core/tests/test_console.cpp @@ -0,0 +1,130 @@ +/// @file test_console.cpp +/// @brief Plan 8.4: nothing a user prints is posted from a thread the host did not nominate. +// SPDX-License-Identifier: MIT +// Copyright 2022-2026 Timothy Place. + +#include +#include +#include +#include +#include + +#include "support.h" + +using namespace tap::python; +using namespace tap::python::test; + +namespace { + + /// Nominate the calling thread as the host's main thread for the console, counting the calls + /// for queued lines; put back the battery's default (every thread posts at once) when it goes. + struct nominated_main_thread { + std::atomic ready{0}; + + nominated_main_thread() { + const auto main = std::this_thread::get_id(); + set_console_threading([main] { return std::this_thread::get_id() == main; }, [this] { ++ready; }); + } + ~nominated_main_thread() { set_console_threading({}, {}); } + }; + + /// Run Python source on a thread of its own, to completion. + void run_elsewhere(const std::string& source) { + std::thread other{[&] { REQUIRE(run(source)); }}; + other.join(); + } + +} // namespace + +SCENARIO("A line printed off the main thread is posted by the host's flush, not by the printing thread (plan 8.4)") { + ensure_runtime(); + nominated_main_thread host; + console().clear(); + + run_elsewhere("print('from another thread')"); + THEN("it has not reached the sink, and the host was told to flush") { + CHECK_FALSE(console().contains("from another thread", log_level::info)); + CHECK(host.ready.load() >= 1); + } + THEN("flush_console() posts it") { + flush_console(); + CHECK(console().contains("from another thread", log_level::info)); + } + THEN("a line printed on the main thread still posts at once") { + REQUIRE(run("print('from the main thread')")); + CHECK(console().contains("from the main thread", log_level::info)); + } + THEN("a partial line flushed by Python off the main thread is queued whole") { + run_elsewhere("import sys\nprint('no newline yet', end='', flush=True)"); + CHECK_FALSE(console().contains("no newline yet", log_level::info)); + flush_console(); + CHECK(console().contains("no newline yet", log_level::info)); + } +} + +SCENARIO("Two threads printing a line in pieces never produce a mixed line (plan 8.4)") { + ensure_runtime(); + nominated_main_thread host; + console().clear(); + REQUIRE(run("import sys\n_switch_interval = sys.getswitchinterval()\nsys.setswitchinterval(1e-6)")); + + const auto pieces = [](const char letter) { + return std::string{"for _ in range(40):\n for _ in range(60):\n print('"} + letter + + "', end='')\n print()\n"; + }; + std::thread a{[&] { REQUIRE(run(pieces('a'))); }}; + std::thread b{[&] { REQUIRE(run(pieces('b'))); }}; + a.join(); + b.join(); + REQUIRE(run("import sys\nsys.setswitchinterval(_switch_interval)")); + flush_console(); + + std::size_t lines = 0; + for (const auto& line : console().lines()) { + if (line.text.find_first_of("ab") == std::string::npos) { + continue; + } + ++lines; + const auto first = line.text[0]; + CHECK(line.text == std::string(60, first)); // all one letter, the whole line + } + CHECK(lines == 80); +} + +SCENARIO("A flood printed off the main thread never blocks it, and the overflow is counted (plan 8.4)") { + ensure_runtime(); + nominated_main_thread host; + console().clear(); + + const auto before = std::chrono::steady_clock::now(); + run_elsewhere("for i in range(10000):\n print('flood', i)"); + const auto took = std::chrono::steady_clock::now() - before; + CHECK(took < std::chrono::seconds{5}); // bounded by Python, not by the console + + flush_console(); + std::size_t flood = 0; + bool dropped_line = false; + for (const auto& line : console().lines()) { + if (line.text.rfind("flood ", 0) == 0) { + ++flood; + } + else if (line.text.find("console line(s) were dropped") != std::string::npos) { + dropped_line = true; + CHECK(line.text.find(std::to_string(10000 - flood)) != std::string::npos); + } + } + CHECK(flood == detail::console_queue::k_slots); // what the queue holds; the rest dropped + CHECK(dropped_line); +} + +SCENARIO("A line longer than a queue slot is cut with an ellipsis, not lost (plan 8.4)") { + ensure_runtime(); + nominated_main_thread host; + console().clear(); + run_elsewhere("print('x' * 2000)"); + flush_console(); + const auto lines = console().lines(); + REQUIRE(lines.size() == 1); + CHECK(lines[0].text.size() == detail::console_queue::k_line_bytes); + CHECK(lines[0].text.ends_with("\u2026")); +} diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index de36f04..ed7cdba 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -660,7 +660,7 @@ down rather than discovered again. *This plan was itself audited before being ad passes on the worker. Release, ASan/UBSan and TSan clean. The `worker` runtime test gains `hang 1` → `mode direct` → `hang 0`, expecting one "interrupted" line and two errors in all — step timing to confirm in the Mac session (8.8), with the stack measurement the item asks for. -- [ ] **8.4 Nothing of the user's prints on the audio thread either (A5).** `runtime_options` +- [x] **8.4 Nothing of the user's prints on the audio thread either (A5).** `runtime_options` gains `is_main_thread` (the host's predicate — `systhread_ismainthread` in Max; the default, with no host predicate, is "always", which keeps the core battery's synchronous `console()` checks as they are) and `console_ready` (real-time safe; the object sets its `m_reports` qelem). @@ -676,7 +676,18 @@ down rather than discovered again. *This plan was itself audited before being ad (`linecache` reads the file once). Tests: `print()` from an audio thread reaches the sink only on the nominated thread; two threads printing half-lines never produce a mixed line; a flood of 10,000 lines drops with a count and never blocks the producer (timed). CHANGELOG: console output - from `process()` is deferred and coalesced. + from `process()` is deferred and coalesced. *Done:* `runtime_options` gained `is_main_thread` and + `console_ready` (and `set_console_threading()` for a test or a host that decides later); + `detail::console_queue` is Vyukov's bounded queue, 256 slots of 480 bytes (a longer line is cut + with an ellipsis), with a dropped count; lines are assembled in `thread_local` buffers and + dispatched whole; `flush_console()` drains to the sink in order and adds the dropped line. The + Max object nominates `systhread_ismainthread` and sets one process-wide qelem that calls + `flush_console()`. Four scenarios in the new `test_console.cpp`: a line from another thread + reaches the sink only through `flush_console()` (and a `flush=True` partial line queues whole); + two threads printing pieces give 80 unmixed lines at a 1 µs switch interval; a flood of 10,000 + keeps 256 and counts 9,744 dropped, in bounded time; a 2,000-character line is cut. The + battery's other console checks run on the test's main thread with no predicate and stand as + they were. - [ ] **8.5 Helper modules follow the class file (A7b).** When a load *executes* the class file (the loader says so), it first drops from `sys.modules` every module whose `__file__` is a `.py` under `scripts_directory()` — *revised:* **except the class modules themselves** diff --git a/source/projects/tap.python_tilde/tap.python_tilde.h b/source/projects/tap.python_tilde/tap.python_tilde.h index d4528bc..a6f2167 100644 --- a/source/projects/tap.python_tilde/tap.python_tilde.h +++ b/source/projects/tap.python_tilde/tap.python_tilde.h @@ -186,7 +186,13 @@ class python : public object, public vector_operator<> { return; } - const auto status = runtime::initialize({home, m_scripts_dir, console_line}); + runtime::runtime_options options; + options.home = home; + options.scripts_dir = m_scripts_dir; + options.console = console_line; + options.is_main_thread = is_main_thread; // Python's output posts from Max's main thread only (8.4) + options.console_ready = console_ready; + const auto status = runtime::initialize(options); if (!status.ok) { cerr << "failed to start Python from '" << home.string() << "': " << status.error << " (run scripts/install-runtime to install the runtime)" << endl; @@ -503,6 +509,18 @@ class python : public object, public vector_operator<> { #endif } + /// Whether this is Max's main thread: where Python's output may be posted at once. On any other + /// thread — the audio thread, a worker, the scheduler — the core queues the line (plan 8.4). + static bool is_main_thread() { return c74::max::systhread_ismainthread() != 0; } + + /// Called on the printing thread when the core queued a line: sets the process-wide qelem that + /// has the lines posted from the main thread. Real-time safe (qelem_set). + static void console_ready() { + static c74::max::t_qelem* s_flush = + c74::max::qelem_new(nullptr, reinterpret_cast(+[](void*) { runtime::flush_console(); })); + c74::max::qelem_set(s_flush); + } + /// Python's print() output and tracebacks, for every instance. static void console_line(const runtime::log_level level, const std::string_view text) { const std::string line{text}; From 0375f90a7c0493ddf656066a8f66fa5b4358625c Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 16:29:25 +0000 Subject: [PATCH 08/11] 8.5: helper modules follow the class file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before the loader executes a changed class file it calls forget_helpers(): every source module imported from the scripts folder — by normcase(abspath(__file__)) against the folder initialize() hands the support module, so no file-system call per module — is dropped from sys.modules, except the class modules themselves (_tap_python_*, which the type hints and the loader's cache depend on) and anything that is not a .py (a compiled extension cannot be imported twice). The fresh class then imports the helper's current source. PyConfig's write_bytecode is off, so no .pyc is written for the folder and the stale-bytecode hazard 3.6a closed for class files cannot return for helpers. A helper saved on its own is still not watched; the ReadMe says to save the class file too, with a change. Pinned in test_loading.cpp: the helper's new value after a changed save; another class file's module left in sys.modules and working; the old value after an unchanged save (the documented limit); no __pycache__. Release, ASan/UBSan and TSan clean. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- CHANGELOG.md | 10 +++++++ ReadMe.md | 2 +- core/include/tap/python/runtime.h | 38 ++++++++++++++++++++++- core/tests/test_loading.cpp | 50 +++++++++++++++++++++++++++++++ docs/PRODUCTION-PLAN.md | 11 +++++-- 5 files changed, 107 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35eaf51..5439498 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,16 @@ breaking changes to the contract are allowed where they buy correctness (D5 in ## Unreleased +### Changed — helper modules follow the class file + +- **A helper module in `python/` is imported afresh when a class file that changed is saved.** + Before each execution of a class file the loader drops from `sys.modules` every source module + imported from the folder (not the class modules themselves, nor compiled extensions), so the + fresh class imports the helper's current source; and the interpreter writes no bytecode, so the + stale-`.pyc` hazard fixed for class files in 0.9.0 cannot return for helpers. A save of the + helper alone is still not watched. Helpers used to load once per Max session. (Plan 8.5, audit + A7.) + ### Changed — nothing of yours posts from the audio thread - **What `process()` prints, or warns, is posted from Max's main thread.** A `print()` or a numpy diff --git a/ReadMe.md b/ReadMe.md index 184170a..3ce3372 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -86,7 +86,7 @@ Without a runtime the object still loads, and says in the Max console what is mi ## Writing a class -Python sources live in the package's `python` folder. `[tap.python~ name]` loads `python/name.py` and instantiates the class `name` defined in it (the file, class, and argument must share the same name, which must be a valid Python identifier; with no argument, `default` is loaded). The file is loaded by its path, never through `import`, so a file named like a standard-library module (`random.py`, `json.py`) works and does not shadow that module for anyone else. Other files in the `python` folder can be imported by your class as helper modules (the folder is on `sys.path`, after the standard library) — they are imported once per Max session, as any module is, so a save of a helper is not picked up until Max restarts; what you are live-coding belongs in the class file. Text is UTF-8: the interpreter runs in Python's UTF-8 mode, so `open()` reads and writes UTF-8 unless you pass another `encoding`, whatever the machine's locale. +Python sources live in the package's `python` folder. `[tap.python~ name]` loads `python/name.py` and instantiates the class `name` defined in it (the file, class, and argument must share the same name, which must be a valid Python identifier; with no argument, `default` is loaded). The file is loaded by its path, never through `import`, so a file named like a standard-library module (`random.py`, `json.py`) works and does not shadow that module for anyone else. Other files in the `python` folder can be imported by your class as helper modules (the folder is on `sys.path`, after the standard library) — a helper is imported afresh whenever a class file that changed is saved (the loader forgets every module imported from the folder first), but a save of the helper alone is not watched: save the class file too, with a change, to pick it up. No bytecode is written for anything in the folder. Text is UTF-8: the interpreter runs in Python's UTF-8 mode, so `open()` reads and writes UTF-8 unless you pass another `encoding`, whatever the machine's locale. - **Attributes** — class-level annotated fields (e.g. via `attrs`) become Max attributes. `int` maps to a Max `long`, `float` to `float64`, `bool` to an on/off `long` (your field receives a real `bool`), anything else to a symbol; `Optional[X]` and `X | None` count as `X`. Names starting with `_` are private, and `ClassVar`s are not fields. If a hint cannot be resolved (say a name imported only under `TYPE_CHECKING`), the console says so and the annotation is read as written. - **Messages** — public methods (including classmethods and staticmethods) become Max messages, called according to their signature: parameters with defaults are optional, `*args` takes any number of arguments, and a keyword-only parameter must have a default (a method with a required keyword-only parameter is not exposed). Argument hints (`int`, `float`, `bool`, `str`) drive the conversion from Max atoms; an unannotated parameter receives the atom as it is (an `int`, `float` or `str`). Methods named `int`, `float`, `symbol`, and `bang` map to those standard Max messages. Names Max or the object handle themselves (`filechanged`, `dsp64`, `notify`, `assist`, `loadbang`, `dblclick`, `anything`, …, and the object's own attributes `mode`, `latency` and `latencysamples`) are not exposed — as methods, or as fields; the console names any skipped this way so you can rename it. diff --git a/core/include/tap/python/runtime.h b/core/include/tap/python/runtime.h index aa1337a..cb02211 100644 --- a/core/include/tap/python/runtime.h +++ b/core/include/tap/python/runtime.h @@ -317,6 +317,15 @@ namespace tap::python { // so that a save that breaks a file shared by many objects, or a patch opening many of them, // reports it once, while an object created later still says why it is silent (plan 6.10). // + // Before a class file is executed, every helper module imported from the scripts folder — a + // source module whose __file__ is under _scripts_dir, other than the class modules + // themselves — is dropped from sys.modules, so that the fresh execution imports the helpers + // afresh (plan 8.5). A helper saved on its own is not watched: it is picked up by the next + // save that changes a class file importing it. Compiled extension modules are left alone + // (they cannot be imported twice), and no bytecode is written for anything (initialize() + // sets write_bytecode off), so the stale-.pyc hazard 3.6a closed for class files cannot + // return for helpers. + // // hint_kind(hint) -> str. The name the host maps a type hint by: 'int', 'float', 'bool', 'str', // 'ndarray', 'tuple', ... ('any' for no hint, 'ClassVar' for a class variable). Optional[X] and // X | None are X; an unresolved hint written as a string is read by name. @@ -340,7 +349,7 @@ namespace tap::python { // `except Exception:` cannot swallow it and keep looping; the processor recognizes it and // keeps the class's audio bound — it is the host's interruption, not the class's fault. inline constexpr const char* k_support_source = R"( -import inspect, re, sys, time, types, typing +import inspect, os, re, sys, time, types, typing class WorkerStopped(BaseException): """tap.python~ stopped the worker thread while process() had not returned.""" @@ -348,6 +357,25 @@ class WorkerStopped(BaseException): _cache = {} _failed = {} _REPORT_WINDOW = 2.0 +_scripts_dir = None # set by initialize(): the folder the helpers live in + +def forget_helpers(): + """Drop from sys.modules every source module imported from the scripts folder, other than the + class modules (_tap_python_*), so the next import runs its current source (plan 8.5).""" + if not _scripts_dir: + return [] + root = os.path.normcase(os.path.abspath(_scripts_dir)) + os.sep + forgotten = [] + for name, module in list(sys.modules.items()): + if name.startswith('_tap_python_'): + continue + file = getattr(module, '__file__', None) + if not isinstance(file, str) or not file.endswith('.py'): + continue + if os.path.normcase(os.path.abspath(file)).startswith(root): + del sys.modules[name] + forgotten.append(name) + return forgotten def load(module_name, path): with open(path, 'rb') as f: @@ -357,6 +385,7 @@ def load(module_name, path): return cached[1], False try: code = compile(source, path, 'exec', dont_inherit=True) + forget_helpers() module = types.ModuleType(module_name) module.__file__ = path previous = sys.modules.get(module_name) @@ -705,6 +734,7 @@ def return_shape(hint): PyConfig_InitIsolatedConfig(&config); // ignore environment variables and user site-packages config.install_signal_handlers = 0; // we are a plugin: never steal signal handling from the host config.parse_argv = 0; + config.write_bytecode = 0; // no .pyc for the user's helpers (plan 8.5); the runtime has its own #ifdef _WIN32 PyStatus status = PyConfig_SetString(&config, &config.home, options.home.wstring().c_str()); @@ -734,6 +764,12 @@ def return_shape(hint): } detail::create_support(); + if (detail::support_globals()) { // where the helpers live, for forget_helpers() + if (PyObject* dir = detail::path_to_unicode(options.scripts_dir)) { + PyDict_SetItemString(detail::support_globals(), "_scripts_dir", dir); + Py_DECREF(dir); + } + } // 2.6: hand the GIL to a waiting thread — the audio thread, above all — sooner if (PyObject* set_interval = PySys_GetObject("setswitchinterval")) { // borrowed diff --git a/core/tests/test_loading.cpp b/core/tests/test_loading.cpp index 7146ee3..5cbde3f 100644 --- a/core/tests/test_loading.cpp +++ b/core/tests/test_loading.cpp @@ -121,6 +121,56 @@ SCENARIO("A reload starts from a fresh module: names deleted from the file are g CHECK(all_equal(render(p, 0.0), 0.0)); // importlib.reload would have kept OFFSET = 1.0 } +// 8.5 — helper modules follow the class file + +SCENARIO("A helper module is imported afresh when a class file that changed is executed (plan 8.5)") { + ensure_runtime(); + write_script("helper_values", "VALUE = 1.0\n"); + write_script("uses_helper", "import helper_values\n" + "class uses_helper:\n" + " def process(self, x: float) -> float:\n" + " return helper_values.VALUE\n"); + processor p{"uses_helper"}; + REQUIRE(p.load()); + CHECK(all_equal(render(p, 0.0), 1.0)); + + WHEN("the helper is edited and the class file is saved with a change") { + write_script("helper_values", "VALUE = 2.0\n"); + write_script("uses_helper", "import helper_values\n" + "class uses_helper:\n" + " def process(self, x: float) -> float:\n" + " return helper_values.VALUE # changed\n"); + REQUIRE(p.load()); + THEN("the class runs the helper's new code") { + CHECK(all_equal(render(p, 0.0), 2.0)); + } + THEN("another class file's module is untouched") { + processor other{"gain"}; + REQUIRE(other.load()); + write_script("helper_values", "VALUE = 3.0\n"); + write_script("uses_helper", "import helper_values\n" + "class uses_helper:\n" + " def process(self, x: float) -> float:\n" + " return helper_values.VALUE # changed again\n"); + REQUIRE(p.load()); + CHECK(all_equal(render(p, 0.0), 3.0)); + CHECK(run("import sys\nassert '_tap_python_gain' in sys.modules, 'the class module was dropped'")); + CHECK(all_equal(render(other, 0.5), 0.5)); + } + } + WHEN("the helper is edited but the class file is saved unchanged") { + write_script("helper_values", "VALUE = 4.0\n"); + REQUIRE(p.load()); + THEN("the class is not re-executed, so the old helper stays: the documented limit") { + CHECK(all_equal(render(p, 0.0), 1.0)); + } + } + THEN("no bytecode was written for the helper") { + CHECK_FALSE(std::filesystem::exists(scripts_dir() / "__pycache__")); + CHECK(run("import sys\nassert sys.dont_write_bytecode, 'bytecode writing is on'")); + } +} + // 3.4 — attribute values survive a reload SCENARIO("Attribute values carry over a reload") { diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index ed7cdba..f92c9cc 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -688,7 +688,7 @@ down rather than discovered again. *This plan was itself audited before being ad keeps 256 and counts 9,744 dropped, in bounded time; a 2,000-character line is cut. The battery's other console checks run on the test's main thread with no predicate and stand as they were. -- [ ] **8.5 Helper modules follow the class file (A7b).** When a load *executes* the class file +- [x] **8.5 Helper modules follow the class file (A7b).** When a load *executes* the class file (the loader says so), it first drops from `sys.modules` every module whose `__file__` is a `.py` under `scripts_directory()` — *revised:* **except the class modules themselves** (`_tap_python_*`, which `typing.get_type_hints` and the loader's cache depend on) and anything @@ -704,7 +704,14 @@ down rather than discovered again. *This plan was itself audited before being ad should force re-execution — the watcher would then send a distinct, undocumented message — or whether the folder should be watched. Test: a class importing `helper.py`, the helper edited, the class file changed and loaded → the new helper runs; edited alone → it does not, and the - ReadMe says so. CHANGELOG: helpers reload with the class file. + ReadMe says so. CHANGELOG: helpers reload with the class file. *Done:* `forget_helpers()` in the support + module, called by `load()` before it executes a changed file, compares + `normcase(abspath(__file__))` against the scripts folder `initialize()` hands it (no file-system + calls per module), skips `_tap_python_*` and anything not a `.py`; `PyConfig.write_bytecode = 0`. + One scenario in `test_loading.cpp`: the helper's new value after a changed save, another class + file's module left in `sys.modules` and working, the old value after an unchanged save (the + documented limit), no `__pycache__`. *Decided:* the rule stands as documented — "save the class + file too" — rather than a forced reload on `filechanged` or a folder watch; revisit if users ask. - [ ] **8.6 Type hints that map as a reader expects (A8).** In `hint_kind`: unwrap `Annotated` and `Final` to their first argument; *revised:* a `Union` of several members other than `None` maps to **`any`** — the atom passes as it is, as for an unannotated parameter — rather than to From dbe50303b54bb2d06edf4625288a6e696cb4aaf7 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 16:34:24 +0000 Subject: [PATCH 09/11] 8.6: type hints map as a reader expects In the support module's hint_kind: Annotated[X, ...] and Final[X] unwrap to X, as objects and, by regex, as strings read as written; a union of several kinds maps to 'any', so the value passes as the atom carried it rather than one member being preferred and the others lost (a float | int field was a symbol attribute that stored "" for every number); a type that subclasses bool, int, float or str maps to its base, and numpy's scalar types map by their abstract bases' names in the MRO, without numpy being imported; return_shape sees through Optional and | None, so -> tuple[float, float] | None names two outputs instead of binding one and reporting every sample as not a number. Pinned in test_types.cpp with typed_more.py, typed_strings.py (hints that cannot resolve, read as written, for the string paths) and typed_numpy.py. Release, ASan/UBSan and TSan clean; ReadMe and CHANGELOG record the mapping. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- CHANGELOG.md | 10 ++++ ReadMe.md | 2 +- core/include/tap/python/runtime.h | 78 ++++++++++++++++++++++++++++-- core/tests/python/typed_more.py | 13 +++++ core/tests/python/typed_numpy.py | 11 +++++ core/tests/python/typed_strings.py | 17 +++++++ core/tests/test_types.cpp | 74 ++++++++++++++++++++++++++++ docs/PRODUCTION-PLAN.md | 9 +++- 8 files changed, 207 insertions(+), 7 deletions(-) create mode 100644 core/tests/python/typed_more.py create mode 100644 core/tests/python/typed_numpy.py create mode 100644 core/tests/python/typed_strings.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 5439498..8af050b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,16 @@ breaking changes to the contract are allowed where they buy correctness (D5 in ## Unreleased +### Changed — type hints map as a reader expects + +- **`Annotated[X, …]` and `Final[X]` fields are `X`; a union of several kinds passes the atom as + it is; numpy scalar types map to the kinds they hold; `Optional[tuple[…]]` names its outputs.** + A `float | int` field used to become a symbol attribute that stored `""` for any number; now it + takes the value as the atom carried it (an `int`, `float` or `str`), as an unannotated parameter + does. `Annotated` and `Final` used to be symbols; `np.float64` and friends too; and + `-> tuple[float, float] | None` bound one output and then reported every sample as not a number. + The same holds for hints read as written. (Plan 8.6, audit A8.) + ### Changed — helper modules follow the class file - **A helper module in `python/` is imported afresh when a class file that changed is saved.** diff --git a/ReadMe.md b/ReadMe.md index 3ce3372..2488ab3 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -88,7 +88,7 @@ Without a runtime the object still loads, and says in the Max console what is mi Python sources live in the package's `python` folder. `[tap.python~ name]` loads `python/name.py` and instantiates the class `name` defined in it (the file, class, and argument must share the same name, which must be a valid Python identifier; with no argument, `default` is loaded). The file is loaded by its path, never through `import`, so a file named like a standard-library module (`random.py`, `json.py`) works and does not shadow that module for anyone else. Other files in the `python` folder can be imported by your class as helper modules (the folder is on `sys.path`, after the standard library) — a helper is imported afresh whenever a class file that changed is saved (the loader forgets every module imported from the folder first), but a save of the helper alone is not watched: save the class file too, with a change, to pick it up. No bytecode is written for anything in the folder. Text is UTF-8: the interpreter runs in Python's UTF-8 mode, so `open()` reads and writes UTF-8 unless you pass another `encoding`, whatever the machine's locale. -- **Attributes** — class-level annotated fields (e.g. via `attrs`) become Max attributes. `int` maps to a Max `long`, `float` to `float64`, `bool` to an on/off `long` (your field receives a real `bool`), anything else to a symbol; `Optional[X]` and `X | None` count as `X`. Names starting with `_` are private, and `ClassVar`s are not fields. If a hint cannot be resolved (say a name imported only under `TYPE_CHECKING`), the console says so and the annotation is read as written. +- **Attributes** — class-level annotated fields (e.g. via `attrs`) become Max attributes. `int` maps to a Max `long`, `float` to `float64`, `bool` to an on/off `long` (your field receives a real `bool`), anything else to a symbol; `Optional[X]`, `X | None`, `Annotated[X, …]` and `Final[X]` count as `X`, a subclass of one of those types (`np.float64`, an `IntEnum`) as its base, and numpy's scalar types (`np.int64`, `np.bool_`) as the kind they hold. A union of several kinds (`float | int`, `str | float`) is a symbol attribute in Max whose value reaches your field as the atom carried it — an `int`, `float` or `str` — like an unannotated parameter. Names starting with `_` are private, and `ClassVar`s are not fields. If a hint cannot be resolved (say a name imported only under `TYPE_CHECKING`), the console says so and the annotation is read as written. - **Messages** — public methods (including classmethods and staticmethods) become Max messages, called according to their signature: parameters with defaults are optional, `*args` takes any number of arguments, and a keyword-only parameter must have a default (a method with a required keyword-only parameter is not exposed). Argument hints (`int`, `float`, `bool`, `str`) drive the conversion from Max atoms; an unannotated parameter receives the atom as it is (an `int`, `float` or `str`). Methods named `int`, `float`, `symbol`, and `bang` map to those standard Max messages. Names Max or the object handle themselves (`filechanged`, `dsp64`, `notify`, `assist`, `loadbang`, `dblclick`, `anything`, …, and the object's own attributes `mode`, `latency` and `latencysamples`) are not exposed — as methods, or as fields; the console names any skipped this way so you can rename it. - **Audio** — a method `process()` runs on the signal, in one of two forms chosen by its type hint: - `process(self, x: np.ndarray) -> np.ndarray` is called **once per signal vector** with a numpy array of the input and must return an array of the same length (any numeric dtype, or a list; it is converted). This is the form to use for anything that must run in real time — see `python/numpy_gain.py`, and `python/numpy_allpass.py` for a filter with feedback. The input array is reused from one call to the next: copy it if you want to keep it. diff --git a/core/include/tap/python/runtime.h b/core/include/tap/python/runtime.h index cb02211..b7907ea 100644 --- a/core/include/tap/python/runtime.h +++ b/core/include/tap/python/runtime.h @@ -327,8 +327,12 @@ namespace tap::python { // return for helpers. // // hint_kind(hint) -> str. The name the host maps a type hint by: 'int', 'float', 'bool', 'str', - // 'ndarray', 'tuple', ... ('any' for no hint, 'ClassVar' for a class variable). Optional[X] and - // X | None are X; an unresolved hint written as a string is read by name. + // 'ndarray', 'tuple', ... ('any' for no hint, 'ClassVar' for a class variable). Optional[X], + // X | None, Annotated[X, ...] and Final[X] are X; a union of several kinds is 'any' (the value + // passes as the atom carried it: preferring one member would lose the others, plan 8.6); a + // subclass of float, int, bool or str is its base, and numpy's scalar types map by their + // abstract bases (floating, integer, bool) without numpy being imported here; an unresolved + // hint written as a string is read by name, with the same unwrapping. // // class_hints(cls) -> ([(name, kind)], error). The class's annotated fields, base classes // first. If typing.get_type_hints fails (e.g. a name imported only under TYPE_CHECKING), falls @@ -414,6 +418,47 @@ def load(module_name, path): return module, True _OPTIONAL = re.compile(r'(?:typing\.)?Optional\[(.*)\]') +_WRAPPED = re.compile(r'(?:typing\.)?(?:Annotated|Final)\[(.*)\]') +_NUMPY_KINDS = {'floating': 'float', 'integer': 'int', 'bool': 'bool', 'bool_': 'bool', 'str_': 'str'} + +def _first_argument(text): + """The first of a hint's comma-separated arguments, as written: 'float' of 'float, "meta"'.""" + depth = 0 + for i, c in enumerate(text): + if c in '[(': + depth += 1 + elif c in '])': + depth -= 1 + elif c == ',' and depth == 0: + return text[:i].strip() + return text.strip() + +def _split_union(text): + """The members of 'a | b | None' other than None, as written.""" + parts, depth, start = [], 0, 0 + for i, c in enumerate(text): + if c in '[(': + depth += 1 + elif c in '])': + depth -= 1 + elif c == '|' and depth == 0: + parts.append(text[start:i]) + start = i + 1 + parts.append(text[start:]) + return [p.strip() for p in parts if p.strip() not in ('None', '')] + +def _unwrap(hint): + """Annotated[X, ...] and Final[X] are X, however deep.""" + while True: + origin = typing.get_origin(hint) + if origin is typing.Annotated or origin is typing.Final: + hint = typing.get_args(hint)[0] + else: + return hint + +def _one_kind(kinds): + kinds = set(kinds) + return kinds.pop() if len(kinds) == 1 else 'any' def hint_kind(hint): if hint is inspect.Parameter.empty: @@ -423,18 +468,32 @@ def hint_kind(hint): match = _OPTIONAL.fullmatch(text) if match: return hint_kind(match.group(1)) - parts = [p.strip() for p in text.split('|') if p.strip() != 'None'] + match = _WRAPPED.fullmatch(text) + if match: + return hint_kind(_first_argument(match.group(1))) + parts = _split_union(text) if len(parts) == 1 and parts[0] != text: return hint_kind(parts[0]) + if len(parts) > 1: + return _one_kind(hint_kind(p) for p in parts) return text.split('[')[0].split('.')[-1] + hint = _unwrap(hint) origin = typing.get_origin(hint) if origin is typing.ClassVar: return 'ClassVar' if origin is typing.Union or origin is types.UnionType: args = [a for a in typing.get_args(hint) if a is not type(None)] - return hint_kind(args[0]) if len(args) == 1 else 'str' + return hint_kind(args[0]) if len(args) == 1 else _one_kind(hint_kind(a) for a in args) if origin is not None: return getattr(origin, '__name__', 'str') + if isinstance(hint, type): + for base, kind in ((bool, 'bool'), (int, 'int'), (float, 'float'), (str, 'str')): + if issubclass(hint, base): + return kind + for cls in hint.__mro__: # numpy's scalars, by their abstract bases, without importing numpy + kind = _NUMPY_KINDS.get(cls.__name__) + if kind is not None and cls.__module__.startswith('numpy'): + return kind return getattr(hint, '__name__', 'str') def class_hints(cls): @@ -488,6 +547,12 @@ def return_shape(hint): return 1, 'any' if isinstance(hint, str): text = hint.strip() + match = _OPTIONAL.fullmatch(text) + if match: + return return_shape(match.group(1)) + parts = _split_union(text) + if len(parts) == 1 and parts[0] != text: + return return_shape(parts[0]) for prefix in ('typing.Tuple[', 'Tuple[', 'tuple['): if text.startswith(prefix) and text.endswith(']'): parts = [p.strip() for p in text[len(prefix):-1].split(',')] @@ -497,6 +562,11 @@ def return_shape(hint): if text in ('tuple', 'Tuple', 'typing.Tuple'): return -1, 'any' return 1, hint_kind(hint) + hint = _unwrap(hint) + if typing.get_origin(hint) is typing.Union or typing.get_origin(hint) is types.UnionType: + args = [a for a in typing.get_args(hint) if a is not type(None)] + if len(args) == 1: + return return_shape(args[0]) # Optional[tuple[...]] is the tuple if hint is tuple or typing.get_origin(hint) is tuple: args = typing.get_args(hint) if not args or Ellipsis in args or args == ((),): diff --git a/core/tests/python/typed_more.py b/core/tests/python/typed_more.py new file mode 100644 index 0000000..a9639f9 --- /dev/null +++ b/core/tests/python/typed_more.py @@ -0,0 +1,13 @@ +# Test fixture: hint shapes a reader expects to map as their payload does (plan 8.6, audit A8). +from typing import Annotated, Final, Optional + + +class typed_more: + noted: Annotated[float, "metadata"] = 0.5 # float + fixed: Final[int] = 3 # int + either: float | int = 1.0 # several kinds: the atom as it comes + text_or_number: str | float = "a" # likewise + maybe_noted: Optional[Annotated[float, "m"]] = 2.0 # float + + def process(self, x: float) -> Optional[tuple[float, float]]: # two outputs + return x, -x diff --git a/core/tests/python/typed_numpy.py b/core/tests/python/typed_numpy.py new file mode 100644 index 0000000..01230d6 --- /dev/null +++ b/core/tests/python/typed_numpy.py @@ -0,0 +1,11 @@ +# Test fixture: numpy scalar types as field hints map to the kinds they hold (plan 8.6). +import numpy as np + + +class typed_numpy: + level: np.float64 = np.float64(0.5) + count: np.int64 = np.int64(2) + flag: np.bool_ = np.bool_(True) + + def process(self, x: float) -> float: + return x * float(self.level) diff --git a/core/tests/python/typed_strings.py b/core/tests/python/typed_strings.py new file mode 100644 index 0000000..2f1442e --- /dev/null +++ b/core/tests/python/typed_strings.py @@ -0,0 +1,17 @@ +# Test fixture: the same shapes written as strings that cannot be resolved (a name imported only +# for type checking), so the annotations are read as written (plan 3.3, 8.6). +from __future__ import annotations + +from typing import TYPE_CHECKING, Annotated, Optional + +if TYPE_CHECKING: + from nowhere import Meta # noqa: F401 — never importable: forces the as-written path + + +class typed_strings: + noted: Annotated[float, Meta] = 0.5 + either: float | int = 1.0 + maybe: Optional[Annotated[int, Meta]] = 2 + + def process(self, x: float) -> tuple[float, float] | None: + return x, x diff --git a/core/tests/test_types.cpp b/core/tests/test_types.cpp index 6234391..6c898fa 100644 --- a/core/tests/test_types.cpp +++ b/core/tests/test_types.cpp @@ -5,9 +5,12 @@ // Copyright 2022-2026 Timothy Place. #include +#include #include #include +#include + #include "support.h" using namespace tap::python; @@ -167,6 +170,77 @@ SCENARIO("Hints that cannot be resolved are reported, and the annotations as wri CHECK(all_equal(render(p, 1.0), 0.5)); } +// 8.6 — hint shapes a reader expects to map as their payload does + +SCENARIO("Annotated, Final, mixed unions and Optional tuples map as their payload does (plan 8.6)") { + ensure_runtime(); + const auto [name, strings] = GENERATE(std::pair{"typed_more", false}, std::pair{"typed_strings", true}); + log_capture log; + processor p{name, log.sink()}; + forget_loaded(name); // this load runs the file, so logs what is true of the class + REQUIRE(p.load()); + + std::vector> fields; + for (const auto& a : p.attributes()) { + fields.emplace_back(a.name, a.type); + } + if (!strings) { + CHECK(fields + == std::vector>{{"noted", value_type::real}, + {"fixed", value_type::integer}, + {"either", value_type::any}, + {"text_or_number", value_type::any}, + {"maybe_noted", value_type::real}}); + } + else { + CHECK(fields + == std::vector>{ + {"noted", value_type::real}, {"either", value_type::any}, {"maybe", value_type::integer}}); + CHECK(log.contains("could not resolve the type hints", log_level::error)); // read as written + } + THEN("an Optional tuple return names its outputs") { + CHECK(p.output_count() == 2); + std::vector in{1.0, 2.0}; + std::vector a(2, 12345.0); + std::vector b(2, 12345.0); + const double* ins[1] = {in.data()}; + double* outs[2] = {a.data(), b.data()}; + p.process(ins, 1, outs, 2, 2); + CHECK(a == std::vector{1.0, 2.0}); + CHECK(b == (strings ? std::vector{1.0, 2.0} : std::vector{-1.0, -2.0})); + } + THEN("a field of several kinds takes the value as the atom carried it") { + REQUIRE(p.set_attribute("either", 2.5)); + CHECK(std::get(*p.get_attribute("either", value_type::any)) == "2.5"); + REQUIRE(p.set_attribute("either", std::int64_t{3})); + CHECK(std::get(*p.get_attribute("either", value_type::any)) == "3"); + } +} + +SCENARIO("numpy scalar types as field hints map to the kinds they hold (plan 8.6)") { + ensure_runtime(); + { + gil_lock lock; + PyObject* numpy = PyImport_ImportModule("numpy"); + if (!numpy) { + PyErr_Clear(); + if (std::getenv("TAP_PYTHON_TEST_REQUIRE_EXAMPLES")) { + FAIL("numpy is not importable by the embedded interpreter"); + } + SKIP("numpy is not importable by the embedded interpreter"); + } + Py_DECREF(numpy); + } + processor p{"typed_numpy"}; + REQUIRE(p.load()); + REQUIRE(p.attributes().size() == 3); + CHECK(p.attributes()[0].type == value_type::real); + CHECK(p.attributes()[1].type == value_type::integer); + CHECK(p.attributes()[2].type == value_type::boolean); + REQUIRE(p.set_attribute("level", 0.25)); + CHECK(all_equal(render(p, 1.0), 0.25)); +} + SCENARIO("process() binds only as a plain instance method") { ensure_runtime(); diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index f92c9cc..dd0aec8 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -712,7 +712,7 @@ down rather than discovered again. *This plan was itself audited before being ad file's module left in `sys.modules` and working, the old value after an unchanged save (the documented limit), no `__pycache__`. *Decided:* the rule stands as documented — "save the class file too" — rather than a forced reload on `filechanged` or a folder watch; revisit if users ask. -- [ ] **8.6 Type hints that map as a reader expects (A8).** In `hint_kind`: unwrap `Annotated` +- [x] **8.6 Type hints that map as a reader expects (A8).** In `hint_kind`: unwrap `Annotated` and `Final` to their first argument; *revised:* a `Union` of several members other than `None` maps to **`any`** — the atom passes as it is, as for an unannotated parameter — rather than to one preferred member, since preferring `float` for `str | float` would silently lose strings as @@ -722,7 +722,12 @@ down rather than discovered again. *This plan was itself audited before being ad `np.bool_`) are recognized by their `__mro__` names without the support module importing numpy. In `return_shape`: see through `Optional[tuple[…]]` to the tuple. Test: `typed_more.py` in `test_types.cpp` with the audit's table. CHANGELOG: the mapping changes (a `float | int` field - was a symbol attribute). + was a symbol attribute). *Done:* `_unwrap()` for `Annotated`/`Final` (objects and, by regex, strings); + unions of several kinds → `'any'` (`_one_kind`); `isinstance(hint, type)` then `issubclass` + against bool, int, float, str in that order, then numpy's abstract bases by `__mro__` name and + module; `return_shape` sees through `Optional`/`| None` (objects and strings). Fixtures + `typed_more.py`, `typed_strings.py` (hints that cannot resolve, read as written) and + `typed_numpy.py`; two scenarios in `test_types.cpp`. - [ ] **8.7 Windows: paths meet Max as UTF-8 (A9).** Every `path.string()` handed to a Max call or a console line (`locatefile_extended` in the constructor, the core's "No file …" and "Failed to load …") goes through `u8string()` (C++20: `reinterpret_cast` its `c_str()`). Verify From 348bea72c57f0027361ff441d570fb4fe0491072 Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 16:36:31 +0000 Subject: [PATCH 10/11] 8.7: paths meet Max and the console as UTF-8 detail::utf8(path) converts through u8string(): on Windows, std::filesystem::path::string() gives the ANSI code page where Max's path API and console expect UTF-8, so the file watcher failed for a package under a folder with a non-ASCII name and hot reload was silently off. It is used for the watcher's file name, the object's runtime messages and the core's "No file" and "Failed to load" lines; the Python-facing conversions already took wide strings. To verify on Windows in the runbook step; the same bytes on POSIX, where the battery passes unchanged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- CHANGELOG.md | 5 +++++ core/include/tap/python/processor.h | 4 ++-- core/include/tap/python/runtime.h | 8 ++++++++ docs/PRODUCTION-PLAN.md | 9 +++++++-- source/projects/tap.python_tilde/tap.python_tilde.h | 9 +++++---- .../projects/tap.python_tilde/tap.python_tilde_package.h | 5 +++-- 6 files changed, 30 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8af050b..c210e82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,6 +63,11 @@ breaking changes to the contract are allowed where they buy correctness (D5 in ### Fixed +- **Windows: a package under a folder with a non-ASCII name is watched.** The path handed to Max's + file watcher, and the paths in the console's "No file" and "Failed to load" lines, were converted + with the ANSI code page on Windows, where Max expects UTF-8; for a user whose `Documents` folder + has an accented character the watcher failed and hot reload was silently off. Every path shown + to Max or the console is UTF-8 now. (Plan 8.7, audit A9.) - **Samples computed before `process()` raised are sanitized.** A `process()` that returned NaN for part of a vector and then raised let those NaNs through; the contract says non-finite output is replaced with 0.0, and now it is on that path too. (Plan 8.2, audit A2.) diff --git a/core/include/tap/python/processor.h b/core/include/tap/python/processor.h index 247bb23..37b4009 100644 --- a/core/include/tap/python/processor.h +++ b/core/include/tap/python/processor.h @@ -175,7 +175,7 @@ namespace tap::python { } std::error_code ec; if (!std::filesystem::is_regular_file(path, ec)) { - log(log_level::error, "No file " + path.string()); + log(log_level::error, "No file " + detail::utf8(path)); release_binding(); return false; } @@ -188,7 +188,7 @@ namespace tap::python { if (!module) { if (!take_reported_load_failure()) { // 6.10: once per failing save, not per object report_exception(); - log(log_level::error, "Failed to load " + path.string()); + log(log_level::error, "Failed to load " + detail::utf8(path)); } release_binding(); return false; diff --git a/core/include/tap/python/runtime.h b/core/include/tap/python/runtime.h index b7907ea..aaf590b 100644 --- a/core/include/tap/python/runtime.h +++ b/core/include/tap/python/runtime.h @@ -656,6 +656,14 @@ def return_shape(hint): return PyModule_Create(&s_console_moduledef); } + /// A path as UTF-8 text, for a console line or a host API that takes UTF-8 (Max's do). On + /// Windows std::filesystem::path::string() gives the ANSI code page instead, which broke the + /// file watcher for a package under a folder with a non-ASCII name (plan 8.7, audit A9). + inline std::string utf8(const std::filesystem::path& path) { + const auto text = path.u8string(); + return std::string{text.begin(), text.end()}; + } + inline PyObject* path_to_unicode(const std::filesystem::path& path) { #ifdef _WIN32 return PyUnicode_FromWideChar(path.wstring().c_str(), -1); diff --git a/docs/PRODUCTION-PLAN.md b/docs/PRODUCTION-PLAN.md index dd0aec8..d1e4001 100644 --- a/docs/PRODUCTION-PLAN.md +++ b/docs/PRODUCTION-PLAN.md @@ -728,12 +728,17 @@ down rather than discovered again. *This plan was itself audited before being ad module; `return_shape` sees through `Optional`/`| None` (objects and strings). Fixtures `typed_more.py`, `typed_strings.py` (hints that cannot resolve, read as written) and `typed_numpy.py`; two scenarios in `test_types.cpp`. -- [ ] **8.7 Windows: paths meet Max as UTF-8 (A9).** Every `path.string()` handed to a Max call or +- [x] **8.7 Windows: paths meet Max as UTF-8 (A9).** Every `path.string()` handed to a Max call or a console line (`locatefile_extended` in the constructor, the core's "No file …" and "Failed to load …") goes through `u8string()` (C++20: `reinterpret_cast` its `c_str()`). Verify in the runbook's Windows step with the package under a folder named with a non-ASCII character: the watcher starts, a save reloads. A bug fix with no contract change: the one item here that - may follow 1.0. + may follow 1.0. *Done (code):* `detail::utf8(path)` in `runtime.h` (`u8string()`), used for + the watcher's file name, the object's three runtime messages, and the core's two file + diagnostics; `path_to_unicode()` and `PyConfig` already took wide strings on Windows. The core + tests' `.string()` expectations are the same bytes on POSIX. *To verify* in the runbook's Windows + step (8.8): the package under a folder named with a non-ASCII character, the watcher starts, a + save reloads. - [ ] **8.8 The Mac session for this phase.** Runtime tests for 8.2 (`faults`) and 8.3 (`worker`); 8.3's stack measurement; the macOS half of 8.7 is not needed (POSIX paths are UTF-8). Then repeat the runbook's step 4 on the release packages, and tag `v0.11.0`. diff --git a/source/projects/tap.python_tilde/tap.python_tilde.h b/source/projects/tap.python_tilde/tap.python_tilde.h index a6f2167..8f6ade6 100644 --- a/source/projects/tap.python_tilde/tap.python_tilde.h +++ b/source/projects/tap.python_tilde/tap.python_tilde.h @@ -176,7 +176,7 @@ class python : public object, public vector_operator<> { #endif if (!std::filesystem::exists(home)) { - cerr << "No Python runtime found at " << home.string() + cerr << "No Python runtime found at " << runtime::detail::utf8(home) << " — run scripts/install-runtime from the package root to install it." << endl; return; } @@ -194,7 +194,7 @@ class python : public object, public vector_operator<> { options.console_ready = console_ready; const auto status = runtime::initialize(options); if (!status.ok) { - cerr << "failed to start Python from '" << home.string() << "': " << status.error + cerr << "failed to start Python from '" << runtime::detail::utf8(home) << "': " << status.error << " (run scripts/install-runtime to install the runtime)" << endl; return; } @@ -233,8 +233,9 @@ class python : public object, public vector_operator<> { return; } const auto watched_file = m_scripts_dir / (m_python_source + ".py"); + const auto watched_utf8 = runtime::detail::utf8(watched_file); // Max's paths are UTF-8 (8.7) char filename[c74::max::MAX_PATH_CHARS]{}; - std::strncpy(filename, watched_file.string().c_str(), c74::max::MAX_PATH_CHARS - 1); + std::strncpy(filename, watched_utf8.c_str(), c74::max::MAX_PATH_CHARS - 1); short path_id{}; c74::max::t_fourcc filetype{}; if (c74::max::locatefile_extended(filename, &path_id, &filetype, nullptr, 0) == 0) { @@ -242,7 +243,7 @@ class python : public object, public vector_operator<> { m_file_watch = std::make_unique(maxobj(), path_id, filename); } else { - cerr << "Unable to watch " << watched_file.string() << " for changes." << endl; + cerr << "Unable to watch " << watched_utf8 << " for changes." << endl; } } diff --git a/source/projects/tap.python_tilde/tap.python_tilde_package.h b/source/projects/tap.python_tilde/tap.python_tilde_package.h index 73bfcfc..3cda027 100644 --- a/source/projects/tap.python_tilde/tap.python_tilde_package.h +++ b/source/projects/tap.python_tilde/tap.python_tilde_package.h @@ -90,13 +90,14 @@ namespace tap::python { /// missing — instead of Max refusing to load it at all. On failure `error` says what to do. inline bool runtime_library_loadable([[maybe_unused]] const std::filesystem::path& home, [[maybe_unused]] std::string& error) { + using detail::utf8; // paths in console text as UTF-8 (plan 8.7) #if defined(WIN_VERSION) && defined(TAP_PYTHON_DLL) // Load it from the package by full path (its own dependencies from its folder); the // delay-load helper then finds the loaded module by name on the first Python call. A runtime // installed while Max is running is picked up by the next object created. const auto dll = home / TAP_PYTHON_DLL; if (LoadLibraryExW(dll.c_str(), nullptr, LOAD_WITH_ALTERED_SEARCH_PATH) == nullptr) { - error = "could not load " + dll.string() + " (Windows error " + std::to_string(GetLastError()) + error = "could not load " + utf8(dll) + " (Windows error " + std::to_string(GetLastError()) + ") — run scripts/install-runtime.ps1 from the package root to install the runtime"; return false; } @@ -106,7 +107,7 @@ namespace tap::python { // through a volatile, so the compiler cannot assume a function's address is non-null.) auto* volatile entry = &Py_InitializeFromConfig; if (entry == nullptr) { - error = "the Python runtime was not found in " + (home / "lib").string() + error = "the Python runtime was not found in " + utf8(home / "lib") + " when Max loaded tap.python~ — run scripts/install-runtime.sh from the package root, " "then restart Max"; return false; From 413c0f2a5a82adbec7daee09a8234a79cbfb82ae Mon Sep 17 00:00:00 2001 From: Timothy Place Date: Thu, 1 Oct 2026 18:22:49 +0000 Subject: [PATCH 11/11] Fix the Windows build: windows.h's min/max macros, and the ellipsis as UTF-8 bytes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit worker.h (8.3) includes windows.h for _beginthreadex, and windows.h defines min and max as function-like macros: std::min(…), std::max(…) and numeric_limits::max() in the header stopped parsing under MSVC, and every later error in the job was cascade from those. NOMINMAX is defined before the include (the SDK and min-api are cross-platform headers that cannot depend on those macros), every min/max call in the core headers is parenthesized so they compile even in a translation unit that saw windows.h first, and is included for the errno the Windows branch reports. Checked here by compiling worker.h with min and max defined as macros before it, under GCC and clang. The console's ellipsis (8.4) was written as a … escape, which MSVC encodes in its execution charset rather than UTF-8; it is now the three UTF-8 bytes, in the queue, the dropped-lines message and the test. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_014sfhCxUoBQmLNnYBSn1ozG --- core/include/tap/python/processor.h | 6 +++--- core/include/tap/python/runtime.h | 7 ++++--- core/include/tap/python/value.h | 4 ++-- core/include/tap/python/worker.h | 14 ++++++++++---- core/tests/test_console.cpp | 2 +- 5 files changed, 20 insertions(+), 13 deletions(-) diff --git a/core/include/tap/python/processor.h b/core/include/tap/python/processor.h index 37b4009..5f9b7b8 100644 --- a/core/include/tap/python/processor.h +++ b/core/include/tap/python/processor.h @@ -890,7 +890,7 @@ namespace tap::python { /// `function` and `instance`. void process_samples(PyObject* function, PyObject* instance, const channels& io, const std::size_t frame_count) { - const std::size_t written = std::min(io.outputs, io.host_outputs); + const std::size_t written = (std::min)(io.outputs, io.host_outputs); PyObject* call_args[1 + k_max_channels]{instance}; for (std::size_t i = 0; i < frame_count; ++i) { // every input of this sample is read before any output of it is written (in place) @@ -960,7 +960,7 @@ namespace tap::python { /// a tuple of as many arrays as it declares outputs. Caller holds the GIL and references to /// `function` and `instance`. void process_block(PyObject* function, PyObject* instance, const channels& io, const std::size_t frame_count) { - const std::size_t written = std::min(io.outputs, io.host_outputs); + const std::size_t written = (std::min)(io.outputs, io.host_outputs); // allocates only when the vector size differs from the one prepare() announced if (!ensure_block_buffers(frame_count, io.inputs)) { silence(io.out, 0, written, 0, frame_count); @@ -1086,7 +1086,7 @@ namespace tap::python { if (m_block_size == size && m_block_inputs.size() >= count) { return true; } - const auto wanted = std::max(count, m_block_size == size ? m_block_inputs.size() : std::size_t{0}); + const auto wanted = (std::max)(count, m_block_size == size ? m_block_inputs.size() : std::size_t{0}); std::vector arrays; arrays.reserve(wanted); diff --git a/core/include/tap/python/runtime.h b/core/include/tap/python/runtime.h index aaf590b..f2400a9 100644 --- a/core/include/tap/python/runtime.h +++ b/core/include/tap/python/runtime.h @@ -197,8 +197,9 @@ namespace tap::python { std::memcpy(cell->text.data(), text.data(), text.size()); } else { - constexpr std::string_view k_ellipsis = "\u2026"; - const auto kept = k_line_bytes - k_ellipsis.size(); + constexpr std::string_view k_ellipsis = + "\xE2\x80\xA6"; // U+2026 as UTF-8 bytes, whatever the compiler's charset + const auto kept = k_line_bytes - k_ellipsis.size(); std::memcpy(cell->text.data(), text.data(), kept); std::memcpy(cell->text.data() + kept, k_ellipsis.data(), k_ellipsis.size()); cell->length = static_cast(k_line_bytes); @@ -738,7 +739,7 @@ def return_shape(hint): if (const auto dropped = state.queue.take_dropped(); dropped != 0) { std::lock_guard lock{state.mutex}; if (state.sink) { - state.sink(log_level::error, "\u2026 and " + std::to_string(dropped) + state.sink(log_level::error, "\xE2\x80\xA6 and " + std::to_string(dropped) + " console line(s) were dropped: printed faster than the main " "thread could post them"); } diff --git a/core/include/tap/python/value.h b/core/include/tap/python/value.h index b704db2..d146507 100644 --- a/core/include/tap/python/value.h +++ b/core/include/tap/python/value.h @@ -63,7 +63,7 @@ namespace tap::python { } if (const auto* d = std::get_if(&v)) { constexpr auto k_lowest = static_cast(std::numeric_limits::lowest()); - constexpr auto k_highest = static_cast(std::numeric_limits::max()); + constexpr auto k_highest = static_cast((std::numeric_limits::max)()); if (!std::isfinite(*d)) { return std::int64_t{0}; } @@ -71,7 +71,7 @@ namespace tap::python { return std::numeric_limits::lowest(); } if (*d >= k_highest) { - return std::numeric_limits::max(); + return (std::numeric_limits::max)(); } return static_cast(*d); } diff --git a/core/include/tap/python/worker.h b/core/include/tap/python/worker.h index 32a504d..69276a6 100644 --- a/core/include/tap/python/worker.h +++ b/core/include/tap/python/worker.h @@ -58,7 +58,12 @@ #include #ifdef _WIN32 +#include + #include +#ifndef NOMINMAX +#define NOMINMAX // windows.h's min/max macros would break std::min/std::max in every header after this +#endif #include #else #include @@ -242,7 +247,7 @@ namespace tap::python { ? static_cast(std::ceil(k_backlog_seconds * sample_rate / static_cast(frames))) : std::size_t{0}; auto next = - std::make_shared(inputs, outputs, frames, vectors, vectors + std::max(k_min_backlog, backlog)); + std::make_shared(inputs, outputs, frames, vectors, vectors + (std::max)(k_min_backlog, backlog)); ring* r = next.get(); const auto period = std::isfinite(sample_rate) && sample_rate > 0.0 ? static_cast(frames) / sample_rate : 0.0; @@ -337,7 +342,7 @@ namespace tap::python { silence(outputs, output_count, 0, frame_count); return; } - const auto frames = std::min(frame_count, r->vector_size); + const auto frames = (std::min)(frame_count, r->vector_size); // this vector's inputs, unless the worker is a whole ring behind const auto k = r->written.load(std::memory_order_relaxed); // written only here @@ -369,7 +374,7 @@ namespace tap::python { const auto index = m % r->slot_count; auto& slot = r->slots[index]; if (r->done.load(std::memory_order_acquire) > m && slot.out_seq.load(std::memory_order_acquire) == m) { - given = std::min(slot.frames.load(std::memory_order_relaxed), frame_count); + given = (std::min)(slot.frames.load(std::memory_order_relaxed), frame_count); for (std::size_t c = 0; c < output_count; ++c) { if (c < r->outputs) { std::copy_n(r->output(index, c), given, outputs[c]); @@ -405,7 +410,8 @@ namespace tap::python { } private: - static constexpr std::uint64_t k_none = std::numeric_limits::max(); + // (parenthesized, as every min/max here: a windows.h included before this header defines them as macros) + static constexpr std::uint64_t k_none = (std::numeric_limits::max)(); static constexpr double k_backlog_seconds = 0.25; static constexpr std::size_t k_min_backlog = 16; diff --git a/core/tests/test_console.cpp b/core/tests/test_console.cpp index 139700e..f77f19c 100644 --- a/core/tests/test_console.cpp +++ b/core/tests/test_console.cpp @@ -126,5 +126,5 @@ SCENARIO("A line longer than a queue slot is cut with an ellipsis, not lost (pla const auto lines = console().lines(); REQUIRE(lines.size() == 1); CHECK(lines[0].text.size() == detail::console_queue::k_line_bytes); - CHECK(lines[0].text.ends_with("\u2026")); + CHECK(lines[0].text.ends_with("\xE2\x80\xA6")); // U+2026 in UTF-8 }