From ff052248349f0ea2cd6b01ecc9487ed281ba86e9 Mon Sep 17 00:00:00 2001 From: ewowi Date: Thu, 22 Oct 2026 14:44:52 +0200 Subject: [PATCH 1/5] Free internal RAM on the ESP32: 150 KB and a 108 KB block on the S3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A board with PSRAM now keeps at least 100 KB of internal RAM free, with a 50 KB block, for WiFi, the web server and the UI. On the S3 that went from 66 KB to about 150 KB, and LCD-MM from 75 to over 100 fps. A classic ESP32 with the UI and preview open keeps 40 to 55 KB instead of 12 to 23. KPI: 256lights | Desktop:2366KB | tick:7/9/8/2/8/8/285/7/7/10us(FPS:142857/111111/125000/500000/125000/125000/3508/142857/142857/100000) | ESP32:2299KB | src:335(84750) | test:249(58033) | functions-over:349 Commit: src +371/-347 lines | test +758/-130 lines | code findings 1672 -> 1663 (deeply nested -1, duplicated block -8) | duplicated lines 3379 -> 3338 (-41) **Core** - platform::InternalRamObject and kInternalFreeFloor (100 KB): a type a cache-safe interrupt reads lives in internal RAM; everything else may live in PSRAM. - HttpServer streams embedded files from 8 send slots and admits a send only above kSendHeadroom, for embedded files, the preview and the state stream. - DevicesModule saves only when the list changes, not on a status refresh. - The FFT twiddle table is sized to the block (4 KB on the classic, built in on the S3) and freed with the microphone. - WiFi NVS off (CONFIG_ESP_WIFI_NVS_ENABLED=n). **Light domain** - LCD-MM ring sized by its lead in rows: ringAuto takes 3-row buffers for 54 rows of lead, never past the internal floor; autoDepth is a pure function the tests pin. - The ring refuses a frame in PSRAM with a status rather than crash at the next flash save; the refill reads its stride from the snapshot, not the source buffer. - The refill encodes faster than the wire: unrolled shift-data emit, Correction::plainRgb fast path; the parallel snapshot split is removed and the prime is serial. - ringDbg reports gap@slice, min lead and the refill core, reset per read. - The ring's settings live on the driver, so switching the peripheral keeps them. **UI** - The status bar's 🧠 and 🧱 show free internal RAM and its largest block. **Scripts/MoonDeck** - run_live_scenario: free_internal and min_stack_free floors, and a browser step (open, refresh, close) through Playwright. - gen_help lists the controls a peripheral declares on the driver card, and reads an accessor's member when neither the named member nor the call carries a comment. **Tests** - scenario_RAM_floor: S3 and P4 contracts at 100 KB free, a 50 KB block and 256 bytes of stack, idle, with the UI open, refreshed and closed. - Ring: refusal with the frame in PSRAM (desktop setTestPsramRange seam), a refill reads nothing of the source, autoDepth, embedded-send streaming, overflow and memory wait; swap and ring setup extracted into helpers. - improv smoke test skipped: needs a board on USB; the product owner's call. **Docs/CI** - mooncore "Internal RAM", performance "Internal RAM", led-drivers "Sizing the MoonI80 ring" with the ringDbg field table, two troubleshooting entries, a MIGRATING entry, testing floors, lessons. - Shipped plans move to docs/work/past/plans at their PR; eight moved, past/plans/index.md and the plan-to-PR table removed. **Reviews** (Fable, staged diff) - Refill read the source Buffer in PSRAM: done, stride from the snapshot, pinned. - PSRAM refusal untested: done, desktop seam and test. - Ring guides over the card's word caps: done, moved to led-drivers.md. - lastSaved help regressed through the accessor rule: done, the call's comment wins. - Per-task heap column behind CONFIG_HEAP_TASK_TRACKING: done, removed. - Hand-rolled operator new/delete: done, inherits InternalRamObject. - Stale fork-join references, mmSnap, : done. - FFT comment held on the S3 only: done. - autoDepth guard at the caller: done, inside. - past/index.md date order: moot, table removed. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- docs/contributing/documentation-standards.md | 2 +- docs/explanation/architecture/mooncore.md | 16 +- docs/help/help.json | 30 ++ docs/reference/MIGRATING.md | 18 + docs/reference/hardware/led-drivers.md | 28 ++ docs/reference/metrics/code.json | 8 +- docs/reference/metrics/docgen.json | 4 +- docs/reference/metrics/prose.json | 4 +- docs/reference/metrics/repo-health.json | 70 ++-- docs/reference/metrics/repo-health.md | 93 +++-- docs/reference/performance.md | 29 ++ docs/reference/testing.md | 8 +- docs/reference/troubleshooting.md | 8 + docs/work/future/backlog-core.md | 14 +- docs/work/future/backlog-light.md | 6 +- docs/work/future/moonwalk.md | 2 +- docs/work/future/processes-inventory.md | 2 +- docs/work/past/index.md | 30 +- docs/work/past/lessons.md | 13 +- ...0260915 - Native HUB75 output (shipped).md | 32 +- ... point as Network submodules (shipped).md} | 2 +- ...dec as runtime configuration (shipped).md} | 0 ... Safe mode and factory reset (shipped).md} | 0 ...onLive Coord3D and RGB types (shipped).md} | 4 +- ...te by subscription, streamed (shipped).md} | 0 ...elp in the UI, from the docs (shipped).md} | 10 +- .../Plan-20261020 - Test engine (shipped).md} | 8 +- docs/work/past/plans/index.md | 7 - ... ESP32-C3 and the Athom bulb (partial).md} | 0 ...> Plan-20261010 - Teensy 4.1 (partial).md} | 0 ...iting, masks and transitions (partial).md} | 0 ...61013 - Audio reactive live performance.md | 2 +- ...ms, every frame from outside (partial).md} | 2 + ... - Processes, described once (partial).md} | 4 +- ...1 - Internal RAM on the ESP32 (shipped).md | 33 ++ docs/work/present/index.md | 16 +- esp32/sdkconfig.defaults | 14 + esp32/sdkconfig.defaults.task-cpu-stats | 4 +- moondeck/docs/gen_help.py | 45 ++- moondeck/docs/gen_tests.py | 4 +- moondeck/scenario/run_live_scenario.py | 136 +++++--- moonwalk/09-plan-setup.yaml | 2 +- src/core/system/DevicesModule.h | 18 +- src/core/system/HttpServerModule.cpp | 35 +- src/core/system/HttpServerModule.h | 21 +- src/core/system/TasksModule.h | 2 +- src/light/drivers/Correction.h | 10 + src/light/drivers/Drivers.h | 4 +- src/light/drivers/I80Peripheral.h | 45 +-- src/light/drivers/LedPeripheral.h | 6 +- src/light/drivers/MoonI80Peripheral.h | 247 ++++--------- src/light/drivers/ParallelLedDriver.h | 110 ++++-- src/light/drivers/ParallelSlots.h | 63 ++-- src/platform/desktop/platform_desktop.cpp | 21 +- src/platform/esp32/platform_esp32.cpp | 3 +- src/platform/esp32/platform_esp32_hub75.cpp | 2 +- src/platform/esp32/platform_esp32_i2s.cpp | 17 +- src/platform/esp32/platform_esp32_i80.cpp | 2 +- src/platform/esp32/platform_esp32_improv.cpp | 4 +- .../esp32/platform_esp32_moon_i80.cpp | 49 +-- src/platform/esp32/platform_esp32_parlio.cpp | 19 +- src/platform/platform.h | 36 +- src/ui/app.js | 4 +- test/python/test_gen_help.py | 50 +++ test/python/test_live_scenario_runner.py | 26 ++ test/scenarios/device/scenario_RAM_floor.json | 324 ++++++++++++++++++ .../core/unit_DevicesModule_discovery.cpp | 19 + .../unit_HttpServerModule_connections.cpp | 140 ++++++++ .../core/unit_HttpServerModule_subscribe.cpp | 16 + test/unit/core/unit_TasksModule.cpp | 8 +- test/unit/light/unit_Correction.cpp | 17 + test/unit/light/unit_MoonI80Peripheral.cpp | 33 +- .../unit_ParallelLedDriver_pinexpander.cpp | 10 +- .../light/unit_ParallelLedDriver_ring.cpp | 136 +++----- .../light/unit_ParallelLedDriver_swap.cpp | 105 ++++-- test/unit/light/unit_ParallelSlots.cpp | 4 +- 77 files changed, 1615 insertions(+), 703 deletions(-) rename docs/work/{present => past/plans}/Plan-20260915 - Native HUB75 output (shipped).md (78%) rename docs/work/{present/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped, partly verified).md => past/plans/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped).md} (98%) rename docs/work/{present/Plan-20261005 - The audio codec as runtime configuration (in PR 129, M3 unverified).md => past/plans/Plan-20261005 - The audio codec as runtime configuration (shipped).md} (100%) rename docs/work/{present/Plan-20261009 - Safe mode and factory reset.md => past/plans/Plan-20261009 - Safe mode and factory reset (shipped).md} (100%) rename docs/work/{present/Plan-20261011 - MoonLive Coord3D and RGB types.md => past/plans/Plan-20261011 - MoonLive Coord3D and RGB types (shipped).md} (89%) rename docs/work/{present/Plan-20261013 - State by subscription, streamed.md => past/plans/Plan-20261013 - State by subscription, streamed (shipped).md} (100%) rename docs/work/{present/Plan-20261014 - Help in the UI, from the docs.md => past/plans/Plan-20261014 - Help in the UI, from the docs (shipped).md} (92%) rename docs/work/{present/Plan-20261020 - Test engine.md => past/plans/Plan-20261020 - Test engine (shipped).md} (93%) delete mode 100644 docs/work/past/plans/index.md rename docs/work/present/{Plan-20261008 - ESP32-C3 and the Athom bulb.md => Plan-20261008 - ESP32-C3 and the Athom bulb (partial).md} (100%) rename docs/work/present/{Plan-20261010 - Teensy 4.1.md => Plan-20261010 - Teensy 4.1 (partial).md} (100%) rename docs/work/present/{Plan-20261011 - Layer compositing, masks and transitions.md => Plan-20261011 - Layer compositing, masks and transitions (partial).md} (100%) rename docs/work/present/{Plan-20261013 - Streams, every frame from outside.md => Plan-20261013 - Streams, every frame from outside (partial).md} (98%) rename docs/work/present/{Plan-20261015 - Processes, described once.md => Plan-20261015 - Processes, described once (partial).md} (97%) create mode 100644 docs/work/present/Plan-20261021 - Internal RAM on the ESP32 (shipped).md create mode 100644 test/scenarios/device/scenario_RAM_floor.json diff --git a/CLAUDE.md b/CLAUDE.md index cd41eebd8..240580afd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -84,7 +84,7 @@ Every activity is a [MoonWalk](docs/moonwalk.md) process, the one home of its st ### Main and branch -**The product owner creates every branch**, and main stays releasable: [the branch](docs/moonwalk.md#branch). A change goes from [a spec and a plan](docs/moonwalk.md#pick-spec-plan) to a pull request the plan describes. **Deleting the plan is the product owner's call**, because "the code is written" is not "the plan is realized". +**The product owner creates every branch**, and main stays releasable: [the branch](docs/moonwalk.md#branch). A change goes from [a spec and a plan](docs/moonwalk.md#pick-spec-plan) to a pull request the plan describes. **Moving the plan to `docs/work/past/plans/` is the product owner's call**, because "the code is written" is not "the plan is realized". **A contributed pull request is one topic, max 400 changed lines and ten files**, tests included, based on main: [Contribute a pull request](docs/moonwalk.md#contribute-pr). The contributor vouches for every line, agent-written or not, answers reviewers in their own words, and says what an agent wrote and what was tested. **A contribution is reviewed and finished as [Review and finish a contribution](docs/moonwalk.md#review-contribution) describes**. The contributor fixes their own code first and the maintainer finishes, case by case. The merge keeps their commits and authorship. diff --git a/docs/contributing/documentation-standards.md b/docs/contributing/documentation-standards.md index 9f44d6f92..6418ab8fa 100644 --- a/docs/contributing/documentation-standards.md +++ b/docs/contributing/documentation-standards.md @@ -126,7 +126,7 @@ Every page serves one of four reader needs, and only one. This is [Diátaxis](ht The test for any page is the cell it sits in. A tutorial that stops to explain, or a reference that starts to teach, is two pages: move the other half to where it belongs. -Two kinds of page sit outside the grid on purpose. **The rules**, [coding-standards.md](coding-standards.md) and this page in `contributing/` and [CLAUDE.md](../../CLAUDE.md) at the repo root, are for contributors and bind every change. `legal/` sits outside it too. **Work not yet in the code** lives under `docs/work/`, and nothing there describes the system as it is. `future` is what does not exist, `present` is being built and deleted at its PR, `past` is what shipped. +Two kinds of page sit outside the grid on purpose. **The rules**, [coding-standards.md](coding-standards.md) and this page in `contributing/` and [CLAUDE.md](../../CLAUDE.md) at the repo root, are for contributors and bind every change. `legal/` sits outside it too. **Work not yet in the code** lives under `docs/work/`, and nothing there describes the system as it is. `future` is what does not exist, `present` is being built and moves to `past` at its PR, `past` is what shipped. Two scales below a page. **A module** has one reference card and one header (see [Module pages](#module-pages)); **a line of code** carries its own reason in a comment (see [Comments](#comments)). diff --git a/docs/explanation/architecture/mooncore.md b/docs/explanation/architecture/mooncore.md index ad2152113..1e6050b7e 100644 --- a/docs/explanation/architecture/mooncore.md +++ b/docs/explanation/architecture/mooncore.md @@ -101,7 +101,7 @@ The light domain plugs into the UI at three points. **A fixed top-level tree**: Only abstract what you need: - **Time**: `millis()`, `micros()`. Monotonic, microsecond resolution. (`esp_timer` / `std::chrono`) -- **Memory**: `alloc(size)`, `free(ptr)`. Prefers PSRAM on ESP32, falls back to regular heap. `freeHeap()`, `maxAllocBlock()` for diagnostics. (`heap_caps_malloc` / `std::malloc`) +- **Memory**: `alloc(size)`, `free(ptr)`. Prefers PSRAM on ESP32, falls back to regular heap; `allocInternal(size)` where only internal RAM will do (see [Internal RAM](#internal-ram)). `freeHeap()`, `maxAllocBlock()` for diagnostics. (`heap_caps_malloc` / `std::malloc`) - **Executable memory**: `allocExec(size)` / `freeExec(ptr, size)` allocate memory the CPU can *fetch and execute* from, and `writeExec(dst, src, len)` copies emitted machine code into it safely. Used by the MoonLive live-script engine (below) to place the native code it compiles. All the W^X and instruction-cache quirks live behind these three functions. On ESP32 that is IRAM via `MALLOC_CAP_EXEC`, with 32-bit-aligned stores and a cache sync so the core fetches fresh code. On POSIX desktops it is an `mmap` `PROT_EXEC` page, with `MAP_JIT` and a write-protect toggle on macOS-arm64. On Windows it is a `VirtualAlloc` `PAGE_EXECUTE_READWRITE` page plus `FlushInstructionCache`. (`heap_caps_malloc(MALLOC_CAP_EXEC)` / `mmap` / `VirtualAlloc`) - **Networking**: `UdpSocket` for ArtNet send. `TcpConnection` / `TcpServer` for HTTP + WebSocket; `TcpConnection::writeSome` is a non-blocking partial write (returns bytes written, 0 = would-block) so a backpressured browser can't stall the render loop. (lwIP sockets / BSD sockets) - **Scheduling**: `yield()` (cooperative yield to OS/RTOS), `delayMs(ms)` (blocking sleep, off-path only), `delayUs(us)` (a microsecond busy-wait, only for sub-millisecond hardware timing a driver owns, such as the WS2812 300 µs inter-frame latch in `RmtLedDriver`), and `reboot()`. General pacing uses the non-blocking `millis()` gate instead. (`vTaskDelay` / `esp_rom_delay_us` / `esp_restart` on ESP32; `std::this_thread::sleep_for` / `std::exit` on desktop) @@ -109,6 +109,20 @@ Only abstract what you need: Abstractions are added when a concrete implementation needs them, not pre-designed. +### Internal RAM + +On an ESP32 internal RAM is the scarce memory. WiFi, lwIP, the task stacks and every DMA buffer need it, while a board with PSRAM has megabytes of the other kind. **Whatever can live in PSRAM does.** Internal RAM is kept for what cannot leave it: + +- **What a DMA engine reads**, the LED drivers' buffers. +- **What a cache-safe interrupt reads.** PSRAM sits behind the flash cache, and a flash write on either core switches that cache off, even in the middle of an interrupt on the other core. So the interrupt's data comes from `platform::allocInternal`, and an object it reads inherits `platform::InternalRamObject`, whose `new` places it there. `ParallelLedDriver` and `MoonI80Peripheral` are the examples; the LCD-MM ring refuses to run when what its refill reads fell back to PSRAM, rather than fault at the next save. +- **The code and stacks of those paths.** + +Everything else asks for PSRAM first: a module through `MoonModule`'s `operator new`, a buffer through `platform::alloc`, a plain `malloc` through `CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=0`. WiFi, lwIP and TLS do so through their options in `esp32/sdkconfig.defaults`, where each option carries its reason. + +**The floor.** `platform::kInternalFreeFloor`, 100 KB, is the internal RAM a board with PSRAM keeps free for WiFi, HTTP and the UI. Its largest free block stays at 50 KB, for a TLS handshake or a state send. What would take the rest respects it. LCD-MM's `ringAuto` sizes its ring by the lead the wire needs and stops at the floor. The HTTP server queues a send on a socket only while internal RAM stays above half of it, so a slow client waits rather than draining the board. `scenario_RAM_floor` holds the floor with the UI open on the S3 and the P4. + +**A board without PSRAM** runs the same calls, which fall back to internal RAM. Its network buffers then share that RAM, so it holds no floor, and the send back-pressure is what keeps an open UI on a weak link from exhausting it. The measurements are in [performance.md](../../reference/performance.md#internal-ram). + **Platform boundary (hard rule).** All `#ifdef`, `#if defined`, platform-specific `#include`s, and hardware API calls live exclusively in `src/platform/`. Everything outside `src/platform/` compiles on every target without modification. Compile-time platform branching uses `if constexpr` on `platform_config.h` flags, never a preprocessor `#ifdef`. **Domain boundary.** A file in `src/core` includes nothing from `src/light`: core declares an interface, and the light domain implements and registers it. Both boundaries are rules of [the code check](../../reference/tests.md#check_code), each at zero, and a crossing raises the count the ratchet refuses. `LightOutput` is the domain boundary's shape: the light domain's `Drivers` implements it, and the WLED shim, MQTT and MoonStats read the device shape and the palettes through it. diff --git a/docs/help/help.json b/docs/help/help.json index fd3bbad81..50f7b9e2f 100644 --- a/docs/help/help.json +++ b/docs/help/help.json @@ -3702,6 +3702,12 @@ } ] }, + "clockPin": { + "line": "The i80 write strobe (WR): the pixel clock, and the expander's shift clock." + }, + "dcPin": { + "line": "The i80 bus's data/command select, which `LCD-IDF` mandates though WS2812 ignores it." + }, "pinExpander": { "line": "A 74HCT595 per data pin: each pin then drives 8 strands.", "more": "It buys pins, not memory: the frame grows eightfold, about 1.1 KB per light, up to 64 strands. `i80` stops at about 96 lights per strand, `MoonI80` drives 128.", @@ -3715,6 +3721,30 @@ "latchPin": { "line": "The 74HCT595 latch (RCLK) pin, needed with the expander." }, + "shiftOverclock": { + "line": "Shift the '595 at 26.67 MHz, not 20, for short wires; off if a panel scrambles." + }, + "useRing": { + "line": "Stream the frame as a ring of small internal buffers, or send it whole from one block." + }, + "ringSnapshot": { + "line": "Copy each frame before the ring sends it, so a frame never mixes with the next." + }, + "ringAuto": { + "line": "Size the ring by itself and show the numbers it picked in the controls below." + }, + "ringRows": { + "line": "Lights per DMA buffer: the ring's grain, and the RAM against interrupt-rate lever." + }, + "ringBufs": { + "line": "How many buffers the DMA circulates, each `ringRows` rows of internal RAM." + }, + "ringPadUs": { + "line": "Inter-buffer zero-pad in µs, buying refill deadline at a linear frame-time cost." + }, + "ringDbg": { + "line": "The ring's live counters, once a second; `ab` counts lost frames, 0 on a healthy ring." + }, "loopbackTest": { "line": "TX to RX self-test: send a known pattern, bit-verify the capture." }, diff --git a/docs/reference/MIGRATING.md b/docs/reference/MIGRATING.md index 3184c0a87..fe6b6d26c 100644 --- a/docs/reference/MIGRATING.md +++ b/docs/reference/MIGRATING.md @@ -24,12 +24,30 @@ For a device still on old firmware (no Backup button yet), the [installer page]( | *re-set a control* | One value resets to its default; set it again in the UI if you had changed it. | | *re-add a module* | The module vanishes from the tree on boot; add it again and re-enter its controls. | | *update a file* | An on-device file must be edited or replaced. | +| *install over USB* | The update cannot come over the air: one install from the installer page with a cable, after which updates over the air work again. | | *erase flash* | A full flash erase is required (the heaviest, a full reconfigure follows). | --- ## Unreleased +### LCD-MM's ring is smaller, and refuses PSRAM + +**Action: *nothing*, unless `ringSnapshot` was off on a board with PSRAM: turn it on.** + +`ringAuto` now sizes the ring by the lead the wire needs, 3-row buffers, where it took every byte of internal RAM above 64 KB. On a 3,840-light S3 that is 21 small buffers instead of 28 large ones, a ring of 37 KB where it took about 112 KB. Turn `ringAuto` off to set `ringRows` and `ringBufs` by hand as before. +The ring refuses to run when its refill would read PSRAM, since a flash save can cut PSRAM off in the middle of a refill and crash the board. With `ringSnapshot` off it reads the live frame, which a board with PSRAM keeps there, so the card shows `ring refused: … turn ringSnapshot on`. + +### ESP32-S3 boards read their flash over four lines + +**Action: *install over USB* once, on an `esp32s3-n16r8`, `esp32s3-n8r8` or `esp32s3-zero` device; the others need *nothing*.** + +These three firmwares read code from flash over four data lines (QIO) where they used two (DIO), which makes an S3's tick about 16% faster. +The flash mode is the bootloader's, and an update over the air replaces only the app, so ESP-IDF refuses an app whose mode differs from the running one. +The Firmware card then reports `ESP_ERR_OTA_SPI_MODE_MISMATCH`. +Install this release once over USB from the [installer page](https://moonmodules.org/MoonLight/install/), which writes the bootloader too. +The partition table is unchanged, so with *Erase chip first* left unticked the config stays; take a backup first anyway. + ### Rings 241 places a few lights one unit over **Action: *nothing*.** diff --git a/docs/reference/hardware/led-drivers.md b/docs/reference/hardware/led-drivers.md index 7bc28ce9a..a8d192a85 100644 --- a/docs/reference/hardware/led-drivers.md +++ b/docs/reference/hardware/led-drivers.md @@ -38,6 +38,34 @@ On an S3, P4 or S31 with `i80` or `MoonI80` as the `peripheral`, `pinExpander` o The expander buys pins, not memory: the register is serial-in, so the DMA frame grows eightfold, about 1.1 KB per light. `i80` holds the whole frame, which past about 96 lights per strand falls to PSRAM too slow for the expander's clock. `MoonI80` streams the frame and drives 128 reliably. +## Sizing the MoonI80 ring + +`MoonI80` streams the frame through a ring of small buffers in internal RAM, refilled from an interrupt as the wire drains each one. The refill encodes faster than the wire drains, so the ring only absorbs the moments the interrupt is held off, up to about 1.1 ms. What prevents a lost frame is the ring's lead over the wire in rows, not its number of buffers. On a 3,840-light expander rig, 45 rows ran clean and 40 lost frames. + +`ringAuto` cuts the ring into 3-row buffers and takes enough for 54 rows of lead. It stops where free internal RAM would fall under 100 KB, and the status says when that cuts it short. With it off, `ringRows` and `ringBufs` are yours to set, at about 600 bytes a row on an 8-pin expander. + +`ringSnapshot` copies each frame into internal RAM before the ring sends it, 3 bytes per RGB light (11.5 KB for 3,840 lights). Off, the ring reads the live frame, and refuses where that frame is in PSRAM. A flash save on the other core switches PSRAM off in the middle of a refill. + +`ringDbg` shows the ring's counters once a second. `ab` and `ml` say whether the depth holds. + +| Field | Meaning | +|---|---| +| `sn`, `lv` | where the snapshot and the live frame live: `I` internal RAM, `P` PSRAM | +| `sl/bf` | slices per frame, over buffers | +| `co`, `cr` | refills deferred while the flash cache was off, and their longest run | +| `ab` | stalls abandoned, each a lost frame: 0 on a healthy ring | +| `dn`, `ld` | frames done, and the last drain | +| `lt` | slices refilled after their drain began; the lead prevents it, so it stays 0 even while frames are lost | +| `tx` | the frame's wire time, in µs | +| `ipb`, `ci` | DMA items per buffer, and consumed | +| `tn`, `de` | the chain terminator, and descriptor errors | +| `enc`, `ea` | the slowest and the average refill, in µs | +| `sg`, `se` | gather and emit cycles per row | +| `tw`, `ts`, `tp` | the tick's wait, snapshot and prime, in µs | +| `gap@` | the longest gap between refills within a frame in µs, and the slice it ended at | +| `ml` | the fewest slices the refill stayed ahead of the drain | +| `c` | the core the refill runs on | + ## Testing the wire with the loopback `loopbackTest` sends a known pattern out of a strand and reads it back through a jumper to `loopbackRxPin`, with its verdict in the status field. A `PASS` means the chip and the peripheral are clean, so flicker that remains is [signal integrity](../troubleshooting.md#stray-pixels-or-wrong-colors-on-lights-that-should-be-off). diff --git a/docs/reference/metrics/code.json b/docs/reference/metrics/code.json index f8d07970b..1a0315887 100644 --- a/docs/reference/metrics/code.json +++ b/docs/reference/metrics/code.json @@ -1,12 +1,12 @@ { "#define used as a constant": 1, - "(duplicated lines)": 3379, - "(total)": 1672, + "(duplicated lines)": 3338, + "(total)": 1663, "blocking call on the render path": 194, "complex function": 296, - "deeply nested": 111, + "deeply nested": 110, "double in firmware code": 2, - "duplicated block": 752, + "duplicated block": 744, "large file": 25, "large stack frame": 33, "long function": 49, diff --git a/docs/reference/metrics/docgen.json b/docs/reference/metrics/docgen.json index fefbbcfae..c99555f82 100644 --- a/docs/reference/metrics/docgen.json +++ b/docs/reference/metrics/docgen.json @@ -1,11 +1,11 @@ { "(appendix lines)": 4376, - "(total)": 3051, + "(total)": 3047, "appendix sections nothing refers to": 199, "member deep dives": 39, "multi-line comment blocks": 307, "over-long sentences": 76, - "over-wide comment lines": 1541, + "over-wide comment lines": 1537, "over-wide doc lines": 34, "split paragraphs": 855 } diff --git a/docs/reference/metrics/prose.json b/docs/reference/metrics/prose.json index f20310b1a..de524eabd 100644 --- a/docs/reference/metrics/prose.json +++ b/docs/reference/metrics/prose.json @@ -1,6 +1,6 @@ { - "(total)": 2336, - "MoonLight.EmDash": 1117, + "(total)": 2334, + "MoonLight.EmDash": 1115, "MoonLight.LinkText": 7, "MoonLight.NegatedHeading": 6, "MoonLight.SelfReference": 17, diff --git a/docs/reference/metrics/repo-health.json b/docs/reference/metrics/repo-health.json index 7255af929..c4f4f5a80 100644 --- a/docs/reference/metrics/repo-health.json +++ b/docs/reference/metrics/repo-health.json @@ -1,20 +1,20 @@ { - "commit": "80e19948", + "commit": "08677e00", "flash": { - "esp32s3-n16r8": 2474896, - "esp32": 2281504, + "esp32s3-n16r8": 2475376, + "esp32": 2283840, "esp32p4rev1-eth": 2404800, "esp32p4rev1-eth-wifi": 2657056, "esp32s3-n8r8": 2437792, "esp32s31": 2744032, - "esp32-16mb": 2356048, + "esp32-16mb": 2399680, "esp32-eth": 1681040, "esp32-wrover": 1843760, "qemu": 1383648, "esp32p4rev3-eth": 2423840, "esp32s3-zero": 2224816, "esp32-pico": 2107168, - "desktop-macos": 2423016, + "desktop-macos": 2423304, "desktop-windows": 1925120, "esp32c3": 2335072, "desktop-linux": 2548000, @@ -24,16 +24,16 @@ "measured": { "esp32p4rev1-eth": "2026-10-19", "esp32s31": "2026-10-13", - "esp32": "2026-10-21", + "esp32": "2026-10-22", "esp32-pico": "2026-09-09", - "esp32s3-n16r8": "2026-10-21", + "esp32s3-n16r8": "2026-10-22", "esp32s3-n8r8": "2026-10-13", "esp32s3-zero": "2026-10-13", - "esp32-16mb": "2026-10-13", + "esp32-16mb": "2026-10-22", "esp32p4rev1-eth-wifi": "2026-10-19", "esp32-eth": "2026-09-11", "esp32p4rev3-eth": "2026-10-21", - "desktop-macos": "2026-10-21", + "desktop-macos": "2026-10-22", "desktop-windows": "2026-09-30", "esp32c3": "2026-10-10", "desktop-linux": "2026-10-12", @@ -365,6 +365,20 @@ "last": "2026-10-14" } }, + "RAM_floor": { + "esp32s3-n16r8": { + "p50": 11732, + "p95": 12527, + "n": 3, + "last": "2026-10-22" + }, + "esp32p4rev1-eth": { + "p50": 2168, + "p95": 2168, + "n": 1, + "last": "2026-10-22" + } + }, "perf_full": { "desktop-macos": { "p50": 256, @@ -822,53 +836,53 @@ } }, "loc": { - "core": 30448, - "light": 31568, - "platform": 21750, + "core": 30547, + "light": 31485, + "platform": 21960, "ui": 12957, - "test": 71673, - "moondeck": 31938 + "test": 72271, + "moondeck": 32063 }, "comments": { "core": { - "lines": 6336, - "ratio": 0.231 + "lines": 6346, + "ratio": 0.23 }, "light": { - "lines": 6666, + "lines": 6643, "ratio": 0.237 }, "platform": { - "lines": 4104, - "ratio": 0.212 + "lines": 4135, + "ratio": 0.211 }, "ui": { "lines": 3489, "ratio": 0.285 }, "test": { - "lines": 7277, - "ratio": 0.118 + "lines": 7318, + "ratio": 0.117 }, "moondeck": { - "lines": 4460, + "lines": 4462, "ratio": 0.16 } }, "tests": { - "cases": 2643, - "scenarios": 32 + "cases": 2674, + "scenarios": 33 }, "docs": { "md_files": 162, - "md_lines": 30528, + "md_lines": 30683, "plans_files": 53, - "backlog_lines": 4031, - "lessons_lines": 535, + "backlog_lines": 4063, + "lessons_lines": 546, "claude_md_lines": 119 }, "complexity": { - "functions": 5171, + "functions": 5183, "over_threshold": 349, "worst_ccn": 128 } diff --git a/docs/reference/metrics/repo-health.md b/docs/reference/metrics/repo-health.md index c96038956..c3fe5b21c 100644 --- a/docs/reference/metrics/repo-health.md +++ b/docs/reference/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `80e19948`. Generated by [`moondeck/check/repo_health.py`](../../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `08677e00`. Generated by [`moondeck/check/repo_health.py`](../../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/reference/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -9,14 +9,14 @@ Current state only; the trend is this file's git history (`git log -p docs/refer | Metric | v6.0.0 | Now | Change | |---|---:|---:|---:| | flash desktop-linux | - | 2,488 KB | new | -| flash desktop-macos | 1,963 KB | 2,366 KB | +403.2 KB (+20.5%) | -| flash esp32 | 2,061 KB | 2,228 KB | +166.8 KB (+8.1%) | -| flash esp32-16mb | 2,012 KB | 2,301 KB | +288.8 KB (+14.4%) | +| flash desktop-macos | 1,963 KB | 2,367 KB | +403.5 KB (+20.6%) | +| flash esp32 | 2,061 KB | 2,230 KB | +169.1 KB (+8.2%) | +| flash esp32-16mb | 2,012 KB | 2,343 KB | +331.4 KB (+16.5%) | | flash esp32c3 | - | 2,280 KB | new | | flash esp32p4rev1-eth | 2,022 KB | 2,348 KB | +326.7 KB (+16.2%) | | flash esp32p4rev1-eth-wifi | 2,304 KB | 2,595 KB | +291.2 KB (+12.6%) | | flash esp32p4rev3-eth | 2,025 KB | 2,367 KB | +341.8 KB (+16.9%) | -| flash esp32s3-n16r8 | 2,111 KB | 2,417 KB | +305.4 KB (+14.5%) | +| flash esp32s3-n16r8 | 2,111 KB | 2,417 KB | +305.9 KB (+14.5%) | | flash esp32s3-n32r16 | - | 2,426 KB | new | | flash esp32s3-n8r8 | 2,038 KB | 2,381 KB | +342.4 KB (+16.8%) | | flash esp32s3-zero | 1,977 KB | 2,173 KB | +195.9 KB (+9.9%) | @@ -24,14 +24,14 @@ Current state only; the trend is this file's git history (`git log -p docs/refer | flash teensy41 | - | 1,242 KB | new | | tick Effects_pipeline_builds_and_renders (desktop-macos) | 7 µs | 7 µs | +0 (+0.0%) | | tick Layouts_resize_reallocates_live (desktop-macos) | 12 µs | 10 µs | -2 (-16.7%) | -| lines core | 22,299 | 30,448 | +8,149 (+36.5%) | -| lines light | 31,054 | 31,568 | +514 (+1.7%) | -| lines platform | 16,798 | 21,750 | +4,952 (+29.5%) | +| lines core | 22,299 | 30,547 | +8,248 (+37.0%) | +| lines light | 31,054 | 31,485 | +431 (+1.4%) | +| lines platform | 16,798 | 21,960 | +5,162 (+30.7%) | | lines ui | 11,394 | 12,957 | +1,563 (+13.7%) | -| lines test | 57,759 | 71,673 | +13,914 (+24.1%) | -| lines moondeck | 29,325 | 31,938 | +2,613 (+8.9%) | -| unit tests | 2,120 | 2,643 | +523 (+24.7%) | -| scenarios | 21 | 32 | +11 (+52.4%) | +| lines test | 57,759 | 72,271 | +14,512 (+25.1%) | +| lines moondeck | 29,325 | 32,063 | +2,738 (+9.3%) | +| unit tests | 2,120 | 2,674 | +554 (+26.1%) | +| scenarios | 21 | 33 | +12 (+57.1%) | Against the release `v6.0.0`, the fixed point a user compares with; the tables below compare against the last commit. A firmware not rebuilt since `v6.0.0` is left out, and a measure the release did not record yet, such as the code check, joins from the next one. @@ -39,25 +39,25 @@ Against the release `v6.0.0`, the fixed point a user compares with; the tables b | Target | Flash | Capacity | Used | Built | |---|---:|---:|---:|:--:| -| desktop-linux | 2,488 KB | - | - | **STALE 9d** | -| desktop-macos | 2,366 KB | - | - | yes | -| desktop-windows | 1,880 KB | - | - | **STALE 21d** | -| esp32 | 2,228 KB | 2,496 KB | 89% | yes | -| esp32-16mb | 2,301 KB | 4,096 KB | 56% | **STALE 8d** | -| esp32-eth | 1,642 KB | 2,496 KB | 66% | **STALE 40d** | -| esp32-pico | 2,058 KB | 3,072 KB | 67% | **STALE 42d** | +| desktop-linux | 2,488 KB | - | - | **STALE 10d** | +| desktop-macos | 2,367 KB (+288 B) ⚠ | - | - | yes | +| desktop-windows | 1,880 KB | - | - | **STALE 22d** | +| esp32 | 2,230 KB (+2 KB) ⚠ | 2,496 KB | 89% | yes | +| esp32-16mb | 2,343 KB (+43 KB) ⚠ | 4,096 KB | 57% | yes | +| esp32-eth | 1,642 KB | 2,496 KB | 66% | **STALE 41d** | +| esp32-pico | 2,058 KB | 3,072 KB | 67% | **STALE 43d** | | esp32-wrover | 1,801 KB | 2,496 KB | 72% | carried (age?) | -| esp32c3 | 2,280 KB | 2,496 KB | 91% | **STALE 11d** | -| esp32p4rev1-eth | 2,348 KB | 4,096 KB | 57% | carried 2d | -| esp32p4rev1-eth-wifi | 2,595 KB | 4,096 KB | 63% | carried 2d | -| esp32p4rev3-eth | 2,367 KB | 4,096 KB | 58% | yes | -| esp32s3-n16r8 | 2,417 KB (+80 B) ⚠ | 4,096 KB | 59% | yes | -| esp32s3-n32r16 | 2,426 KB | 4,096 KB | 59% | carried 5d | -| esp32s3-n8r8 | 2,381 KB | 3,072 KB | 77% | **STALE 8d** | -| esp32s3-zero | 2,173 KB | 2,496 KB | 87% | **STALE 8d** | -| esp32s31 | 2,680 KB | 4,096 KB | 65% | **STALE 8d** | +| esp32c3 | 2,280 KB | 2,496 KB | 91% | **STALE 12d** | +| esp32p4rev1-eth | 2,348 KB | 4,096 KB | 57% | carried 3d | +| esp32p4rev1-eth-wifi | 2,595 KB | 4,096 KB | 63% | carried 3d | +| esp32p4rev3-eth | 2,367 KB | 4,096 KB | 58% | carried 1d | +| esp32s3-n16r8 | 2,417 KB (+480 B) ⚠ | 4,096 KB | 59% | yes | +| esp32s3-n32r16 | 2,426 KB | 4,096 KB | 59% | carried 6d | +| esp32s3-n8r8 | 2,381 KB | 3,072 KB | 77% | **STALE 9d** | +| esp32s3-zero | 2,173 KB | 2,496 KB | 87% | **STALE 9d** | +| esp32s31 | 2,680 KB | 4,096 KB | 65% | **STALE 9d** | | qemu | 1,351 KB | - | - | carried (age?) | -| teensy41 | 1,242 KB | 3,456 KB | 36% | carried 6d | +| teensy41 | 1,242 KB | 3,456 KB | 36% | **STALE 7d** | `Built: yes` was measured this run. `carried (age?)` was not rebuilt either and predates this record, so its age is unknown: it dates itself on the next build. `carried Nd` was NOT rebuilt and its number is N days old, so an absent delta says nothing about the change. **STALE** marks a carry older than 7 days: the number has gone unchecked long enough that growth will surface later as one jump, blamed on whichever commit happens to rebuild that target. `Used` is against the app slot in the firmware's own partition table. @@ -65,15 +65,11 @@ Against the release `v6.0.0`, the fixed point a user compares with; the tables b | Target | Net | New | Removed | Grown | Shrunk | Other | Since | |---|---:|---:|---:|---:|---:|---:|---| -| esp32 | +0.0 KB | +0.0 KB | +0.0 KB | +0.0 KB | +0.0 KB | +0.0 KB | `80e19948` | -| esp32p4rev3-eth | +0.0 KB | +0.0 KB | +0.0 KB | +0.0 KB | +0.0 KB | +0.0 KB | `80e19948` | -| esp32s3-n16r8 | +0.1 KB | +0.0 KB | +0.0 KB | +0.1 KB | +0.0 KB | -0.0 KB | `80e19948` | +| esp32-16mb | +42.6 KB | +312.2 KB | -275.1 KB | +9.4 KB | -4.9 KB | +1.0 KB | `08677e00` | Largest moves: -- **esp32**: none -- **esp32p4rev3-eth**: none -- **esp32s3-n16r8**: `mm::ui::file10` +83 B +- **esp32-16mb**: `mm::ui::file1` +153,235 B, `mm::ui::appJs` -151,132 B, `mm::ui::file2` +28,697 B, `mm::ui::styleCss` -27,548 B, `mm::ui::file6` +27,418 B What this commit's flash went to, symbol by symbol against the measurement at `Since`. New is new functionality, removed and shrunk are what a cleanup saved, and other is what no symbol owns, such as merged strings and alignment. A rename reads as removed plus new, which the largest moves expose. @@ -106,6 +102,7 @@ What this commit's flash went to, symbol by symbol against the measurement at `S | NetworkSend_one_sender_many_receivers | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 3,847 ? | - | - | | Network_hardware_reconfigures_live | 21 ? | 2,834 ? | - | 8,692 ? | - | - | - | - | 2,147 ? | 237 ? | - | - | - | - | - | - | - | - | | Palettes_a_script_drives_the_effects | 7 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | +| RAM_floor | - | - | - | 11,732 ? | 2,168 ? | - | - | - | - | - | - | - | - | - | - | - | - | - | | State_one_document_adds_an_effect_with_its_controls | 7 | - | - | 2,505 ? | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | perf_full | 256 | 592 ? | 5,313 | 25,340 ? | 11,711 ? | 13,373 ? | - | - | 6,797 ? | 2,438 ? | 5,469,660 ? | - | - | - | - | - | - | - | | perf_light | 16 | 35 ? | 2,510 | 3,044 | 1,117 ? | 1,378 ? | - | - | 662 ? | 780 ? | 5,385,763 ? | - | - | - | - | - | - | - | @@ -114,7 +111,7 @@ What this commit's flash went to, symbol by symbol against the measurement at `S Microseconds. `?` marks a cell backed by fewer than 4 samples, which is a first impression rather than a percentile; several are months old and were captured during a network reconfigure, so they read as whole milliseconds. `-` means that target has never run that scenario. -**Coverage: 115/378 cells measured (30%), 28 of them with 4+ samples (7%).** The blanks are the point: a regression on a target that has never run a scenario cannot be DETECTED in it, and the target cannot be compared against the others. Filling the matrix means running the scenario suite on each bench board, which is a standing task rather than a one-off. +**Coverage: 117/396 cells measured (29%), 28 of them with 4+ samples (7%).** The blanks are the point: a regression on a target that has never run a scenario cannot be DETECTED in it, and the target cannot be compared against the others. Filling the matrix means running the scenario suite on each bench board, which is a standing task rather than a one-off. ### desktop-macos: isolated scenarios (p50 of the sample window) @@ -138,25 +135,25 @@ These build a bare pipeline with no optional modules, so a change here is a chan | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 30,448 | 6,336 | 23.1 % | -| light | 31,568 | 6,666 | 23.7 % | -| platform | 21,750 | 4,104 | 21.2 % | -| ui | 12,957 (+2) ⚠ | 3,489 | 28.5 % | -| test | 71,673 (+9) ⚠ | 7,277 | 11.8 % | -| moondeck | 31,938 (+117) ⚠ | 4,460 | 16.0 % | +| core | 30,547 (+99) ⚠ | 6,346 | 23.0 % (−0.1 %) ✓ | +| light | 31,485 (−83) ✓ | 6,643 | 23.7 % | +| platform | 21,960 (+210) ⚠ | 4,135 | 21.1 % (−0.1 %) ✓ | +| ui | 12,957 | 3,489 | 28.5 % | +| test | 72,271 (+598) ⚠ | 7,318 | 11.7 % (−0.1 %) ✓ | +| moondeck | 32,063 (+125) ⚠ | 4,462 | 16.0 % | ## Tests | Kind | Count | |---|---:| -| unit cases | 2,643 | -| scenarios | 32 | +| unit cases | 2,674 (+31) ✓ | +| scenarios | 33 (+1) ✓ | ## Complexity | Metric | Value | |---|---:| -| functions | 5,171 | +| functions | 5,183 (+12) ✓ | | over threshold | 349 | | worst CCN | 128 | @@ -165,9 +162,9 @@ These build a bare pipeline with no optional modules, so a change here is a chan | Metric | Value | |---|---:| | markdown files | 162 | -| markdown lines | 30,528 (+47) ⚠ | +| markdown lines | 30,683 (+155) ⚠ | | plan files | 53 | -| backlog lines | 4,031 | -| lessons lines | 535 | +| backlog lines | 4,063 (+32) ⚠ | +| lessons lines | 546 (+11) ⚠ | | CLAUDE.md lines | 119 | diff --git a/docs/reference/performance.md b/docs/reference/performance.md index 7534f4c86..63e814a63 100644 --- a/docs/reference/performance.md +++ b/docs/reference/performance.md @@ -300,6 +300,35 @@ The exec block is the emitted machine code, so its size varies by ISA. `lines.ml --- +## Internal RAM + +What the ESP32's scarce memory holds and what freed it, measured 2026-10-21 and 2026-10-22 with BouncingBalls held fixed. The rule behind it is in [MoonCore](../explanation/architecture/mooncore.md#internal-ram); `scenario_RAM_floor` measures it on any board. + +**The S3** (shiffy, `esp32s3-n16r8`, 3,840 lights through 2 pins and the 74HCT595 expander on LCD-MM), with a browser on the UI: + +| | Free internal | Largest block | fps | +|---|---|---|---| +| Start, `ringAuto` taking all but 64 KB (28 buffers of 7 rows) | 66 KB | 30 KB | 75 | +| WiFi, lwIP and TLS buffers in PSRAM, WiFi's code in flash | 110 KB | 31 KB | 72 | +| ESP-IDF's 32 KB DMA reserve pool off (it capped the block at its own size) | 111 KB | 59 KB | 72 | +| Every plain `malloc` asking PSRAM first, smaller stacks | 132 KB | 88 KB | 71 | +| `ringAuto` sizing by lead, 3-row buffers (3 × 21, a 37 KB ring) | 145 KB | 94 KB | 81 | +| The prime's core-0 helper removed | 150 to 155 KB | 108 KB | 102 to 107 | + +**The ring's depth** (`ringAuto` off, 40 s each, frames lost): what counts is the lead over the wire in rows. Before the encode sped up, 7 × 10 and 3 × 24 ran clean and 7 × 8 lost a frame in five. After it, 3 × 18 and 2 × 26 run clean, 3 × 14 loses one every few seconds, and 1-row buffers send nothing. + +| Per row of 64 strands | Before | After | +|---|---|---| +| Emit (transpose and store) | 5,800 cycles | 3,650 (its eight stores unrolled to constant shifts, and the empty second byte of a 16-bit bus skipped) | +| Gather (correction per light) | 1,850 cycles | 1,300 (a plain RGB strip skips every optional fixture role) | +| Refill of a slice, against its wire time | 220 µs vs 207 (7 rows) | 63 µs vs 89 (3 rows) | + +**The P4** (64 lights) holds 368 to 373 KB free with a 192 KB block through the same scenario. + +**The classic ESP32 without PSRAM** (the Olimex, WiFi, 64 lights) rests at about 70 KB free. A page load over a slow link once took it to 3.6 KB. The RAM went to 34 TCP segments of 1,536 bytes waiting for acknowledgement and 8 of WiFi's receive buffers. With the sends' back-pressure an open UI and preview hold 40 to 55 KB with a 40 KB block, where they held 12 to 23 KB. On the classic, WiFi's code in flash frees only the code-only region (22 to 55 KB, MoonLive's scripts use it), no data RAM. + +**Network receive** is unaffected by WiFi's code in flash. In `scenario_NetworkReceive_two_boards`, the P4 sending to the S3, every row matched the run with it in IRAM. That is 38 to 40 frames a second up to 4,096 lights, in Art-Net, E1.31 and DDP. + ## Multi-pin LED driving (all three peripherals, 128×128 grid) The rows name the `ParallelLedDriver` backends by the class names they were measured under: `MultiPinLedDriver` is the `i80` peripheral, `MoonLedDriver` is `MoonI80`, and `ParlioLedDriver` is `Parlio`. The `peripheral` control selects the backend at one vtable dispatch per frame, so the timings hold. diff --git a/docs/reference/testing.md b/docs/reference/testing.md index ca3a1a656..4fa76ab50 100644 --- a/docs/reference/testing.md +++ b/docs/reference/testing.md @@ -336,6 +336,12 @@ The runner compares each measurement to the contract and fails if the device mis - `tick_us` is a **ceiling**, the device must deliver this tick or faster. Going faster than contract is good news, never a failure. - `free_heap` is a **floor**, the device must deliver at least this much free heap. Growing heap is good news. +- `max_alloc_block`, `free_internal` and `min_stack_free` are **floors** a step opts into, one for each of these: + - the largest free block, which an allocator needing one piece depends on; + - the internal RAM WiFi, HTTP and the UI need on a board with PSRAM, where `free_heap` counts the PSRAM too; + - the headroom of the tightest task stack, which the runner names. + + The first two take the heap tolerance; a stack's headroom is exact. `scenario_RAM_floor` holds them with the UI open in a real browser, through the live runner's `browser` step (`action` open, refresh or close, with a `settle` in seconds). - Both are **hand-set promises**, not auto-captured last readings. Renegotiating a contract requires `--update-contract --reason "..."`, see below. - `set_by` records when the contract was last (re)negotiated; `reason` records why. Both stamped automatically by `--update-contract`. @@ -425,7 +431,7 @@ Every `scenario_*.json` carries top-level metadata plus a `description` per step } ``` -`name` matches the filename stem exactly; `module` drives the doc grouping and the MoonDeck filter; `also` lists peer modules. `mode` is `construct` or `mutate` (see above). `fixture` is required for `mutate` scenarios that need to run in-process. `reset` (optional but recommended for mutate scenarios) normalizes controls before measurement so contract assertions aren't coupled to run order. `contract[]` carries the per-target performance contract, see § Performance contracts. `observed[]` carries the published summary of the observations per metric (tick_us, free_heap, max_alloc_block), written on request; see § Persistent observations. Per-step `description` is recommended (a missing one falls back to the step's `name`/`op` in the generated view). Optional top-level `"live_only": true` keeps the scenario out of the in-process runner. Optional per-step `"measure": true` triggers a measurement after the step, checked against the step's `contract`. +`name` matches the filename stem exactly; `module` drives the doc grouping and the MoonDeck filter; `also` lists peer modules. `mode` is `construct` or `mutate` (see above). `fixture` is required for `mutate` scenarios that need to run in-process. `reset` (optional but recommended for mutate scenarios) normalizes controls before measurement so contract assertions aren't coupled to run order. `contract[]` carries the per-target performance contract, see § Performance contracts. `observed[]` carries the published summary of the observations per metric (tick_us, free_heap, max_alloc_block, and on a live device free_internal and min_stack_free), written on request; see § Persistent observations. Per-step `description` is recommended (a missing one falls back to the step's `name`/`op` in the generated view). Optional top-level `"live_only": true` keeps the scenario out of the in-process runner. Optional per-step `"measure": true` triggers a measurement after the step, checked against the step's `contract`. Unknown JSON keys are ignored by both runners (C++ and Python), so adding a new field is safe. diff --git a/docs/reference/troubleshooting.md b/docs/reference/troubleshooting.md index 8bc62a4c5..44a111c5d 100644 --- a/docs/reference/troubleshooting.md +++ b/docs/reference/troubleshooting.md @@ -44,6 +44,14 @@ Confirm the firmware is innocent before reaching for the soldering iron: [Diagno How the bench found this is in [lessons.md](../work/past/lessons.md). +## The lights stutter once a second + +At **logLevel** `Info` or above, the device writes a status line to its serial console once a second, from the loop that renders the frames. On an ESP32-S3 driving a large setup, that write stalls the loop long enough to see. Set **logLevel** on the System card to `Warn`: the line stops and the stutter with it. The first minute after boot always prints, since the installer reads the device's address from that line. + +## Free memory is low with the interface open + +The status bar's 🧠 is free internal RAM and 🧱 its largest block, the memory WiFi, the web server and the LED drivers share. A board with PSRAM keeps 100 KB free and a 50 KB block. A classic ESP32 without PSRAM runs with less. An open interface over a weak WiFi link takes more there, since what the board sends waits in that RAM until the browser acknowledges it. Ethernet, a better WiFi link or a closed preview gives it back. Where the RAM goes and what each board holds is in [performance.md](performance.md#internal-ram). + ## Colors are wrong everywhere Red where you expect green usually means the channel order does not match the strip. Set it on the driver: the [fixture profile](../moonmodules/light/supporting.md#fixtureprofiles) names which channel carries which color. diff --git a/docs/work/future/backlog-core.md b/docs/work/future/backlog-core.md index 5da703968..f2fdce1da 100644 --- a/docs/work/future/backlog-core.md +++ b/docs/work/future/backlog-core.md @@ -2,6 +2,14 @@ Forward-looking to-build items for the **core / infrastructure** domain (`src/core/`, `src/platform/`, build, CI, network, persistence, UI). The light-domain counterpart is [backlog-light.md](backlog-light.md); items that genuinely span both are in [backlog-mixed.md](backlog-mixed.md). Index + overview: [README.md](index.md). Completed items are removed. +### An update's connection that would not open until a reset (2026-10-22, from the internal RAM plan) + +On 3.0.0 a board could not open the update's connection (`ESP_ERR_HTTP_CONNECT`) until it was reset, with 80 KB of internal RAM free. TLS has lived in PSRAM since then (`CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC`), which may be what cured it; it has not recurred. Recorded so a recurrence is recognized: the free internal RAM and largest block at that moment are the first readings to take. + +### The HTTP server is audited for the two PHP request-handling bug classes (2026-10-21) + +PHP's critical advisory of 2026-10-21 (CVE-2026-109618, CVE-2026-109619) names two bug classes any C++ server can have. **Out of memory mid-request**: an allocation fails while a request is half built, and cleanup or a later step treats the partial state as valid. That path is live here: a classic ESP32 fell to 916 bytes free during a page load. **State outliving its request**: a reused connection slot (`wsClients_`, `streams_`) keeps a pointer, length or buffer from its previous client. The audit is a read-only pass over `HttpServerModule` and the platform TCP layer, every allocation-failure path and every slot reuse, and each finding is pinned by a desktop test that forces the failure. + ### Headers link the docs by page, not by host (2026-10-21, from the PR 150 review) Seventeen headers under `src/` carry absolute `https://moonmodules.org/MoonLight/...` links, so a docs move edits firmware sources and marks every firmware stale, as PR 150's move from `how-to/` to `reference/` did. A header names the page site-relative (`reference/hardware/panel-cards.md#...`) and `gen_help.py` resolves the host once, so the next docs move touches no source file. @@ -260,7 +268,7 @@ The "public member has no `///`" rule reads a declaration's shape with a regex a ## A capture device on Linux feeds the Picture service (robwomp, 2026-10-21) -The [Streams plan](../present/Plan-20261013%20-%20Streams,%20every%20frame%20from%20outside.md) brings capture to the ESP32: a USB grabber on the P4 ([#151](https://github.com/MoonModules/MoonLight/pull/151), [#152](https://github.com/MoonModules/MoonLight/pull/152)) and a MIPI-CSI camera. +The [Streams plan](../present/Plan-20261013%20-%20Streams%2C%20every%20frame%20from%20outside%20(partial).md) brings capture to the ESP32: a USB grabber on the P4 ([#151](https://github.com/MoonModules/MoonLight/pull/151), [#152](https://github.com/MoonModules/MoonLight/pull/152)) and a MIPI-CSI camera. On the desktop build a capture device is read through Video4Linux (V4L2), which covers USB grabbers and the HDMI and CSI inputs of boards such as Rockchip SBCs. It is one more source of the picture family: the platform layer reads frames, and the Picture service takes them as it takes every other source. robwomp has such boards and offered to test on Discord. @@ -335,7 +343,7 @@ declared rather than for a buffer to fill, and to time out on stall rather than - **Live RMII Ethernet reconfigure** — runtime PHY/pin config shipped (`ethType` + pin controls in NetworkModule, per-board defaults in `deviceModels.json`, `platform::setEthConfig`/`ethInit` dispatch). W5500 (SPI) on S3 applies **live** — `ethStop()` tears down the SPI bus and `ethInit()` re-runs on the next `loop1s()` with no reboot. RMII (classic/P4 internal EMAC) still saves config and asks for a restart to apply, because the EMAC bring-up is fiddlier to hot-cycle cleanly. Make RMII live too: a hot `esp_eth_stop` + EMAC/netif teardown + re-init on config change, matching the W5500 path, so every interface honours the no-reboot principle. - **GCC below 16 needs four warnings demoted, and nothing exercises those versions** - `-Wnull-dereference`, `-Wrestrict`, `-Wstringop-overflow` and `-Wformat-truncation` fire on provably correct code from GCC 12 through 15 (five of the twelve inside libstdc++ and glibc headers, unreachable from our source), so CMakeLists demotes them to non-fatal there and keeps them fatal on 16+. That unblocks CI and from-source builds on Debian and Raspberry Pi OS alike, but it is a suppression, not an understanding: nobody routinely compiles with 12-15, so a REAL instance of one of these on those versions is now a warning nobody reads. Revisit when the runner's default GCC reaches 16, at which point the whole block can be deleted. - **Installer UX polish** — clear "Pre-release (beta)" warning on RC/latest picks, yank-by-asset-tag instead of yank-by-release-deletion. -- **Offer MoonLight/MoonLight as a library**: a downstream sketch where another firmware/app consumes the light pipeline (or a subset) as an embeddable dependency rather than running the whole binary. `library.json` is already a PlatformIO *library* manifest, so the seed exists. When this is designed, give it a small public **identity surface**: one runtime constant the consumer reads (a `kProjectName`, likely a `ProjectInfo` bundle of name + version + url) that the network wire-strings (ArtNet/E1.31 source-name + CID), the UI banner, and any "About" string all *derive from*: the one place a consumer queries "what am I embedding." This is the genuine home for the name-centralisation that the rename ([the MoonLight plan](../past/plans/Plan-20260922%20-%20MoonLight,%20from%20v5.0.0%20to%20the%20rename%20(shipped).md)) deliberately *didn't* do: the rename is a one-time sweep (a constant would just split it), but a library consumer references the identity ongoing and widely, which is the test a constant must pass. Build it *then*, against the real library API, not speculatively now. +- **Offer MoonLight/MoonLight as a library**: a downstream sketch where another firmware/app consumes the light pipeline (or a subset) as an embeddable dependency rather than running the whole binary. `library.json` is already a PlatformIO *library* manifest, so the seed exists. When this is designed, give it a small public **identity surface**: one runtime constant the consumer reads (a `kProjectName`, likely a `ProjectInfo` bundle of name + version + url) that the network wire-strings (ArtNet/E1.31 source-name + CID), the UI banner, and any "About" string all *derive from*: the one place a consumer queries "what am I embedding." This is the genuine home for the name-centralisation that the rename ([the MoonLight plan](../past/plans/Plan-20260922%20-%20MoonLight%2C%20from%20v5.0.0%20to%20the%20rename%20(shipped).md)) deliberately *didn't* do: the rename is a one-time sweep (a constant would just split it), but a library consumer references the identity ongoing and widely, which is the test a constant must pass. Build it *then*, against the real library API, not speculatively now. - **HTTP: a request split across packets, or a body, that arrives a few ms late is dropped, intermittently (2026-08-20).** `handleConnection` runs SYNCHRONOUSLY inside `tick20ms`, so its waits are kept short to protect the render loop. A connection with no bytes yet is parked and served on a later tick (2026-10-09), but a request whose headers arrive in two parts gets **~5 ms** for the rest, and a body **~50 ms** (`empties > 50`). A client that misses either budget gets a request processed as truncated, or a `400 {"error":"incomplete request body"}`. Parking these too needs a buffer per parked connection. @@ -859,7 +867,7 @@ Today the eth-only build profile compiles WiFi out (`MM_NO_WIFI`). Turning WiFi **Hardware-limit tail (not covered by the pin check).** Pin-uniqueness rejects the common case but not the controller-count limit: the S3 has **2 I2S controllers** regardless of pins, so a 3rd mic on distinct pins passes the pin check yet fails `i2s_new_channel` at runtime. That tail is already handled — the platform I2S init returns false on failure (no panic, module stays `inited_=false`); verified live (4 pinned AudioModules → error spam, no crash). So scope = pin-uniqueness check + the existing graceful-degrade; don't try to make the pin check also model controller counts. -**Related:** the shipped "disabling releases resources" work (see [past plans](../past/plans/index.md)) — a disabled module freeing its pins is what lets the same GPIO be reassigned live without a conflict-reject. +**Related:** the shipped "disabling releases resources" work (see [past plans](../past/index.md)) — a disabled module freeing its pins is what lets the same GPIO be reassigned live without a conflict-reject. ### PinsModule — strict reject-on-add mode (the one remaining increment) diff --git a/docs/work/future/backlog-light.md b/docs/work/future/backlog-light.md index c7d52a2fb..a2f4a4e35 100644 --- a/docs/work/future/backlog-light.md +++ b/docs/work/future/backlog-light.md @@ -20,6 +20,10 @@ Forward-looking to-build items for the **light domain** (`src/light/`: drivers, ## Effects +### The ring's refill jitter (2026-10-22, from the internal RAM plan) + +LCD-MM's refill interrupt goes about 1.1 ms without firing almost every second, where it fires every 89 µs at 3-row slices, so `ringAuto` keeps 54 rows of lead (3 × 21 buffers, 37 KB on shiffy) where hpwit's S3 driver runs on 10. Measured with `ringDbg`'s `gap` (within a frame, `@` the slice it ended at), `ml` (the fewest slices of lead) and `c` (the interrupt's core): not the flash cache (`co` stays 0), not the core (registered on core 1 it was the same, and it broke the cache guard, so it stays on core 0), not `mmSnap` (removed), and it ends mid-frame as often as at the idle tail. The next measurement is the gap against the frame's own events, the arm and the snapshot, then other interrupts on core 0. A lead of 10 rows would free most of the ring's internal RAM. + ### A preview GIF for every MoonLive script (2026-10-19) The 140 compiled effects each show a GIF on their catalog card (`docs/assets/light/effects/.gif`, 25 MB in all); the 49 MoonLive effect scripts show none. @@ -896,7 +900,7 @@ The LED-driver increments **shipped**: increment 1 (RMT/WS2812B single-strand on **Why it waits.** It is fourteen effects' worth of change across audio-reactive and simulation families, each needing its own judgement about whether to clear, fade, or seed differently — not a mechanical sweep. `unit_Effects_gridsweep.cpp` already measures it (`afterFirst`) and asserts only the settled frame, so the number is visible without blocking. -- **A scripted modifier that animates every frame** (2026-10-11): `MoonLiveModifier` compiles a `modifyLive()` entry, so a script moves lights per frame with `t` (scroll, wobble, sweep) as the compiled Rotate does. Geometry only: dimming is a blend mode or an effect's `mulRGB`, per [the layer compositing plan](../present/Plan-20261011%20-%20Layer%20compositing,%20masks%20and%20transitions.md). Built once a script needs it; the cost to measure is one compiled-script call per light per frame, on an S3 at its largest grid. +- **A scripted modifier that animates every frame** (2026-10-11): `MoonLiveModifier` compiles a `modifyLive()` entry, so a script moves lights per frame with `t` (scroll, wobble, sweep) as the compiled Rotate does. Geometry only: dimming is a blend mode or an effect's `mulRGB`, per [the layer compositing plan](../present/Plan-20261011%20-%20Layer%20compositing%2C%20masks%20and%20transitions%20(partial).md). Built once a script needs it; the cost to measure is one compiled-script call per light per frame, on an S3 at its largest grid. - **A scripted modifier needs a way to drop a light** (2026-08-10). A coordinate slot is a byte, so a script that computes past 255 wraps: `shift.mlv` with a large `amount` lands lights back at the left edge instead of walking them off it. The Layer already drops an out-of-bounds position, but a script has no way to SAY out-of-bounds — every value it can write is a valid coordinate. Needs a sentinel the binding recognizes (or a wider coordinate slot), at which point the "walks off the edge" behavior a scroll modifier wants becomes expressible. - **A scripted layout loses its control values across a reboot** (2026-08-11). Persistence SAVES them correctly (`/.config/Layouts.json` holds `"0.width":64`), but on boot they are loaded into a control set that does not exist yet: a script's controls are created by `defineControls`, which can only publish what the ENGINE declared, and the engine does not compile until `prepare()` — Scheduler phase 4, after the phase-2b re-bind. So the loaded value has nowhere to land and the script's declared default wins; a saved 64×64 grid comes back 16×16. Verified on the P4 bench. diff --git a/docs/work/future/moonwalk.md b/docs/work/future/moonwalk.md index bbd5854d9..fe9fc622e 100644 --- a/docs/work/future/moonwalk.md +++ b/docs/work/future/moonwalk.md @@ -1,6 +1,6 @@ # MoonWalk -Step 2 of [Processes, described once](../present/Plan-20261015%20-%20Processes%2C%20described%20once.md): MoonWalk, the model every process is written in and the map that shows them, chosen against [the inventory](processes-inventory.md) and approved on 2026-10-16. The schema is [moonwalk/moonwalk.schema.json](../../../moonwalk/moonwalk.schema.json). +Step 2 of [Processes, described once](../present/Plan-20261015%20-%20Processes%2C%20described%20once%20(partial).md): MoonWalk, the model every process is written in and the map that shows them, chosen against [the inventory](processes-inventory.md) and approved on 2026-10-16. The schema is [moonwalk/moonwalk.schema.json](../../../moonwalk/moonwalk.schema.json). ## What it must do diff --git a/docs/work/future/processes-inventory.md b/docs/work/future/processes-inventory.md index 4949c2fa6..9e2e6693c 100644 --- a/docs/work/future/processes-inventory.md +++ b/docs/work/future/processes-inventory.md @@ -1,6 +1,6 @@ # Processes inventory -Step 1 of [Processes, described once](../present/Plan-20261015%20-%20Processes%2C%20described%20once.md), the Extract phase: every process MoonLight has, recorded by its content rather than the format it is stored in, counted on 2026-10-16 by three read-only Researchers. Part 1 is the end user's journey, Part 2 the contributor's, product owner's, agents' and CI's, and Part 3 the executable step lists: the scenarios and the MoonTube clips. +Step 1 of [Processes, described once](../present/Plan-20261015%20-%20Processes%2C%20described%20once%20(partial).md), the Extract phase: every process MoonLight has, recorded by its content rather than the format it is stored in, counted on 2026-10-16 by three read-only Researchers. Part 1 is the end user's journey, Part 2 the contributor's, product owner's, agents' and CI's, and Part 3 the executable step lists: the scenarios and the MoonTube clips. ## What it found diff --git a/docs/work/past/index.md b/docs/work/past/index.md index 8117330f1..50bb3cc66 100644 --- a/docs/work/past/index.md +++ b/docs/work/past/index.md @@ -4,7 +4,7 @@ Shipped plans and dated records. **A shipped plan is a working document, not testimony.** Whatever its PR already carries is duplication and can be trimmed or deleted. What earns keeping is the part no PR and no commit holds: the alternatives considered and rejected, and the reason the chosen shape won. -`plans/` holds the plans that shipped before the current rule, when a plan was archived rather than folded into its PR. They map onto merged PRs, and the mapping is the next thing to write here. From now on a plan ends at its PR ([CLAUDE.md, Branch](../../../CLAUDE.md#main-and-branch)), so this folder stops growing. +`plans/` holds every plan that left `present`, each named with its status, `(shipped)`, `(superseded)`, `(cancelled)` or `(attempted, reverted)`. A plan moves here in its own PR ([CLAUDE.md, Branch](../../../CLAUDE.md#main-and-branch)). The prior-version inventories beside it state what was true at a moment and stay unrewritten. Release notes live on [the GitHub release](https://github.com/MoonModules/MoonLight/releases) itself, which is their one home: a copy here drifts from what shipped, and the links in one pinned to an older tag went stale exactly that way. @@ -15,31 +15,3 @@ The prior-version inventories beside it state what was true at a moment and stay The surveys of v1, v2 and MoonLight that once sat here are gone. They were written to decide what to harvest into v3, that decision shipped, and each described code that lives in its own repository and in git. The LED-driver and shift-register design analyses moved to [future](../future/index.md), where the other research documents sit: they were written to decide what to build, and that is what future holds. - -## Plans and their PRs - -21 plans, each mapped to the merged pull request that carries the same design plus the diff that implemented it. The mapping is derived from title words and merge date, so it is a starting point rather than a record: a `?` means no confident match, and any row is worth checking before it is trusted. 4 need a hand. - -| Date | Plan | PR | -|---|---|---| -| 2026-08-13 | [MoonLive on a stack machine: the frame is where values live](plans/Plan-20260813%20-%20MoonLive%20on%20a%20stack%20machine%20%E2%80%94%20the%20frame%20is%20where%20values%20live%20%28shipped%29.md) | [#63](https://github.com/MoonModules/MoonLight/pull/63) | -| 2026-08-17 | [MoonLive scripts are classes](plans/Plan-20260817%20-%20MoonLive%20scripts%20are%20classes%20%28shipped%29.md) | [#65](https://github.com/MoonModules/MoonLight/pull/65) | -| 2026-08-18 | [A file editor control and a filesystem change seam](plans/Plan-20260818%20-%20A%20file%20editor%20control%20and%20a%20filesystem%20change%20seam%20%28shipped%29.md) | [#67](https://github.com/MoonModules/MoonLight/pull/67) | -| 2026-08-21 | [MoonLive on Windows](plans/Plan-20260821%20-%20MoonLive%20on%20Windows%20%28x86_64%20host-JIT%20backend%29%20%28shipped%29.md) | [#73](https://github.com/MoonModules/MoonLight/pull/73) | -| 2026-08-21 | [Particles in MoonLive, plus a fade builtin](plans/Plan-20260821%20-%20Particles%20in%20MoonLive%2C%20plus%20a%20fade%20builtin%20%28shipped%29.md) | [#71](https://github.com/MoonModules/MoonLight/pull/71) | -| 2026-08-22 | [Raw L2 Ethernet on Windows](plans/Plan-20260822%20-%20Raw%20L2%20Ethernet%20on%20Windows%20%28shipped%29.md) | [#73](https://github.com/MoonModules/MoonLight/pull/73) | -| 2026-08-23 | [A Windows installer, and settings that persist](plans/Plan-20260823%20-%20A%20Windows%20installer%2C%20and%20settings%20that%20persist%20%28shipped%29.md) | [#74](https://github.com/MoonModules/MoonLight/pull/74) | -| 2026-08-23 | [Five types for MoonLive scripts](plans/Plan-20260823%20-%20Five%20types%20for%20MoonLive%20scripts%20%28shipped%29.md) | [#77](https://github.com/MoonModules/MoonLight/pull/77) | -| 2026-08-24 | [NDI output](plans/Plan-20260824%20-%20NDI%20output%20%28shipped%29.md) | [#80](https://github.com/MoonModules/MoonLight/pull/80) | -| 2026-08-25 | [A lossy channel for the preview](plans/Plan-20260825%20-%20A%20lossy%20channel%20for%20the%20preview%20%28shipped%29.md) | [#81](https://github.com/MoonModules/MoonLight/pull/81) | -| 2026-08-25 | [Client-driven preview adaptation](plans/Plan-20260825%20-%20Client-driven%20preview%20adaptation%20%28superseded%29.md) | [#81](https://github.com/MoonModules/MoonLight/pull/81) | -| 2026-08-25 | [Lean preview transport](plans/Plan-20260825%20-%20Lean%20preview%20transport%20%28shipped%29.md) | [#81](https://github.com/MoonModules/MoonLight/pull/81) | -| 2026-08-27 | [HLS on ESP32-P4](plans/Plan-20260827%20-%20HLS%20on%20ESP32-P4%20%28shipped%29.md) | [#78](https://github.com/MoonModules/MoonLight/pull/78) | -| 2026-08-27 | [HLS streaming driver](plans/Plan-20260827%20-%20HLS%20streaming%20driver%20%28shipped%29.md) | [#85](https://github.com/MoonModules/MoonLight/pull/85) | -| 2026-08-27 | [Raw-L2 interface dropdown](plans/Plan-20260827%20-%20Raw-L2%20interface%20dropdown%20%28shipped%29.md) | [#85](https://github.com/MoonModules/MoonLight/pull/85) | -| 2026-08-27 | [Sprites and flying toasters](plans/Plan-20260827%20-%20Sprites%20and%20flying%20toasters%20%28shipped%29.md) | [#86](https://github.com/MoonModules/MoonLight/pull/86) | -| 2026-08-30 | [Ship the MoonLive script library](plans/Plan-20260830%20-%20Ship%20the%20MoonLive%20script%20library%20%28shipped%29.md) | [#89](https://github.com/MoonModules/MoonLight/pull/89) | -| 2026-08-31 | [Scripts declare dimensions and tags](plans/Plan-20260831%20-%20Scripts%20declare%20dimensions%20and%20tags%20%28shipped%29.md) | [#89](https://github.com/MoonModules/MoonLight/pull/89) | -| 2026-09-22 | [MoonLight, from v5.0.0 to the rename](plans/Plan-20260922%20-%20MoonLight%2C%20from%20v5.0.0%20to%20the%20rename%20%28shipped%29.md) | [#119](https://github.com/MoonModules/MoonLight/pull/119) | -| 2026-09-22 | [RTSP video out on the P4 and the desktop](plans/Plan-20260922%20-%20RTSP%20video%20out%20on%20the%20P4%20and%20the%20desktop%20%28shipped%29.md) | [#110](https://github.com/MoonModules/MoonLight/pull/110) | -| 2026-10-02 | [One way to send to many receivers](plans/Plan-20261002%20-%20One%20way%20to%20send%20to%20many%20receivers%20%28shipped%29.md) | [#123](https://github.com/MoonModules/MoonLight/pull/123) | diff --git a/docs/work/past/lessons.md b/docs/work/past/lessons.md index 16cd0a1f6..95fae7669 100644 --- a/docs/work/past/lessons.md +++ b/docs/work/past/lessons.md @@ -3,7 +3,7 @@ Hard-won debugging lessons and gotchas, recorded with the code that proved them. The PR-merge *carry-forward* gate writes new entries here (CLAUDE.md § Lifecycle Events). This is the **lesson** record: a bug, its root cause, and the fix. Two neighbours hold the other genres, and a lesson belongs in whichever fits: - A durable **rule** that graduated from a lesson lives in [CLAUDE.md](../../../CLAUDE.md) or [coding-standards.md](../../contributing/coding-standards.md), not here. -- The forward-looking **design intent** of a feature is a [plan](plans/index.md). +- The forward-looking **design intent** of a feature is a [plan](index.md). Entries run oldest-first by branch. A lesson fully absorbed into a rule doc or the code is pruned (per *Mandatory subtraction*): the git history is the permanent record, this file is the working narrative on top. @@ -533,3 +533,14 @@ Two tests were written and both passed against the unfixed code before one final Three weeks without a Windows run left three build breaks waiting, while CI's `build-windows-x64` job was green on the same commit. They were not one bug: `if (anyKnownChip)` and `if (i2sLanes > 0)` are compile-time constants that **this** MSVC (19.44.35227) reports as C4127 and the toolset `windows-latest` pins does not, and `/WX` turns either into C2220. A non-ASCII literal raised C4566, which is not a toolset question at all but the system code page: MSVC alone assumes it for both the source and execution charset, so `/utf-8` settles it the way GCC and clang already behave. On the Python side the same code page is the default for `read_text`, `write_text`, `open` and `subprocess(text=True)`, and `PYTHONIOENCODING` masks only stdio, so a UTF-8 doc read crashes a script whose output looked fine. **The rule: a platform in CI is one toolchain on that platform, not the platform.** `windows-latest` is a moving pin, so green there says the code built with that toolset on that runner image, which is a weaker claim than it reads as. `if constexpr` is the textbook answer to C4127 and says what was meant anyway; explicit `encoding="utf-8"` is the answer everywhere Python touches a file. The second half of the lesson is cheaper than the first: the drift is silent and the cost scales with how long the platform goes unexercised, so the fix is to run it, not to reason about it. + +## Lessons from the internal RAM plan (2026-10-22) + +- **Measure at the low point, not on average.** Free RAM read once a second said 70 KB on a classic board whose page load dropped it to 3.6 KB; a task sampling every tick and recording the heap's block sizes at each new low named the owner (1,536-byte TCP segments) in one run. +- **A block size is a fingerprint.** 32 KB blocks were ESP-IDF's DMA reserve pool, 1,536 bytes a TCP segment, 1,664 WiFi's receive buffers, 4,096 the ring's DMA nodes; matching sizes to owners attributed most of the heap without a tracer. +- **The obvious owner can be wrong.** Thirty-two queued packets matched WiFi's transmit queue of 32 exactly, and halving that queue changed nothing: they were TCP's. +- **A tick figure is wall-clock.** Control read 1.4 ms per tick because a helper task on its core preempted it; its own work was 150 µs. The helper itself, meant to halve the ring's prime, cost 12 fps by preempting the effects. +- **A guard that holds on one core can fail on the other.** A cache-safe interrupt's cache-off check held on the core doing flash writes; moved to the other core, the cache went off mid-refill and the board panicked. Whatever such an interrupt reads must be internal, whichever core it runs on. +- **Check the compiler's output before optimizing the algorithm.** Removing branches and a whole wasted transpose gained nothing measurable; the disassembly showed the store loop's variable shifts costing more than the transpose, and unrolling them gained 37%. +- **A depth that runs clean is the measurement, not free RAM.** `ringAuto` took 28 buffers because RAM allowed it; 21 smaller ones run clean, and the RAM went back to WiFi and the UI. +- **Two fix attempts, then stop.** The ring's 1.1 ms of jitter survived two; it is in the backlog with its measurements rather than chased further. diff --git a/docs/work/present/Plan-20260915 - Native HUB75 output (shipped).md b/docs/work/past/plans/Plan-20260915 - Native HUB75 output (shipped).md similarity index 78% rename from docs/work/present/Plan-20260915 - Native HUB75 output (shipped).md rename to docs/work/past/plans/Plan-20260915 - Native HUB75 output (shipped).md index 97e07a006..a5ed92f31 100644 --- a/docs/work/present/Plan-20260915 - Native HUB75 output (shipped).md +++ b/docs/work/past/plans/Plan-20260915 - Native HUB75 output (shipped).md @@ -4,7 +4,7 @@ Answers [issue #102](https://github.com/MoonModules/MoonLight/issues/102). Phase ## The gap -MoonLight drives HUB75 panels one way: [PanelCardDriver](../../../src/light/drivers/PanelCardDriver.h) emits ColorLight frames over raw Ethernet to a 5A-75B/E receiving card, which does the HUB75 driving. That path is excellent above roughly 16,384 pixels and it is what the [panel-cards guide](../../reference/hardware/panel-cards.md) documents. +MoonLight drives HUB75 panels one way: [PanelCardDriver](../../../../src/light/drivers/PanelCardDriver.h) emits ColorLight frames over raw Ethernet to a 5A-75B/E receiving card, which does the HUB75 driving. That path is excellent above roughly 16,384 pixels and it is what the [panel-cards guide](../../../reference/hardware/panel-cards.md) documents. Below that size it is the wrong shape. The user buys a receiving card, configures it with LEDvision (a Windows tool, not ours), and runs a dedicated Ethernet link, all to light one 64×64 panel that a $10 board could drive from its own pins. WLED does this natively; MoonLight does not. That is the gap, and it is the whole of Phase 1. @@ -18,7 +18,7 @@ frame bytes = (height / 2) × width × bitDepth Measured against the peripherals MoonLight already has: -Measured from the encoder's own formula (`Hub75Geometry::frameBytes`, pinned by [unit_Hub75Slots.cpp](../../../test/unit/light/unit_Hub75Slots.cpp)), at 1/32 scan: +Measured from the encoder's own formula (`Hub75Geometry::frameBytes`, pinned by [unit_Hub75Slots.cpp](../../../../test/unit/light/unit_Hub75Slots.cpp)), at 1/32 scan: | Panels | Geometry | 4-bit | 6-bit | Parlio (65,535 B cap) | LCD_CAM (internal RAM) | |---|---|---:|---:|:--|:--:| @@ -28,9 +28,9 @@ Measured from the encoder's own formula (`Hub75Geometry::frameBytes`, pinned by Two bytes a slot, since the bus word is 16 bits. **Four panels miss the Parlio cap even at 4-bit, by one byte**, so four panels are an LCD_CAM job at any depth, and Parlio carries one panel at up to 6-bit. The encoder offers 2 to 6 bits; 8-bit waits on repeating the upper planes (backlog). -**This is the fact the issue does not account for.** #102 proposes the P4-Nano as lead candidate because Espressif's reference uses PARLIO, but [platform_esp32_parlio.cpp:238](../../../src/platform/esp32/platform_esp32_parlio.cpp) records the hardware cap: `kParlioMaxTransferBytes = 0x7FFFF / 8` = 65,535 bytes, width-invariant. PARLIO carries one panel comfortably, reaches its limit at four, and cannot do sixteen at any useful depth. +**This is the fact the issue does not account for.** #102 proposes the P4-Nano as lead candidate because Espressif's reference uses PARLIO, but [platform_esp32_parlio.cpp:238](../../../../src/platform/esp32/platform_esp32_parlio.cpp) records the hardware cap: `kParlioMaxTransferBytes = 0x7FFFF / 8` = 65,535 bytes, width-invariant. PARLIO carries one panel comfortably, reaches its limit at four, and cannot do sixteen at any useful depth. -The route that scales is the one the [shift-register analysis](../future/shift-register-driver-analysis.md) already established for a structurally identical problem: **i80/LCD_CAM, whose DMA reaches PSRAM**, proven on this exact path to 16,384 lights. `hasLcdCam` is true on S3, P4 and S31. +The route that scales is the one the [shift-register analysis](../../future/shift-register-driver-analysis.md) already established for a structurally identical problem: **i80/LCD_CAM, whose DMA reaches PSRAM**, proven on this exact path to 16,384 lights. `hasLcdCam` is true on S3, P4 and S31. ### GPIO requirements @@ -43,21 +43,21 @@ HUB75 is pin-hungry in a way the memory table hides. One port needs: | Control | `clk lat oe` | 3 | | | **Total** | **12-14** | -Against the per-chip free sets in [gpio-usage.md](../../reference/hardware/gpio-usage.md), which exclude flash/PSRAM, input-only, USB, UART0 and strapping pins: +Against the per-chip free sets in [gpio-usage.md](../../../reference/hardware/gpio-usage.md), which exclude flash/PSRAM, input-only, USB, UART0 and strapping pins: | Chip | Usable output GPIOs | 1/16 scan (13 pins) | 1/32 scan (14 pins) | |---|---:|:--:|:--:| | **ESP32-S3** (N16R8) | 16 | ✅ 3 spare | ✅ 2 spare | | **ESP32-P4** (P4-NANO) | 20 | ✅ 7 spare | ✅ 6 spare | -| **ESP32-S31** | board-specific | take from the [coreboard reference](../../reference/hardware/esp32-s31-coreboard.md) | | +| **ESP32-S31** | board-specific | take from the [coreboard reference](../../../reference/hardware/esp32-s31-coreboard.md) | | | **ESP32 classic** | 13 | ❌ | ❌ | **This settles the target list, and it removes a chip the issue implies.** The classic ESP32 has 13 usable output GPIOs and would need every one for a 1/16-scan port, leaving nothing for the LED strand, a button or a mic; a 1/32 panel does not fit at all. gpio-usage.md already records the same squeeze for 16-lane i80 ("a 16-lane set must borrow strap pins, and GPIO 12 is the flash-voltage strap: driving it at reset can brick the boot"). So HUB75 is an **S3, P4 and S31 feature**, and the classic is out on pins before memory is even considered. Two consequences for the plan: -- **The board catalog entries in step 5 are load-bearing, not polish.** An S3 has exactly 3 spare pins at 1/16 scan, so a user picking 13 by hand will land on octal-PSRAM (33-37), USB (19-20), UART0 (43-44) or a strap (0, 3, 45, 46). [PinsModule](../../moonmodules/core/system.md) flags a conflict after the fact; a catalog entry prevents it. -- **Defaults stay unset.** Even with the pins available, a default would guess the user's wiring. Same rule the LED drivers follow, and the P4's own history is the argument: its first LED-driver default landed on strapping pins ([lessons.md](../past/lessons.md)). +- **The board catalog entries in step 5 are load-bearing, not polish.** An S3 has exactly 3 spare pins at 1/16 scan, so a user picking 13 by hand will land on octal-PSRAM (33-37), USB (19-20), UART0 (43-44) or a strap (0, 3, 45, 46). [PinsModule](../../../moonmodules/core/system.md) flags a conflict after the fact; a catalog entry prevents it. +- **Defaults stay unset.** Even with the pins available, a default would guess the user's wiring. Same rule the LED drivers follow, and the P4's own history is the argument: its first LED-driver default landed on strapping pins ([lessons.md](../lessons.md)). ### Refresh @@ -69,7 +69,7 @@ frame time = (height / 2) x bitDepth x line time refresh = 1 / frame time ``` -Predicted at a 20 MHz shift clock (the rate the i80 path already runs, per the [shift-register analysis](../future/shift-register-driver-analysis.md)), ignoring latch overhead, so these are ceilings rather than promises: +Predicted at a 20 MHz shift clock (the rate the i80 path already runs, per the [shift-register analysis](../../future/shift-register-driver-analysis.md)), ignoring latch overhead, so these are ceilings rather than promises: | Geometry | 4-bit | 6-bit | 8-bit | |---|---:|---:|---:| @@ -85,11 +85,11 @@ Predicted at a 20 MHz shift clock (the rate the i80 path already runs, per the [ ### The driver -`Hub75Driver : DriverBase`, registered in [main.cpp](../../../src/main.cpp) beside the others, gated on `platform::hasLcdCam || platform::parlioLanes` so it is offered only where it can run — the same inert-on-wrong-chip rule [ParlioLedDriver](../../../src/light/drivers/ParlioLedDriver.h) follows. That gate happens to exclude the classic ESP32, which § GPIO requirements shows could not host a port anyway. +`Hub75Driver : DriverBase`, registered in [main.cpp](../../../../src/main.cpp) beside the others, gated on `platform::hasLcdCam || platform::parlioLanes` so it is offered only where it can run — the same inert-on-wrong-chip rule [ParlioLedDriver](../../../../src/light/drivers/ParlioLedDriver.h) follows. That gate happens to exclude the classic ESP32, which § GPIO requirements shows could not host a port anyway. It is **not** a `LedPeripheral` backend. That seam is shaped around the parallel WS2812 orchestrator (`supportsPinExpander`, `lanesAvailable`, slot encoding), and HUB75 shares none of it: different wire protocol, different buffer shape, different control set. Reusing it would mean a base class whose every method one subclass ignores, which is the abstraction CLAUDE.md's minimalism rule exists to prevent. What they legitimately share is the **platform DMA seam**, and that is where the sharing belongs. -Controls, all defaulting to unset because a soldered pin must never be guessed ([lessons.md](../past/lessons.md)): +Controls, all defaulting to unset because a soldered pin must never be guessed ([lessons.md](../lessons.md)): | Control | Why | |---|---| @@ -101,7 +101,7 @@ Controls, all defaulting to unset because a soldered pin must never be guessed ( | `sPWM` | The [WLED-MM sPWM commit](https://github.com/MoonModules/WLED-MM/commit/778ae558) case | | `peripheral` | Which silicon block drives the panel, where the chip has more than one | -**Geometry is not a driver control.** [PanelLayout](../../../src/light/layouts/PanelLayout.h) and [PanelsLayout](../../../src/light/layouts/PanelsLayout.h) already model a serpentine panel and an M×N tiling of them, with configurable axis order and per-axis direction. A layout emits coordinates and the driver owns pins: that separation is the architecture, and HUB75 must not re-litigate it. The issue asks for "UI features to set panel dimensions and arrangement", and the honest answer is that MoonLight already has them. +**Geometry is not a driver control.** [PanelLayout](../../../../src/light/layouts/PanelLayout.h) and [PanelsLayout](../../../../src/light/layouts/PanelsLayout.h) already model a serpentine panel and an M×N tiling of them, with configurable axis order and per-axis direction. A layout emits coordinates and the driver owns pins: that separation is the architecture, and HUB75 must not re-litigate it. The issue asks for "UI features to set panel dimensions and arrangement", and the honest answer is that MoonLight already has them. ### The platform seam @@ -115,7 +115,7 @@ bool hub75Transmit(Hub75Handle& h, uint8_t buffer); void hub75Deinit(Hub75Handle& h); ``` -Two backends behind it, and **the user picks which**, through a `peripheral` select exactly like the one [ParallelLedDriver](../../../src/light/drivers/ParallelLedDriver.h) already carries. Each backend self-registers at static init, and `main.cpp` links it only where its SOC macro says the silicon exists, so the dropdown's options are per-chip rather than a fixed list: LCD_CAM on an S3 or S31, LCD_CAM and PARLIO on a P4, nothing on a classic. +Two backends behind it, and **the user picks which**, through a `peripheral` select exactly like the one [ParallelLedDriver](../../../../src/light/drivers/ParallelLedDriver.h) already carries. Each backend self-registers at static init, and `main.cpp` links it only where its SOC macro says the silicon exists, so the dropdown's options are per-chip rather than a fixed list: LCD_CAM on an S3 or S31, LCD_CAM and PARLIO on a P4, nothing on a classic. **An earlier draft of this plan had the platform choose, on memory alone. That was wrong, and the reason is contention rather than capacity.** A P4 has both peripherals and only one of each. A user driving WS2812 strips from PARLIO needs HUB75 on LCD_CAM; another user wants the reverse. Both are correct, the difference is what else is plugged into that board, and the platform cannot know it. The sibling claim guard stops two drivers colliding on one block, but it cannot guess which driver should win — that is the user's call, and a select is how the repo already asks it. @@ -127,7 +127,7 @@ A desktop stub returning false from everything keeps `mm_tests` and every non-HU The one genuinely new piece: rendered RGB → bit-plane-major scan buffer. Per bit plane, per scan row, pack six colour bits per pixel-column into the parallel word alongside the row address and the latch/OE timing. -This is the same class of work as [ParallelSlots.h](../../../src/light/drivers/ParallelSlots.h) (serialise a frame through a parallel bus under a DMA ceiling) and is testable the same way: a host unit test that walks a known pattern through the encoder and asserts the exact bytes, including a walking-one per colour line. The '595 investigation's hardest lesson was that an encoder proven byte-for-byte on the host still left a wall-visible artifact, so the test pins the encoder and hardware verification stays the PO's. +This is the same class of work as [ParallelSlots.h](../../../../src/light/drivers/ParallelSlots.h) (serialise a frame through a parallel bus under a DMA ceiling) and is testable the same way: a host unit test that walks a known pattern through the encoder and asserts the exact bytes, including a walking-one per colour line. The '595 investigation's hardest lesson was that an encoder proven byte-for-byte on the host still left a wall-visible artifact, so the test pins the encoder and hardware verification stays the PO's. ## End states considered @@ -146,7 +146,7 @@ This is the same class of work as [ParallelSlots.h](../../../src/light/drivers/P 3. **`Hub75Driver`.** Controls, pin claims through the existing `PinsModule` registry, status line reporting geometry and achieved refresh, registration in `main.cpp`. 4. **A backend registry, then the PARLIO backend.** The seam built in step 1 picks its peripheral internally; this turns it into an interface the driver selects, mirroring `LedPeripheral` without reusing it (that one is WS2812-shaped — same pattern, own interface). Then PARLIO behind it, and the `peripheral` select on the card. This REVISES step 1 rather than only adding to it. 5. **Board catalog entries**: Adafruit S3 Portal, Waveshare RGB Matrix, Lilygo S3, ESP32-S3-16MB-PSRAM. Catalog data in `deviceModels.json`, not driver work, and it is what makes the feature findable. -6. **Docs**: a how-to beside [panel-cards.md](../../reference/hardware/panel-cards.md), the driver's catalog card, and a line in panel-cards.md saying which of the two paths a given wall wants. +6. **Docs**: a how-to beside [panel-cards.md](../../../reference/hardware/panel-cards.md), the driver's catalog card, and a line in panel-cards.md saying which of the two paths a given wall wants. Steps 1-3 are the shippable unit. Steps 4-6 each stand alone. @@ -178,7 +178,7 @@ The PO's MoonHub75 (a LILYGO T7-S3 on the passive adapter) drove a 64×64 1/32-s ## Risks - **Refresh at depth may disappoint on 16 panels.** The encoder cost is linear in bit planes and the scan is serial. **Settled by publishing the prediction, not by measuring first**: § Refresh computes the ceiling per geometry and depth, the driver reports what it actually achieves, and the docs name PanelCard as the path above whatever that turns out to be. A user meets the limit with an explanation rather than a surprise. -- **The [48×256 white-flash](../future/backlog-light.md) is open on the i80 path.** Six theories ruled out, cause unfound. A HUB75 driver rides the same peripheral, so it may inherit it. **Settled by shipping and watching**: Phase 1 targets 1-4 panels, a size at which the flash has never been seen, so it does not block. If a tester on a large HUB75 wall reports it, that is the seventh theory the backlog entry is missing — the flash following a non-WS2812 encoder through the same peripheral would exonerate the WS2812 slot encoding, which six hardware tests could not do. +- **The [48×256 white-flash](../../future/backlog-light.md) is open on the i80 path.** Six theories ruled out, cause unfound. A HUB75 driver rides the same peripheral, so it may inherit it. **Settled by shipping and watching**: Phase 1 targets 1-4 panels, a size at which the flash has never been seen, so it does not block. If a tester on a large HUB75 wall reports it, that is the seventh theory the backlog entry is missing — the flash following a non-WS2812 encoder through the same peripheral would exonerate the WS2812 slot encoding, which six hardware tests could not do. - **Pin count is the tightest constraint on the classic ESP32 and shapes the target list.** See § GPIO requirements: the classic cannot host a full port without a strap pin, so it is excluded rather than half-supported. ## What this plan does not do diff --git a/docs/work/present/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped, partly verified).md b/docs/work/past/plans/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped).md similarity index 98% rename from docs/work/present/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped, partly verified).md rename to docs/work/past/plans/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped).md index 9811b275c..7943462fd 100644 --- a/docs/work/present/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped, partly verified).md +++ b/docs/work/past/plans/Plan-20261003 - Ethernet, WiFi and access point as Network submodules (shipped).md @@ -150,7 +150,7 @@ Waiting for more devices and a cable: ## Out of scope at first -The advanced settings users will ask for sooner or later are listed in the backlog's [advanced network settings](../future/backlog-core.md) item: roaming, enterprise WiFi, pinning to one access point, power save, the country code, the access point's address, and IPv6 beyond link-local. +The advanced settings users will ask for sooner or later are listed in the backlog's [advanced network settings](../../future/backlog-core.md) item: roaming, enterprise WiFi, pinning to one access point, power save, the country code, the access point's address, and IPv6 beyond link-local. ## Decided diff --git a/docs/work/present/Plan-20261005 - The audio codec as runtime configuration (in PR 129, M3 unverified).md b/docs/work/past/plans/Plan-20261005 - The audio codec as runtime configuration (shipped).md similarity index 100% rename from docs/work/present/Plan-20261005 - The audio codec as runtime configuration (in PR 129, M3 unverified).md rename to docs/work/past/plans/Plan-20261005 - The audio codec as runtime configuration (shipped).md diff --git a/docs/work/present/Plan-20261009 - Safe mode and factory reset.md b/docs/work/past/plans/Plan-20261009 - Safe mode and factory reset (shipped).md similarity index 100% rename from docs/work/present/Plan-20261009 - Safe mode and factory reset.md rename to docs/work/past/plans/Plan-20261009 - Safe mode and factory reset (shipped).md diff --git a/docs/work/present/Plan-20261011 - MoonLive Coord3D and RGB types.md b/docs/work/past/plans/Plan-20261011 - MoonLive Coord3D and RGB types (shipped).md similarity index 89% rename from docs/work/present/Plan-20261011 - MoonLive Coord3D and RGB types.md rename to docs/work/past/plans/Plan-20261011 - MoonLive Coord3D and RGB types (shipped).md index 4553a6d5e..31dd9f3f3 100644 --- a/docs/work/present/Plan-20261011 - MoonLive Coord3D and RGB types.md +++ b/docs/work/past/plans/Plan-20261011 - MoonLive Coord3D and RGB types (shipped).md @@ -1,10 +1,10 @@ # Plan: MoonLive `Coord3D` and `RGB` types -Most of what a script handles is a position or a color, and today both are loose integers: a coordinate is three values or an index the script computes by hand, and a color is three bytes that cannot travel together. This plan adds the two predefined types the [language roadmap](../future/backlog-moonlive.md) names as item 4b, so a script reads as its compiled counterpart does: `addRGB(i, getRGB(j))`. It lands now because every script written before it is a script to revisit after it. +Most of what a script handles is a position or a color, and today both are loose integers: a coordinate is three values or an index the script computes by hand, and a color is three bytes that cannot travel together. This plan adds the two predefined types the [language roadmap](../../future/backlog-moonlive.md) names as item 4b, so a script reads as its compiled counterpart does: `addRGB(i, getRGB(j))`. It lands now because every script written before it is a script to revisit after it. ## The rule it follows -The vocabulary follows the compiled modules ([MoonLive architecture](../../explanation/architecture/moonlive.md)): `Coord3D` and `RGB` are the C++ types of the same names, and every builtin taking or returning one is a one-line call into the `draw` function that does the same for a compiled effect. +The vocabulary follows the compiled modules ([MoonLive architecture](../../../explanation/architecture/moonlive.md)): `Coord3D` and `RGB` are the C++ types of the same names, and every builtin taking or returning one is a one-line call into the `draw` function that does the same for a compiled effect. ## The design: a struct is a named fixed-size array diff --git a/docs/work/present/Plan-20261013 - State by subscription, streamed.md b/docs/work/past/plans/Plan-20261013 - State by subscription, streamed (shipped).md similarity index 100% rename from docs/work/present/Plan-20261013 - State by subscription, streamed.md rename to docs/work/past/plans/Plan-20261013 - State by subscription, streamed (shipped).md diff --git a/docs/work/present/Plan-20261014 - Help in the UI, from the docs.md b/docs/work/past/plans/Plan-20261014 - Help in the UI, from the docs (shipped).md similarity index 92% rename from docs/work/present/Plan-20261014 - Help in the UI, from the docs.md rename to docs/work/past/plans/Plan-20261014 - Help in the UI, from the docs (shipped).md index b7a5d9a59..7a19327d7 100644 --- a/docs/work/present/Plan-20261014 - Help in the UI, from the docs.md +++ b/docs/work/past/plans/Plan-20261014 - Help in the UI, from the docs (shipped).md @@ -114,7 +114,7 @@ The device serves the UI; the **browser** fetches the help whenever it has inter - **Without internet**, as on a device's own access point, it uses the cache; with no cache either, the cards show as they do today. - **Help per module** opens from the card's title in its levels: the line, then the summary, the picture and the controls, then the details, then the links to the source and the tests. - **Help per control** is Material's helper text, one persistent line under the field, never a hover-only tooltip, which Nielsen Norman Group's guidelines warn hides what people need and fails on touch. The pilot compared it with Pickering's ⓘ toggletip and kept the line, since the status bar's ⓘ already shows or hides all help at once. A control with more opens it from its line, behind the module help's arrow. -- **Off until asked**: the ⓘ in the status bar turns help on, and the first time it asks before downloading, since the request shows GitHub, which hosts the docs site, the browser's address. Privacy by default, GDPR article 25's term; nothing is fetched before the yes. The [privacy policy](../../legal/privacy-policy.md) says so. +- **Off until asked**: the ⓘ in the status bar turns help on, and the first time it asks before downloading, since the request shows GitHub, which hosts the docs site, the browser's address. Privacy by default, GDPR article 25's term; nothing is fetched before the yes. The [privacy policy](../../../legal/privacy-policy.md) says so. ## Steps @@ -125,13 +125,13 @@ The change is gradual, the textbook strangler fig: the new path grows module by 3. ✅ **Layouts, the second catalog.** All seventeen layouts carry their help in their headers, and `layouts.md` is generated from `help.json`: its intro and clip from `LayoutBase` through `@video`, its cards from the headers, and its details from `MoonLiveLayout`'s `@guide`. The layouts credit what they draw on, Boustrophedon for serpentine wiring, polar coordinates for the rings and a conical spiral, through the attribution catalog. A layout is built from others as objects again, its public settings set and placed through `LightEmitter::place`, the relay the Layouts container uses for its children: CarLights and Rings241 from rings, which gain a center, Tubes from columns, Panels from panels and GridBlacks from a grid. Done in 436f37e9. 4. ✅ **Layouts composed in the tree.** A layout built from other layouts in code, as CarLights is from rings, has a runtime twin: the Layouts container placing each child at a position, so a user builds a car's lights from Ring children without code. To analyze: whether the position is the child's own control, such as a ring's center, or the container's per child, and how the bounding box and the light numbering follow. Done in 436f37e9: the position is the child's own control, Ring's `centerX` and `centerY`, so a user places rings side by side under the Layouts container. 5. ✅ **Effects, the third catalog.** All seventy effects carry their help in their headers, and `effects.md` is generated from `help.json`: its intro from `EffectBase`, its cards from the headers, and its details from each effect's `@guide`. Each header gets the step 13 sweeps as it moves, as the layouts did: `@attribution` for its `Author:` and `Prior art:` lines, no predecessor links, one comment kind per declaration, a line per control and a trimmed `@technical`. Done in f2e1b2ee. -6. ✅ **Drivers, the fourth catalog.** All eleven drivers carry their help in their headers, and `drivers.md` is generated from `help.json`: its intro and its two clips from `DriverBase`, its cards from the headers, and the shared controls read through `DriverBase`'s `defineControls`, as the virtual call resolves it. The card's details sections move by who reads them. In the app the user is at the card with a setting in hand, so the help is a glance: what the driver is, what to pick, the one fix when it does not work. On the website the user is sorting something out, which is a how-to page: [choosing and wiring an LED driver](../../reference/hardware/led-drivers.md), [driving a HUB75 panel](../../reference/hardware/hub75-panels.md), [watching the output as video](../../reference/watching-the-output.md) and the panel-card tutorial. So a control's longer help goes to a `/// @guide` line on its member, the choice to a `@guide` section of the module, which never explains a control, tables, steps and reasons to the how-to page the guide links, the mechanism to `@technical`, credit to `@attribution`, and what the code or another page holds is cut. `check_docgen` caps the app help on every generated catalog: 40 words for a control's longer help, 60 for a guide section, 250 for the whole. Done in ef20e444. +6. ✅ **Drivers, the fourth catalog.** All eleven drivers carry their help in their headers, and `drivers.md` is generated from `help.json`: its intro and its two clips from `DriverBase`, its cards from the headers, and the shared controls read through `DriverBase`'s `defineControls`, as the virtual call resolves it. The card's details sections move by who reads them. In the app the user is at the card with a setting in hand, so the help is a glance: what the driver is, what to pick, the one fix when it does not work. On the website the user is sorting something out, which is a how-to page: [choosing and wiring an LED driver](../../../reference/hardware/led-drivers.md), [driving a HUB75 panel](../../../reference/hardware/hub75-panels.md), [watching the output as video](../../../reference/watching-the-output.md) and the panel-card tutorial. So a control's longer help goes to a `/// @guide` line on its member, the choice to a `@guide` section of the module, which never explains a control, tables, steps and reasons to the how-to page the guide links, the mechanism to `@technical`, credit to `@attribution`, and what the code or another page holds is cut. `check_docgen` caps the app help on every generated catalog: 40 words for a control's longer help, 60 for a guide section, 250 for the whole. Done in ef20e444. 7. ✅ **System, Services and Supporting, the last catalogs.** Their modules carry their help in their headers by step 6's rules, and the three pages are generated: Services from its container's class comment, which is then no card, and System and Supporting from their own markdown kept as the intro the deck follows. Supporting's classes that are no module become one linked list. Control's details move into the control-surface, presets and home-automation how-tos, and Audio's into a new audio-input how-to. The separate deck pages go, every catalog page being a deck. Done in 18d069d1. 8. ✅ **The header is the technical reference.** Doxygen, npx moxygen and `gen_api` go, with their installs in the docs job and the 262 pages they generate. The header is what a developer reads, in the repo or on GitHub, so a second rendering of it is a copy to keep in step. Every link into `moxygen/` becomes a link to the header, which the docs build already rewrites to its file on GitHub, and the help's Technical link goes, Source being the same file. `check_docgen` follows: an `@xref` still names a heading in its own file, read as text, and the rules that measured the generated pages (a page nothing links to, an appendix section no page links) go or read the header alone. The supporting lists link headers the same way. Done. 9. ✅ **The migration**, module by module, each card deleted as its header takes over, so the site and the app stayed complete throughout. All 145 modules take their help from their headers, and a card's details section moved as step 6 sets out. The card bridge in `gen_help` is gone, `help.json` coming out byte for byte the same without it. Done. -10. ✅ **The checks.** `check_docgen` checks the help in headers. Done: a stale `help.json` fails it, drawn cards meet the card limits, a control line may not state its default or range, every module and every control has its help line, the app help stays within its caps, a control's guide does not repeat its line, the module guide explains no control, a function carrying a macro such as `MM_NONBLOCKING` has its `///`, and every tag emoji groups at least two modules or factory scripts; `check_specs` counts a module documented when the help reads its header. Reading declarations from the clang AST rather than by pattern is in [the core backlog](../future/backlog-core.md). +10. ✅ **The checks.** `check_docgen` checks the help in headers. Done: a stale `help.json` fails it, drawn cards meet the card limits, a control line may not state its default or range, every module and every control has its help line, the app help stays within its caps, a control's guide does not repeat its line, the module guide explains no control, a function carrying a macro such as `MM_NONBLOCKING` has its `///`, and every tag emoji groups at least two modules or factory scripts; `check_specs` counts a module documented when the help reads its header. Reading declarations from the clang AST rather than by pattern is in [the core backlog](../../future/backlog-core.md). 11. ✅ **The end state.** The `?` button is gone, the help's Docs link taking its place. A catalog page writes each module under its deck as plain markdown from `help.json`, its name bold rather than a heading so the table of contents lists no card, which search indexes and a reader without JavaScript reads. `split_blocks`, `fill_cards_from_help`, the table renderer, `update_module_docs.py` and the table styles are gone. `check_docgen`'s card rules read `help.json`: the picture and its format, a control line's width, and a line stating its default or range. `check_specs` counts a module documented when the help reads its header, so its page lookup and drift guards are gone. -12. ✅ **The attribution catalog as a deck.** [The attribution catalog](../../reference/attributions.md) is a card deck drawn from its own table. The table stays the one home, which search indexes and a reader without JavaScript sees, and the deck script turns its rendered rows into cards. A card shows the entry's picture, name and what it is known for, and its back the modules that name it. Its chips are a Kind column: 🧑 a person, 🏢 a company or shop, 🧰 a project or library, 📜 a standard or protocol, 🧮 an algorithm, 🎨 an art form or culture. 🧰 replaces the planned 📦, which the app's chips use for a compiled module. One card builder and one deck serve both kinds of card. +12. ✅ **The attribution catalog as a deck.** [The attribution catalog](../../../reference/attributions.md) is a card deck drawn from its own table. The table stays the one home, which search indexes and a reader without JavaScript sees, and the deck script turns its rendered rows into cards. A card shows the entry's picture, name and what it is known for, and its back the modules that name it. Its chips are a Kind column: 🧑 a person, 🏢 a company or shop, 🧰 a project or library, 📜 a standard or protocol, 🧮 an algorithm, 🎨 an art form or culture. 🧰 replaces the planned 📦, which the app's chips use for a compiled module. One card builder and one deck serve both kinds of card. 13. 🚧 **The sweeps**, the last commits before this branch merges: each a mechanical commit of its own, apart from behavioral changes, and under 100 files so a review still reads it whole. In order of size, counted on 2026-10-15: | # | Sweep | Status or size | @@ -144,5 +144,5 @@ The change is gradual, the textbook strangler fig: the new path grows module by | 5 ✅ | One comment kind per declaration, and its rule | 561 notes in 92 headers, class and file scope; `check_docgen` reports a `//` on a declaration | | 6 ✅ | `@defgroup` goes | 137 files and three generators; a file's lead is its first `///` block before any code | | 7 ✅ | `@moreinfo` becomes `@technical` | 273 files: the marker, the generators, `check_docgen` and `check_clang_query`, the standards and the tests; the past plans keep the old name as history | - | 8 → | `@technical` trimmed, Guides that add | moved to [the processes plan](Plan-20261015%20-%20Processes%2C%20described%20once.md), step 12 | + | 8 → | `@technical` trimmed, Guides that add | moved to [the processes plan](../../present/Plan-20261015%20-%20Processes%2C%20described%20once%20(partial).md), step 12 | 14. ✅ **Publishing.** Each release attaches its `help.json` as `MoonLight-help.json`, which the Pages deploy mirrors beside the release's binaries, and the app reads the copy of the release its device runs, falling back to the site's `help/help.json`. Both answered after the merge's run, and the fallback to the branch's copy is gone. diff --git a/docs/work/present/Plan-20261020 - Test engine.md b/docs/work/past/plans/Plan-20261020 - Test engine (shipped).md similarity index 93% rename from docs/work/present/Plan-20261020 - Test engine.md rename to docs/work/past/plans/Plan-20261020 - Test engine (shipped).md index 172a64fd4..f4d30c52a 100644 --- a/docs/work/present/Plan-20261020 - Test engine.md +++ b/docs/work/past/plans/Plan-20261020 - Test engine (shipped).md @@ -2,7 +2,7 @@ Plan, for the product owner's approval (2026-10-20). Automated tests are one of MoonLight's core strengths, and they run as several separate machines: C++ unit tests, scenarios on the desktop and on a live device, JS and Python tests, and the Python checks (docgen, prose, code, specs). Each has its own runner, its own output and its own report. This plan finds one model above them, so that what exists, what ran and what it found are described once and read by everyone it concerns: the help, MoonWalk, MoonDeck, the docs and CI. -It starts from [Processes, described once](Plan-20261015%20-%20Processes%2C%20described%20once.md), step 17: the map should show which tests cover a step, and the help which tests cover a module and a control. That needs a machine-readable index of the tests, which does not exist. The index is this plan's first result, and the rest follows from it. +It starts from [Processes, described once](../../present/Plan-20261015%20-%20Processes%2C%20described%20once%20(partial).md), step 17: the map should show which tests cover a step, and the help which tests cover a module and a control. That needs a machine-readable index of the tests, which does not exist. The index is this plan's first result, and the rest follows from it. ## Why @@ -52,7 +52,7 @@ What the inventory settled: ## The model (step 2, approved 2026-10-20) -[The schema](../../../moondeck/docs/tests.schema.json) holds the records. Three, all JSON; JUnit and SARIF are what runners and GitHub exchange, and the working model is ours: +[The schema](../../../../moondeck/docs/tests.schema.json) holds the records. Three, all JSON; JUnit and SARIF are what runners and GitHub exchange, and the working model is ours: 1. **The registry, `tests.json`**: every test by a derived id (`unit:test/unit/core/unit_X.cpp::the digest matches`, `scenario:`, `scenario:#`, `python:`, `js:`, `check:`), with its kind, size, title, file, the **subjects** it is about, the **controls** it sets or expects, the **steps** it proves, and the call that runs it. Generated like `help.json`. 2. **A run**, under `build/`, never committed: the test id, the git tree id it saw, a status, seconds, and **measurements** (target, metric, value, window). A runner that writes JUnit is read into it. @@ -84,8 +84,8 @@ No runner is a prerequisite. doctest, the scenario runner, pytest and each check Each step ends in something that works and can be seen, and nothing waits for a later step to be useful. 1. ✅ **Inventory**, done 2026-10-20: above. -2. ✅ **The model**, approved 2026-10-20 with the derived id: this section and [the schema](../../../moondeck/docs/tests.schema.json). -3. ✅ **`tests.json` and a drawn page on the website**, done 2026-10-20. `moondeck/docs/gen_tests.py` writes the sharded registry (2,607 tests, 195 subjects, an index of 13 KB and a largest shard of 235 KB, 44 KB gzipped), the docs build publishes it as `tests/index.json` and a shard per subject, and `docs/assets/tests.js` draws [the test page](../../reference/tests.md): subjects grouped by the help page of the module (Light · Effects, Core · Services) or by the domain of the tests for internals, with counts, a search, kind filters, each test with its file and line and the command that runs it, and a scenario's steps with the contract and observed range of each measured step, with every test's name in the page as plain text for search. `unit-tests.md` and `scenario-tests.md` are gone, redirected, and the help's Tests links point at the new page. The build found and the fix removed eleven unit tests duplicated in `unit_MoonLiveLayout.cpp`. MoonDeck still draws its own scenario views from `_test_metadata.py` and the perf table in `generate_test_docs.py`, which step 6 replaces with the registry. +2. ✅ **The model**, approved 2026-10-20 with the derived id: this section and [the schema](../../../../moondeck/docs/tests.schema.json). +3. ✅ **`tests.json` and a drawn page on the website**, done 2026-10-20. `moondeck/docs/gen_tests.py` writes the sharded registry (2,607 tests, 195 subjects, an index of 13 KB and a largest shard of 235 KB, 44 KB gzipped), the docs build publishes it as `tests/index.json` and a shard per subject, and `docs/assets/tests.js` draws [the test page](../../../reference/tests.md): subjects grouped by the help page of the module (Light · Effects, Core · Services) or by the domain of the tests for internals, with counts, a search, kind filters, each test with its file and line and the command that runs it, and a scenario's steps with the contract and observed range of each measured step, with every test's name in the page as plain text for search. `unit-tests.md` and `scenario-tests.md` are gone, redirected, and the help's Tests links point at the new page. The build found and the fix removed eleven unit tests duplicated in `unit_MoonLiveLayout.cpp`. MoonDeck still draws its own scenario views from `_test_metadata.py` and the perf table in `generate_test_docs.py`, which step 6 replaces with the registry. 4. ✅ **Help and MoonWalk read it**, done 2026-10-20. A control's help lists the scenarios that set or expect it (46 controls, each link opening that test on the page, proved in the device app as well as on the site's decks), scenarios declare their instances' types once (`instances`, and the generator refuses a step naming an undeclared one), and a step shows "Tested by" for the tests that name it (`steps_proved` in a scenario, `// @step` above a unit test) and for the scenarios that share its control. The build refuses a test naming a step no process has. The unit tests ran 2,585 cases with none failing and the scenario sweep 17 passed with none failing, after the new keys went in. *Result: the links in the help and the map, on the site and in the device's app.* 5. ✅ **The gaps**, done 2026-10-20: the page's Gaps view lists the 25 device-changing steps no test covers, the 18 help modules no test is about, and the 9 controls a scenario sets that the help does not know. *Result: the list of what to test next.* 6. ✅ **Runs**, done 2026-10-20. MoonDeck's runners keep each test's last result as a run, in the model's shape (the derived id, a status, seconds, the git tree it saw), under `build/moondeck-logs/`, never committed (`moondeck/docs/test_runs.py`). The page shows it beside the test, "passed in 0 ms, on this tree" or "on an older tree", where the file exists; the public site shows none until CI records its own (step 10). Recorded from: doctest's JUnit (followed live, so MoonDeck's log still streams; subcase runs count for their case, and a shared `.inc` body is one test per target), pytest's and node's JUnit through their flags, each scenario's `PASS`, `FAIL` or `SKIP` from `run_scenario.py`, and MoonDeck's record of each check script. The registry now holds every kind, 3,526 tests: unit, scenario, live, Python, JS and the 15 checks as `check: