Free internal RAM on the ESP32; live JSON in MoonLive services; a change rebuilds only what it reshapes

CLAUDE.md

@@ -84,7 +84,7 @@ Every activity is a [MoonWalk](docs/moonwalk.md) process, the one home of its st
 
 ### Main and branch
 
-**The product owner creates every branch**, and main stays releasable: [the branch](docs/moonwalk.md#branch). A change goes from [a spec and a plan](docs/moonwalk.md#pick-spec-plan) to a pull request the plan describes. **Deleting the plan is the product owner's call**, because "the code is written" is not "the plan is realized".
+**The product owner creates every branch**, and main stays releasable: [the branch](docs/moonwalk.md#branch). A change goes from [a spec and a plan](docs/moonwalk.md#pick-spec-plan) to a pull request the plan describes. **Moving the plan to `docs/work/past/plans/` is the product owner's call**, because "the code is written" is not "the plan is realized".
 
 **A contributed pull request is one topic, max 400 changed lines and ten files**, tests included, based on main: [Contribute a pull request](docs/moonwalk.md#contribute-pr). The contributor vouches for every line, agent-written or not, answers reviewers in their own words, and says what an agent wrote and what was tested. **A contribution is reviewed and finished as [Review and finish a contribution](docs/moonwalk.md#review-contribution) describes**. The contributor fixes their own code first and the maintainer finishes, case by case. The merge keeps their commits and authorship.
 

CMakeLists.txt

@@ -131,6 +131,7 @@ find_program(UV_EXECUTABLE NAMES uv REQUIRED
 add_library(mm_core STATIC
     src/core/module/Control.cpp
     src/core/util/JsonUtil.cpp
+    src/core/util/JsonFetch.cpp
     src/core/util/parse.cpp
     src/core/util/ScratchBuffer.cpp
     src/core/system/HttpServerModule.cpp
@@ -262,12 +263,9 @@ add_custom_target(ui_embed DEPENDS ${CMAKE_SOURCE_DIR}/src/ui/ui_embedded.h)
 # library while the device holds only what someone actually picked. Globbed, so the DEPENDS is the
 # directory contents rather than a file list that would drift; CONFIGURE_DEPENDS re-globs when a
 # script is added or removed.
-file(GLOB MOONLIVE_SCRIPTS CONFIGURE_DEPENDS
-     ${CMAKE_SOURCE_DIR}/moonlive/effects/*.mle
-     ${CMAKE_SOURCE_DIR}/moonlive/layouts/*.mll
-     ${CMAKE_SOURCE_DIR}/moonlive/modifiers/*.mlm
-     ${CMAKE_SOURCE_DIR}/moonlive/services/*.mls
-     ${CMAKE_SOURCE_DIR}/moonlive/palettes/*.mlp)
+file(GLOB_RECURSE MOONLIVE_SCRIPTS CONFIGURE_DEPENDS
+     ${CMAKE_SOURCE_DIR}/moonlive/*.mle ${CMAKE_SOURCE_DIR}/moonlive/*.mll ${CMAKE_SOURCE_DIR}/moonlive/*.mlm
+     ${CMAKE_SOURCE_DIR}/moonlive/*.mls ${CMAKE_SOURCE_DIR}/moonlive/*.mlp)
 # The file LIST itself is a dependency, not just each file's timestamp: DEPENDS notices an edited
 # script but not a DELETED one, so removing a script left it in the catalog and the device went on
 # offering a name that no longer exists upstream. The stamp is written at configure time from the

docs/contributing/documentation-standards.md

@@ -126,7 +126,7 @@ Every page serves one of four reader needs, and only one. This is [Diátaxis](ht
 
 The test for any page is the cell it sits in. A tutorial that stops to explain, or a reference that starts to teach, is two pages: move the other half to where it belongs.
 
-Two kinds of page sit outside the grid on purpose. **The rules**, [coding-standards.md](coding-standards.md) and this page in `contributing/` and [CLAUDE.md](../../CLAUDE.md) at the repo root, are for contributors and bind every change. `legal/` sits outside it too. **Work not yet in the code** lives under `docs/work/`, and nothing there describes the system as it is. `future` is what does not exist, `present` is being built and deleted at its PR, `past` is what shipped.
+Two kinds of page sit outside the grid on purpose. **The rules**, [coding-standards.md](coding-standards.md) and this page in `contributing/` and [CLAUDE.md](../../CLAUDE.md) at the repo root, are for contributors and bind every change. `legal/` sits outside it too. **Work on its way to the code** lives under `docs/work/`, apart from the pages that describe the system. `future` holds what is planned, `present` what is being built, which moves to `past` at its PR, and `past` the closed plans, shipped or cancelled.
 
 Two scales below a page. **A module** has one reference card and one header (see [Module pages](#module-pages)); **a line of code** carries its own reason in a comment (see [Comments](#comments)).
 

docs/explanation/architecture/mooncore.md

@@ -101,14 +101,28 @@ The light domain plugs into the UI at three points. **A fixed top-level tree**:
 Only abstract what you need:
 
 - **Time**: `millis()`, `micros()`. Monotonic, microsecond resolution. (`esp_timer` / `std::chrono`)
-- **Memory**: `alloc(size)`, `free(ptr)`. Prefers PSRAM on ESP32, falls back to regular heap. `freeHeap()`, `maxAllocBlock()` for diagnostics. (`heap_caps_malloc` / `std::malloc`)
+- **Memory**: `alloc(size)`, `free(ptr)`. Prefers PSRAM on ESP32, falls back to regular heap; `allocInternal(size)` where only internal RAM will do (see [Internal RAM](#internal-ram)). `freeHeap()`, `maxAllocBlock()` for diagnostics. (`heap_caps_malloc` / `std::malloc`)
 - **Executable memory**: `allocExec(size)` / `freeExec(ptr, size)` allocate memory the CPU can *fetch and execute* from, and `writeExec(dst, src, len)` copies emitted machine code into it safely. Used by the MoonLive live-script engine (below) to place the native code it compiles. All the W^X and instruction-cache quirks live behind these three functions. On ESP32 that is IRAM via `MALLOC_CAP_EXEC`, with 32-bit-aligned stores and a cache sync so the core fetches fresh code. On POSIX desktops it is an `mmap` `PROT_EXEC` page, with `MAP_JIT` and a write-protect toggle on macOS-arm64. On Windows it is a `VirtualAlloc` `PAGE_EXECUTE_READWRITE` page plus `FlushInstructionCache`. (`heap_caps_malloc(MALLOC_CAP_EXEC)` / `mmap` / `VirtualAlloc`)
 - **Networking**: `UdpSocket` for ArtNet send. `TcpConnection` / `TcpServer` for HTTP + WebSocket; `TcpConnection::writeSome` is a non-blocking partial write (returns bytes written, 0 = would-block) so a backpressured browser can't stall the render loop. (lwIP sockets / BSD sockets)
 - **Scheduling**: `yield()` (cooperative yield to OS/RTOS), `delayMs(ms)` (blocking sleep, off-path only), `delayUs(us)` (a microsecond busy-wait, only for sub-millisecond hardware timing a driver owns, such as the WS2812 300 µs inter-frame latch in `RmtLedDriver`), and `reboot()`. General pacing uses the non-blocking `millis()` gate instead. (`vTaskDelay` / `esp_rom_delay_us` / `esp_restart` on ESP32; `std::this_thread::sleep_for` / `std::exit` on desktop)
 - **Platform config**: `platform_config.h` per platform: compile-time constants like `hasPsram` and `hasWiFi`. Each platform provides its own version; `types.h` includes it without `#ifdef`. Core code branches on these via `if constexpr` (such as NetworkModule drops its WiFi cascade when `hasWiFi` is false), so the dead branch is removed from the binary with no `#ifdef` outside `src/platform/`.
 
 Abstractions are added when a concrete implementation needs them, not pre-designed.
 
+### Internal RAM
+
+On an ESP32 internal RAM is the scarce memory. WiFi, lwIP, the task stacks and every DMA buffer need it, while a board with PSRAM has megabytes of the other kind. **Whatever can live in PSRAM does.** Internal RAM is kept for what cannot leave it:
+
+- **What a DMA engine reads**, the LED drivers' buffers.
+- **What a cache-safe interrupt reads.** PSRAM sits behind the flash cache, and a flash write on either core switches that cache off, even in the middle of an interrupt on the other core. So the interrupt's data comes from `platform::allocInternal`, and an object it reads inherits `platform::InternalRamObject`, whose `new` places it there. `ParallelLedDriver` and `MoonI80Peripheral` are the examples; the LCD-MM ring refuses to run when what its refill reads fell back to PSRAM, rather than fault at the next save.
+- **The code and stacks of those paths.**
+
+Everything else asks for PSRAM first: a module through `MoonModule`'s `operator new`, a buffer through `platform::alloc`, a plain `malloc` through `CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=0`. WiFi, lwIP and TLS do so through their options in `esp32/sdkconfig.defaults`, where each option carries its reason.
+
+**The floor.** `platform::kInternalFreeFloor`, 100 KB, is the internal RAM a board with PSRAM keeps free for WiFi, HTTP and the UI. Its largest free block stays at 50 KB, for a TLS handshake or a state send. What would take the rest respects it. LCD-MM's `ringAuto` sizes its ring by the lead the wire needs and stops at the floor. The HTTP server queues a send on a socket only while internal RAM stays above half of it, so a slow client waits rather than draining the board. `scenario_RAM_floor` holds the floor with the UI open on the S3 and the P4.
+
+**A board without PSRAM** runs the same calls, which fall back to internal RAM. Its network buffers then share that RAM, so it holds no floor, and the send back-pressure is what keeps an open UI on a weak link from exhausting it. The measurements are in [performance.md](../../reference/performance.md#internal-ram).
+
 **Platform boundary (hard rule).** All `#ifdef`, `#if defined`, platform-specific `#include`s, and hardware API calls live exclusively in `src/platform/`. Everything outside `src/platform/` compiles on every target without modification. Compile-time platform branching uses `if constexpr` on `platform_config.h` flags, never a preprocessor `#ifdef`. **Domain boundary.** A file in `src/core` includes nothing from `src/light`: core declares an interface, and the light domain implements and registers it.
 
 Both boundaries are rules of [the code check](../../reference/tests.md#check_code), each at zero, and a crossing raises the count the ratchet refuses. `LightOutput` is the domain boundary's shape: the light domain's `Drivers` implements it, and the WLED shim, MQTT and MoonStats read the device shape and the palettes through it.

docs/explanation/architecture/moonlight.md

@@ -199,7 +199,7 @@ A modifier is a coordinate transform, applied in one of two ways (the fold contr
 
 The blend+map step walks each layer in turn: reads each logical light, uses that layer's LUT to find the physical position(s), blends the color into the physical output buffer. This is where logical space meets physical space.
 
-Each mapping LUT is a flat, contiguous lookup table allocated outside the hot path. It is built in `Layer::prepare()` and rebuilt whenever a Layout or Modifier control changes (the controls' `affectsPrepare` returns true) or a Modifier/Layout child is added/removed/replaced/moved; both triggers flow through the same core mechanism, see [Event triggering between modules](moonmodule.md#event-triggering-between-modules).
+Each mapping LUT is a flat, contiguous lookup table allocated outside the hot path. It is built in `Layer::prepare()`. A Layout or Modifier control change rebuilds it, a Modifier's reaching its own layer alone, and so does adding, removing, replacing or moving a Modifier or Layout child. Both triggers flow through the same core mechanism: [Event triggering between modules](moonmodule.md#event-triggering-between-modules).
 
 The LUT supports four mapping types:
 

docs/explanation/architecture/moonmodule.md

@@ -121,7 +121,7 @@ Control values and each module's `enabled` flag are persisted to flash so settin
 
 - **Storage**: one file per top-level module under `/.config/<TypeName>.json`, holding that module's [state document](../../reference/integrating.md#setting-everything-at-once), the shape a preset and `PATCH /api/state` carry, with the module's secrets in it since the file never leaves the device. Children are keyed by name, so a file applies through the same engine as every other document, as a stored file: a type this build lacks and a key no control matches are skipped and logged, and a value clamps to its control's bounds.
 - **Lifecycle**: the five phases the diagram above draws, read from persistence's side. `defineControls` binds every module's control set. The load hook then applies each file, creating children with their controls bound and setting values twice around a `rebuildControls`, so a control set that depends on a value (a driver's `peripheral`) restores. Each module's own `setup()` runs with the values already in place, and `applyState` sizes the buffers. The fifth phase sets the controls a MoonLive script declares as it compiles, which the load held until then. Modules themselves know nothing about persistence; they bind their variables.
-- **Save trigger**: HttpServerModule marks the target module dirty on every successful control mutation. FilesystemModule debounces 2 s in `tick1s()`, walks the tree, writes any subtree containing a dirty descendant via atomic write-and-rename.
+- **Save trigger**: `Scheduler::setControl` marks the target module dirty on every successful control mutation, except a live one. A live write is to a control declared live, or one a service script makes, which drives a value rather than configures it. FilesystemModule debounces 2 s in `tick1s()`, walks the tree, and writes any subtree holding a dirty descendant, with an atomic write-and-rename.
 - **Conditional controls**: every conditional control is always bound; the module sets a `hidden` flag (`controls_.setHidden(i, …)`) to tell the UI not to render it. The load path can therefore find persisted values regardless of the live conditional state.
 - **Code-wired children survive a stale file**: some children aren't created by the user; `main.cpp`'s boot wiring attaches them (`ImprovProvisioningModule` under `NetworkModule`; `NetworkSendDriver`, `PreviewDriver` under their parents). Each such child calls `markWiredByCode()` after `addChild()`, a one-bit flag meaning *"I belong here because the code put me here, not because a saved file or a user asked for me."* The problem it solves is a trim on load. A stored document says its children are exactly the ones it lists, so a child that exists in code but is absent from an older file, written before that child was added, would be dropped. The flag tells the apply step to keep it, and to keep its type where the file names another. Children added through the HTTP API or recreated from a file stay unmarked; those follow the file's tree shape exactly, so UI deletes still take effect.
 
@@ -168,10 +168,10 @@ Both shapes extend to any future producer/consumer pair (a sensor owning a state
 A control changes, or the module tree is mutated (a child added, deleted, replaced, moved), and other modules may need to react. The framework provides a three-tier split so each change costs only as much as it has to, from cheapest to most expensive:
 
 1. **`onControlChanged(controlName)`**: runs on *every* control change, but only on the module whose own control changed. A cheap, in-place, per-control reaction that touches nothing else: recompute a small derived table, re-bind a socket. Default no-op.
-2. **`affectsPrepare(controlName)`**: a gate, default `false`. A module returns `true` only for controls that change the size or shape of its derived state (and thus may ripple to other modules); for controls that tweak a value in place it stays `false`. When `true`, the framework runs the tree-wide rebuild; when `false`, it doesn't.
+2. **`affectsPrepare(controlName)`**: a gate, default `false`. A module returns `true` only for a control that changes the size or shape of its derived state, which may ripple to other modules. A control that tweaks a value in place stays `false`. When `true`, the framework runs the rebuild; when `false`, it doesn't. How far the rebuild reaches is `prepareScope()`: the whole tree by default, the module's container for a modifier, whose mapping belongs to its layer alone.
 3. **`prepare()`**: the module (re)builds its derived state (buffers, tables) for the current control values. Reached via `Scheduler::prepareTree()`, the coordinator-driven sweep that walks every module's `prepare`.
 
-`Scheduler::prepareTree()` fires from two triggers: a tier-2 gate returning true after a control change, **and** any tree mutation (HTTP add/delete/replace/move handlers all call it unconditionally, since a structural change is rare and unambiguously needs a rebuild). Both triggers funnel through the same sweep; each module's `prepare` is idempotent (such as an effect only reallocs when its grid count changed), so over-rebuilding is wasted work, not a correctness hazard.
+A rebuild runs at the next frame boundary. Up to four scopes are pending per frame, and a fifth widens the rebuild to the tree. After a scoped rebuild every top module still gets `onTreePrepared()`. A module reading what the scope reshaped then prepares itself again when that changed, as `Drivers` does with the first layer's shape it sized from. `Scheduler::prepareTree()` fires from two triggers: a tier-2 gate whose scope is the tree, **and** any tree mutation. The HTTP add, delete, replace and move handlers call it unconditionally, since a structural change is rare and needs a rebuild. Both triggers funnel through the same sweep. Each module's `prepare` is idempotent, an effect reallocating only when its grid count changed, so over-rebuilding wastes work without breaking anything.
 
 **`quiesce()`, the structural path's thread guard.** A module may hand work to another thread: `Drivers` ticks its Driver children on a core-1 task, see [Parallelism](#parallelism). That makes a *structural* mutation dangerous, and switching a module on or off with it. `addChild` reallocates the child array a worker may be walking, and `removeChild` is followed by the caller's `release()` and `deleteTree()`, which frees a module a worker may be inside `tick()` on. So `MoonModule` declares `virtual void quiesce()` (default no-op) and **core calls it on the parent before every child-array mutation** (`addChild` / `removeChild` / `replaceChildAt`); a module owning a worker overrides it to park that worker. The control path already funnels through `applyState()`/`prepareTree()`, where the owner quiesces itself, this is the same rule extended to the sibling (structural) path, and it lives in core so no HTTP handler has to remember it (CLAUDE.md § *when core already owns a mechanism for one path, extend it to the sibling path*). Deleting a driver from the UI mid-encode is therefore safe by construction, not by handler discipline.