MultiObserver

MultiObserver is an open-source observer extension for MeshCore Repeater. It is designed as a reusable, application-level library that adds Wi-Fi and dual-broker MQTT observation to a MeshCore repeater without replacing or forking MeshCore core functionality.

The observer receives repeater traffic, publishes packet and status telemetry to up to two independently configured MQTT brokers, and exposes persistent network configuration through the standard MeshCore CLI. It supports secure MQTT-over-WebSocket connections (WSS/TLS), making it suitable for community or private telemetry backends.

Compatibility

MultiObserver is installed into a clean MeshCore repeater source tree and has been validated with the Heltec_v3_repeater environment for:

MeshCore Repeater release	Status
1.15.0	Installer verified and firmware builds successfully
1.16.0	Installer verified and firmware builds successfully
1.17.1	Installer verified and firmware builds successfully

The integration deliberately keeps MeshCore core code intact. The installer adds only the minimal build and application hooks; MOBridge is the thin adapter between the repeater application and the MultiObserver library.

What MultiObserver provides

• Two independent MQTT broker profiles with separate host, port, transport, username, password and enable/disable settings.
• Wi-Fi and MQTT configuration through the MeshCore CLI, stored persistently on the device.
• MQTT over TCP or WSS/TLS, with connection startup gated on usable Wi-Fi and valid system time.
• Packet, RAW and periodic status publication to every enabled broker.
• Serial diagnostics for Wi-Fi, SNTP, MQTT state changes and received/transmitted packets.
• Heltec V3 integration: Wi-Fi IP shown on the OLED, an EastMesh-style cached battery measurement, and a non-blocking white LED pulse for packet activity.
• A compact Wi-Fi/MQTT watchdog with a configurable startup grace period and\r?\n recovery timings that avoid repeated ESP reboot loops.\r?\n- An encrypted private AlertChannel that uses MeshCore's native outbound queue for startup, outage and recovery notifications.
• Optional Remote CLI access through that private channel, backed by the repeater's existing command handler and outbound queue.

Current status

The V2 implementation is active and builds successfully for the supported MeshCore Repeater releases listed above.

Implemented runtime mechanisms include:

• persistent WiFi and dual-MQTT WSS/TLS configuration through the MeshCore CLI
• explicit WiFi, SNTP and MQTT state machines with transition diagnostics
• MQTT connection gating on both WiFi connectivity and sane system time
• bounded reconnect backoff, heap headroom checks and ESP-TLS error reporting
• packet, RAW and periodic status publication to both enabled brokers
• one-line RX/TX packet diagnostics in the serial monitor
• EastMesh-style 60-second battery sampling cache for Heltec V3 GPIO37
• non-blocking visible packet pulse on the Heltec V3 white GPIO35 LED
• WiFi IP address on the Heltec OLED status screen
• staged Wi-Fi, MQTT1 and MQTT2 supervision with one-shot reboot attribution
• 128-bit and 256-bit private MeshCore AlertChannel keys
• private-channel Remote CLI without a duplicate parser or radio queue

Integration architecture:

• MeshCore 1.17.1 core remains unchanged in src/
• repeater integration lives in examples/simple_repeater/
• MultiObserver functionality lives in lib/MultiObserver/
• MOBridge provides the application-level adapter between the repeater and MultiObserver

Repository layout

MultiObserver/
├── examples/
│   └── simple_repeater/
│       ├── MOBridge.cpp
│       └── MOBridge.h
├── installer/
│   ├── mo_installer/
│   │   ├── cli.py
│   │   ├── common.py
│   │   ├── patches.py
│   │   ├── storage.py
│   │   ├── validation.py
│   │   └── workflow.py
│   ├── tests/
│   ├── install.py
│   ├── multiobserver_cpp17.py
│   └── verify.py
├── lib/
│   └── MultiObserver/
│       ├── library.json
│       └── src/
│           ├── MO.cpp
│           ├── MO.h
│           ├── MOConfig.cpp
│           ├── MOConfig.h
│           ├── MOWifi.cpp
│           ├── MOWifi.h
│           ├── MOWifiPrefs.cpp
│           ├── MOWifiPrefs.h
│           ├── MOMQTT.cpp
│           ├── MOMQTT.h
│           ├── MOMQTTPrefs.cpp
│           ├── MOMQTTPrefs.h
│           ├── MOCli.cpp
│           ├── MOCli.h
│           ├── MOEtap2Prefs.cpp
│           ├── MOEtap2Prefs.h
│           ├── MOAlertChannel.cpp
│           ├── MOAlertChannel.h
│           ├── MOWatchdog.cpp
│           └── MOWatchdog.h
├── install.ps1
└── Run-Installer.cmd

Design principles

• preserve MeshCore compatibility
• make the smallest safe integration changes
• avoid unnecessary dependencies
• keep WiFi and MQTT lifecycle deterministic
• keep runtime configuration persistent
• do not duplicate existing MeshCore functionality without a reason
• keep secrets and local credentials out of Git
• keep install.py as a thin entry point and isolate patching, validation, backup and workflow responsibilities

License

See LICENSE.

Build-system integration

For Heltec_v3_repeater, the installer adds the local MultiObserver library, enables the ESP32 certificate bundle and compiles the extension as C++17. The extension uses the ESP-IDF MQTT client, so the unused Arduino WiFiClientSecure library is excluded while the framework WiFi library remains available to MOWifi.

The installer verifies that all 18 MeshCore 1.17.1 Heltec V3 environments are preserved and creates a rollback backup before changing the target tree.

The installer never keeps a prepared device identity in the MO source tree. When identity arguments are supplied, it writes the generated identity and name directly into the selected MeshCore target's data/ directory.

Quick start (Windows / Heltec V3)

This is the recommended complete path for a new device. Commands marked MultiObserver directory are run in the cloned MultiObserver repository. Commands marked prepared MeshCore directory are run in the MeshCore source tree selected by the installer.

Prerequisites

• A clean MeshCore repeater source tree. MeshCore 1.15.0, 1.16.0 and 1.17.1 have been installed and compiled successfully with this release.
• Python 3 and PlatformIO (py -3 -m platformio).
• A USB-connected Heltec V3 board when you are ready to flash it.

0. Get MultiObserver

Open PowerShell and clone the repository where you keep your projects:

git clone https://github.com/beepoo304/MultiObserver.git
cd .\MultiObserver

If you already downloaded or cloned MultiObserver, simply open PowerShell in that directory instead.

1. Apply MultiObserver to MeshCore

In the MultiObserver directory, start the interactive installer:

.\Run-Installer.cmd

When prompted, enter:

1. The full path to a clean MeshCore Repeater source tree (for example, C:\Projects\MeshCore-repeater-v1.17.1).
2. The repeater name.
3. The 64-character public identity key.
4. The 128-character private identity key.

The installer creates a rollback backup, adds the minimal application/build hooks, prepares the MeshCore filesystem data (data/identity/_main.id and data/prefs.json), then verifies the resulting tree. Keep the identity keys private: never commit or paste them into a public issue.

For non-interactive or troubleshooting use:

python installer/install.py <MeshCore-path> --source-root <MultiObserver-path>
python installer/verify.py <MeshCore-path>

2. Build the prepared MeshCore tree

In PowerShell, change to the prepared MeshCore directory selected in step 1. Replace the example path with your own:

cd "C:\Projects\MeshCore-repeater-v1.17.1"
py -3 -m platformio run -e Heltec_v3_repeater

Wait for SUCCESS before continuing.

3. Flash firmware and configuration

With the Heltec V3 connected by USB, and still in the prepared MeshCore directory, upload both the firmware and the filesystem:

py -3 -m platformio run -e Heltec_v3_repeater -t upload
py -3 -m platformio run -e Heltec_v3_repeater -t uploadfs

To completely erase the connected board first:

py -3 -m platformio run -e Heltec_v3_repeater -t erase

upload flashes the program. uploadfs writes the identity, repeater name and persistent configuration prepared by the installer; it is therefore also required after an erase. Do not publish identity keys, Wi-Fi passwords or MQTT credentials in a public repository.

4. Open the serial monitor

Still in the prepared MeshCore directory, open the serial monitor at 115200 baud:

py -3 -m platformio device monitor -b 115200

Press Ctrl+C to leave the monitor.

5. Configure Wi-Fi and brokers

Use the MeshCore CLI in the serial monitor to configure Wi-Fi and each broker. The values below are placeholders: use your own network and broker values.

set wifi.ssid YourWiFiName
set wifi.pwd YourWiFiPassword

set mqtt1.on
set mqtt1.host mqtt.example.org
set mqtt1.port 443
set mqtt1.transport wss
set mqtt1.username YourMqttUser
set mqtt1.pwd YourMqttPassword

set mqtt2.on
set mqtt2.host backup-mqtt.example.org
set mqtt2.port 8883
set mqtt2.transport tcp
set mqtt2.username YourMqttUser
set mqtt2.pwd YourMqttPassword

Verify the current configuration and connection state with:

get wifi.status
get mqtt1.status
get mqtt2.status

Configuration is retained on the device. Restart Wi-Fi or an individual broker after changing settings when needed:

restart.wifi
restart.mqtt1
restart.mqtt2

MultiObserver CLI command reference

Enter these commands in the MeshCore serial monitor. All settings are saved persistently. Commands and values are case-sensitive where shown.

For a standalone operational reference containing all 41 MultiObserver commands, replies, validation rules and a safe first-configuration sequence, see CLI_REFERENCE.md.

Wi-Fi

Command	Description
set wifi.ssid <SSID>	Save the Wi-Fi network name and reconnect.
set wifi.pwd <password>	Save the Wi-Fi password and reconnect.
get wifi.ssid	Show the configured Wi-Fi network name.
get wifi.status	Show Wi-Fi state, IP address and diagnostic details.
restart.wifi	Restart Wi-Fi using the saved credentials.

MQTT broker 1

Command	Description
set mqtt1.on	Enable broker 1 and start connecting.
set mqtt1.off	Disable broker 1 and disconnect it.
set mqtt1.host <hostname>	Set the broker hostname.
set mqtt1.port <1-65535>	Set the broker port.
set mqtt1.transport tcp	Use direct MQTT over TCP.
set mqtt1.transport wss	Use secure MQTT over WebSocket (WSS/TLS).
set mqtt1.username <username>	Set the MQTT username.
set mqtt1.pwd <password>	Set the MQTT password.
get mqtt1.status	Show enabled/disabled state, connection state, host, failures and last error.
restart.mqtt1	Restart broker 1 using the saved settings.

MQTT broker 2

Broker 2 uses the same commands as broker 1. Replace mqtt1 with mqtt2:

Command	Description
set mqtt2.on / set mqtt2.off	Enable or disable broker 2.
set mqtt2.host <hostname>	Set the broker hostname.
set mqtt2.port <1-65535>	Set the broker port.
set mqtt2.transport tcp	Use direct MQTT over TCP.
set mqtt2.transport wss	Use secure MQTT over WebSocket (WSS/TLS).
set mqtt2.username <username>	Set the MQTT username.
set mqtt2.pwd <password>	Set the MQTT password.
get mqtt2.status	Show broker 2 connection and diagnostic state.
restart.mqtt2	Restart broker 2 using the saved settings.

Observer location / IATA code

Command	Description
set mqtt.iata <ABC>	Set the three-letter uppercase IATA location code used by MQTT telemetry.
get mqtt.iata	Show the saved IATA location code.

For example, use set mqtt.iata KTW for Katowice. The code must contain exactly three uppercase letters.

Watchdog

The watchdog starts after a configurable 120-300 second grace period. Wi-Fi health is local: connected station, assigned address and gateway. MQTT status uses the active broker connection; delayed publish acknowledgements do not falsely show a connected broker as down.

Command	Description
get wdg.status	Show WDG mode, phase, Wi-Fi/MQTT health and pending reboot state.
set wdg.on	Enable WDG persistently and start a fresh grace period.
set wdg.off	Disable WDG, cancel recovery and a pending WDG reboot.
get wdg.grace	Show the saved startup grace period in seconds.
set wdg.grace <120-300>	Save a valid grace period and restart WDG grace.
restart.wdg	Reset the WDG state machine and grace timer without rebooting the ESP.

Wi-Fi is checked every 5 seconds. If it remains unhealthy for 30 seconds, WDG restarts Wi-Fi once; the Wi-Fi manager then retries every 3 minutes. A Wi-Fi outage alone never reboots the ESP.

MQTT1 and MQTT2 are handled independently while Wi-Fi is healthy. When a broker disconnects, WDG restarts only that MQTT client and makes up to three connection attempts, spaced 3 minutes apart. If all three fail, it queues ESP restart.mqtt1 or ESP restart.mqtt2, waits 30 seconds, then reboots once. The affected broker mask is saved before reboot so a failed recovery cannot cause a reboot loop. After the startup grace period, a broker still down reports mqtt1.down or mqtt2.down and retries every 5 minutes. When it reconnects, WDG queues one matching mqtt1.restore or mqtt2.restore alert, then returns that broker to the normal 3-minute retry interval. Recovery cancels only that broker's pending reboot request.

WDG keeps no persistent event log. get wdg.status reports MQTT UP or DOWN from the actual connection state, not from the age of the last publish acknowledgement.

Connection recovery fix (2026-10-08)

Disconnected/error MQTT clients are now stopped and destroyed by the main loop before a new client is created at the scheduled retry deadline. The previous code left the handle alive while requiring a null handle to reconnect. Retry backoff remains 10 seconds up to 5 minutes. Broker handshakes remain serialized to limit TLS memory use.

MQTT callbacks only enqueue bounded event copies. The main loop owns state changes, publication and client destruction. This also avoids cross-client MQTT locks when both brokers connect together. Events from retired clients are ignored; queue overflow triggers controlled recovery of the affected client.

Runtime regression tests compile the actual MOMQTT.cpp and MOWatchdog.cpp against fake MQTT, Wi-Fi, storage and time: python tests/runtime/run.py. Set CXX or pass --compiler; Zig is also supported. These tests exercise reconnection, duplicate error events, timeouts, both brokers, Wi-Fi/NTP loss, memory gating, stale callbacks, queue overflow, timer rollover, reboot ownership, storage failure and bounded post-reboot cooldown. They do not replace device tests.

For an existing Heltec V3 installation, update firmware only using the unmerged firmware.bin through start ota and http://192.168.4.1/update. Select Firmware, never Filesystem. Do not erase flash, upload a new filesystem, provision new identity files or change the partition table. This fix does not change preference formats or the existing partition layout. Heltec V3 uses ESP32 Wi-Fi OTA, not Nordic Bluetooth DFU ZIP packages.

Private AlertChannel

Create a private channel in a MeshCore client and copy its secret key. Standard MeshCore private channels normally expose a 32-hex-character (128-bit) key; 64-hex-character (256-bit) keys are accepted as well. Treat this key as a secret. MultiObserver never publishes it to serial diagnostics.

Command	Description
get channel.status	Show whether AlertChannel is off, ready or missing a key/sender.
set channel.on	Enable AlertChannel persistently.
set channel.off	Disable AlertChannel persistently.
get channel.key	Return the saved channel key over the authenticated MeshCore CLI.
set channel.key <32-or-64-hex>	Save a private MeshCore channel key.
test.channel	Queue test alertchannel, time and concise WDG state (maximum 130 characters).

After each grace period AlertChannel attempts to queue AlertChannel newStart with repeater time. Alert messages are encrypted as MeshCore group datagrams and handed directly to the repeater's existing scoped outbound queue. MultiObserver does not maintain a second queue, retry alerts, wait for ACKs or control the radio directly.

test.channel returns QUEUED when MeshCore's native outbound queue accepts the encrypted message. This is intentionally not reported as SENT or DELIVERED: actual airtime remains under MeshCore control and AlertChannel does not wait for a delivery acknowledgement.

Human-readable AlertChannel timestamps use the local repeater time. The default is Central European time with automatic CET/CEST daylight-saving changes. Override the POSIX timezone at build time with -D MULTIOBSERVER_TZ=\"your-POSIX-TZ\". MeshCore packet epochs and MQTT timestamps remain UTC.

How AlertChannel works

1. set channel.key validates and persistently stores a 128-bit or 256-bit MeshCore private-channel secret. The key is never printed in serial logs.
2. set channel.on enables delivery. A missing/invalid key, disabled channel or unavailable MeshCore sender makes an alert fail immediately and produces only a short diagnostic; the watchdog itself never blocks.
3. AC adds the repeater-clock date and time and limits the complete alert text to 130 characters.
4. The thin MOBridge passes the decoded key and text to MyMesh.
5. MyMesh calculates the standard channel hash and calls MeshCore's createGroupDatagram(PAYLOAD_TYPE_GRP_TXT, ...).
6. The encrypted packet is submitted with sendFloodScoped(...), so ordering, airtime and actual transmission remain owned by MeshCore's normal outbound dispatcher.

AC intentionally does not know whether the radio later transmitted the packet. There is no AC retry, delivery ACK or parallel queue. This prevents an alert failure from creating a second watchdog/reboot loop and keeps MultiObserver a thin application layer over MeshCore.

Remote CLI over LoRa

Remote CLI uses the configured AlertChannel key and is off by default:

set rcli.on
get rcli.status
set rcli.off

When RCLI READY is shown, send any normal repeater or MultiObserver CLI command as a message on that private channel, for example get status or get wifi.status. The existing MeshCore command handler creates one reply, which is returned through the native outbound queue. MultiObserver adds no second command table, response queue or retransmission mechanism.

The channel key is the authorization boundary. Every person who has it can run configuration-changing and restart commands, so use a dedicated secret channel.

EastMesh mechanism reference

EastMesh v2026.8.2 is used as a behavior reference, not as the integration architecture. MultiObserver matches its essential V1 network mechanisms: WiFi state transitions, SNTP readiness, MQTT network gating, event/error diagnostics, connection backoff and one-broker-at-a-time startup. EastMesh-only features such as gateway ARP watchdogs, WiFi channel hints and WSS preflight remain outside the current V1 scope.

The Heltec fixes reproduce the observed EastMesh behavior without moving business logic into MeshCore core: the installer adds only verified application/build hooks, while MOBridge remains the thin adapter to MO.