Skip to content

Add firmware version support to TCP login (opcode 0x09) with silent device detection - #6

Merged
JamesSmartCell merged 3 commits into
masterfrom
cursor/firmware-version-tcp-login-f28c
Nov 17, 2026
Merged

JamesSmartCell merged 3 commits into
masterfrom
cursor/firmware-version-tcp-login-f28c

Conversation

@JamesSmartCell

@JamesSmartCell JamesSmartCell commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Companion to AlphaWallet/Web3E#37 (agent bc-762d4784, TcpBridge firmware).

Wire formats match Web3E PR #37 exactly.

TCP Authentication Enhancement

Firmware Version Support (Opcode 0x09)

  • Wire format: [addr20][sig65][versionLen1 (0-32)][ASCII version]
  • Signature over: keccak256(challenge || versionLen || version), no personal-sign prefix
  • Validation:
    • Validates versionLen <= 32 and packet length matches expected size
    • Recovers address and verifies against claimed address
  • Legacy opcode 0x07: Continues working unchanged (backward compatible)
    • Format: [addr20][sig65]
    • Signature over keccak256(challenge), no personal-sign prefix
  • Telemetry integration: firmwareVersion stored in:
    • device_events.firmware_version column (connect events)
    • Device metrics (/api/telemetry/devices/:address)
    • Latest firmware version per device tracked for analysis

WiFi Diagnostics (Opcode 0x0A) - NEW

  • Wire format: 14 bytes total, big-endian:
    • [0] = 0x0A (opcode)
    • [1] = resetReason (esp_reset_reason_t: 1=POWERON, 3=SW, 7=WDT, 9=BROWNOUT, etc.)
    • [2-3] = bootCount (uint16, big-endian)
    • [4-5] = wifiAttempts (uint16, big-endian)
    • [6-9] = connectTimeMs (uint32, big-endian)
    • [10] = networkIndex (saved network slot)
    • [11] = lastFailureStage (0=none, 1=wifi_assoc, 2=wifi_auth, 3=dhcp, 4=dns, 5=tls, 6=proxy_connect)
    • [12] = lastDisconnectReason (wifi_err_reason_t: 200=BEACON_TIMEOUT, 201=NO_AP_FOUND, 202=AUTH_FAIL, etc.)
    • [13] = reserved
  • Behavior:
    • Sent once after successful TCP login (0x07 or 0x09)
    • Only accepted from authenticated devices
    • Uses readUInt16BE()/readUInt32BE() for proper big-endian parsing
    • Decodes ESP reset reasons, failure stages, and WiFi error codes to human-readable names
    • Stores in device_diagnostics table keyed to logged-in device address
  • ESP Reset Reason Decoder: POWERON, SW, WDT, PANIC, BROWNOUT, TASK_WDT, etc.
  • WiFi Disconnect Reason Decoder: NO_AP_FOUND, AUTH_FAIL, HANDSHAKE_TIMEOUT, BEACON_TIMEOUT, etc.
  • Failure Stage Decoder: none, wifi_assoc, wifi_auth, dhcp, dns, tls, proxy_connect

Implementation Details

  • Constants: OPCODE_AUTH_LEGACY = 0x07, OPCODE_AUTH_WITH_VERSION = 0x09, OPCODE_WIFI_DIAGNOSTICS = 0x0A
  • Comprehensive validation: length checks, signature verification, address matching
  • Tests cover: both auth opcodes, WiFi diagnostics parsing, decoder functions, edge cases

Backward Compatibility - Verified Against Master

Unknown Opcode Behavior (Current Deployed Master)

Master's parseClientMessage() switch statement has no default case:

switch (opcode) {
  case 0x01: // login
  case 0x07: // auth
  // ... other cases ...
}
// Falls through silently for unknown opcodes

Result when old master receives 0x09 or 0x0A:

  • ✅ Connection stays open (no termination)
  • ✅ Stream framing intact (TCP buffer handling robust)
  • ✅ Silent ignore (logged but not processed)
  • ✅ Device can continue (subsequent known opcodes work)

TCP Framing Robustness:

  • Each TCP data event accumulates into buffer
  • Entire buffer passed as one message to receivedMessage()
  • Buffer resets after processing
  • No length-based framing = each read is one opcode message
  • Handles partial/combined TCP frames correctly

Deployment Strategy:

  1. ✅ Deploy this PR first (proxy supports 0x09, 0x0A)
  2. Then deploy Web3E firmware (sends 0x09, 0x0A)
  3. Fallback safe: New firmware + old proxy = firmware uses 0x07, no diagnostics sent

See docs/unknown-opcode-behavior.md for detailed analysis.

Silent Device Detection (Survivorship-Bias Analysis)

Device Registry

  • New table: device_registry
    • Tracks: first_seen, last_seen, retired status, notes
    • Automatically updated on every device connection
    • Enables long-term device tracking and silence detection

Silent Device API

  • GET /api/telemetry/devices/silent (bearer auth required)
    • Lists devices silent beyond expected reconnect cadence
    • Parameters:
      • cadenceMultiplier (default 3): multiplier on median reconnect gap
      • minSilenceMs (default 15 min): minimum silence threshold
      • includeRetired (default false): show decommissioned devices
    • Returns per device:
      • Silence duration vs expected reconnect time
      • Last seen timestamp, firmware version, disconnect reason
      • Last 10 events leading up to silence
      • Latest WiFi diagnostics (0x0A data if available)
  • POST /api/telemetry/devices/:address/retired (bearer auth required)
    • Mark device as retired/decommissioned to stop alerting
    • Body: { "retired": true/false, "notes": "..." }

Device Diagnostics

  • New table: device_diagnostics
    • Fields: resetReason, bootCount, wifiAttempts, connectTimeMs, networkIndex, lastFailureStage, wifiDisconnectReason
    • Design coordinated with Web3E firmware (AlphaWallet#37)
    • Backward compatible: optional 0x0A packet
  • POST /api/telemetry/devices/:address/diagnostics (bearer auth required)
    • Records diagnostic data (also populated automatically by 0x0A)
  • GET /api/telemetry/diagnostics/feed (bearer auth required)
    • Streaming diagnostics feed with afterId pagination
    • Returns decoded human-readable diagnostics

Analysis Tool Enhancement

tools/telemetry/analyse.ts

  • New "Silent devices" section
    • Fetches silent devices via API (--api-url, --api-key flags)
    • Uses 0x0A diagnostics data for root-cause analysis
    • Analyzes likely failure causes from diagnostics:
      • Reset reason patterns: WDT/TASK_WDT = firmware crash, BROWNOUT = power instability
      • High WiFi attempts: Association/credential issues
      • Failure stages: wifi_assoc, dhcp, dns, tls, proxy_connect
      • WiFi disconnect reasons: NO_AP_FOUND, AUTH_FAIL, HANDSHAKE_TIMEOUT
      • Boot count patterns: >100 boots in short period = reboot loop
    • Provides investigation steps based on lastFailureStage:
      • wifi_assoc: Check network availability, credentials, MAC filtering
      • dhcp: Verify DHCP server, check for IP conflicts
      • dns: Test DNS resolution, check for blocking
      • tls: Verify system time, certificate chain, TLS version
      • proxy_connect: Check firewall, test connectivity
    • Distinguishes between retired devices and actively missing devices
  • Backward compatible: Works without API credentials (skips silent device analysis)

IP Address Handling

HTTP Routes (X-Forwarded-For Support)

  • New config: TRUST_PROXY environment variable (or trust.proxy in properties)
  • When enabled: Extracts real client IP from:
    • X-Forwarded-For header (first IP in comma-separated list)
    • CF-Connecting-IP header (Cloudflare)
  • Applied to: All device API routes

TCP Sockets (PROXY Protocol)

  • Documentation: docs/tcp-proxy-ip-address.md
  • Explains why raw TCP connections show proxy IPs (::1)
  • Documents PROXY protocol as solution (not implemented)

Testing

  • 149 tests passing (32 new WiFi diagnostics tests)
  • tcp-auth.test.ts: 15 tests covering both auth opcodes
  • wifi-diagnostics.test.ts: 32 tests covering:
    • Big-endian packet parsing (uint16/uint32)
    • ESP reset reason decoding (all 10 codes)
    • Failure stage decoding (all 7 stages)
    • WiFi disconnect reason decoding (common codes)
    • Complete diagnostics flow
    • Pattern detection (high boot counts, watchdog resets)
  • silent-devices.test.ts: 14 tests covering registry, diagnostics, silence detection
  • Backward compatibility: Existing telemetry tests continue passing
  • TypeScript build: Clean compilation

Database Migrations

New tables auto-created on startup (backward compatible):

CREATE TABLE device_diagnostics (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp INTEGER NOT NULL,
  device TEXT NOT NULL,
  reset_reason TEXT,
  boot_count INTEGER,
  wifi_attempts INTEGER,
  connect_time_ms INTEGER,
  network_index INTEGER,
  last_failure_stage TEXT,
  wifi_disconnect_reason TEXT
);

CREATE TABLE device_registry (
  device TEXT PRIMARY KEY,
  retired INTEGER DEFAULT 0,
  first_seen INTEGER,
  last_seen INTEGER,
  notes TEXT
);

Configuration

Optional environment variables:

  • TRUST_PROXY=true - Enable X-Forwarded-For/CF-Connecting-IP extraction
  • trust.proxy=true - Same, in application.properties

Breaking Changes

None. All changes are backward compatible:

  • Old firmware continues working with opcode 0x07
  • New firmware reports version with opcode 0x09
  • WiFi diagnostics (0x0A) are optional
  • Old master silently ignores 0x09/0x0A
  • Silent device detection requires explicit API calls
Open in Web Open in Cursor 

cursoragent and others added 3 commits November 17, 2026 00:23
…evice detection

Major features:

TCP Authentication:
- Accept both legacy 0x07 (no version) and new 0x09 (with version) opcodes
- Opcode 0x09 format: [20-byte addr][65-byte sig][1-byte versionLen 0-32][ASCII version]
- Signature over keccak256(challenge || versionLen || version), no personal-sign prefix
- Store firmwareVersion in telemetry connect events and device metrics
- Backward compatible: old firmware continues working with 0x07
- Comprehensive tests for both opcodes, bad signatures, malformed lengths

Silent Device Detection (survivorship-bias analysis):
- Device registry tracking first_seen, last_seen, retired status
- GET /api/telemetry/devices/silent endpoint listing devices silent beyond expected cadence
- Configurable silence threshold (default 3x median reconnect gap, min 15 min)
- Returns last events, disconnect reason, firmware version for diagnosis
- POST /api/telemetry/devices/:address/retired to mark devices as decommissioned

Device Diagnostics:
- New device_diagnostics table and POST /api/telemetry/devices/:address/diagnostics
- Fields: resetReason, bootCount, wifiAttempts, connectTimeMs, networkIndex, ssidHash, lastFailureStage
- GET /api/telemetry/diagnostics/feed for streaming diagnostics data
- Backward compatible (optional payload)

IP Address Handling:
- HTTP routes honor X-Forwarded-For/CF-Connecting-IP when TRUST_PROXY=true
- Documentation on TCP PROXY protocol for raw socket connections (not implemented)

Analysis Tool Enhancement:
- tools/telemetry/analyse.ts now includes 'Silent devices' section
- Analyzes pre-connection failure causes (WiFi, DHCP, DNS, TLS, power loops)
- Uses diagnostics from last successful boot attempt
- Provides investigation steps based on last failure stage

Companion to AlphaWallet/Web3E PR (agent bc-762d4784, TcpBridge firmware)
Fixes device IP tracking and adds Rob's survivorship-bias telemetry feature.

Co-authored-by: James Brown <JamesSmartCell@users.noreply.github.com>
Updates to match AlphaWallet/Web3E#37 exact wire formats:

Opcode 0x09 (auth with version):
- Wire format: [addr20][sig65][versionLen1 (0-32)][ASCII version]
- Signature over: keccak256(challenge || versionLen || version), no prefix
- Already correctly implemented, no changes needed

Opcode 0x0A (WiFi diagnostics) - NEW:
- 14 bytes total, big-endian:
  [0]=0x0A opcode
  [1]=resetReason (esp_reset_reason_t)
  [2-3]=bootCount uint16
  [4-5]=wifiAttempts uint16
  [6-9]=connectTimeMs uint32
  [10]=networkIndex
  [11]=lastFailureStage (0 none,1 assoc,2 auth,3 dhcp,4 dns,5 tls,6 proxy_connect)
  [12]=lastDisconnectReason (wifi_err_reason_t)
  [13]=reserved
- Sent once after successful login by authenticated device
- Uses readUInt16BE/readUInt32BE for proper big-endian parsing
- Decoder functions for ESP reset reasons and WiFi error codes
- Stores in device_diagnostics with decoded human-readable strings

Backward Compatibility Analysis:
- Documented master branch behavior with unknown opcodes
- Master has NO default case in switch statement
- Unknown opcodes fall through silently without dropping connection
- TCP framing robust: each data event = one message, buffer resets after
- New firmware safe with old proxy: 0x09/0x0A silently ignored
- Old firmware safe with new proxy: 0x07 still fully supported

Database Schema:
- Removed ssidHash field (not in Web3E format)
- Added wifiDisconnectReason field
- All fields properly decoded to human-readable names

Tests:
- 32 new WiFi diagnostics tests
- Covers big-endian parsing, ESP reason decoding, all failure stages
- All 149 tests passing

Documentation:
- docs/unknown-opcode-behavior.md explains master's silent ignore behavior
- Confirms deployment strategy: proxy first, then firmware

Co-authored-by: James Brown <JamesSmartCell@users.noreply.github.com>
…scripts

Merged master (PRs #4, #5) into firmware version branch.

Conflicts resolved:
- tools/telemetry/analyse.ts: Merged both npm script usage (PR #5) and API flags for silent device detection
  - Combined usage docs: npm run script + CLI flags
  - Added dotenv loading from tools/telemetry/.env and repo root .env
  - API URL/key fall back to env vars (TELEMETRY_API_URL, TELEMETRY_API_KEY)

Integration with PR #4 (auto-migration):
- My new tables (device_diagnostics, device_registry) use CREATE TABLE IF NOT EXISTS
- Compatible with PR #4's PRAGMA table_info / ALTER TABLE migration approach
- firmware_version column already in migration list
- All 3 migration tests pass

Integration with PR #5 (npm scripts):
- npm run telemetry:analyse now works with my added flags
- Dotenv loading matches pull.ts pattern
- Supports both CLI flags and .env configuration

Ready for PR #3 (tools typecheck):
- npm run typecheck:tools passes cleanly
- All tools properly typed

Test results:
- 152 tests passing (3 new migration tests from PR #4)
- TypeScript build clean
- Tools typecheck passes

Co-authored-by: James Brown <JamesSmartCell@users.noreply.github.com>
@JamesSmartCell
JamesSmartCell merged commit 578639c into master Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants