Repository navigation
Add firmware version support to TCP login (opcode 0x09) with silent device detection - #6
Merged
Conversation
…evice detection Major features: TCP Authentication: - Accept both legacy 0x07 (no version) and new 0x09 (with version) opcodes - Opcode 0x09 format: [20-byte addr][65-byte sig][1-byte versionLen 0-32][ASCII version] - Signature over keccak256(challenge || versionLen || version), no personal-sign prefix - Store firmwareVersion in telemetry connect events and device metrics - Backward compatible: old firmware continues working with 0x07 - Comprehensive tests for both opcodes, bad signatures, malformed lengths Silent Device Detection (survivorship-bias analysis): - Device registry tracking first_seen, last_seen, retired status - GET /api/telemetry/devices/silent endpoint listing devices silent beyond expected cadence - Configurable silence threshold (default 3x median reconnect gap, min 15 min) - Returns last events, disconnect reason, firmware version for diagnosis - POST /api/telemetry/devices/:address/retired to mark devices as decommissioned Device Diagnostics: - New device_diagnostics table and POST /api/telemetry/devices/:address/diagnostics - Fields: resetReason, bootCount, wifiAttempts, connectTimeMs, networkIndex, ssidHash, lastFailureStage - GET /api/telemetry/diagnostics/feed for streaming diagnostics data - Backward compatible (optional payload) IP Address Handling: - HTTP routes honor X-Forwarded-For/CF-Connecting-IP when TRUST_PROXY=true - Documentation on TCP PROXY protocol for raw socket connections (not implemented) Analysis Tool Enhancement: - tools/telemetry/analyse.ts now includes 'Silent devices' section - Analyzes pre-connection failure causes (WiFi, DHCP, DNS, TLS, power loops) - Uses diagnostics from last successful boot attempt - Provides investigation steps based on last failure stage Companion to AlphaWallet/Web3E PR (agent bc-762d4784, TcpBridge firmware) Fixes device IP tracking and adds Rob's survivorship-bias telemetry feature. Co-authored-by: James Brown <JamesSmartCell@users.noreply.github.com>
Updates to match AlphaWallet/Web3E#37 exact wire formats: Opcode 0x09 (auth with version): - Wire format: [addr20][sig65][versionLen1 (0-32)][ASCII version] - Signature over: keccak256(challenge || versionLen || version), no prefix - Already correctly implemented, no changes needed Opcode 0x0A (WiFi diagnostics) - NEW: - 14 bytes total, big-endian: [0]=0x0A opcode [1]=resetReason (esp_reset_reason_t) [2-3]=bootCount uint16 [4-5]=wifiAttempts uint16 [6-9]=connectTimeMs uint32 [10]=networkIndex [11]=lastFailureStage (0 none,1 assoc,2 auth,3 dhcp,4 dns,5 tls,6 proxy_connect) [12]=lastDisconnectReason (wifi_err_reason_t) [13]=reserved - Sent once after successful login by authenticated device - Uses readUInt16BE/readUInt32BE for proper big-endian parsing - Decoder functions for ESP reset reasons and WiFi error codes - Stores in device_diagnostics with decoded human-readable strings Backward Compatibility Analysis: - Documented master branch behavior with unknown opcodes - Master has NO default case in switch statement - Unknown opcodes fall through silently without dropping connection - TCP framing robust: each data event = one message, buffer resets after - New firmware safe with old proxy: 0x09/0x0A silently ignored - Old firmware safe with new proxy: 0x07 still fully supported Database Schema: - Removed ssidHash field (not in Web3E format) - Added wifiDisconnectReason field - All fields properly decoded to human-readable names Tests: - 32 new WiFi diagnostics tests - Covers big-endian parsing, ESP reason decoding, all failure stages - All 149 tests passing Documentation: - docs/unknown-opcode-behavior.md explains master's silent ignore behavior - Confirms deployment strategy: proxy first, then firmware Co-authored-by: James Brown <JamesSmartCell@users.noreply.github.com>
…scripts Merged master (PRs #4, #5) into firmware version branch. Conflicts resolved: - tools/telemetry/analyse.ts: Merged both npm script usage (PR #5) and API flags for silent device detection - Combined usage docs: npm run script + CLI flags - Added dotenv loading from tools/telemetry/.env and repo root .env - API URL/key fall back to env vars (TELEMETRY_API_URL, TELEMETRY_API_KEY) Integration with PR #4 (auto-migration): - My new tables (device_diagnostics, device_registry) use CREATE TABLE IF NOT EXISTS - Compatible with PR #4's PRAGMA table_info / ALTER TABLE migration approach - firmware_version column already in migration list - All 3 migration tests pass Integration with PR #5 (npm scripts): - npm run telemetry:analyse now works with my added flags - Dotenv loading matches pull.ts pattern - Supports both CLI flags and .env configuration Ready for PR #3 (tools typecheck): - npm run typecheck:tools passes cleanly - All tools properly typed Test results: - 152 tests passing (3 new migration tests from PR #4) - TypeScript build clean - Tools typecheck passes Co-authored-by: James Brown <JamesSmartCell@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to AlphaWallet/Web3E#37 (agent bc-762d4784, TcpBridge firmware).
Wire formats match Web3E PR #37 exactly.
TCP Authentication Enhancement
Firmware Version Support (Opcode 0x09)
[addr20][sig65][versionLen1 (0-32)][ASCII version]keccak256(challenge || versionLen || version), no personal-sign prefixversionLen <= 32and packet length matches expected size[addr20][sig65]keccak256(challenge), no personal-sign prefixfirmwareVersionstored in:device_events.firmware_versioncolumn (connect events)/api/telemetry/devices/:address)WiFi Diagnostics (Opcode 0x0A) - NEW
[0]= 0x0A (opcode)[1]= resetReason (esp_reset_reason_t: 1=POWERON, 3=SW, 7=WDT, 9=BROWNOUT, etc.)[2-3]= bootCount (uint16, big-endian)[4-5]= wifiAttempts (uint16, big-endian)[6-9]= connectTimeMs (uint32, big-endian)[10]= networkIndex (saved network slot)[11]= lastFailureStage (0=none, 1=wifi_assoc, 2=wifi_auth, 3=dhcp, 4=dns, 5=tls, 6=proxy_connect)[12]= lastDisconnectReason (wifi_err_reason_t: 200=BEACON_TIMEOUT, 201=NO_AP_FOUND, 202=AUTH_FAIL, etc.)[13]= reservedreadUInt16BE()/readUInt32BE()for proper big-endian parsingdevice_diagnosticstable keyed to logged-in device addressImplementation Details
OPCODE_AUTH_LEGACY = 0x07,OPCODE_AUTH_WITH_VERSION = 0x09,OPCODE_WIFI_DIAGNOSTICS = 0x0ABackward Compatibility - Verified Against Master
Unknown Opcode Behavior (Current Deployed Master)
Master's
parseClientMessage()switch statement has no default case:Result when old master receives 0x09 or 0x0A:
TCP Framing Robustness:
dataevent accumulates into bufferreceivedMessage()Deployment Strategy:
See
docs/unknown-opcode-behavior.mdfor detailed analysis.Silent Device Detection (Survivorship-Bias Analysis)
Device Registry
device_registryfirst_seen,last_seen,retiredstatus, notesSilent Device API
/api/telemetry/devices/silent(bearer auth required)cadenceMultiplier(default 3): multiplier on median reconnect gapminSilenceMs(default 15 min): minimum silence thresholdincludeRetired(default false): show decommissioned devices/api/telemetry/devices/:address/retired(bearer auth required){ "retired": true/false, "notes": "..." }Device Diagnostics
device_diagnosticsresetReason,bootCount,wifiAttempts,connectTimeMs,networkIndex,lastFailureStage,wifiDisconnectReason/api/telemetry/devices/:address/diagnostics(bearer auth required)/api/telemetry/diagnostics/feed(bearer auth required)afterIdpaginationAnalysis Tool Enhancement
tools/telemetry/analyse.ts
--api-url,--api-keyflags)IP Address Handling
HTTP Routes (X-Forwarded-For Support)
TRUST_PROXYenvironment variable (ortrust.proxyin properties)X-Forwarded-Forheader (first IP in comma-separated list)CF-Connecting-IPheader (Cloudflare)TCP Sockets (PROXY Protocol)
docs/tcp-proxy-ip-address.md::1)Testing
Database Migrations
New tables auto-created on startup (backward compatible):
Configuration
Optional environment variables:
TRUST_PROXY=true- Enable X-Forwarded-For/CF-Connecting-IP extractiontrust.proxy=true- Same, in application.propertiesBreaking Changes
None. All changes are backward compatible: