diff --git a/docs/tcp-proxy-ip-address.md b/docs/tcp-proxy-ip-address.md new file mode 100644 index 0000000..4598fa2 --- /dev/null +++ b/docs/tcp-proxy-ip-address.md @@ -0,0 +1,65 @@ +# TCP Connection IP Address Handling + +## Current State + +Device IP addresses shown in telemetry often display as `::1` (localhost IPv6) because traffic arrives via a local reverse proxy or tunnel before reaching SmartLayerProxy. + +## HTTP API Routes: X-Forwarded-For Support + +For HTTP routes, the proxy can now honor standard forwarding headers when configured: + +- Set environment variable `TRUST_PROXY=true` (or `trust.proxy=true` in application.properties) +- SmartLayerProxy will extract the real client IP from: + - `X-Forwarded-For` header (takes first IP from comma-separated list) + - `CF-Connecting-IP` header (Cloudflare) + +**Example:** +```bash +export TRUST_PROXY=true +``` + +This applies to all HTTP-based device API routes: +- `GET /api/getEthAddress` +- `GET /api/:Address/:method` +- `GET /api/async/:Address/:method` +- `GET /api/bridge/:Address/:data` +- `GET /api/bridge/login` + +## TCP Socket Connections: PROXY Protocol + +For raw TCP connections (port 8082), the current implementation cannot read IP forwarding information because: + +1. TCP sockets don't have HTTP headers +2. The socket's `remoteAddress` reflects the immediate peer (the reverse proxy/tunnel) + +### Potential Solution: PROXY Protocol + +The **PROXY protocol** (v1 or v2) prepends connection metadata (real client IP, port) to the TCP stream before application data. This is supported by: + +- HAProxy +- nginx (with `proxy_protocol` directive) +- AWS ELB/NLB +- Cloudflare Spectrum + +**Implementation would require:** + +1. Detecting and parsing PROXY protocol header on new TCP connections +2. Extracting the real client IP before processing authentication opcodes +3. Stripping the PROXY header from the data stream + +**Example PROXY v1 header:** +``` +PROXY TCP4 192.168.1.100 10.0.0.1 50123 8082\r\n +``` + +This feature is **not implemented** in the current release. TCP connections will continue to show the tunnel/proxy IP address until PROXY protocol support is added. + +### Workarounds + +- For development/testing: disable the tunnel and connect directly +- For production: accept that TCP telemetry shows proxy IPs, or implement PROXY protocol support + +## References + +- [PROXY protocol specification](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) +- [nginx proxy_protocol](https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/) diff --git a/docs/unknown-opcode-behavior.md b/docs/unknown-opcode-behavior.md new file mode 100644 index 0000000..42a8e6f --- /dev/null +++ b/docs/unknown-opcode-behavior.md @@ -0,0 +1,93 @@ +# Unknown Opcode Handling in ASyncTCPService + +## Current Master Behavior (Pre-PR) + +The current deployed `master` branch's `ASyncTCPService.parseClientMessage()` uses a `switch` statement **without a default case**: + +```typescript +switch (opcode) { + case 0x01: // login + case 0x02: // challenge + case 0x03: // auth with personal sign + case 0x07: // auth without personal sign + case 0x04: // return values + case 0x05: // API response + case 0x08: // HTTP bridge response + // ... handlers ... +} +// No default case - falls through silently +``` + +### Impact on Unknown Opcodes + +When master receives an **unknown opcode** (e.g., 0x09 or 0x0A from new firmware): + +1. ✅ **Connection stays open** - No termination +2. ✅ **Stream framing intact** - TCP buffer read as single message (see below) +3. ✅ **Silent ignore** - Message logged but not processed +4. ✅ **Device can continue** - Subsequent known opcodes work normally + +### TCP Framing Robustness + +The `TCPClient` implementation in master is **robust to partial and combined frames**: + +```typescript +socket.on('data', (data: Buffer) => { + buffer = Buffer.concat([buffer, data]); + // ... length checks ... + this.serverCallback.receivedMessage(this.clientIndex, buffer); + buffer = Buffer.alloc(0); // Reset after processing +}); +``` + +**Key characteristics:** +- Each TCP `data` event accumulates into a buffer +- The entire buffer (up to `MAX_TCP_PACKET = 4096` bytes) is passed as one message +- Buffer resets after `receivedMessage()` completes +- **No length-based framing** - each TCP read is one opcode message + +### Backward Compatibility Conclusion + +✅ **New firmware (0x09, 0x0A) is safe with old master servers:** + +- 0x09 auth with version: Ignored, but firmware should fall back to 0x07 +- 0x0A diagnostics: Silently ignored, connection remains open + +**However:** Since master doesn't implement 0x09, **firmware MUST send 0x07 as fallback** if the server doesn't respond to 0x09, or use 0x07 exclusively until this PR is deployed. + +## This PR's Changes + +### Opcode 0x09 - Auth with Firmware Version +- **Format:** `[addr20][sig65][versionLen1 (0-32)][ASCII version]` +- **Signature over:** `keccak256(challenge || versionLen || version)` +- **Behavior:** Validates and stores firmware version; falls back gracefully if malformed + +### Opcode 0x0A - WiFi Diagnostics +- **Format:** 14 bytes, big-endian: + - `[0]` = 0x0A + - `[1]` = resetReason (esp_reset_reason_t) + - `[2-3]` = bootCount uint16 + - `[4-5]` = wifiAttempts uint16 + - `[6-9]` = connectTimeMs uint32 + - `[10]` = networkIndex + - `[11]` = lastFailureStage (0-6) + - `[12]` = lastDisconnectReason (wifi_err_reason_t) + - `[13]` = reserved +- **Behavior:** + - Only accepted from authenticated devices + - Decodes ESP reset reasons and WiFi error codes to names + - Stores in `device_diagnostics` table for analysis + - Silently ignores if unauthenticated or malformed + +### Unknown Opcode Handling +- **No default case added** - maintains backward compatibility +- Unknown opcodes still fall through silently +- Future opcodes can be added without breaking old clients + +## Deployment Strategy + +1. **Deploy this PR first** (proxy supports 0x09, 0x0A) +2. **Then deploy Web3E firmware** (sends 0x09, 0x0A) +3. **Fallback safe:** New firmware + old proxy = firmware uses 0x07, no diagnostics + +**Migration window:** Once this PR deploys, new firmware can start sending 0x09/0x0A immediately. Old firmware continues working indefinitely with 0x07. diff --git a/src/__tests__/silent-devices.test.ts b/src/__tests__/silent-devices.test.ts new file mode 100644 index 0000000..29b7537 --- /dev/null +++ b/src/__tests__/silent-devices.test.ts @@ -0,0 +1,291 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { TelemetryService } from '../service/TelemetryService'; +import { unlinkSync, existsSync } from 'fs'; + +const TEST_DB_PATH = './test-silent-devices.db'; + +describe('Silent Device Detection', () => { + let telemetry: TelemetryService; + + beforeEach(() => { + if (existsSync(TEST_DB_PATH)) { + unlinkSync(TEST_DB_PATH); + } + TelemetryService.resetInstance(); + telemetry = TelemetryService.getInstance(TEST_DB_PATH, 30); + }); + + afterEach(() => { + telemetry.close(); + if (existsSync(TEST_DB_PATH)) { + unlinkSync(TEST_DB_PATH); + } + }); + + describe('Device Registry', () => { + it('should track first and last seen times', () => { + const device = '0xabc123'; + const now = Date.now(); + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + + const registry = telemetry['db'].prepare('SELECT * FROM device_registry WHERE device = ?').get(device) as any; + expect(registry).toBeDefined(); + expect(registry.first_seen).toBeGreaterThanOrEqual(now); + expect(registry.last_seen).toBeGreaterThanOrEqual(now); + expect(registry.retired).toBe(0); + }); + + it('should update last seen on subsequent connections', async () => { + const device = '0xabc123'; + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + const first = telemetry['db'].prepare('SELECT * FROM device_registry WHERE device = ?').get(device) as any; + + await new Promise(resolve => setTimeout(resolve, 10)); + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + const second = telemetry['db'].prepare('SELECT * FROM device_registry WHERE device = ?').get(device) as any; + + expect(second.first_seen).toBe(first.first_seen); + expect(second.last_seen).toBeGreaterThan(first.last_seen); + }); + + it('should mark device as retired', () => { + const device = '0xabc123'; + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + telemetry.markDeviceRetired(device, true, 'Replaced with new hardware'); + + const registry = telemetry['db'].prepare('SELECT * FROM device_registry WHERE device = ?').get(device) as any; + expect(registry.retired).toBe(1); + expect(registry.notes).toBe('Replaced with new hardware'); + }); + + it('should unretire device', () => { + const device = '0xabc123'; + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + telemetry.markDeviceRetired(device, true); + telemetry.markDeviceRetired(device, false); + + const registry = telemetry['db'].prepare('SELECT * FROM device_registry WHERE device = ?').get(device) as any; + expect(registry.retired).toBe(0); + }); + }); + + describe('Device Diagnostics', () => { + it('should record device diagnostics', () => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'POWERON', + bootCount: 42, + wifiAttempts: 3, + connectTimeMs: 1500, + networkIndex: 0, + lastFailureStage: 'none', + wifiDisconnectReason: 'NO_AP_FOUND', + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag).toBeDefined(); + expect(diag?.resetReason).toBe('POWERON'); + expect(diag?.bootCount).toBe(42); + expect(diag?.wifiAttempts).toBe(3); + expect(diag?.connectTimeMs).toBe(1500); + expect(diag?.networkIndex).toBe(0); + expect(diag?.lastFailureStage).toBe('none'); + expect(diag?.wifiDisconnectReason).toBe('NO_AP_FOUND'); + }); + + it('should return latest diagnostics for device', async () => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'POWERON', + bootCount: 1, + }); + + await new Promise(resolve => setTimeout(resolve, 5)); + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'SW', + bootCount: 2, + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag?.resetReason).toBe('SW'); + expect(diag?.bootCount).toBe(2); + }); + + it('should handle partial diagnostics', () => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'WDT', + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag?.resetReason).toBe('WDT'); + expect(diag?.bootCount).toBeNull(); + }); + }); + + describe('Silent Device Detection', () => { + it('should detect device that has gone silent', () => { + const device = '0xabc123'; + const now = Date.now(); + const oneHourAgo = now - 60 * 60 * 1000; + + telemetry['db'].prepare( + 'INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 0)' + ).run(device, oneHourAgo, oneHourAgo); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, protocol) VALUES (?, ?, ?, ?)' + ).run(oneHourAgo, device, 'connect', 'TCP'); + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 30 * 60 * 1000, + cadenceMultiplier: 2, + }); + + expect(silentDevices.length).toBeGreaterThan(0); + const silent = silentDevices.find(d => d.address === device); + expect(silent).toBeDefined(); + expect(silent?.silentDurationMs).toBeGreaterThan(30 * 60 * 1000); + }); + + it('should not detect recently active device', () => { + const device = '0xabc123'; + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 15 * 60 * 1000, + }); + + const silent = silentDevices.find(d => d.address === device); + expect(silent).toBeUndefined(); + }); + + it('should exclude retired devices by default', () => { + const device = '0xabc123'; + const oneHourAgo = Date.now() - 60 * 60 * 1000; + + telemetry['db'].prepare( + 'INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 1)' + ).run(device, oneHourAgo, oneHourAgo); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, protocol) VALUES (?, ?, ?, ?)' + ).run(oneHourAgo, device, 'connect', 'TCP'); + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 15 * 60 * 1000, + includeRetired: false, + }); + + const silent = silentDevices.find(d => d.address === device); + expect(silent).toBeUndefined(); + }); + + it('should include retired devices when requested', () => { + const device = '0xabc123'; + const oneHourAgo = Date.now() - 60 * 60 * 1000; + + telemetry['db'].prepare( + 'INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 1)' + ).run(device, oneHourAgo, oneHourAgo); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, protocol) VALUES (?, ?, ?, ?)' + ).run(oneHourAgo, device, 'connect', 'TCP'); + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 15 * 60 * 1000, + includeRetired: true, + }); + + const silent = silentDevices.find(d => d.address === device); + expect(silent).toBeDefined(); + expect(silent?.retired).toBe(true); + }); + + it('should include last disconnect reason', () => { + const device = '0xabc123'; + const oneHourAgo = Date.now() - 60 * 60 * 1000; + + telemetry['db'].prepare( + 'INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 0)' + ).run(device, oneHourAgo, oneHourAgo); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, protocol) VALUES (?, ?, ?, ?)' + ).run(oneHourAgo, device, 'connect', 'TCP'); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, disconnect_reason) VALUES (?, ?, ?, ?)' + ).run(oneHourAgo + 1000, device, 'disconnect', 'timeout'); + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 15 * 60 * 1000, + }); + + const silent = silentDevices.find(d => d.address === device); + expect(silent?.lastDisconnectReason).toBe('timeout'); + }); + + it('should include last diagnostics', () => { + const device = '0xabc123'; + const oneHourAgo = Date.now() - 60 * 60 * 1000; + + telemetry['db'].prepare( + 'INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 0)' + ).run(device, oneHourAgo, oneHourAgo); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, protocol) VALUES (?, ?, ?, ?)' + ).run(oneHourAgo, device, 'connect', 'TCP'); + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'WDT', + bootCount: 100, + }); + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 15 * 60 * 1000, + }); + + const silent = silentDevices.find(d => d.address === device); + expect(silent?.lastDiagnostics?.resetReason).toBe('WDT'); + expect(silent?.lastDiagnostics?.bootCount).toBe(100); + }); + + it('should sort by silence duration descending', () => { + const device1 = '0xabc001'; + const device2 = '0xabc002'; + const twoHoursAgo = Date.now() - 2 * 60 * 60 * 1000; + const oneHourAgo = Date.now() - 60 * 60 * 1000; + + for (const [device, time] of [[device1, twoHoursAgo], [device2, oneHourAgo]]) { + telemetry['db'].prepare( + 'INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 0)' + ).run(device, time, time); + + telemetry['db'].prepare( + 'INSERT INTO device_events (timestamp, device, event_type, protocol) VALUES (?, ?, ?, ?)' + ).run(time, device, 'connect', 'TCP'); + } + + const silentDevices = telemetry.getSilentDevices({ + minSilenceMs: 15 * 60 * 1000, + }); + + expect(silentDevices.length).toBeGreaterThanOrEqual(2); + expect(silentDevices[0].address).toBe(device1); + expect(silentDevices[1].address).toBe(device2); + }); + }); +}); diff --git a/src/__tests__/tcp-auth.test.ts b/src/__tests__/tcp-auth.test.ts new file mode 100644 index 0000000..803717d --- /dev/null +++ b/src/__tests__/tcp-auth.test.ts @@ -0,0 +1,280 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { ASyncTCPService } from '../service/ASyncTCPService'; +import { TelemetryService } from '../service/TelemetryService'; +import { ethers } from 'ethers'; +import { unlinkSync, existsSync } from 'fs'; + +const TEST_DB_PATH = './test-tcp-auth.db'; + +const OPCODE_AUTH_LEGACY = 0x07; +const OPCODE_AUTH_WITH_VERSION = 0x09; + +describe('TCP Authentication with Firmware Version', () => { + let telemetry: TelemetryService; + let wallet: ethers.HDNodeWallet; + let address: string; + + beforeEach(() => { + if (existsSync(TEST_DB_PATH)) { + unlinkSync(TEST_DB_PATH); + } + TelemetryService.resetInstance(); + telemetry = TelemetryService.getInstance(TEST_DB_PATH, 30); + + wallet = ethers.Wallet.createRandom(); + address = wallet.address.toLowerCase(); + }); + + afterEach(() => { + telemetry.close(); + if (existsSync(TEST_DB_PATH)) { + unlinkSync(TEST_DB_PATH); + } + }); + + function buildAuthPacket(opcode: number, challenge: Buffer, firmwareVersion?: string): Buffer { + const addrBytes = Buffer.from(address.substring(2), 'hex'); + + let preimage: Buffer; + if (opcode === OPCODE_AUTH_WITH_VERSION && firmwareVersion !== undefined) { + const versionBytes = Buffer.from(firmwareVersion, 'utf-8'); + const versionLen = versionBytes.length; + preimage = Buffer.concat([challenge, Buffer.from([versionLen]), versionBytes]); + } else { + preimage = challenge; + } + + const msgHash = ethers.keccak256(preimage); + const sig = wallet.signingKey.sign(msgHash); + + const r = Buffer.from(sig.r.substring(2), 'hex'); + const s = Buffer.from(sig.s.substring(2), 'hex'); + const v = Buffer.from([sig.v]); + const sigBytes = Buffer.concat([r, s, v]); + + if (opcode === OPCODE_AUTH_WITH_VERSION && firmwareVersion !== undefined) { + const versionBytes = Buffer.from(firmwareVersion, 'utf-8'); + const versionLen = versionBytes.length; + return Buffer.concat([addrBytes, sigBytes, Buffer.from([versionLen]), versionBytes]); + } + + return Buffer.concat([addrBytes, sigBytes]); + } + + describe('Legacy Auth (opcode 0x07)', () => { + it('should accept legacy auth without firmware version', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const authPacket = buildAuthPacket(OPCODE_AUTH_LEGACY, challenge); + + expect(authPacket.length).toBe(85); + + const recoveredAddr = authPacket.subarray(0, 20); + expect('0x' + recoveredAddr.toString('hex').toLowerCase()).toBe(address); + }); + + it('should store null firmware version for legacy auth', () => { + telemetry.recordDeviceConnection(address, 'TCP', '192.168.1.1'); + + const metrics = telemetry.getDeviceMetrics(address); + expect(metrics).toBeDefined(); + expect(metrics?.firmwareVersion).toBeNull(); + }); + + it('should reject short legacy auth packet', () => { + const shortPacket = Buffer.alloc(84); + expect(shortPacket.length).toBeLessThan(85); + }); + + it('should reject bad signature for legacy auth', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const authPacket = buildAuthPacket(OPCODE_AUTH_LEGACY, challenge); + + authPacket[20] ^= 0xFF; + + const addrBytes = authPacket.subarray(0, 20); + const sig = authPacket.subarray(20, 85); + const msgHash = ethers.keccak256(challenge); + + let recovered = ''; + try { + const r = '0x' + Buffer.from(sig.slice(0, 32)).toString('hex'); + const s = '0x' + Buffer.from(sig.slice(32, 64)).toString('hex'); + let v = sig[64]; + if (v < 27) v += 27; + const sigObj = ethers.Signature.from({ r, s, v }); + recovered = ethers.recoverAddress(msgHash, sigObj); + } catch { + recovered = ''; + } + + const claimed = '0x' + addrBytes.toString('hex').toLowerCase(); + expect(recovered.toLowerCase()).not.toBe(claimed); + }); + }); + + describe('New Auth with Firmware Version (opcode 0x08)', () => { + it('should accept auth with firmware version', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const firmwareVersion = 'v2.1.0'; + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, firmwareVersion); + + expect(authPacket.length).toBe(85 + 1 + firmwareVersion.length); + + const versionLen = authPacket[85]; + expect(versionLen).toBe(firmwareVersion.length); + + const versionBytes = authPacket.subarray(86, 86 + versionLen); + expect(versionBytes.toString('utf-8')).toBe(firmwareVersion); + }); + + it('should verify signature over challenge || versionLen || version', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const firmwareVersion = 'v2.1.0'; + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, firmwareVersion); + + const addrBytes = authPacket.subarray(0, 20); + const sig = authPacket.subarray(20, 85); + const versionLen = authPacket[85]; + const versionBytes = authPacket.subarray(86, 86 + versionLen); + + const preimage = Buffer.concat([challenge, Buffer.from([versionLen]), versionBytes]); + const msgHash = ethers.keccak256(preimage); + + const r = '0x' + Buffer.from(sig.slice(0, 32)).toString('hex'); + const s = '0x' + Buffer.from(sig.slice(32, 64)).toString('hex'); + let v = sig[64]; + if (v < 27) v += 27; + const sigObj = ethers.Signature.from({ r, s, v }); + const recovered = ethers.recoverAddress(msgHash, sigObj); + + const claimed = '0x' + addrBytes.toString('hex').toLowerCase(); + expect(recovered.toLowerCase()).toBe(claimed); + }); + + it('should store firmware version in telemetry', () => { + const firmwareVersion = 'v2.1.0'; + telemetry.recordDeviceConnection(address, 'TCP', '192.168.1.1', firmwareVersion); + + const metrics = telemetry.getDeviceMetrics(address); + expect(metrics).toBeDefined(); + expect(metrics?.firmwareVersion).toBe(firmwareVersion); + }); + + it('should handle various firmware version formats', () => { + const versions = ['v1.0', 'v2.1.3-beta', '1.2.3', 'dev-build-20231201', 'a']; + + versions.forEach(version => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, version); + + const versionLen = authPacket[85]; + expect(versionLen).toBe(version.length); + + const versionBytes = authPacket.subarray(86, 86 + versionLen); + expect(versionBytes.toString('utf-8')).toBe(version); + }); + }); + + it('should reject version length exceeding max', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const tooLongVersion = 'a'.repeat(33); + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, tooLongVersion); + + const versionLen = authPacket[85]; + expect(versionLen).toBeGreaterThan(32); + }); + + it('should reject truncated version data', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const firmwareVersion = 'v2.1.0'; + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, firmwareVersion); + + const truncated = authPacket.subarray(0, authPacket.length - 2); + expect(truncated.length).toBe(85 + 1 + firmwareVersion.length - 2); + + const versionLen = truncated[85]; + expect(truncated.length).toBeLessThan(86 + versionLen); + }); + + it('should reject short packet for version auth', () => { + const shortPacket = Buffer.alloc(85); + expect(shortPacket.length).toBeLessThan(86); + }); + + it('should reject bad signature with version', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const firmwareVersion = 'v2.1.0'; + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, firmwareVersion); + + authPacket[20] ^= 0xFF; + + const addrBytes = authPacket.subarray(0, 20); + const sig = authPacket.subarray(20, 85); + const versionLen = authPacket[85]; + const versionBytes = authPacket.subarray(86, 86 + versionLen); + + const preimage = Buffer.concat([challenge, Buffer.from([versionLen]), versionBytes]); + const msgHash = ethers.keccak256(preimage); + + let recovered = ''; + try { + const r = '0x' + Buffer.from(sig.slice(0, 32)).toString('hex'); + const s = '0x' + Buffer.from(sig.slice(32, 64)).toString('hex'); + let v = sig[64]; + if (v < 27) v += 27; + const sigObj = ethers.Signature.from({ r, s, v }); + recovered = ethers.recoverAddress(msgHash, sigObj); + } catch { + recovered = ''; + } + + const claimed = '0x' + addrBytes.toString('hex').toLowerCase(); + expect(recovered.toLowerCase()).not.toBe(claimed); + }); + + it('should accept empty firmware version', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const firmwareVersion = ''; + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, firmwareVersion); + + expect(authPacket.length).toBe(85 + 1 + 0); + + const versionLen = authPacket[85]; + expect(versionLen).toBe(0); + }); + + it('should accept max length firmware version', () => { + const challenge = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const firmwareVersion = 'a'.repeat(32); + const authPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge, firmwareVersion); + + expect(authPacket.length).toBe(85 + 1 + 32); + + const versionLen = authPacket[85]; + expect(versionLen).toBe(32); + + const versionBytes = authPacket.subarray(86, 86 + versionLen); + expect(versionBytes.toString('utf-8')).toBe(firmwareVersion); + }); + }); + + describe('Backward Compatibility', () => { + it('should accept both legacy and new auth from same device', () => { + const challenge1 = Buffer.from('1234567890abcdef1234567890abcdef', 'hex'); + const legacyPacket = buildAuthPacket(OPCODE_AUTH_LEGACY, challenge1); + expect(legacyPacket.length).toBe(85); + + const challenge2 = Buffer.from('fedcba0987654321fedcba0987654321', 'hex'); + const newPacket = buildAuthPacket(OPCODE_AUTH_WITH_VERSION, challenge2, 'v2.1.0'); + expect(newPacket.length).toBeGreaterThan(85); + + telemetry.recordDeviceConnection(address, 'TCP', '192.168.1.1'); + const metrics1 = telemetry.getDeviceMetrics(address); + expect(metrics1?.firmwareVersion).toBeNull(); + + telemetry.recordDeviceConnection(address, 'TCP', '192.168.1.1', 'v2.1.0'); + const metrics2 = telemetry.getDeviceMetrics(address); + expect(metrics2?.firmwareVersion).toBe('v2.1.0'); + }); + }); +}); diff --git a/src/__tests__/wifi-diagnostics.test.ts b/src/__tests__/wifi-diagnostics.test.ts new file mode 100644 index 0000000..02d7b23 --- /dev/null +++ b/src/__tests__/wifi-diagnostics.test.ts @@ -0,0 +1,251 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { TelemetryService } from '../service/TelemetryService'; +import { unlinkSync, existsSync } from 'fs'; + +const TEST_DB_PATH = './test-wifi-diagnostics.db'; + +const OPCODE_WIFI_DIAGNOSTICS = 0x0A; + +describe('WiFi Diagnostics (opcode 0x0A)', () => { + let telemetry: TelemetryService; + + beforeEach(() => { + if (existsSync(TEST_DB_PATH)) { + unlinkSync(TEST_DB_PATH); + } + TelemetryService.resetInstance(); + telemetry = TelemetryService.getInstance(TEST_DB_PATH, 30); + }); + + afterEach(() => { + telemetry.close(); + if (existsSync(TEST_DB_PATH)) { + unlinkSync(TEST_DB_PATH); + } + }); + + describe('Packet Format', () => { + it('should parse 14-byte diagnostics packet (big-endian)', () => { + const packet = Buffer.alloc(14); + packet[0] = OPCODE_WIFI_DIAGNOSTICS; + packet[1] = 1; // POWERON + packet.writeUInt16BE(42, 2); // bootCount + packet.writeUInt16BE(3, 4); // wifiAttempts + packet.writeUInt32BE(1500, 6); // connectTimeMs + packet[10] = 0; // networkIndex + packet[11] = 0; // lastFailureStage (none) + packet[12] = 201; // lastDisconnectReason (NO_AP_FOUND) + packet[13] = 0; // reserved + + expect(packet.length).toBe(14); + expect(packet[0]).toBe(0x0A); + + const resetReason = packet[1]; + const bootCount = packet.readUInt16BE(2); + const wifiAttempts = packet.readUInt16BE(4); + const connectTimeMs = packet.readUInt32BE(6); + const networkIndex = packet[10]; + const lastFailureStage = packet[11]; + const lastDisconnectReason = packet[12]; + + expect(resetReason).toBe(1); + expect(bootCount).toBe(42); + expect(wifiAttempts).toBe(3); + expect(connectTimeMs).toBe(1500); + expect(networkIndex).toBe(0); + expect(lastFailureStage).toBe(0); + expect(lastDisconnectReason).toBe(201); + }); + + it('should handle large values correctly', () => { + const packet = Buffer.alloc(14); + packet[0] = OPCODE_WIFI_DIAGNOSTICS; + packet[1] = 10; // SDIO + packet.writeUInt16BE(65535, 2); // max uint16 + packet.writeUInt16BE(1000, 4); + packet.writeUInt32BE(4294967295, 6); // max uint32 + packet[10] = 5; + packet[11] = 6; + packet[12] = 23; + packet[13] = 0; + + expect(packet.readUInt16BE(2)).toBe(65535); + expect(packet.readUInt32BE(6)).toBe(4294967295); + }); + + it('should detect short packet', () => { + const shortPacket = Buffer.alloc(13); + shortPacket[0] = OPCODE_WIFI_DIAGNOSTICS; + + expect(shortPacket.length).toBeLessThan(14); + }); + }); + + describe('ESP Reset Reason Decoding', () => { + const resetReasons = [ + { code: 1, name: 'POWERON' }, + { code: 2, name: 'EXT' }, + { code: 3, name: 'SW' }, + { code: 4, name: 'PANIC' }, + { code: 5, name: 'INT_WDT' }, + { code: 6, name: 'TASK_WDT' }, + { code: 7, name: 'WDT' }, + { code: 8, name: 'DEEPSLEEP' }, + { code: 9, name: 'BROWNOUT' }, + { code: 10, name: 'SDIO' }, + ]; + + it.each(resetReasons)('should decode reset reason $code as $name', ({ code, name }) => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + resetReason: name, + bootCount: 1, + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag?.resetReason).toBe(name); + }); + }); + + describe('Failure Stage Decoding', () => { + const stages = [ + { code: 0, name: 'none' }, + { code: 1, name: 'wifi_assoc' }, + { code: 2, name: 'wifi_auth' }, + { code: 3, name: 'dhcp' }, + { code: 4, name: 'dns' }, + { code: 5, name: 'tls' }, + { code: 6, name: 'proxy_connect' }, + ]; + + it.each(stages)('should decode failure stage $code as $name', ({ code, name }) => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + lastFailureStage: name, + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag?.lastFailureStage).toBe(name); + }); + }); + + describe('WiFi Disconnect Reason Decoding', () => { + const commonReasons = [ + { code: 1, name: 'UNSPECIFIED' }, + { code: 15, name: 'FOURWAY_HANDSHAKE_TIMEOUT' }, + { code: 23, name: 'IEEE_802_1X_AUTH_FAILED' }, + { code: 200, name: 'BEACON_TIMEOUT' }, + { code: 201, name: 'NO_AP_FOUND' }, + { code: 202, name: 'AUTH_FAIL' }, + { code: 203, name: 'ASSOC_FAIL' }, + { code: 204, name: 'HANDSHAKE_TIMEOUT' }, + { code: 205, name: 'CONNECTION_FAIL' }, + ]; + + it.each(commonReasons)('should decode wifi disconnect reason $code as $name', ({ code, name }) => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + wifiDisconnectReason: name, + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag?.wifiDisconnectReason).toBe(name); + }); + }); + + describe('Complete Diagnostics Flow', () => { + it('should store complete diagnostics from 0x0A packet', () => { + const device = '0xabc123'; + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'POWERON', + bootCount: 5, + wifiAttempts: 2, + connectTimeMs: 2500, + networkIndex: 0, + lastFailureStage: 'none', + wifiDisconnectReason: 'NO_AP_FOUND', + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag).toBeDefined(); + expect(diag?.resetReason).toBe('POWERON'); + expect(diag?.bootCount).toBe(5); + expect(diag?.wifiAttempts).toBe(2); + expect(diag?.connectTimeMs).toBe(2500); + expect(diag?.networkIndex).toBe(0); + expect(diag?.lastFailureStage).toBe('none'); + expect(diag?.wifiDisconnectReason).toBe('NO_AP_FOUND'); + }); + + it('should track diagnostics across multiple boots', async () => { + const device = '0xabc123'; + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'POWERON', + bootCount: 1, + wifiAttempts: 1, + connectTimeMs: 1200, + networkIndex: 0, + lastFailureStage: 'none', + wifiDisconnectReason: 'NO_AP_FOUND', + }); + + await new Promise(resolve => setTimeout(resolve, 5)); + + telemetry.recordDeviceConnection(device, 'TCP', '192.168.1.1'); + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'SW', + bootCount: 2, + wifiAttempts: 3, + connectTimeMs: 3500, + networkIndex: 0, + lastFailureStage: 'dhcp', + wifiDisconnectReason: 'AUTH_FAIL', + }); + + const diag = telemetry.getLatestDiagnostics(device); + expect(diag?.resetReason).toBe('SW'); + expect(diag?.bootCount).toBe(2); + expect(diag?.lastFailureStage).toBe('dhcp'); + }); + + it('should identify problematic patterns from diagnostics', () => { + const device = '0xabc123'; + + telemetry.recordDeviceDiagnostics(device, { + resetReason: 'WDT', + bootCount: 150, + wifiAttempts: 10, + connectTimeMs: 45000, + networkIndex: 0, + lastFailureStage: 'wifi_assoc', + wifiDisconnectReason: 'AUTH_FAIL', + }); + + const diag = telemetry.getLatestDiagnostics(device); + + if (diag?.resetReason?.includes('WDT')) { + expect(diag.resetReason).toMatch(/WDT/); + } + + if (diag && diag.bootCount && diag.bootCount > 100) { + expect(diag.bootCount).toBeGreaterThan(100); + } + + if (diag && diag.wifiAttempts && diag.wifiAttempts > 5) { + expect(diag.wifiAttempts).toBeGreaterThan(5); + } + + if (diag && diag.connectTimeMs && diag.connectTimeMs > 30000) { + expect(diag.connectTimeMs).toBeGreaterThan(30000); + } + }); + }); +}); diff --git a/src/api/routes.ts b/src/api/routes.ts index bc85231..6df8b0f 100644 --- a/src/api/routes.ts +++ b/src/api/routes.ts @@ -24,6 +24,18 @@ function normalizeAddress(addr: string): string { return '0x' + addr.replace(/^0x/i, '').toLowerCase(); } +function getClientIp(req: Request): string { + const config = loadConfig(); + if (config.trustProxy) { + const forwardedFor = req.get('x-forwarded-for') || req.get('cf-connecting-ip'); + if (forwardedFor) { + const ips = forwardedFor.split(',').map(ip => ip.trim()); + return ips[0]; + } + } + return req.socket.remoteAddress ?? ''; +} + function parseQueryParams(req: Request): Record { const params: Record = {}; for (const [key, val] of Object.entries(req.query)) { @@ -149,6 +161,49 @@ export function createRouter(service: ASyncService, tcpService: ASyncTCPService) } }); + router.get('/telemetry/diagnostics/feed', telemetryAuthMiddleware, (req, res) => { + try { + const afterId = parseInt(req.query.afterId as string) || 0; + const limit = Math.min( + parseInt(req.query.limit as string) || 1000, + config.telemetryFeedMaxLimit + ); + + const stmt = telemetry['db'].prepare(` + SELECT + id, timestamp, device, + reset_reason as resetReason, + boot_count as bootCount, + wifi_attempts as wifiAttempts, + connect_time_ms as connectTimeMs, + network_index as networkIndex, + last_failure_stage as lastFailureStage, + wifi_disconnect_reason as wifiDisconnectReason + FROM device_diagnostics + WHERE id > ? + ORDER BY id ASC + LIMIT ? + `); + + const rows = stmt.all(afterId, limit) as any[]; + + const maxIdStmt = telemetry['db'].prepare('SELECT MAX(id) as maxId FROM device_diagnostics'); + const { maxId } = maxIdStmt.get() as any; + + const hasMore = rows.length === limit && maxId > (rows[rows.length - 1]?.id || 0); + + res.json({ + rows, + maxId: maxId || 0, + hasMore, + serverTime: Date.now(), + }); + } catch (err) { + console.error('[Telemetry] Diagnostics feed error:', err); + res.status(500).json({ error: 'Internal error' }); + } + }); + router.post('/telemetry/ack', telemetryAuthMiddleware, async (req, res) => { try { const { upToId, signature, message, dryRun } = req.body || {}; @@ -619,6 +674,69 @@ export function createRouter(service: ASyncService, tcpService: ASyncTCPService) } }); + router.get('/telemetry/devices/silent', telemetryAuthMiddleware, (req, res) => { + try { + const cadenceMultiplier = req.query.cadenceMultiplier + ? parseFloat(req.query.cadenceMultiplier as string) + : 3; + const minSilenceMs = req.query.minSilenceMs + ? parseInt(req.query.minSilenceMs as string) + : 15 * 60 * 1000; + const includeRetired = req.query.includeRetired === 'true'; + + if (cadenceMultiplier < 0 || minSilenceMs < 0) { + res.status(400).json({ error: 'Invalid parameters' }); + return; + } + + const silentDevices = telemetry.getSilentDevices({ + cadenceMultiplier, + minSilenceMs, + includeRetired, + }); + + res.json({ + devices: silentDevices, + params: { cadenceMultiplier, minSilenceMs, includeRetired }, + serverTime: Date.now(), + }); + } catch (err) { + console.error(err); + res.status(500).json({ error: 'Internal error' }); + } + }); + + router.post('/telemetry/devices/:address/retired', telemetryAuthMiddleware, (req, res) => { + try { + const address = normalizeAddress(req.params.address); + const { retired, notes } = req.body || {}; + + if (typeof retired !== 'boolean') { + res.status(400).json({ error: 'Missing or invalid retired boolean' }); + return; + } + + telemetry.markDeviceRetired(address, retired, notes); + res.json({ success: true, address, retired, notes: notes || null }); + } catch (err) { + console.error(err); + res.status(500).json({ error: 'Internal error' }); + } + }); + + router.post('/telemetry/devices/:address/diagnostics', telemetryAuthMiddleware, (req, res) => { + try { + const address = normalizeAddress(req.params.address); + const diagnostics = req.body || {}; + + telemetry.recordDeviceDiagnostics(address, diagnostics); + res.json({ success: true, address }); + } catch (err) { + console.error(err); + res.status(500).json({ error: 'Internal error' }); + } + }); + // ========== Camera upload (called by Remote Camera device) ========== router.post('/camera/upload', express.raw({ type: 'image/jpeg', limit: '3mb' }), async (req, res) => { try { @@ -653,7 +771,8 @@ export function createRouter(service: ASyncService, tcpService: ASyncTCPService) async function handleApiCall(req: Request, res: Response, asyncPath: boolean): Promise { const address = normalizeAddress(req.params.Address); const method = req.params.method; - const clientDesignator = req.socket.remoteAddress + '-' + address + method; + const clientIp = getClientIp(req); + const clientDesignator = clientIp + '-' + address + method; console.log('[TS-Proxy] ADDRESS:', address, 'METHOD:', method); const argMap = parseQueryParams(req); @@ -695,7 +814,7 @@ export function createRouter(service: ASyncService, tcpService: ASyncTCPService) router.all('/getEthAddress', async (req, res) => { try { - const ipAddress = req.socket.remoteAddress ?? ''; + const ipAddress = getClientIp(req); const [udpAddrs, tcpAddrs] = await Promise.all([ service.getDeviceAddress(ipAddress), tcpService.getDeviceAddress(ipAddress), @@ -712,12 +831,13 @@ export function createRouter(service: ASyncService, tcpService: ASyncTCPService) try { const address = normalizeAddress(req.params.Address); const method = req.params.data; - const clientDesignator = req.socket.remoteAddress + '-' + address + method; + const clientIp = getClientIp(req); + const clientDesignator = clientIp + '-' + address + method; console.log('ADDRESS:', address); console.log('METHOD:', method); console.log('Designator:', clientDesignator); - const response = tcpService.handleDeviceConnection(address, method, req.socket.remoteAddress ?? ''); + const response = tcpService.handleDeviceConnection(address, method, clientIp); res.status(201).send(response); } catch (err) { console.error(err); @@ -727,8 +847,9 @@ export function createRouter(service: ASyncService, tcpService: ASyncTCPService) router.all('/bridge/login', async (req, res) => { try { - console.log('Login:', req.socket.remoteAddress); - const response = tcpService.getLoginChallenge(req.socket.remoteAddress ?? ''); + const clientIp = getClientIp(req); + console.log('Login:', clientIp); + const response = tcpService.getLoginChallenge(clientIp); res.status(201).send(response); } catch (err) { console.error(err); diff --git a/src/config.ts b/src/config.ts index fe80966..9ea1d3c 100644 --- a/src/config.ts +++ b/src/config.ts @@ -13,6 +13,7 @@ export interface Config { telemetryAckEnabled: boolean; telemetryFeedMaxLimit: number; firmwareUploadMaxSizeMb: number; + trustProxy: boolean; } function loadProperties(): Record { @@ -72,5 +73,6 @@ export function loadConfig(): Config { firmwareUploadMaxSizeMb: parseInt( process.env.FIRMWARE_UPLOAD_MAX_SIZE_MB || props['firmware.upload.max.size.mb'] || '10', 10 ), + trustProxy: (process.env.TRUST_PROXY || props['trust.proxy'] || 'false').toLowerCase() === 'true', }; } diff --git a/src/service/ASyncTCPService.ts b/src/service/ASyncTCPService.ts index 9ff1521..b21c821 100644 --- a/src/service/ASyncTCPService.ts +++ b/src/service/ASyncTCPService.ts @@ -14,6 +14,76 @@ const UNAUTH_GRACE_MS = 30 * 1000; const MAX_TCP_CLIENTS = 64; const HEX_RE = /^[0-9a-fA-F]*$/; +const OPCODE_AUTH_LEGACY = 0x07; +const OPCODE_AUTH_WITH_VERSION = 0x09; +const OPCODE_WIFI_DIAGNOSTICS = 0x0A; +const MAX_FIRMWARE_VERSION_LEN = 32; + +function decodeEspResetReason(code: number): string { + const reasons: Record = { + 1: 'POWERON', + 2: 'EXT', + 3: 'SW', + 4: 'PANIC', + 5: 'INT_WDT', + 6: 'TASK_WDT', + 7: 'WDT', + 8: 'DEEPSLEEP', + 9: 'BROWNOUT', + 10: 'SDIO', + }; + return reasons[code] || `UNKNOWN(${code})`; +} + +function decodeFailureStage(code: number): string { + const stages: Record = { + 0: 'none', + 1: 'wifi_assoc', + 2: 'wifi_auth', + 3: 'dhcp', + 4: 'dns', + 5: 'tls', + 6: 'proxy_connect', + }; + return stages[code] || `unknown(${code})`; +} + +function decodeWifiDisconnectReason(code: number): string { + const reasons: Record = { + 1: 'UNSPECIFIED', + 2: 'AUTH_EXPIRE', + 3: 'AUTH_LEAVE', + 4: 'ASSOC_EXPIRE', + 5: 'ASSOC_TOOMANY', + 6: 'NOT_AUTHED', + 7: 'NOT_ASSOCED', + 8: 'ASSOC_LEAVE', + 9: 'ASSOC_NOT_AUTHED', + 10: 'DISASSOC_PWRCAP_BAD', + 11: 'DISASSOC_SUPCHAN_BAD', + 13: 'IE_INVALID', + 14: 'MIC_FAILURE', + 15: 'FOURWAY_HANDSHAKE_TIMEOUT', + 16: 'GROUP_KEY_UPDATE_TIMEOUT', + 17: 'IE_IN_4WAY_DIFFERS', + 18: 'GROUP_CIPHER_INVALID', + 19: 'PAIRWISE_CIPHER_INVALID', + 20: 'AKMP_INVALID', + 21: 'UNSUPP_RSN_IE_VERSION', + 22: 'INVALID_RSN_IE_CAP', + 23: 'IEEE_802_1X_AUTH_FAILED', + 24: 'CIPHER_SUITE_REJECTED', + 200: 'BEACON_TIMEOUT', + 201: 'NO_AP_FOUND', + 202: 'AUTH_FAIL', + 203: 'ASSOC_FAIL', + 204: 'HANDSHAKE_TIMEOUT', + 205: 'CONNECTION_FAIL', + 206: 'AP_TSF_RESET', + }; + return reasons[code] || `unknown(${code})`; +} + function createChallenge(): Buffer { return Buffer.from(require('crypto').randomBytes(16)); } @@ -204,21 +274,54 @@ export class ASyncTCPService implements TCPCallback { case 0x02: return; case 0x03: - case 0x07: { - if (message.length < 85) { - this.dropUnauth(client, index, 'short auth'); - return; + case OPCODE_AUTH_LEGACY: + case OPCODE_AUTH_WITH_VERSION: { + let firmwareVersion: string | undefined = undefined; + let preimage: Buffer; + + if (opcode === OPCODE_AUTH_WITH_VERSION) { + if (message.length < 86) { + this.dropUnauth(client, index, 'short auth with version'); + return; + } + const versionLen = message[85]; + if (versionLen > MAX_FIRMWARE_VERSION_LEN) { + this.dropUnauth(client, index, 'version length exceeds max'); + return; + } + if (message.length < 86 + versionLen) { + this.dropUnauth(client, index, 'truncated version'); + return; + } + const versionBytes = message.subarray(86, 86 + versionLen); + firmwareVersion = versionBytes.toString('utf-8'); + + const challenge = client.getChallenge(); + if (!challenge.length) { + this.dropUnauth(client, index, 'auth before challenge'); + return; + } + + preimage = Buffer.concat([challenge, Buffer.from([versionLen]), versionBytes]); + } else { + if (message.length < 85) { + this.dropUnauth(client, index, 'short auth'); + return; + } + const challenge = client.getChallenge(); + if (!challenge.length) { + this.dropUnauth(client, index, 'auth before challenge'); + return; + } + preimage = challenge; } + const addrBytes = message.subarray(0, 20); const sig = message.subarray(20, 85); - const challenge = client.getChallenge(); - if (!challenge.length) { - this.dropUnauth(client, index, 'auth before challenge'); - return; - } + let recoveredAddr = ''; try { - const msgHash = opcode === 0x03 ? getEthereumMessageHash(challenge) : ethers.keccak256(challenge); + const msgHash = opcode === 0x03 ? getEthereumMessageHash(preimage) : ethers.keccak256(preimage); recoveredAddr = recoverAddressFromSignature(msgHash, new Uint8Array(sig)); } catch (err) { console.error('Recover failed #' + index, err instanceof Error ? err.message : err); @@ -242,19 +345,59 @@ export class ASyncTCPService implements TCPCallback { return; } if (authed && client.getAddress().toLowerCase() === addrLower) { - console.log('Refresh auth #' + index, addrLower); + console.log('Refresh auth #' + index, addrLower, firmwareVersion ? `fw:${firmwareVersion}` : ''); this.touchPin(addrLower, ip); return; } client.setAddress(recoveredAddr); this.addToClientMap(recoveredAddr, index); this.touchPin(addrLower, ip); - this.telemetry.recordDeviceConnection(addrLower, 'TCP', ip); + this.telemetry.recordDeviceConnection(addrLower, 'TCP', ip, firmwareVersion); return; } this.dropUnauth(client, index, 'bad signature'); return; } + case OPCODE_WIFI_DIAGNOSTICS: { + if (!authed) { + console.log('Ignore diagnostics from unauthenticated #' + index); + return; + } + if (message.length < 13) { + console.log('Short diagnostics packet #' + index, message.length, 'bytes'); + return; + } + + const resetReason = message[0]; + const bootCount = message.readUInt16BE(1); + const wifiAttempts = message.readUInt16BE(3); + const connectTimeMs = message.readUInt32BE(5); + const networkIndex = message[9]; + const lastFailureStage = message[10]; + const lastDisconnectReason = message[11]; + + const addr = client.getAddress().toLowerCase(); + console.log( + 'WiFi diagnostics #' + index, addr, + 'reset:', decodeEspResetReason(resetReason), + 'boots:', bootCount, + 'wifiAttempts:', wifiAttempts, + 'connectMs:', connectTimeMs, + 'failStage:', decodeFailureStage(lastFailureStage), + 'wifiDisc:', decodeWifiDisconnectReason(lastDisconnectReason) + ); + + this.telemetry.recordDeviceDiagnostics(addr, { + resetReason: decodeEspResetReason(resetReason), + bootCount, + wifiAttempts, + connectTimeMs, + networkIndex, + lastFailureStage: decodeFailureStage(lastFailureStage), + wifiDisconnectReason: decodeWifiDisconnectReason(lastDisconnectReason), + }); + return; + } case 0x04: return; case 0x05: diff --git a/src/service/TelemetryService.ts b/src/service/TelemetryService.ts index 367c0e3..a260800 100644 --- a/src/service/TelemetryService.ts +++ b/src/service/TelemetryService.ts @@ -42,6 +42,33 @@ export interface DeviceMetrics { firmwareVersion?: string; } +export interface DeviceDiagnostics { + id: number; + timestamp: number; + device: string; + resetReason?: string; + bootCount?: number; + wifiAttempts?: number; + connectTimeMs?: number; + networkIndex?: number; + lastFailureStage?: string; + wifiDisconnectReason?: string; +} + +export interface SilentDevice { + address: string; + firstSeen: number; + lastSeen: number; + lastFirmwareVersion?: string; + lastDisconnectReason?: DisconnectReason; + medianReconnectGapMs?: number; + silentDurationMs: number; + expectedReconnectMs: number; + lastEvents: DeviceEvent[]; + lastDiagnostics?: DeviceDiagnostics; + retired: boolean; +} + export interface SystemMetrics { totalDevices: number; totalApiCalls: number; @@ -115,6 +142,30 @@ export class TelemetryService { CREATE INDEX IF NOT EXISTS idx_timestamp ON device_events(timestamp); CREATE INDEX IF NOT EXISTS idx_event_type ON device_events(event_type); CREATE INDEX IF NOT EXISTS idx_id ON device_events(id); + + CREATE TABLE IF NOT EXISTS device_diagnostics ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp INTEGER NOT NULL, + device TEXT NOT NULL, + reset_reason TEXT, + boot_count INTEGER, + wifi_attempts INTEGER, + connect_time_ms INTEGER, + network_index INTEGER, + last_failure_stage TEXT, + wifi_disconnect_reason TEXT + ); + + CREATE INDEX IF NOT EXISTS idx_diagnostics_device ON device_diagnostics(device, timestamp DESC); + CREATE INDEX IF NOT EXISTS idx_diagnostics_timestamp ON device_diagnostics(timestamp); + + CREATE TABLE IF NOT EXISTS device_registry ( + device TEXT PRIMARY KEY, + retired INTEGER DEFAULT 0, + first_seen INTEGER, + last_seen INTEGER, + notes TEXT + ); `); // Migrate schema: add missing columns (idempotent) @@ -175,6 +226,19 @@ export class TelemetryService { VALUES (?, ?, 'connect', ?, ?, ?) `); stmt.run(now, normalized, protocol, ipAddress || null, firmwareVersion || null); + + this.updateDeviceRegistry(normalized, now); + } + + private updateDeviceRegistry(device: string, timestamp: number): void { + const existing = this.db.prepare('SELECT * FROM device_registry WHERE device = ?').get(device) as any; + + if (existing) { + this.db.prepare('UPDATE device_registry SET last_seen = ? WHERE device = ?').run(timestamp, device); + } else { + this.db.prepare('INSERT INTO device_registry (device, first_seen, last_seen, retired) VALUES (?, ?, ?, 0)') + .run(device, timestamp, timestamp); + } } recordDeviceDisconnect(address: string, reason: DisconnectReason): void { @@ -424,12 +488,159 @@ export class TelemetryService { }; } + recordDeviceDiagnostics( + address: string, + diagnostics: { + resetReason?: string; + bootCount?: number; + wifiAttempts?: number; + connectTimeMs?: number; + networkIndex?: number; + lastFailureStage?: string; + wifiDisconnectReason?: string; + } + ): void { + const normalized = address.toLowerCase(); + const now = Date.now(); + + const stmt = this.db.prepare(` + INSERT INTO device_diagnostics ( + timestamp, device, reset_reason, boot_count, wifi_attempts, + connect_time_ms, network_index, last_failure_stage, wifi_disconnect_reason + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + `); + + stmt.run( + now, + normalized, + diagnostics.resetReason ?? null, + diagnostics.bootCount ?? null, + diagnostics.wifiAttempts ?? null, + diagnostics.connectTimeMs ?? null, + diagnostics.networkIndex ?? null, + diagnostics.lastFailureStage ?? null, + diagnostics.wifiDisconnectReason ?? null + ); + } + + getLatestDiagnostics(address: string): DeviceDiagnostics | undefined { + const normalized = address.toLowerCase(); + const row = this.db.prepare(` + SELECT + id, timestamp, device, + reset_reason as resetReason, + boot_count as bootCount, + wifi_attempts as wifiAttempts, + connect_time_ms as connectTimeMs, + network_index as networkIndex, + last_failure_stage as lastFailureStage, + wifi_disconnect_reason as wifiDisconnectReason + FROM device_diagnostics + WHERE device = ? + ORDER BY timestamp DESC + LIMIT 1 + `).get(normalized) as any; + + return row || undefined; + } + + getSilentDevices(options: { + cadenceMultiplier?: number; + minSilenceMs?: number; + includeRetired?: boolean; + } = {}): SilentDevice[] { + const { + cadenceMultiplier = 3, + minSilenceMs = 15 * 60 * 1000, + includeRetired = false, + } = options; + + const now = Date.now(); + const devices: SilentDevice[] = []; + + const registryFilter = includeRetired ? '' : 'WHERE retired = 0'; + const registryDevices = this.db.prepare(` + SELECT device, first_seen, last_seen, retired FROM device_registry ${registryFilter} + `).all() as any[]; + + for (const reg of registryDevices) { + const device = reg.device; + const lastSeen = reg.last_seen; + const silentDurationMs = now - lastSeen; + + const metrics = this.getDeviceMetrics(device); + if (!metrics) continue; + + const medianGap = metrics.medianReconnectTime || 0; + const expectedReconnect = Math.max(medianGap * cadenceMultiplier, minSilenceMs); + + if (silentDurationMs > expectedReconnect) { + const lastDisconnect = this.db.prepare(` + SELECT disconnect_reason FROM device_events + WHERE device = ? AND event_type = 'disconnect' + ORDER BY timestamp DESC LIMIT 1 + `).get(device) as any; + + const lastEvents = this.db.prepare(` + SELECT + id, timestamp, device, event_type as eventType, protocol, + disconnect_reason as disconnectReason, api_method as apiMethod, + api_success as apiSuccess, api_response_time as apiResponseTime, + ip_address as ipAddress, firmware_version as firmwareVersion + FROM device_events + WHERE device = ? + ORDER BY timestamp DESC LIMIT 10 + `).all(device) as any[]; + + const lastDiagnostics = this.getLatestDiagnostics(device); + + devices.push({ + address: device, + firstSeen: reg.first_seen, + lastSeen: lastSeen, + lastFirmwareVersion: metrics.firmwareVersion || undefined, + lastDisconnectReason: lastDisconnect?.disconnect_reason, + medianReconnectGapMs: medianGap || undefined, + silentDurationMs, + expectedReconnectMs: expectedReconnect, + lastEvents: lastEvents.map(e => ({ + ...e, + apiSuccess: e.apiSuccess !== null && e.apiSuccess !== undefined ? e.apiSuccess === 1 : undefined, + })), + lastDiagnostics, + retired: reg.retired === 1, + }); + } + } + + return devices.sort((a, b) => b.silentDurationMs - a.silentDurationMs); + } + + markDeviceRetired(address: string, retired: boolean, notes?: string): void { + const normalized = address.toLowerCase(); + + const existing = this.db.prepare('SELECT * FROM device_registry WHERE device = ?').get(normalized); + + if (existing) { + const stmt = this.db.prepare('UPDATE device_registry SET retired = ?, notes = ? WHERE device = ?'); + stmt.run(retired ? 1 : 0, notes || null, normalized); + } else { + const now = Date.now(); + const stmt = this.db.prepare( + 'INSERT INTO device_registry (device, retired, notes, first_seen, last_seen) VALUES (?, ?, ?, ?, ?)' + ); + stmt.run(normalized, retired ? 1 : 0, notes || null, now, now); + } + } + close(): void { this.db.close(); } reset(): void { this.db.exec('DELETE FROM device_events'); + this.db.exec('DELETE FROM device_diagnostics'); + this.db.exec('DELETE FROM device_registry'); this.activeSessions.clear(); this.systemStartTime = Date.now(); } diff --git a/tools/telemetry/analyse.ts b/tools/telemetry/analyse.ts index 11a5f90..ca33ada 100644 --- a/tools/telemetry/analyse.ts +++ b/tools/telemetry/analyse.ts @@ -7,18 +7,35 @@ * - Signature/auth failures * - Timeouts vs keepalive issues * - Failing/slow API methods + * - Silent devices (expected but not connecting) * - Attribution: proxy vs Web3E TcpBridge/UdpBridge vs device firmware * * Usage: - * npm run telemetry:analyse [-- data/telemetry-YYYYMMDD.jsonl] - * OR: ts-node tools/telemetry/analyse.ts [data/telemetry-YYYYMMDD.jsonl] + * npm run telemetry:analyse [-- data/telemetry-YYYYMMDD.jsonl] [--api-url=http://localhost:8081] [--api-key=...] + * OR: ts-node tools/telemetry/analyse.ts [data/telemetry-YYYYMMDD.jsonl] [--api-url=...] [--api-key=...] + * + * For silent device analysis, provide --api-url and --api-key flags. + * These can also be set in tools/telemetry/.env as TELEMETRY_API_URL and TELEMETRY_API_KEY. * * Outputs markdown report to stdout */ +import { config } from 'dotenv'; import { readFileSync, readdirSync, existsSync } from 'fs'; import { join } from 'path'; +// Try loading .env from multiple locations +const envPaths = [ + join(__dirname, '.env'), // tools/telemetry/.env + join(__dirname, '..', '..', '.env'), // repo root .env +]; +for (const envPath of envPaths) { + if (existsSync(envPath)) { + config({ path: envPath }); + break; + } +} + interface Event { id: number; timestamp: number; @@ -60,6 +77,35 @@ interface DeviceAnalysis { firmwareVersion?: string; } +interface SilentDevice { + address: string; + firstSeen: number; + lastSeen: number; + lastFirmwareVersion?: string; + lastDisconnectReason?: string; + medianReconnectGapMs?: number; + silentDurationMs: number; + expectedReconnectMs: number; + lastEvents: Event[]; + lastDiagnostics?: { + resetReason?: string; + bootCount?: number; + wifiAttempts?: number; + connectTimeMs?: number; + networkIndex?: number; + ssidHash?: string; + lastFailureStage?: string; + }; + retired: boolean; +} + +interface SilentDeviceAnalysis { + device: SilentDevice; + silentDays: number; + likelyFailureCause: string; + diagnosticClues: string[]; +} + function parseJsonlFile(path: string): Event[] { const content = readFileSync(path, 'utf-8'); return content @@ -179,7 +225,82 @@ function analyzeDevice(events: Event[]): DeviceAnalysis { }; } -function generateMarkdownReport(analyses: Map): string { +function analyzeSilentDevice(device: SilentDevice): SilentDeviceAnalysis { + const silentDays = device.silentDurationMs / (24 * 60 * 60 * 1000); + const diagnosticClues: string[] = []; + let likelyFailureCause = 'Unknown - no diagnostics available'; + + if (device.lastDiagnostics) { + const diag = device.lastDiagnostics; + + if (diag.resetReason) { + diagnosticClues.push(`Last reset: ${diag.resetReason}`); + if (diag.resetReason.match(/power|brownout|watchdog/i)) { + likelyFailureCause = 'Power instability or firmware crash (watchdog/brownout reset)'; + } + } + + if (diag.wifiAttempts && diag.wifiAttempts > 5) { + diagnosticClues.push(`High WiFi attempts: ${diag.wifiAttempts}`); + likelyFailureCause = 'WiFi association/credential issues'; + } + + if (diag.lastFailureStage) { + diagnosticClues.push(`Last failure at: ${diag.lastFailureStage}`); + if (diag.lastFailureStage.match(/dns/i)) { + likelyFailureCause = 'DNS resolution failure (network/router config)'; + } else if (diag.lastFailureStage.match(/tls|ssl|cert/i)) { + likelyFailureCause = 'TLS/certificate validation failure'; + } else if (diag.lastFailureStage.match(/tcp|connect/i)) { + likelyFailureCause = 'TCP connection failure (firewall/proxy unreachable)'; + } else if (diag.lastFailureStage.match(/dhcp/i)) { + likelyFailureCause = 'DHCP failure (network infrastructure)'; + } + } + + if (diag.connectTimeMs && diag.connectTimeMs > 30000) { + diagnosticClues.push(`Slow connect: ${(diag.connectTimeMs / 1000).toFixed(1)}s`); + } + + if (diag.bootCount !== undefined) { + diagnosticClues.push(`Boot count: ${diag.bootCount}`); + if (diag.bootCount > 100 && silentDays < 7) { + likelyFailureCause = 'Reboot loop (likely firmware crash or power cycling)'; + } + } + } + + if (device.lastDisconnectReason) { + diagnosticClues.push(`Last disconnect: ${device.lastDisconnectReason}`); + if (device.lastDisconnectReason === 'auth_failure' || device.lastDisconnectReason === 'signature_failure') { + likelyFailureCause = 'Authentication failure (credential mismatch or firmware bug)'; + } else if (device.lastDisconnectReason === 'timeout') { + likelyFailureCause = 'Network timeout (intermittent connectivity or keepalive failure)'; + } + } + + const recentConnectFailures = device.lastEvents.filter(e => + e.eventType === 'disconnect' && + (e.disconnectReason === 'auth_failure' || e.disconnectReason === 'signature_failure') + ).length; + + if (recentConnectFailures >= 2 && diagnosticClues.length === 0) { + likelyFailureCause = 'Repeated authentication failures suggest credential/firmware issue'; + } + + if (diagnosticClues.length === 0 && silentDays > 30) { + likelyFailureCause = 'Device likely decommissioned or permanently offline'; + } + + return { + device, + silentDays, + likelyFailureCause, + diagnosticClues, + }; +} + +function generateMarkdownReport(analyses: Map, silentDevices?: SilentDeviceAnalysis[]): string { let md = '# Telemetry Analysis Report\n\n'; md += `Generated: ${new Date().toISOString()}\n\n`; md += `Total devices analyzed: ${analyses.size}\n\n`; @@ -190,7 +311,101 @@ function generateMarkdownReport(analyses: Map): string { ); md += `- Devices with issues: ${withIssues.length}\n`; md += `- Devices with reconnect storms: ${Array.from(analyses.values()).filter(a => a.reconnectStorm).length}\n`; - md += `- Devices with auth failures: ${Array.from(analyses.values()).filter(a => a.authFailures > 0).length}\n\n`; + md += `- Devices with auth failures: ${Array.from(analyses.values()).filter(a => a.authFailures > 0).length}\n`; + if (silentDevices) { + md += `- Silent devices (expected but not connecting): ${silentDevices.filter(d => !d.device.retired).length}\n`; + } + md += '\n'; + + if (silentDevices && silentDevices.length > 0) { + md += '## Silent Devices\n\n'; + md += '**Survivorship-bias analysis:** Devices that should be connecting based on historical patterns but have gone silent.\n\n'; + + const activelyMissing = silentDevices.filter(d => !d.device.retired); + + if (activelyMissing.length === 0) { + md += '*All silent devices are marked as retired. No action needed.*\n\n'; + } else { + md += `**${activelyMissing.length} device(s) need attention:**\n\n`; + + for (const analysis of activelyMissing) { + const dev = analysis.device; + md += `### 🔴 \`${dev.address}\`\n\n`; + md += `**Silent for:** ${analysis.silentDays.toFixed(1)} days (expected reconnect: ${(dev.expectedReconnectMs / 1000 / 60).toFixed(0)} minutes)\n\n`; + md += `**Last seen:** ${new Date(dev.lastSeen).toISOString()}\n\n`; + + if (dev.lastFirmwareVersion) { + md += `**Last firmware:** ${dev.lastFirmwareVersion}\n\n`; + } + + md += `**Likely cause:** ${analysis.likelyFailureCause}\n\n`; + + if (analysis.diagnosticClues.length > 0) { + md += '**Diagnostic clues:**\n'; + for (const clue of analysis.diagnosticClues) { + md += `- ${clue}\n`; + } + md += '\n'; + } + + md += '**Pre-connection failure analysis:**\n'; + if (dev.lastDiagnostics?.lastFailureStage) { + md += `- Last failure stage: **${dev.lastDiagnostics.lastFailureStage}**\n`; + md += '- Investigation steps:\n'; + const stage = dev.lastDiagnostics.lastFailureStage.toLowerCase(); + if (stage.includes('wifi') || stage.includes('assoc')) { + md += ' 1. Check WiFi network availability and signal strength\n'; + md += ' 2. Verify WiFi credentials in device config\n'; + md += ' 3. Check for MAC address filtering or network policy changes\n'; + } else if (stage.includes('dhcp')) { + md += ' 1. Verify DHCP server is running and has available leases\n'; + md += ' 2. Check for IP address conflicts\n'; + md += ' 3. Review router logs for DHCP denials\n'; + } else if (stage.includes('dns')) { + md += ' 1. Verify DNS server configuration in network\n'; + md += ' 2. Test DNS resolution from another device on same network\n'; + md += ' 3. Check for DNS blocking/filtering\n'; + } else if (stage.includes('tls') || stage.includes('ssl')) { + md += ' 1. Check device system time (TLS requires accurate time)\n'; + md += ' 2. Verify certificate chain is valid\n'; + md += ' 3. Check for TLS version mismatch (device may need firmware update)\n'; + } else if (stage.includes('tcp') || stage.includes('connect')) { + md += ' 1. Verify proxy/server is reachable from device network\n'; + md += ' 2. Check firewall rules blocking outbound connections\n'; + md += ' 3. Test connectivity with curl/telnet from same network\n'; + } + } else { + md += '- No diagnostic data from next boot attempt (device never reconnected)\n'; + md += '- Possible causes: power loss, physical damage, complete network loss\n'; + } + md += '\n'; + + if (dev.lastEvents.length > 0) { + md += '**Last 3 events:**\n'; + for (const event of dev.lastEvents.slice(0, 3)) { + const time = new Date(event.timestamp).toISOString(); + md += `- ${time}: ${event.eventType}`; + if (event.disconnectReason) md += ` (${event.disconnectReason})`; + if (event.apiMethod) md += ` - ${event.apiMethod}`; + md += '\n'; + } + md += '\n'; + } + + md += '---\n\n'; + } + } + + if (silentDevices.some(d => d.device.retired)) { + const retired = silentDevices.filter(d => d.device.retired); + md += `### Retired Devices (${retired.length})\n\n`; + md += 'These devices are marked as decommissioned and no longer alert:\n\n'; + for (const analysis of retired) { + md += `- \`${analysis.device.address}\` - last seen ${analysis.silentDays.toFixed(0)} days ago\n`; + } + md += '\n'; + } + } md += '## Device Details\n\n'; for (const [device, analysis] of Array.from(analyses.entries()).sort((a: [string, DeviceAnalysis], b: [string, DeviceAnalysis]) => { @@ -248,8 +463,49 @@ function findLatestFile(): string | null { } } +async function fetchSilentDevices(apiUrl: string, apiKey: string): Promise { + const url = `${apiUrl}/api/telemetry/devices/silent`; + + const response = await fetch(url, { + headers: { + 'Authorization': `Bearer ${apiKey}`, + }, + }); + + if (!response.ok) { + throw new Error(`API request failed: ${response.status} ${response.statusText}`); + } + + const data = await response.json() as { devices: SilentDevice[] }; + return data.devices; +} + +function parseArgs(): { inputFile?: string; apiUrl?: string; apiKey?: string } { + const args = process.argv.slice(2); + let inputFile: string | undefined; + let apiUrl: string | undefined; + let apiKey: string | undefined; + + for (const arg of args) { + if (arg.startsWith('--api-url=')) { + apiUrl = arg.substring('--api-url='.length); + } else if (arg.startsWith('--api-key=')) { + apiKey = arg.substring('--api-key='.length); + } else if (!arg.startsWith('--')) { + inputFile = arg; + } + } + + // Fall back to environment variables if not provided via CLI + if (!apiUrl) apiUrl = process.env.TELEMETRY_API_URL; + if (!apiKey) apiKey = process.env.TELEMETRY_API_KEY; + + return { inputFile, apiUrl, apiKey }; +} + async function main() { - let inputFile: string | undefined = process.argv[2]; + const { inputFile: argFile, apiUrl, apiKey } = parseArgs(); + let inputFile = argFile; if (!inputFile) { const latestFile = findLatestFile(); @@ -277,7 +533,19 @@ async function main() { analyses.set(device, analyzeDevice(devEvents.sort((a, b) => a.timestamp - b.timestamp))); } - const report = generateMarkdownReport(analyses); + let silentAnalyses: SilentDeviceAnalysis[] | undefined; + if (apiUrl && apiKey) { + console.error('Fetching silent devices from API...\n'); + try { + const silentDevices = await fetchSilentDevices(apiUrl, apiKey); + console.error(`Found ${silentDevices.length} silent devices\n`); + silentAnalyses = silentDevices.map(analyzeSilentDevice); + } catch (err) { + console.error(`Warning: Could not fetch silent devices: ${err}\n`); + } + } + + const report = generateMarkdownReport(analyses, silentAnalyses); console.log(report); }