Route isolated browser UDP through egress-proxy connect-udp

experiments/udp-egress/.gitignore

@@ -0,0 +1 @@
+__pycache__/

experiments/udp-egress/Dockerfile

@@ -0,0 +1,8 @@
+ARG BASE_IMAGE=kernel-images:isp-base
+FROM ${BASE_IMAGE}
+RUN apt-get update && apt-get install -y --no-install-recommends iproute2 iptables libnss3-tools python3 socat tcpdump && rm -rf /var/lib/apt/lists/*
+COPY experiments/udp-egress/*.py experiments/udp-egress/*.cjs experiments/udp-egress/chromium-launcher.sh /opt/udp-egress/
+RUN chmod +x /opt/udp-egress/chromium-launcher.sh && \
+    sed -i 's|command=/usr/local/bin/chromium-launcher|command=/opt/udp-egress/chromium-launcher.sh|' /etc/supervisor/conf.d/services/chromium.conf && \
+    cd /opt/udp-egress && PYTHONDONTWRITEBYTECODE=1 python3 -m unittest -v test_protocol.py
+COPY experiments/udp-egress/network.conf /etc/supervisor/conf.d/services/udp-egress.conf

experiments/udp-egress/README.md

@@ -0,0 +1,139 @@
+# Isolated browser UDP experiment
+
+This opt-in headful image experiment sends a browser's HTTP(S) and UDP through
+one egress-proxy process: HTTP(S) as an ordinary HTTP proxy (CONNECT, with
+egress-proxy's MITM) and UDP as connect-udp
+([RFC 9298](https://www.rfc-editor.org/rfc/rfc9298)) on the same listener. It
+needs an egress-proxy built with connect-udp support and started with
+`EGRESS_PROXY_UDP_ENABLED=true`. It does not change the normal images. Use only
+a disposable, locally run container.
+
+The container holds no proxy credentials. egress-proxy authorizes both kinds of
+traffic by the container's source address, selects the session's upstream, and
+relays UDP through that upstream's SOCKS5 UDP association.
+
+## Isolation
+
+Chromium runs as the unprivileged `kernel` user in the `browser-udp` network
+namespace. The desktop, live-view server, DevTools bridge, TCP relay and UDP
+shim stay in the container's original namespace.
+
+Chromium's only proxy is `192.0.2.1:18080`, a plain TCP relay to egress-proxy.
+The UDP DNAT rules capture only the browser's veth. Each explicit test
+destination has its own local listener; the shim opens one connect-udp request
+per browser flow and carries the unchanged payload as DATAGRAM capsules.
+Returned payloads pass through normal reverse NAT, without changing STUN
+contents or browser candidates. Unlisted UDP destinations are denied.
+
+This intentionally uses explicit per-test routes rather than general transparent
+UDP interception. On some kernels, ancillary destination information contains the
+post-DNAT address, not the original destination. General interception needs a
+separately validated original-destination mechanism; this experiment does not
+assume that mechanism works.
+
+An independent FORWARD deny rule is installed before the browser gets its default
+route. If the shim stops, packets reach a closed local port. If the DNAT rule
+disappears, forwarding remains denied. If egress-proxy refuses or drops a flow,
+nothing else carries it. The browser namespace has IPv6 disabled when the kernel
+supports it; this is not a dual-stack forwarding test. DevTools uses TCP loopback
+bridges; live-view media is not intercepted.
+
+## Build and run
+
+Build the base image from the repository root using the normal headful Dockerfile,
+then build this experiment:
+
+```sh
+GITHUB_TOKEN="$(gh auth token)" DOCKER_BUILDKIT=1 \
+  docker build --secret id=github_token,env=GITHUB_TOKEN \
+  -t kernel-images:udp-base -f images/chromium-headful/Dockerfile .
+docker build --build-arg BASE_IMAGE=kernel-images:udp-base \
+  -t kernel-images:udp-experiment -f experiments/udp-egress/Dockerfile .
+```
+
+Create a directory outside the checkout containing `egress.json`, using
+`egress.example.json` as the schema, and egress-proxy's MITM CA certificate as
+`mitm-ca.crt` (certificate only, never the key). `egress_host` must be the
+egress-proxy address the container reaches; egress-proxy sees the container's
+address as the source. Set `udp_targets` to the controlled STUN receivers'
+public IPv4 addresses and ports, plus the public DNS resolver used by the
+browser namespace (`8.8.8.8`). Configure it before starting the container.
+
+egress-proxy must authorize the container's address as a claimed instance whose
+route resolves to a Byteful ISP upstream. Give the container a fixed address on
+a dedicated network so that mapping stays valid across restarts. Keep HTTP, API
+and DevTools ports on loopback and publish only the live-view media range:
+
+```sh
+docker network create --subnet 172.30.0.0/24 udp-egress-net
+docker run --rm --privileged --tmpfs /dev/shm:size=2g \
+  --name udp-egress-test --network udp-egress-net --ip 172.30.0.10 \
+  -p 127.0.0.1:18081:8080 -p 127.0.0.1:19222:9222 \
+  -p 127.0.0.1:10001:10001 -p 59000-59100:59000-59100/udp \
+  -v "$CONFIG_DIR:/run/udp-egress:ro" \
+  -e WITHDOCKER=false -e ENABLE_WEBRTC=true \
+  -e NEKO_WEBRTC_EPR=59000-59100 -e NEKO_WEBRTC_NAT1TO1="$VIEWER_REACHABLE_IP" \
+  -e CHROMIUM_FLAGS="--no-sandbox --no-zygote --disable-dev-shm-usage --disable-quic --proxy-server=http://192.0.2.1:18080 --webrtc-ip-handling-policy=disable_non_proxied_udp" \
+  kernel-images:udp-experiment
+```
+
+The normal launcher still supplies the display and DevTools settings. To enable
+WebRTC UDP, set the policy in `/chromium/flags` and restart only the Chromium
+supervisor program. Keep the shim and deny rule in place first, and do not relax
+the policy in a normal browser image. `PATCH /chromium/flags` appends a second
+policy rather than replacing the first, so write the file directly:
+
+```sh
+docker exec udp-egress-test sh -c \
+  'mkdir -p /chromium &&
+   printf "{\"flags\":[\"--webrtc-ip-handling-policy=default\"]}" > /chromium/flags &&
+   supervisorctl restart chromium'
+```
+
+## Validation
+
+Use HTTPS and STUN receivers you control, or equivalent independently verifiable
+receiver-side observations. The HTTPS echo endpoint used by `probe.cjs` must
+return JSON containing the observed source address as `ip`. Add a unique query
+string so a cached response cannot stand in for a request.
+
+```sh
+python3 -m unittest discover -s experiments/udp-egress -p 'test_*.py'
+docker exec udp-egress-test node /opt/udp-egress/probe.cjs \
+  "https://receiver.example.test/request?nonce=$RANDOM" stun:203.0.113.20:3478 baseline
+```
+
+Repeat with WebRTC UDP enabled. Require the HTTPS receiver source IP, the STUN
+receiver's XOR-MAPPED-ADDRESS, and the browser's server-reflexive candidate to
+agree, and the HTTPS issuer to be the MITM CA. Correlate the receiver response
+transaction IDs in `/var/log/supervisord/udp-relay` with outgoing requests.
+Candidates alone are not proof of the forwarding path.
+
+Capture the container's external interface and the egress host's while testing.
+No browser UDP should leave the container, and the egress host should exchange
+UDP only with the upstream's relay, never with the STUN receivers. Test all of:
+
+- fresh peer connections and associations;
+- a Chromium restart;
+- `supervisorctl stop udp-relay`, with HTTPS and live view still working;
+- egress-proxy with UDP disabled, then stopped, checking there is no direct path;
+- an egress-proxy restart while an association is open;
+- instance invalidation while an association is open;
+- deletion of only this experiment's DNAT rule, checking FORWARD deny counters;
+- whole-container restart, re-enabling UDP only after the guard is installed;
+- IPv6 behavior, distinguishing an IPv6-disabled kernel from an IPv6-capable test.
+
+Verify live-view video frames advance and the service records a connected ICE
+state. A healthy HTTP endpoint does not establish media health.
+
+## Limits
+
+The shim is intentionally IPv4-only, permits only the explicit public test
+destinations, and caps itself at 32 flows with a 60-second idle timeout. It is
+not a production design for capturing browser UDP. Results for an ISP binding do
+not establish behavior for rotating residential bindings, other providers,
+different kernels, or the deployed stealth browser.
+
+No production credentials belong in captures. Detach the private exposure before
+stopping the container, then remove the runtime directory. Do not leave an
+enabled experiment running unattended.

experiments/udp-egress/chromium-launcher.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+set -eu
+while [ ! -f /run/udp-egress-ready ]; do sleep 0.1; done
+exec ip netns exec browser-udp /usr/local/bin/chromium-launcher "$@"

experiments/udp-egress/common.py

@@ -0,0 +1,98 @@
+import ipaddress
+import json
+import socket
+
+
+def config():
+    with open('/run/udp-egress/egress.json') as f:
+        return json.load(f)
+
+
+def udp_targets(settings):
+    targets = settings['udp_targets']
+    if not isinstance(targets, list) or not 0 < len(targets) <= 16:
+        raise ValueError('configure between 1 and 16 explicit UDP targets')
+    seen = set()
+    for target in targets:
+        address = ipaddress.ip_address(target['host'])
+        port = target['port']
+        if address.version != 4 or not address.is_global:
+            raise ValueError('UDP targets must be public IPv4 addresses')
+        if type(port) is not int or not 1 <= port <= 65535:
+            raise ValueError('invalid UDP target port')
+        endpoint = (str(address), port)
+        if endpoint in seen:
+            raise ValueError('duplicate UDP target')
+        seen.add(endpoint)
+    return targets
+
+
+def encode_varint(value):
+    for size, prefix in ((1, 0), (2, 0x40), (4, 0x80), (8, 0xc0)):
+        if value < 1 << (8 * size - 2):
+            data = bytearray(value.to_bytes(size, 'big'))
+            data[0] |= prefix
+            return bytes(data)
+    raise ValueError('varint too large')
+
+
+def decode_varint(data, offset=0):
+    """Returns (value, next offset), or None when data is incomplete."""
+    if offset >= len(data):
+        return None
+    size = 1 << (data[offset] >> 6)
+    if offset + size > len(data):
+        return None
+    value = data[offset] & 0x3f
+    for byte in data[offset + 1:offset + size]:
+        value = value << 8 | byte
+    return value, offset + size
+
+
+def datagram_capsule(payload):
+    return encode_varint(0) + encode_varint(len(payload) + 1) + b'\x00' + payload
+
+
+def parse_capsules(buffer):
+    """Splits complete capsules off buffer. Returns (datagram payloads, rest)."""
+    payloads = []
+    offset = 0
+    while True:
+        kind = decode_varint(buffer, offset)
+        if kind is None:
+            break
+        length = decode_varint(buffer, kind[1])
+        if length is None or length[1] + length[0] > len(buffer):
+            break
+        body = buffer[length[1]:length[1] + length[0]]
+        offset = length[1] + length[0]
+        context = decode_varint(body)
+        if kind[0] == 0 and context is not None and context[0] == 0:
+            payloads.append(body[context[1]:])
+    return payloads, buffer[offset:]
+
+
+def open_tunnel(settings, destination):
+    """Opens a connect-udp tunnel to destination through egress-proxy."""
+    sock = socket.create_connection((settings['egress_host'], settings['egress_port']), timeout=10)
+    try:
+        host, port = destination
+        sock.sendall((f'GET /.well-known/masque/udp/{host}/{port}/ HTTP/1.1\r\n'
+                      f'Host: {settings["egress_host"]}:{settings["egress_port"]}\r\n'
+                      'Connection: Upgrade\r\nUpgrade: connect-udp\r\n'
+                      'Capsule-Protocol: ?1\r\n\r\n').encode())
+        response = b''
+        while b'\r\n\r\n' not in response:
+            part = sock.recv(4096)
+            if not part or len(response) > 16384:
+                raise OSError('egress closed during connect-udp')
+            response += part
+        head, rest = response.split(b'\r\n\r\n', 1)
+        status = head.split(b'\r\n', 1)[0]
+        if status.split(b' ')[1:2] != [b'101']:
+            raise OSError(f'connect-udp refused: {status.decode(errors="replace")}')
+        sock.settimeout(None)
+        return sock, rest
+    except Exception:
+        sock.close()
+        raise

experiments/udp-egress/egress.example.json

@@ -0,0 +1,8 @@
+{
+  "egress_host": "192.0.2.10",
+  "egress_port": 3129,
+  "udp_targets": [
+    { "host": "8.8.8.8", "port": 53 },
+    { "host": "203.0.113.20", "port": 3478 }
+  ]
+}

experiments/udp-egress/network.conf

@@ -0,0 +1,21 @@
+[program:udp-egress]
+command=python3 -u /opt/udp-egress/network.py
+priority=1
+autostart=true
+autorestart=false
+startsecs=1
+stopasgroup=true
+killasgroup=true
+stdout_logfile=/var/log/supervisord/udp-egress
+redirect_stderr=true
+
+[program:udp-relay]
+command=python3 -u /opt/udp-egress/udp.py
+priority=2
+autostart=true
+autorestart=false
+startsecs=1
+stopasgroup=true
+killasgroup=true
+stdout_logfile=/var/log/supervisord/udp-relay
+redirect_stderr=true

experiments/udp-egress/network.py

@@ -0,0 +1,71 @@
+import pathlib
+import subprocess
+import time
+
+from common import config, udp_targets
+
+
+def run(*args):
+    subprocess.run(args, check=True)
+
+
+def main():
+    ready = pathlib.Path('/run/udp-egress-ready')
+    ready.unlink(missing_ok=True)
+    run('ip', 'netns', 'add', 'browser-udp')
+    run('ip', 'link', 'add', 'udp-host', 'type', 'veth', 'peer', 'name', 'udp-browser')
+    run('ip', 'link', 'set', 'udp-browser', 'netns', 'browser-udp')
+    run('iptables', '-I', 'FORWARD', '1', '-i', 'udp-host', '-j', 'DROP')
+    settings = config()
+    targets = udp_targets(settings)
+    for index, target in enumerate(targets):
+        run('iptables', '-t', 'nat', '-A', 'PREROUTING', '-i', 'udp-host', '-p', 'udp',
+            '-d', target['host'], '--dport', str(target['port']), '-j', 'DNAT',
+            '--to-destination', f'192.0.2.1:{40000 + index}')
+    run('ip', 'addr', 'add', '192.0.2.1/30', 'dev', 'udp-host')
+    run('ip', 'link', 'set', 'udp-host', 'up')
+    prefix = ('ip', 'netns', 'exec', 'browser-udp')
+    if pathlib.Path('/proc/sys/net/ipv6').exists():
+        run(*prefix, 'sysctl', '-w', 'net.ipv6.conf.all.disable_ipv6=1',
+            'net.ipv6.conf.default.disable_ipv6=1')
+    run(*prefix, 'ip', 'link', 'set', 'lo', 'up')
+    run(*prefix, 'ip', 'addr', 'add', '192.0.2.2/30', 'dev', 'udp-browser')
+    run(*prefix, 'ip', 'link', 'set', 'udp-browser', 'up')
+    run(*prefix, 'ip', 'route', 'add', 'default', 'via', '192.0.2.1')
+    resolv = pathlib.Path('/etc/netns/browser-udp')
+    resolv.mkdir(parents=True)
+    (resolv / 'resolv.conf').write_text('nameserver 8.8.8.8\n')
+
+    ca = pathlib.Path('/run/udp-egress/mitm-ca.crt')
+    if ca.exists():
+        nssdb = pathlib.Path('/home/kernel/.pki/nssdb')
+        run('runuser', '-u', 'kernel', '--', 'mkdir', '-p', str(nssdb))
+        if not (nssdb / 'cert9.db').exists():
+            run('runuser', '-u', 'kernel', '--', 'certutil', '-N', '-d', f'sql:{nssdb}', '--empty-password')
+        run('runuser', '-u', 'kernel', '--', 'certutil', '-A', '-d', f'sql:{nssdb}',
+            '-t', 'C,,', '-n', 'egress-proxy-mitm', '-i', str(ca))
+
+    processes = [
+        subprocess.Popen(['socat', 'TCP4-LISTEN:18080,bind=192.0.2.1,reuseaddr,fork',
+                          f'TCP4:{settings["egress_host"]}:{settings["egress_port"]}']),
+        subprocess.Popen(['socat', 'TCP4-LISTEN:9223,bind=127.0.0.1,reuseaddr,fork',
+                          'TCP4:192.0.2.2:9223']),
+        subprocess.Popen([*prefix, 'socat',
+                          'TCP4-LISTEN:9223,bind=192.0.2.2,reuseaddr,fork',
+                          'TCP4:127.0.0.1:9223']),
+    ]
+    time.sleep(1)
+    if any(p.poll() is not None for p in processes):
+        raise RuntimeError('network helper failed during startup')
+    ready.touch()
+    print('browser namespace ready; forwarding outside the relay is denied', flush=True)
+    while True:
+        for p in processes:
+            if p.poll() is not None:
+                ready.unlink(missing_ok=True)
+                raise RuntimeError('network helper exited; namespace forwarding remains denied')
+        time.sleep(1)
+
+
+if __name__ == '__main__':
+    main()