From 8e97303713dd347355d5d5e27c49848261c441a5 Mon Sep 17 00:00:00 2001 From: Alex Leventer Date: Sun, 18 Oct 2026 19:54:14 -0700 Subject: [PATCH 1/2] Add Cloudflare 525, TLS socket, xterm-kitty and regions-vs-AZ docs Search Console shows these queries landing on old community threads or on pages that don't answer them: - Cloudflare error 525 (2,319 impressions/28d, position ~8, 1 click) - "client network socket disconnected before secure TLS connection was established" (467 impressions, position 6.2) - "error opening terminal: xterm-kitty" (325 impressions, position 6.3) - AWS zones vs regions (1,058 impressions on /reference/regions, 0 clicks) Co-Authored-By: Claude Opus 5.5 --- getting-started/troubleshooting.mdx | 25 +++++++++++++++++++++++++ networking/understanding-cloudflare.mdx | 10 ++++++++++ reference/regions.mdx | 12 ++++++++++++ 3 files changed, 47 insertions(+) diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index 06c278d412..cd907fa16a 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -26,6 +26,7 @@ If the error isn't obvious, start here. - [401 Unauthorized](#registry-401-errors) — registry auth failure during deploy - [403 Forbidden](#403-forbidden) — usually your app's CORS config or a third-party block - [520 with Cloudflare](#520-errors-with-cloudflare) — Cloudflare doesn't like the response + - [525 with Cloudflare](/networking/understanding-cloudflare#cloudflare-error-525-ssl-handshake-failed) — TLS handshake between Cloudflare and Fly.io failed - **My deploy failed** - [Build hangs at 'Waiting for depot builder...'](#build-hangs-waiting-for-depot-builder) - [Release command failures](#release-command-failures) @@ -41,6 +42,8 @@ If the error isn't obvious, start here. - [Suspend vs stop (clock skew)](#suspend-vs-stop) - **I can't connect to something** - [Custom domain TLS errors](#custom-domains-and-tls) + - ["Client network socket disconnected before secure TLS connection was established"](#client-network-socket-disconnected-before-secure-tls-connection-was-established) + - ["Error opening terminal: xterm-kitty" in `fly ssh console`](#error-opening-terminal-xterm-kitty) - [Flycast redirect loops or TCP issues](#flycast-internal-load-balancing) - [Outbound TCP connections failing](#outbound-connections) - [POST requests getting 403'd](#cors-issues) @@ -124,6 +127,8 @@ If you're using Cloudflare: - Check your [Cloudflare](/networking/understanding-cloudflare) proxy settings - If 520s are intermittent, they may correlate with specific response headers from your app +Getting a 525 instead? That's a failed TLS handshake between Cloudflare and Fly.io. See [Cloudflare error 525](/networking/understanding-cloudflare#cloudflare-error-525-ssl-handshake-failed). + Note: if Cloudflare itself goes down, your Fly-hosted apps behind Cloudflare go down too. Fly is still running — the CDN in front of it isn't. --- @@ -430,6 +435,14 @@ You need either A **and** AAAA records, or a single CNAME record pointing to Fly **Using Cloudflare?** Most TLS issues on Fly involve domains behind Cloudflare. Read [Understanding Cloudflare](/networking/understanding-cloudflare) before debugging further. For all other setups, see [Custom domains](/networking/custom-domain). +### Client network socket disconnected before secure TLS connection was established {#client-network-socket-disconnected-before-secure-tls-connection-was-established} + +This error comes from Node.js clients (and tools built on them) when the server closes the connection during the TLS handshake. When the server is a Fly.io app, check these in order: + +- **DNS points at the wrong IPs.** If you deleted and recreated the app, or released and allocated new IPs, your domain or a client's DNS cache may still point at the old addresses. Compare `fly ips list` with `dig A example.com` and `dig AAAA example.com`, update your DNS records, and flush any local DNS cache. Recreating an app on every deploy (for example from Terraform) gives it new IPs each time, so deploy to the same app instead. +- **No certificate for the hostname.** Fly.io can only complete a handshake for hostnames it has a certificate for. Run `fly certs show `, and `fly certs add ` if it's missing. +- **No Machine can serve the request.** If every Machine is crashing or failing health checks, the handshake fails. Check `fly status` and [Health checks failing](#health-checks-failing). + ### Flycast (internal load balancing) [Flycast](/networking/flycast) routes traffic between your Fly apps over the private network. Two gotchas: @@ -477,6 +490,18 @@ If POST requests to your app return `403`, it's almost certainly your app's CORS - Make sure your app returns the correct `Access-Control-Allow-Origin`0 headers on preflight (`OPTIONS`) responses - If it works via `curl` but fails in the browser, that confirms it's a CORS issue in your app, not a Fly issue +### Error opening terminal: xterm-kitty {#error-opening-terminal-xterm-kitty} + +``` +Error opening terminal: xterm-kitty. +'xterm-kitty': unknown terminal type. +``` + +`fly ssh console` passes your local `TERM` to the Machine. Terminals like Kitty and Alacritty set `TERM` to a value (`xterm-kitty`, `alacritty`) whose terminfo entry usually isn't installed in your app's image, so programs like `htop`, `clear`, and `vim` can't start. Either fix works: + +- Override `TERM` for the session: `TERM=xterm-256color fly ssh console` +- Install the terminfo entry in your image, for example `apt-get install -y kitty-terminfo` on Debian or Ubuntu + --- ## Database connections diff --git a/networking/understanding-cloudflare.mdx b/networking/understanding-cloudflare.mdx index 897b896909..adc2be5fb6 100644 --- a/networking/understanding-cloudflare.mdx +++ b/networking/understanding-cloudflare.mdx @@ -87,6 +87,16 @@ Custom and ACME certificates can coexist for the same hostname. The custom certi - Always use "Full (strict)" in Cloudflare when connecting to Fly.io. Using "Flexible" mode may cause redirect loops. - Only one application should manage certificates for a subdomain or apex domain. Using more than one can cause conflicts. +### Cloudflare error 525: SSL handshake failed {#cloudflare-error-525-ssl-handshake-failed} + +Error 525 means Cloudflare's proxy couldn't complete a TLS handshake with your Fly.io app. Cloudflare is reachable, and so is Fly.io, but the encrypted connection between them fails. Common causes: + +- **Fly.io has no certificate for the hostname.** With **Full** or **Full (strict)** SSL mode, Cloudflare connects to Fly.io over TLS using your domain name. If you haven't run `fly certs add` for that hostname, or the certificate hasn't been issued yet, the handshake fails. Run `fly certs check example.com` and follow the [recommended setup](#recommended-setup), or import a [Cloudflare Origin Certificate](#using-a-cloudflare-origin-certificate) if issuance is stuck. +- **No Machine is available to serve the request.** If all of your app's Machines are crashing or failing health checks, Fly.io can't serve the connection. Run `fly status` and `fly logs`, and see [Troubleshoot your deployment](/getting-started/troubleshooting#health-checks-failing). +- **Intermittent network resets.** Occasional 525s that clear on their own are usually connection resets between Cloudflare and Fly.io. Run at least two Machines so a single restarting Machine doesn't drop handshakes. + +Don't switch Cloudflare to **Flexible** mode to make a 525 go away. Flexible sends plain HTTP to Fly.io, which causes redirect loops when your app forces HTTPS. + ### ACME-specific issues These issues apply when using Let's Encrypt (ACME) certificates behind Cloudflare's proxy. They do not apply if you are using a [Cloudflare Origin Certificate](#using-a-cloudflare-origin-certificate). diff --git a/reference/regions.mdx b/reference/regions.mdx index 3aa140ab6e..cebcd1ede6 100644 --- a/reference/regions.mdx +++ b/reference/regions.mdx @@ -51,6 +51,18 @@ Learn more about Machine placement and regional capacity in this [guide](/machin When an application instance is started, the three-letter name for the region it's running in is stored in the Machine's `FLY_REGION` environment variable. This, along with other [Runtime Environment](/machines/runtime-environment) information, is visible to your app running on that instance. +## Regions vs availability zones (coming from AWS) {#regions-vs-availability-zones} + +On AWS, a region is a geographic area and an availability zone (AZ) is an isolated group of datacenters inside it. You pick AZs explicitly and spread instances across them for redundancy. + +Fly.io has regions but no availability zone layer to choose from. Each region code in the table above is a single location. Redundancy within a region comes from running Machines and volumes on different physical servers: + +- **Machines:** `fly launch` and `fly deploy` create two Machines for process groups with services, and Fly.io places them on separate hosts when it can. See [App availability and resiliency](/apps/app-availability). +- **Volumes:** [`fly volumes create`](/flyctl/cmd/fly_volumes_create) places each volume in a separate hardware zone (a different server) by default, so one host failure doesn't take out every copy of your data. Volumes don't replicate, so your app still needs to handle replication. +- **Across regions:** For the protection you'd get from multiple AWS regions, run Machines in more than one Fly.io region, for example `fly scale count 2 --region iad,ord`. See [Scaling to multiple regions](/blueprints/resilient-apps-multiple-machines#scaling-to-multiple-regions). + +If you're mapping a multi-AZ AWS setup to Fly.io, think "at least two Machines per region" in place of "one instance per AZ", and add a second region when you need to survive a whole-region outage. + ## Related reading - [Scaling to multiple regions](/blueprints/resilient-apps-multiple-machines#scaling-to-multiple-regions) Read our guide for making your app resilient and globally distributed. From 826448fb40e53322617b30890a47744319bfd980 Mon Sep 17 00:00:00 2001 From: Alex Leventer Date: Sun, 18 Oct 2026 21:29:53 -0700 Subject: [PATCH 2/2] Add ERR_CONNECTION_CLOSED, systemd, SSL access denied, Express trust proxy and VS Code SSH docs Search Console shows these error queries landing on old community threads instead of the docs: - ERR_CONNECTION_CLOSED (16,124 impressions/28d, position 10.8, thread 5489) - "System has not been booted with systemd as init system" (thread 14416) - SSL_ERROR_ACCESS_DENIED_ALERT (thread 3676) - Express trust proxy (thread 6346) - VS Code "Failed to set up dynamic port forwarding connection" (thread 23541) Co-Authored-By: Claude Opus 5.5 --- app-guides/multiple-processes.mdx | 13 +++++++++++++ blueprints/opensshd.mdx | 17 +++++++++++++++++ getting-started/troubleshooting.mdx | 9 +++++++++ networking/custom-domain.mdx | 13 +++++++++++++ networking/request-headers.mdx | 22 ++++++++++++++++++++++ 5 files changed, 74 insertions(+) diff --git a/app-guides/multiple-processes.mdx b/app-guides/multiple-processes.mdx index bd06587142..064607602d 100644 --- a/app-guides/multiple-processes.mdx +++ b/app-guides/multiple-processes.mdx @@ -246,3 +246,16 @@ Another thing people run multiple programs to get is static asset HTTP service; ``` When Fly.io sees that in your `fly.toml`, it'll fish your static assets out of your container and serve them directly from our proxies at the `url_prefix` you specify. This is faster and, of course, doesn't demand you package an HTTP server into your container. + +## System has not been booted with systemd as init system (PID 1). Can't operate. {#system-has-not-been-booted-with-systemd-as-init-system} + +``` +System has not been booted with systemd as init system (PID 1). Can't operate. +``` + +This error comes from `systemctl`, usually called by a vendor install script or by a command like `systemctl start ` in your Dockerfile or entrypoint. On Fly.io, PID 1 in every Machine is Fly's own [init](/getting-started/troubleshooting#the-init-process), not systemd, and you can't replace it. Installing the `systemd` package doesn't change that, so systemd units never run. + +Start your programs without systemd instead: + +- **One program:** run it directly from `CMD` or `ENTRYPOINT`, in the foreground, instead of through `systemctl`. If an install script calls `systemctl`, change it to start the program directly. +- **Several programs:** use [process groups](#process-groups) to run each one in its own Machines, or start them together with a [Bash script](#just-use-bash), [supervisord](#use-supervisord), or a [Procfile manager](#use-a-procfile-manager) as described above. These run under Fly's init rather than replacing it. diff --git a/blueprints/opensshd.mdx b/blueprints/opensshd.mdx index 2833507202..e7e8e663fe 100644 --- a/blueprints/opensshd.mdx +++ b/blueprints/opensshd.mdx @@ -204,6 +204,23 @@ Additionally, if you want to avoid the fingerprint checking, you can add the fol StrictHostKeyChecking no ``` +## Failed to set up dynamic port forwarding connection over SSH to the VS Code server {#failed-to-set-up-dynamic-port-forwarding-vs-code} + +VS Code Remote - SSH shows this error when you connect it to the SSH server that Fly.io runs on port 22 of every Machine, the one `fly ssh console` uses. The Remote - SSH log has a line like: + +``` +Failed to set up socket for dynamic port forward to remote port 37957: Socket closed. TCP port forwarding may be disabled, or the remote server may have crashed. +``` + +VS Code reaches the VS Code Server it installs on the Machine through SSH port forwarding, and Fly.io's built-in SSH server doesn't support port forwarding. It rejects the forwarding channel with `unexpected channel type type=direct-tcpip`. + +To fix it, run OpenSSH alongside the built-in server as this guide describes, and point VS Code at OpenSSH instead: + +- Over the public internet, connect to `.fly.dev` on the external port you mapped in [Map internal port 2222 to external port 22](#map-internal-port-2222-to-external-port-22). +- Over your [WireGuard private network](/networking/private-networking), connect to `.internal` on port 2222, where `sshd` listens. + +You can add either one as a `Host` entry in your SSH config, as shown in the client configuration section above. + ## Related reading - [`flyctl` SSH commands](/flyctl/cmd/fly_ssh) Official reference for how to use `fly ssh`/`fly ssh sftp` to access running machines. diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index cd907fa16a..078dfc1701 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -43,6 +43,7 @@ If the error isn't obvious, start here. - **I can't connect to something** - [Custom domain TLS errors](#custom-domains-and-tls) - ["Client network socket disconnected before secure TLS connection was established"](#client-network-socket-disconnected-before-secure-tls-connection-was-established) + - ["ERR_CONNECTION_CLOSED" in the browser](#err-connection-closed) - ["Error opening terminal: xterm-kitty" in `fly ssh console`](#error-opening-terminal-xterm-kitty) - [Flycast redirect loops or TCP issues](#flycast-internal-load-balancing) - [Outbound TCP connections failing](#outbound-connections) @@ -443,6 +444,14 @@ This error comes from Node.js clients (and tools built on them) when the server - **No certificate for the hostname.** Fly.io can only complete a handshake for hostnames it has a certificate for. Run `fly certs show `, and `fly certs add ` if it's missing. - **No Machine can serve the request.** If every Machine is crashing or failing health checks, the handshake fails. Check `fly status` and [Health checks failing](#health-checks-failing). +### ERR_CONNECTION_CLOSED {#err-connection-closed} + +Chrome shows `ERR_CONNECTION_CLOSED` when the connection to your site closes before a response arrives. When the site is a Fly.io app, check these in order: + +- **No certificate for the hostname.** Fly Proxy closes the connection during the TLS handshake for a hostname your app has no certificate for, and `curl -v https://` fails with `SSL_ERROR_SYSCALL`. Run `fly certs check `, and `fly certs add ` if it's missing. `example.com` and `www.example.com` need [a certificate each](/networking/custom-domain#ssl-error-access-denied-alert). +- **Machines are being replaced.** If the error shows up while you deploy or scale, check which Machines are running with `fly status`, and read a specific Machine's logs with `fly logs --machine `. +- **It works with curl or in a private window.** Then the difference is that browser's saved state for the site, such as cookies. Clear the site's data in the browser and retry. + ### Flycast (internal load balancing) [Flycast](/networking/flycast) routes traffic between your Fly apps over the private network. Two gotchas: diff --git a/networking/custom-domain.mdx b/networking/custom-domain.mdx index 9ef590353f..9dc654d392 100644 --- a/networking/custom-domain.mdx +++ b/networking/custom-domain.mdx @@ -143,6 +143,19 @@ The best way to avoid hitting rate limits is to use staging environments and dom If you're building a platform on top of Fly.io, and you expect that your users will frequently delete and then recreate the same resources within a short window, consider implementing "soft delete" logic into your platform that retains the Fly.io resources for a period of time, negating the need to recreate certs frequently. +### SSL_ERROR_ACCESS_DENIED_ALERT {#ssl-error-access-denied-alert} + +Firefox shows `SSL_ERROR_ACCESS_DENIED_ALERT` when the server rejects the TLS handshake, and curl reports `tlsv1 alert access denied`. On Fly.io, this usually means the browser asked for a hostname your app has no certificate for. + +The common case is `www`. `example.com` and `www.example.com` are separate hostnames, and a certificate for one doesn't cover the other. Add a certificate for each hostname you serve: + +```bash +fly certs add example.com +fly certs add www.example.com +``` + +Then set up [DNS records](#configure-dns-records) for each hostname, and run `fly certs check ` to confirm each certificate is issued. `fly certs list` shows every hostname your app has a certificate for. + ### I use Cloudflare, and there seems to be a problem issuing or validating my Fly.io TLS certificate If you're using Cloudflare's proxying feature (orange cloud), verify domain ownership by adding a [`_fly-ownership` TXT record](#domain-ownership-verification). Once ownership is verified, Fly.io can issue certificates using the HTTP-01 challenge through Cloudflare's proxy. diff --git a/networking/request-headers.mdx b/networking/request-headers.mdx index 6b4c20f3bb..164804f388 100644 --- a/networking/request-headers.mdx +++ b/networking/request-headers.mdx @@ -47,3 +47,25 @@ This header must be treated with caution to avoid spoofing attempts. Follow the ### `Via` **Proxy Route**: This header, added by proxies, shows the path taken, and protocols used, by the connection. MDN has [full documentation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Via) for this header. For example, a connection through the Fly edge may show `2 fly.io` in the field, denoting that version 2 of HTTP was used by the connection as it passed through the Fly Proxy. + +## Express trust proxy on Fly.io {#express-trust-proxy-on-fly-io} + +Express ignores proxy headers until you set `trust proxy`, so behind Fly Proxy `req.ip` is the proxy's address rather than the client's. Fly Proxy appends two addresses to [`X-Forwarded-For`](#x-forwarded-for): the client's IP address, then the IP address of your app that the request arrived on. Anything to the left of those two came from the client and can be spoofed. + +Set `trust proxy` to `2` so that Express takes the second address from the right, which is the client's: + +```js +app.set('trust proxy', 2) +``` + +Don't use `app.set('trust proxy', true)`. With `true`, Express uses the leftmost address in `X-Forwarded-For`, which any client can set. + +If you only need the client IP, you can read the [`Fly-Client-IP`](#fly-client-ip) header instead, which Fly Proxy sets itself: + +```js +const clientIp = req.get('Fly-Client-IP') +``` + +If another proxy, such as Cloudflare, sits in front of Fly.io, `Fly-Client-IP` is that proxy's address and `2` no longer points at the client. Parse [`X-Forwarded-For`](#x-forwarded-for) as described above. + +`trust proxy` also makes Express read `X-Forwarded-Proto` for `req.protocol` and `req.secure`. When a client sends its own `X-Forwarded-Proto`, `X-Forwarded-Port`, or `X-Forwarded-Ssl` header, Fly Proxy passes that value through. The `Fly-Forwarded-Proto`, [`Fly-Forwarded-Port`](#fly-forwarded-port), and `Fly-Forwarded-Ssl` headers carry the values Fly Proxy saw, so check those if your app makes security decisions, like an HTTP-to-HTTPS redirect, based on the protocol.