Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions app-guides/multiple-processes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -246,3 +246,16 @@ Another thing people run multiple programs to get is static asset HTTP service;
```

When Fly.io sees that in your `fly.toml`, it'll fish your static assets out of your container and serve them directly from our proxies at the `url_prefix` you specify. This is faster and, of course, doesn't demand you package an HTTP server into your container.

## System has not been booted with systemd as init system (PID 1). Can't operate. {#system-has-not-been-booted-with-systemd-as-init-system}

```
System has not been booted with systemd as init system (PID 1). Can't operate.
```

This error comes from `systemctl`, usually called by a vendor install script or by a command like `systemctl start <service>` in your Dockerfile or entrypoint. On Fly.io, PID 1 in every Machine is Fly's own [init](/getting-started/troubleshooting#the-init-process), not systemd, and you can't replace it. Installing the `systemd` package doesn't change that, so systemd units never run.

Start your programs without systemd instead:

- **One program:** run it directly from `CMD` or `ENTRYPOINT`, in the foreground, instead of through `systemctl`. If an install script calls `systemctl`, change it to start the program directly.
- **Several programs:** use [process groups](#process-groups) to run each one in its own Machines, or start them together with a [Bash script](#just-use-bash), [supervisord](#use-supervisord), or a [Procfile manager](#use-a-procfile-manager) as described above. These run under Fly's init rather than replacing it.
17 changes: 17 additions & 0 deletions blueprints/opensshd.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,23 @@ Additionally, if you want to avoid the fingerprint checking, you can add the fol
StrictHostKeyChecking no
```

## Failed to set up dynamic port forwarding connection over SSH to the VS Code server {#failed-to-set-up-dynamic-port-forwarding-vs-code}

VS Code Remote - SSH shows this error when you connect it to the SSH server that Fly.io runs on port 22 of every Machine, the one `fly ssh console` uses. The Remote - SSH log has a line like:

```
Failed to set up socket for dynamic port forward to remote port 37957: Socket closed. TCP port forwarding may be disabled, or the remote server may have crashed.
```

VS Code reaches the VS Code Server it installs on the Machine through SSH port forwarding, and Fly.io's built-in SSH server doesn't support port forwarding. It rejects the forwarding channel with `unexpected channel type type=direct-tcpip`.

To fix it, run OpenSSH alongside the built-in server as this guide describes, and point VS Code at OpenSSH instead:

- Over the public internet, connect to `<app-name>.fly.dev` on the external port you mapped in [Map internal port 2222 to external port 22](#map-internal-port-2222-to-external-port-22).
- Over your [WireGuard private network](/networking/private-networking), connect to `<app-name>.internal` on port 2222, where `sshd` listens.

You can add either one as a `Host` entry in your SSH config, as shown in the client configuration section above.

## Related reading

- [`flyctl` SSH commands](/flyctl/cmd/fly_ssh) Official reference for how to use `fly ssh`/`fly ssh sftp` to access running machines.
Expand Down
34 changes: 34 additions & 0 deletions getting-started/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ If the error isn't obvious, start here.
- [401 Unauthorized](#registry-401-errors) — registry auth failure during deploy
- [403 Forbidden](#403-forbidden) — usually your app's CORS config or a third-party block
- [520 with Cloudflare](#520-errors-with-cloudflare) — Cloudflare doesn't like the response
- [525 with Cloudflare](/networking/understanding-cloudflare#cloudflare-error-525-ssl-handshake-failed) — TLS handshake between Cloudflare and Fly.io failed
- **My deploy failed**
- [Build hangs at 'Waiting for depot builder...'](#build-hangs-waiting-for-depot-builder)
- [Release command failures](#release-command-failures)
Expand All @@ -41,6 +42,9 @@ If the error isn't obvious, start here.
- [Suspend vs stop (clock skew)](#suspend-vs-stop)
- **I can't connect to something**
- [Custom domain TLS errors](#custom-domains-and-tls)
- ["Client network socket disconnected before secure TLS connection was established"](#client-network-socket-disconnected-before-secure-tls-connection-was-established)
- ["ERR_CONNECTION_CLOSED" in the browser](#err-connection-closed)
- ["Error opening terminal: xterm-kitty" in `fly ssh console`](#error-opening-terminal-xterm-kitty)
- [Flycast redirect loops or TCP issues](#flycast-internal-load-balancing)
- [Outbound TCP connections failing](#outbound-connections)
- [POST requests getting 403'd](#cors-issues)
Expand Down Expand Up @@ -124,6 +128,8 @@ If you're using Cloudflare:
- Check your [Cloudflare](/networking/understanding-cloudflare) proxy settings
- If 520s are intermittent, they may correlate with specific response headers from your app

Getting a 525 instead? That's a failed TLS handshake between Cloudflare and Fly.io. See [Cloudflare error 525](/networking/understanding-cloudflare#cloudflare-error-525-ssl-handshake-failed).

Note: if Cloudflare itself goes down, your Fly-hosted apps behind Cloudflare go down too. Fly is still running — the CDN in front of it isn't.

---
Expand Down Expand Up @@ -430,6 +436,22 @@ You need either A **and** AAAA records, or a single CNAME record pointing to Fly

**Using Cloudflare?** Most TLS issues on Fly involve domains behind Cloudflare. Read [Understanding Cloudflare](/networking/understanding-cloudflare) before debugging further. For all other setups, see [Custom domains](/networking/custom-domain).

### Client network socket disconnected before secure TLS connection was established {#client-network-socket-disconnected-before-secure-tls-connection-was-established}

This error comes from Node.js clients (and tools built on them) when the server closes the connection during the TLS handshake. When the server is a Fly.io app, check these in order:

- **DNS points at the wrong IPs.** If you deleted and recreated the app, or released and allocated new IPs, your domain or a client's DNS cache may still point at the old addresses. Compare `fly ips list` with `dig A example.com` and `dig AAAA example.com`, update your DNS records, and flush any local DNS cache. Recreating an app on every deploy (for example from Terraform) gives it new IPs each time, so deploy to the same app instead.
- **No certificate for the hostname.** Fly.io can only complete a handshake for hostnames it has a certificate for. Run `fly certs show <hostname>`, and `fly certs add <hostname>` if it's missing.
- **No Machine can serve the request.** If every Machine is crashing or failing health checks, the handshake fails. Check `fly status` and [Health checks failing](#health-checks-failing).

### ERR_CONNECTION_CLOSED {#err-connection-closed}

Chrome shows `ERR_CONNECTION_CLOSED` when the connection to your site closes before a response arrives. When the site is a Fly.io app, check these in order:

- **No certificate for the hostname.** Fly Proxy closes the connection during the TLS handshake for a hostname your app has no certificate for, and `curl -v https://<hostname>` fails with `SSL_ERROR_SYSCALL`. Run `fly certs check <hostname>`, and `fly certs add <hostname>` if it's missing. `example.com` and `www.example.com` need [a certificate each](/networking/custom-domain#ssl-error-access-denied-alert).
- **Machines are being replaced.** If the error shows up while you deploy or scale, check which Machines are running with `fly status`, and read a specific Machine's logs with `fly logs --machine <machine-id>`.
- **It works with curl or in a private window.** Then the difference is that browser's saved state for the site, such as cookies. Clear the site's data in the browser and retry.

### Flycast (internal load balancing)

[Flycast](/networking/flycast) routes traffic between your Fly apps over the private network. Two gotchas:
Expand Down Expand Up @@ -477,6 +499,18 @@ If POST requests to your app return `403`, it's almost certainly your app's CORS
- Make sure your app returns the correct `Access-Control-Allow-Origin`0 headers on preflight (`OPTIONS`) responses
- If it works via `curl` but fails in the browser, that confirms it's a CORS issue in your app, not a Fly issue

### Error opening terminal: xterm-kitty {#error-opening-terminal-xterm-kitty}

```
Error opening terminal: xterm-kitty.
'xterm-kitty': unknown terminal type.
```

`fly ssh console` passes your local `TERM` to the Machine. Terminals like Kitty and Alacritty set `TERM` to a value (`xterm-kitty`, `alacritty`) whose terminfo entry usually isn't installed in your app's image, so programs like `htop`, `clear`, and `vim` can't start. Either fix works:

- Override `TERM` for the session: `TERM=xterm-256color fly ssh console`
- Install the terminfo entry in your image, for example `apt-get install -y kitty-terminfo` on Debian or Ubuntu

---

## Database connections
Expand Down
13 changes: 13 additions & 0 deletions networking/custom-domain.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,19 @@ The best way to avoid hitting rate limits is to use staging environments and dom

If you're building a platform on top of Fly.io, and you expect that your users will frequently delete and then recreate the same resources within a short window, consider implementing "soft delete" logic into your platform that retains the Fly.io resources for a period of time, negating the need to recreate certs frequently.

### SSL_ERROR_ACCESS_DENIED_ALERT {#ssl-error-access-denied-alert}

Firefox shows `SSL_ERROR_ACCESS_DENIED_ALERT` when the server rejects the TLS handshake, and curl reports `tlsv1 alert access denied`. On Fly.io, this usually means the browser asked for a hostname your app has no certificate for.

The common case is `www`. `example.com` and `www.example.com` are separate hostnames, and a certificate for one doesn't cover the other. Add a certificate for each hostname you serve:

```bash
fly certs add example.com
fly certs add www.example.com
```

Then set up [DNS records](#configure-dns-records) for each hostname, and run `fly certs check <hostname>` to confirm each certificate is issued. `fly certs list` shows every hostname your app has a certificate for.

### I use Cloudflare, and there seems to be a problem issuing or validating my Fly.io TLS certificate

If you're using Cloudflare's proxying feature (orange cloud), verify domain ownership by adding a [`_fly-ownership` TXT record](#domain-ownership-verification). Once ownership is verified, Fly.io can issue certificates using the HTTP-01 challenge through Cloudflare's proxy.
Expand Down
22 changes: 22 additions & 0 deletions networking/request-headers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,25 @@ This header must be treated with caution to avoid spoofing attempts. Follow the
### `Via`

**Proxy Route**: This header, added by proxies, shows the path taken, and protocols used, by the connection. MDN has [full documentation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Via) for this header. For example, a connection through the Fly edge may show `2 fly.io` in the field, denoting that version 2 of HTTP was used by the connection as it passed through the Fly Proxy.

## Express trust proxy on Fly.io {#express-trust-proxy-on-fly-io}

Express ignores proxy headers until you set `trust proxy`, so behind Fly Proxy `req.ip` is the proxy's address rather than the client's. Fly Proxy appends two addresses to [`X-Forwarded-For`](#x-forwarded-for): the client's IP address, then the IP address of your app that the request arrived on. Anything to the left of those two came from the client and can be spoofed.

Set `trust proxy` to `2` so that Express takes the second address from the right, which is the client's:

```js
app.set('trust proxy', 2)
```

Don't use `app.set('trust proxy', true)`. With `true`, Express uses the leftmost address in `X-Forwarded-For`, which any client can set.

If you only need the client IP, you can read the [`Fly-Client-IP`](#fly-client-ip) header instead, which Fly Proxy sets itself:

```js
const clientIp = req.get('Fly-Client-IP')
```

If another proxy, such as Cloudflare, sits in front of Fly.io, `Fly-Client-IP` is that proxy's address and `2` no longer points at the client. Parse [`X-Forwarded-For`](#x-forwarded-for) as described above.

`trust proxy` also makes Express read `X-Forwarded-Proto` for `req.protocol` and `req.secure`. When a client sends its own `X-Forwarded-Proto`, `X-Forwarded-Port`, or `X-Forwarded-Ssl` header, Fly Proxy passes that value through. The `Fly-Forwarded-Proto`, [`Fly-Forwarded-Port`](#fly-forwarded-port), and `Fly-Forwarded-Ssl` headers carry the values Fly Proxy saw, so check those if your app makes security decisions, like an HTTP-to-HTTPS redirect, based on the protocol.
10 changes: 10 additions & 0 deletions networking/understanding-cloudflare.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,16 @@ Custom and ACME certificates can coexist for the same hostname. The custom certi
- Always use "Full (strict)" in Cloudflare when connecting to Fly.io. Using "Flexible" mode may cause redirect loops.
- Only one application should manage certificates for a subdomain or apex domain. Using more than one can cause conflicts.

### Cloudflare error 525: SSL handshake failed {#cloudflare-error-525-ssl-handshake-failed}

Error 525 means Cloudflare's proxy couldn't complete a TLS handshake with your Fly.io app. Cloudflare is reachable, and so is Fly.io, but the encrypted connection between them fails. Common causes:

- **Fly.io has no certificate for the hostname.** With **Full** or **Full (strict)** SSL mode, Cloudflare connects to Fly.io over TLS using your domain name. If you haven't run `fly certs add` for that hostname, or the certificate hasn't been issued yet, the handshake fails. Run `fly certs check example.com` and follow the [recommended setup](#recommended-setup), or import a [Cloudflare Origin Certificate](#using-a-cloudflare-origin-certificate) if issuance is stuck.
- **No Machine is available to serve the request.** If all of your app's Machines are crashing or failing health checks, Fly.io can't serve the connection. Run `fly status` and `fly logs`, and see [Troubleshoot your deployment](/getting-started/troubleshooting#health-checks-failing).
- **Intermittent network resets.** Occasional 525s that clear on their own are usually connection resets between Cloudflare and Fly.io. Run at least two Machines so a single restarting Machine doesn't drop handshakes.

Don't switch Cloudflare to **Flexible** mode to make a 525 go away. Flexible sends plain HTTP to Fly.io, which causes redirect loops when your app forces HTTPS.

### ACME-specific issues

These issues apply when using Let's Encrypt (ACME) certificates behind Cloudflare's proxy. They do not apply if you are using a [Cloudflare Origin Certificate](#using-a-cloudflare-origin-certificate).
Expand Down
12 changes: 12 additions & 0 deletions reference/regions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,18 @@ Learn more about Machine placement and regional capacity in this [guide](/machin

When an application instance is started, the three-letter name for the region it's running in is stored in the Machine's `FLY_REGION` environment variable. This, along with other [Runtime Environment](/machines/runtime-environment) information, is visible to your app running on that instance.

## Regions vs availability zones (coming from AWS) {#regions-vs-availability-zones}

On AWS, a region is a geographic area and an availability zone (AZ) is an isolated group of datacenters inside it. You pick AZs explicitly and spread instances across them for redundancy.

Fly.io has regions but no availability zone layer to choose from. Each region code in the table above is a single location. Redundancy within a region comes from running Machines and volumes on different physical servers:

- **Machines:** `fly launch` and `fly deploy` create two Machines for process groups with services, and Fly.io places them on separate hosts when it can. See [App availability and resiliency](/apps/app-availability).
- **Volumes:** [`fly volumes create`](/flyctl/cmd/fly_volumes_create) places each volume in a separate hardware zone (a different server) by default, so one host failure doesn't take out every copy of your data. Volumes don't replicate, so your app still needs to handle replication.
- **Across regions:** For the protection you'd get from multiple AWS regions, run Machines in more than one Fly.io region, for example `fly scale count 2 --region iad,ord`. See [Scaling to multiple regions](/blueprints/resilient-apps-multiple-machines#scaling-to-multiple-regions).

If you're mapping a multi-AZ AWS setup to Fly.io, think "at least two Machines per region" in place of "one instance per AZ", and add a second region when you need to survive a whole-region outage.

## Related reading

- [Scaling to multiple regions](/blueprints/resilient-apps-multiple-machines#scaling-to-multiple-regions) Read our guide for making your app resilient and globally distributed.
Expand Down
Loading