Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ jobs:
run: |
scripts/test-deploy-classifier.sh
scripts/test-deploy.sh
scripts/test-image-deploy.sh
scripts/test-production-repair.sh
scripts/test-deploy-workflow.sh
scripts/test-backup-postgres.sh

- name: Validate CI branch coverage
Expand Down
207 changes: 207 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
name: Production deploy

on:
workflow_run:
workflows: [CI]
branches: [master]
types: [completed]
workflow_dispatch:
inputs:
ref:
description: Commit, branch, or tag to deploy. Defaults to master.
required: false
type: string

permissions:
contents: read
packages: write

concurrency:
group: production-deploy
cancel-in-progress: false

jobs:
resolve:
name: Resolve release source
if: >-
github.event_name == 'workflow_dispatch'
|| (
github.event.workflow_run.conclusion == 'success'
&& github.event.workflow_run.head_branch == 'master'
)
runs-on: ubuntu-24.04
outputs:
source_sha: ${{ steps.source.outputs.source_sha }}
steps:
- name: Check out release source
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.head_sha || inputs.ref || 'master' }}
fetch-depth: 0

- name: Resolve exact commit
id: source
run: echo "source_sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"

image:
name: Build immutable image
needs: resolve
runs-on: ubuntu-24.04
outputs:
image_ref: ${{ steps.image.outputs.image_ref }}
steps:
- name: Check out exact release source
uses: actions/checkout@v4
with:
ref: ${{ needs.resolve.outputs.source_sha }}

- uses: docker/setup-buildx-action@v3

- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}

- name: Build and publish release image with retries
id: build
env:
IMAGE_TAG: ghcr.io/${{ github.repository }}:${{ needs.resolve.outputs.source_sha }}
SOURCE_SHA: ${{ needs.resolve.outputs.source_sha }}
run: |
for attempt in 1 2 3; do
if docker buildx build . \
--build-arg "SOURCE_SHA=$SOURCE_SHA" \
--cache-from type=gha,scope=production-image \
--cache-to type=gha,mode=max,scope=production-image \
--metadata-file /tmp/build-metadata.json \
--push \
--tag "$IMAGE_TAG"; then
digest=$(jq -r '."containerimage.digest" // empty' /tmp/build-metadata.json)
[[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]]
echo "digest=$digest" >> "$GITHUB_OUTPUT"
exit 0
fi

if [ "$attempt" -lt 3 ]; then
echo "Image build attempt $attempt failed. Retrying in 30 seconds." >&2
sleep 30
fi
done
exit 1

- name: Record digest-qualified image
id: image
env:
IMAGE_REPOSITORY: ghcr.io/${{ github.repository }}
IMAGE_DIGEST: ${{ steps.build.outputs.digest }}
run: |
[[ "$IMAGE_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
echo "image_ref=$IMAGE_REPOSITORY@$IMAGE_DIGEST" >> "$GITHUB_OUTPUT"

deploy:
name: Activate production release
needs: [resolve, image]
runs-on: ubuntu-24.04
timeout-minutes: 30
environment: production
env:
APP_DIR: /opt/letscube/runtime
ENV_FILE: /home/www/letscube/.env.prod
PRIVACY_EMAIL_CUTOVER_FILE: /home/www/letscube/.privacy-email-cutover
SERVER_IP: ${{ secrets.SERVER_IP }}
SERVER_USER: www
SOURCE_SHA: ${{ needs.resolve.outputs.source_sha }}
IMAGE_REF: ${{ needs.image.outputs.image_ref }}
steps:
- name: Check out exact release source
uses: actions/checkout@v4
with:
ref: ${{ needs.resolve.outputs.source_sha }}
fetch-depth: 0

- name: Configure deployment SSH
env:
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }}
run: |
install -m 700 -d "$HOME/.ssh"
printf '%s\n' "$DEPLOY_SSH_KEY" > "$HOME/.ssh/id_ed25519"
printf '%s\n' "$DEPLOY_KNOWN_HOSTS" > "$HOME/.ssh/known_hosts"
chmod 600 "$HOME/.ssh/id_ed25519" "$HOME/.ssh/known_hosts"

- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}

- name: Pull and validate the immutable image
run: |
docker pull "$IMAGE_REF"
test "$(docker image inspect "$IMAGE_REF" --format '{{index .RepoDigests 0}}')" = "$IMAGE_REF"
test "$(docker image inspect "$IMAGE_REF" --format '{{index .Config.Labels "org.opencontainers.image.revision"}}')" = "$SOURCE_SHA"
docker tag "$IMAGE_REF" "letscube-app:$SOURCE_SHA"

- name: Enforce the production privacy floor
run: |
cutover=$(ssh -q "$SERVER_USER@$SERVER_IP" \
"test ! -f '$PRIVACY_EMAIL_CUTOVER_FILE' || cat '$PRIVACY_EMAIL_CUTOVER_FILE'")
if [ -n "$cutover" ]; then
[[ "$cutover" =~ ^[0-9a-f]{40}$ ]]
git merge-base --is-ancestor "$cutover" "$SOURCE_SHA"
fi

- name: Stage release configuration
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
ssh -q "$SERVER_USER@$SERVER_IP" "mkdir -p '$remote_release_dir'"
scp -q compose.yml compose.prod.yml nginx.conf \
scripts/deploy-image-release.sh scripts/check-production-health.sh \
scripts/repair-production-services.sh \
scripts/disable-legacy-deploy-cron.sh \
"$SERVER_USER@$SERVER_IP:$remote_release_dir/"

- name: Transfer the release image
run: |
set -o pipefail
for attempt in 1 2 3; do
if docker save "letscube-app:$SOURCE_SHA" | gzip | \
ssh -q "$SERVER_USER@$SERVER_IP" 'gzip -d | docker load'; then
exit 0
fi

if [ "$attempt" -lt 3 ]; then
echo "Image transfer attempt $attempt failed. Retrying in 30 seconds." >&2
sleep 30
fi
done
exit 1

- name: Deploy with bounded automatic retries
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
for attempt in 1 2 3; do
if ssh -q "$SERVER_USER@$SERVER_IP" \
"APP_DIR='$APP_DIR' ENV_FILE='$ENV_FILE' bash '$remote_release_dir/deploy-image-release.sh' '$SOURCE_SHA' '$remote_release_dir'"; then
exit 0
fi

if [ "$attempt" -lt 3 ]; then
echo "Deployment attempt $attempt failed. Retrying the same image in 30 seconds." >&2
sleep 30
fi
done
exit 1

- name: Replace the legacy deployment cron with health repair
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
ssh -q "$SERVER_USER@$SERVER_IP" \
"APP_DIR='$APP_DIR' ENV_FILE='$ENV_FILE' bash '$remote_release_dir/disable-legacy-deploy-cron.sh'"

- name: Remove staged release files
if: always()
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
ssh -q "$SERVER_USER@$SERVER_IP" "rm -rf -- '$remote_release_dir'" || true
8 changes: 7 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
FROM node:22-bookworm-slim AS client-build
WORKDIR /app

RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates openssl \
&& rm -rf /var/lib/apt/lists/*

RUN corepack enable

COPY package.json yarn.lock ./
Expand All @@ -27,7 +31,7 @@ RUN yarn workspace letscube-client build

FROM node:22-bookworm-slim AS server-deps
RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 make g++ \
&& apt-get install -y --no-install-recommends ca-certificates openssl python3 make g++ \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app
Expand All @@ -42,6 +46,8 @@ RUN yarn install --frozen-lockfile --production=true --network-concurrency 1 --n
&& mkdir -p server/node_modules

FROM node:22-bookworm-slim AS runtime
ARG SOURCE_SHA=local
LABEL org.opencontainers.image.revision=${SOURCE_SHA}
ENV NODE_ENV=prod
ENV PORT=8080
ENV SOCKETIO_PORT=9000
Expand Down
7 changes: 7 additions & 0 deletions compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,5 +48,12 @@ services:
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
- ${LETSENCRYPT_DIR:-/etc/letsencrypt}:/etc/letsencrypt:ro
- ${CERTBOT_WWW_DIR:-/var/www/certbot}:/var/www/certbot:ro
healthcheck:
test: ["CMD-SHELL", "wget --quiet --spider --no-check-certificate https://127.0.0.1/health/api || exit 1"]
interval: 30s
start_interval: 2s
timeout: 5s
retries: 5
start_period: 10s
logging: *production-logging
restart: unless-stopped
52 changes: 47 additions & 5 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,50 @@ backfill tool.

## Deployment

Use the Compose deploy script from the active checkout:
Production deploys run through the `Production deploy` GitHub Actions workflow.
The workflow starts after a successful `CI` run on `master`. It performs these
steps:

1. Build one application image from the exact `master` commit.
2. Push the image to GHCR and retain its immutable digest.
3. Transfer that exact image and the production Compose files to the server.
4. Apply migrations, replace Socket.IO before the API, and verify all five
production services.
5. Retry image builds, transfers, and activation up to three times. Each failed
activation restores the previous application images and Compose files before
the next attempt.

The server does not fetch or merge Git commits during this process. A manual
workflow run can deploy a specific commit, branch, or tag. The workflow uses the
GitHub `production` environment, so configure its approval policy to match the
release policy.

The repository needs these GitHub Actions secrets:

- `SERVER_IP`: the production server address.
- `DEPLOY_SSH_KEY`: the private key for the `www` account.
- `DEPLOY_KNOWN_HOSTS`: the pinned SSH host key entry.

The server needs Docker access for `www`, the production `.env.prod` file at
`/home/www/letscube`, and the existing certificate directories. Actions stores
runtime Compose files in `/opt/letscube/runtime`, outside the old Git checkout.
After the first successful image activation, it removes only the old
`scripts/deploy.sh` cron entry. It saves the prior crontab at
`/opt/letscube/legacy-deploy-crontab.backup`. The PostgreSQL backup cron remains
active. A replacement cron checks all five services every ten minutes. It
recreates only missing or unhealthy services from the last successful image.
It does not fetch source code.

The checkout-based script remains available as a temporary recovery path:

```sh
APP_DIR=/opt/letscube ENV_FILE=.env.prod scripts/deploy.sh
APP_DIR=/home/www/letscube ENV_FILE=.env.prod scripts/deploy.sh
```

The script builds an immutable image, starts PostgreSQL and Redis, applies
committed Prisma migrations, then replaces and health-checks Socket.IO and the
API. It does not start or depend on MongoDB.
It records the last successful commit separately from the checkout. A failed
deployment therefore retries on the next run even when Git already moved to the
new commit. It also treats a missing or unhealthy PostgreSQL, Redis, Socket.IO,
API, or nginx container as a full repair deployment.

Before a big-bang cutover, take a PostgreSQL backup and run the MongoDB
backfill tool during a maintenance window:
Expand All @@ -34,6 +69,9 @@ not copy sessions or OAuth tokens; users authenticate again after cutover.

```sh
docker compose -f compose.yml -f compose.prod.yml --env-file .env.prod ps
APP_DIR=/opt/letscube/runtime \
ENV_FILE=/home/www/letscube/.env.prod \
/opt/letscube/runtime/scripts/check-production-health.sh
docker compose -f compose.yml -f compose.prod.yml --env-file .env.prod logs --tail=200 api socket postgres
curl -fsS https://letscube.net/health/api
curl -fsS https://letscube.net/health/socket
Expand All @@ -42,6 +80,10 @@ curl -fsS https://letscube.net/health/socket
Both health endpoints require PostgreSQL. The socket endpoint also requires
Redis. A failed required dependency returns HTTP 503.

nginx also has a container health check. It calls the API health route through
the local TLS listener. This detects a broken proxy configuration and an
unavailable API upstream.

## Backups and restore

Create a PostgreSQL custom-format backup and test restores against a separate
Expand Down
13 changes: 7 additions & 6 deletions docs/release-workflow.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Release Workflow

`dev` is the integration branch. `master` is the reviewed release branch.
Production deployment remains a separately authorized operation described in
[Production operations](operations.md).
Production deployment runs through the protected GitHub `production`
environment described in [Production operations](operations.md).

## Branch Flow

Expand All @@ -11,8 +11,8 @@ Production deployment remains a separately authorized operation described in
3. Treat the resulting `dev` merge commit as the release candidate. Its push
CI and CodeQL push runs must finish before promotion is considered.
4. Open a reviewed promotion from that exact `dev` commit to `master`.
5. After the promotion is merged and its `master` push CI passes, perform the
separately authorized deployment procedure.
5. After the promotion is merged, the successful `master` CI run starts the
production deployment workflow.

Do not bypass `dev` by merging feature work directly to `master`. A new commit
on `dev` after release-candidate validation requires a new validation run.
Expand Down Expand Up @@ -42,10 +42,11 @@ explicit operational authorization required by the operations guide.

## Deployment Handoff

After `master` CI is green, follow the operations guide in order:
After `master` CI is green, GitHub Actions follows the operations guide in
order:

1. complete the [preflight](operations.md#preflight) checks;
2. deploy with the documented procedure;
2. build and deploy the immutable release image;
3. perform the [verification and monitoring](operations.md#verification-and-monitoring)
checks; and
4. use the documented rollback procedure if readiness or verification fails.
Expand Down
Loading
Loading