Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ REACT_APP_WCA_ORIGIN=https://www.worldcubeassociation.org
REACT_APP_WCA_CLIENT_ID=replace-me

# Server config
MONGO_URL=mongodb://mongo:27017/letscube
REDIS_URL=redis://redis:6379
POSTGRES_ENABLED=true
# DATABASE_URL can replace the individual PG* connection fields.
Expand Down Expand Up @@ -51,9 +50,9 @@ METRICS_HASH_SECRET=replace-with-a-different-long-random-secret
LETSENCRYPT_DIR=/etc/letsencrypt
CERTBOT_WWW_DIR=/var/www/certbot

# Mongo backups
# PostgreSQL backups
BACKUP_DIR=/opt/letscube/backups
MONGO_BACKUP_URI=mongodb://127.0.0.1:27017/letscube
PG_BACKUP_URI=postgresql://letscube:replace-with-a-long-random-secret@127.0.0.1:5432/letscube
BACKUP_S3_URI=
AWS_ENDPOINT_URL=
AWS_ACCESS_KEY_ID=
Expand Down
44 changes: 35 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,21 @@ jobs:
- name: Validate production metrics configuration
run: scripts/test-compose-metrics-config.sh

- name: Validate production deployment
run: |
scripts/test-deploy-classifier.sh
scripts/test-deploy.sh
scripts/test-image-deploy.sh
scripts/test-production-repair.sh
scripts/test-deploy-workflow.sh
scripts/test-backup-postgres.sh

- name: Validate CI branch coverage
run: scripts/test-ci-branches.sh

- name: Validate CodeQL branch coverage
run: scripts/test-codeql-branches.sh

- name: Rehearse MongoDB backup and restore scripts
run: scripts/test-mongo-backup-restore.sh

- name: Rehearse a live MongoDB backup and restore
run: scripts/test-mongo-backup-restore-live.sh

server:
name: Server lint and unit tests
runs-on: ubuntu-latest
Expand Down Expand Up @@ -108,6 +111,26 @@ jobs:
cypress:
name: Cypress full-stack smoke
runs-on: ubuntu-latest
env:
PGHOST: 127.0.0.1
PGPORT: 5432
PGDATABASE: letscube
PGUSER: letscube
PGPASSWORD: letscube
services:
postgres:
image: postgres:17-alpine
env:
POSTGRES_DB: letscube
POSTGRES_USER: letscube
POSTGRES_PASSWORD: letscube
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U letscube -d letscube"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Check out repository
uses: actions/checkout@v4
Expand All @@ -131,13 +154,16 @@ jobs:
node-version: 22.17.0

- name: Start backing services
run: docker compose -f compose.yml -f compose.dev.yml up -d mongo redis
run: docker compose -f compose.yml -f compose.dev.yml up -d redis

- name: Apply PostgreSQL migrations
run: yarn workspace letscube-server postgres:migrate

- name: Start server
run: CORS_ORIGINS=http://localhost:3000,http://localhost:8080 LETSCUBE_TEST_AUTH=true SOCIAL_FEATURES_ENABLED=true POSTGRES_ENABLED=false yarn workspace letscube-server node index.js > server.log 2>&1 &
run: CORS_ORIGINS=http://localhost:3000,http://localhost:8080 LETSCUBE_TEST_AUTH=true SOCIAL_FEATURES_ENABLED=true yarn workspace letscube-server node index.js > server.log 2>&1 &

- name: Start socket server
run: SOCIAL_FEATURES_ENABLED=true POSTGRES_ENABLED=false yarn workspace letscube-server node socket/ > socket.log 2>&1 &
run: SOCIAL_FEATURES_ENABLED=true yarn workspace letscube-server node socket/ > socket.log 2>&1 &

- name: Wait for local stack
run: |
Expand Down
207 changes: 207 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
name: Production deploy

on:
workflow_run:
workflows: [CI]
branches: [master]
types: [completed]
workflow_dispatch:
inputs:
ref:
description: Commit, branch, or tag to deploy. Defaults to master.
required: false
type: string

permissions:
contents: read
packages: write

concurrency:
group: production-deploy
cancel-in-progress: false

jobs:
resolve:
name: Resolve release source
if: >-
github.event_name == 'workflow_dispatch'
|| (
github.event.workflow_run.conclusion == 'success'
&& github.event.workflow_run.head_branch == 'master'
)
runs-on: ubuntu-24.04
outputs:
source_sha: ${{ steps.source.outputs.source_sha }}
steps:
- name: Check out release source
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.head_sha || inputs.ref || 'master' }}
fetch-depth: 0

- name: Resolve exact commit
id: source
run: echo "source_sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"

image:
name: Build immutable image
needs: resolve
runs-on: ubuntu-24.04
outputs:
image_ref: ${{ steps.image.outputs.image_ref }}
steps:
- name: Check out exact release source
uses: actions/checkout@v4
with:
ref: ${{ needs.resolve.outputs.source_sha }}

- uses: docker/setup-buildx-action@v3

- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}

- name: Build and publish release image with retries
id: build
env:
IMAGE_TAG: ghcr.io/${{ github.repository }}:${{ needs.resolve.outputs.source_sha }}
SOURCE_SHA: ${{ needs.resolve.outputs.source_sha }}
run: |
for attempt in 1 2 3; do
if docker buildx build . \
--build-arg "SOURCE_SHA=$SOURCE_SHA" \
--cache-from type=gha,scope=production-image \
--cache-to type=gha,mode=max,scope=production-image \
--metadata-file /tmp/build-metadata.json \
--push \
--tag "$IMAGE_TAG"; then
digest=$(jq -r '."containerimage.digest" // empty' /tmp/build-metadata.json)
[[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]]
echo "digest=$digest" >> "$GITHUB_OUTPUT"
exit 0
fi

if [ "$attempt" -lt 3 ]; then
echo "Image build attempt $attempt failed. Retrying in 30 seconds." >&2
sleep 30
fi
done
exit 1

- name: Record digest-qualified image
id: image
env:
IMAGE_REPOSITORY: ghcr.io/${{ github.repository }}
IMAGE_DIGEST: ${{ steps.build.outputs.digest }}
run: |
[[ "$IMAGE_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
echo "image_ref=$IMAGE_REPOSITORY@$IMAGE_DIGEST" >> "$GITHUB_OUTPUT"

deploy:
name: Activate production release
needs: [resolve, image]
runs-on: ubuntu-24.04
timeout-minutes: 30
environment: production
env:
APP_DIR: /opt/letscube/runtime
ENV_FILE: /home/www/letscube/.env.prod
PRIVACY_EMAIL_CUTOVER_FILE: /home/www/letscube/.privacy-email-cutover
SERVER_IP: ${{ secrets.SERVER_IP }}
SERVER_USER: www
SOURCE_SHA: ${{ needs.resolve.outputs.source_sha }}
IMAGE_REF: ${{ needs.image.outputs.image_ref }}
steps:
- name: Check out exact release source
uses: actions/checkout@v4
with:
ref: ${{ needs.resolve.outputs.source_sha }}
fetch-depth: 0

- name: Configure deployment SSH
env:
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }}
run: |
install -m 700 -d "$HOME/.ssh"
printf '%s\n' "$DEPLOY_SSH_KEY" > "$HOME/.ssh/id_ed25519"
printf '%s\n' "$DEPLOY_KNOWN_HOSTS" > "$HOME/.ssh/known_hosts"
chmod 600 "$HOME/.ssh/id_ed25519" "$HOME/.ssh/known_hosts"

- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}

- name: Pull and validate the immutable image
run: |
docker pull "$IMAGE_REF"
test "$(docker image inspect "$IMAGE_REF" --format '{{index .RepoDigests 0}}')" = "$IMAGE_REF"
test "$(docker image inspect "$IMAGE_REF" --format '{{index .Config.Labels "org.opencontainers.image.revision"}}')" = "$SOURCE_SHA"
docker tag "$IMAGE_REF" "letscube-app:$SOURCE_SHA"

- name: Enforce the production privacy floor
run: |
cutover=$(ssh -q "$SERVER_USER@$SERVER_IP" \
"test ! -f '$PRIVACY_EMAIL_CUTOVER_FILE' || cat '$PRIVACY_EMAIL_CUTOVER_FILE'")
if [ -n "$cutover" ]; then
[[ "$cutover" =~ ^[0-9a-f]{40}$ ]]
git merge-base --is-ancestor "$cutover" "$SOURCE_SHA"
fi

- name: Stage release configuration
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
ssh -q "$SERVER_USER@$SERVER_IP" "mkdir -p '$remote_release_dir'"
scp -q compose.yml compose.prod.yml nginx.conf \
scripts/deploy-image-release.sh scripts/check-production-health.sh \
scripts/repair-production-services.sh \
scripts/disable-legacy-deploy-cron.sh \
"$SERVER_USER@$SERVER_IP:$remote_release_dir/"

- name: Transfer the release image
run: |
set -o pipefail
for attempt in 1 2 3; do
if docker save "letscube-app:$SOURCE_SHA" | gzip | \
ssh -q "$SERVER_USER@$SERVER_IP" 'gzip -d | docker load'; then
exit 0
fi

if [ "$attempt" -lt 3 ]; then
echo "Image transfer attempt $attempt failed. Retrying in 30 seconds." >&2
sleep 30
fi
done
exit 1

- name: Deploy with bounded automatic retries
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
for attempt in 1 2 3; do
if ssh -q "$SERVER_USER@$SERVER_IP" \
"APP_DIR='$APP_DIR' ENV_FILE='$ENV_FILE' bash '$remote_release_dir/deploy-image-release.sh' '$SOURCE_SHA' '$remote_release_dir'"; then
exit 0
fi

if [ "$attempt" -lt 3 ]; then
echo "Deployment attempt $attempt failed. Retrying the same image in 30 seconds." >&2
sleep 30
fi
done
exit 1

- name: Replace the legacy deployment cron with health repair
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
ssh -q "$SERVER_USER@$SERVER_IP" \
"APP_DIR='$APP_DIR' ENV_FILE='$ENV_FILE' bash '$remote_release_dir/disable-legacy-deploy-cron.sh'"

- name: Remove staged release files
if: always()
run: |
remote_release_dir="/tmp/letscube-release-$SOURCE_SHA"
ssh -q "$SERVER_USER@$SERVER_IP" "rm -rf -- '$remote_release_dir'" || true
8 changes: 7 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
FROM node:22-bookworm-slim AS client-build
WORKDIR /app

RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates openssl \
&& rm -rf /var/lib/apt/lists/*

RUN corepack enable

COPY package.json yarn.lock ./
Expand All @@ -27,7 +31,7 @@ RUN yarn workspace letscube-client build

FROM node:22-bookworm-slim AS server-deps
RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 make g++ \
&& apt-get install -y --no-install-recommends ca-certificates openssl python3 make g++ \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app
Expand All @@ -42,6 +46,8 @@ RUN yarn install --frozen-lockfile --production=true --network-concurrency 1 --n
&& mkdir -p server/node_modules

FROM node:22-bookworm-slim AS runtime
ARG SOURCE_SHA=local
LABEL org.opencontainers.image.revision=${SOURCE_SHA}
ENV NODE_ENV=prod
ENV PORT=8080
ENV SOCKETIO_PORT=9000
Expand Down
29 changes: 12 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ The server's owner/admin handoff and reconnect guarantees are documented in

## Metrics

The server stores pseudonymous room and authentication events in both the
`metric_events` MongoDB collection and the PostgreSQL `analytics.events` table.
The server stores pseudonymous room and authentication events in the
PostgreSQL `analytics.events` table.
It records room creations, joins, join failures, leaves and visit duration,
accepted result counts, and authentication failures. Peak room users and peak
room solve counts are the maximum `activeUserCount` and `roomSolveCount` values
Expand All @@ -100,33 +100,28 @@ ability to correlate pseudonymous users and rooms across the change.
Metrics never include names, email addresses, WCA IDs, room names, passwords,
access codes, OAuth credentials, chat content, scramble text, or solve times.

## PostgreSQL dual writes
## PostgreSQL storage

New MongoDB writes are mirrored into PostgreSQL without changing application
reads. PostgreSQL receives public identity and preferences, rooms and
participant state, RaceSession projections for normal rooms, attempts, durable
solve results, and sanitized analytics events. OAuth access tokens are deliberately not copied. Writes use
deterministic UUIDs and upserts,
so retries and future backfills are idempotent. Live room saves mirror only the
attempts and results changed by that save; complete room snapshots are reserved
for explicit backfills. Changing a normal room event ends its projected
RaceSession and starts a new one; earlier attempts and solves remain preserved
under their earlier session.
PostgreSQL stores public identity and preferences, rooms and participant state,
RaceSessions, attempts, durable solve results, social data, sessions, and
sanitized analytics events. OAuth access tokens are deliberately not persisted.
Writes use deterministic UUIDs and upserts, so retries remain idempotent. A
one-time MongoDB backfill tool exists only for cutover and is not loaded by the
runtime.

Solve penalties use dedicated boolean columns rather than JSON so histories and
statistics remain compact and index-friendly. `GET /api/solve-history` reads
PostgreSQL session-linked history for the authenticated participant. It is
enabled in development and may be enabled for selected production users with
`FEATURE_SOLVE_HISTORY_USER_IDS`.

Set `POSTGRES_ENABLED=false` to disable mirroring. Production should set
Production should set
`PGHOST`, `PGDATABASE`, `PGUSER`, and `POSTGRES_PASSWORD`, or provide a
`DATABASE_URL`. When runtime traffic uses a pooled connection, set
`DIRECT_DATABASE_URL` to the direct connection used by Prisma Migrate. External
TLS connections can set `PGSSL=true` and provide a CA with `PGSSL_CA`;
certificate verification is enabled by default. PostgreSQL failures are logged
but do not fail the corresponding MongoDB-backed application operation during
this migration phase.
certificate verification is enabled by default. PostgreSQL is required for both
application processes; startup and health checks fail closed if it is unavailable.

Username lookup uses a separately normalized, uniquely indexed key while
preserving display casing. See [the normalized username migration](docs/username-migration.md)
Expand Down
Loading
Loading