Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .claude/skills/hackstack-app/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
name: hackstack-app
description: Add, change, move or review a Hackstack app (anything under apps/, experiments/ or deprecated/) - its docker-compose files, overlays, .env.example, config/ examples and README - following the repo's conventions.
---

# Hackstack app

The conventions are in [AGENTS.md](../../../AGENTS.md) at the repo root. Read it first; this is the checklist for applying it.

## Adding an app

1. **Gather facts from upstream** before writing anything: the published image and its tags, the port it listens on, the uid it runs as, the tools in the image (for the healthcheck), every environment variable it reads, the config files it uses, and whether it needs a database, Redis, MQTT, mail or the Docker socket. If upstream has its own production compose file and `.env.example`, start from those.
2. **Pick the namespace**: `experiments/<app>` unless told otherwise.
3. **Write `docker-compose.yml`** to the AGENTS.md pattern: `${IMAGE_VERSION:-...}`, `container_name`/`hostname` = app name, `restart: unless-stopped`, `env_file: .env`, relative bind mounts under `../../lib|log|run/<app>` and `./config`, commented-out `ports:`, only the networks it needs, a healthcheck.
- Shared PostgreSQL/MariaDB, never a bundled one. Per-app Redis on a private network.
- No host networking unless nothing else works; macvlan overlay if it must appear on the LAN.
4. **Add overlays** for anything a deployment would choose differently (NFS, macvlan, GPU backend, published ports vs proxy networks), enabled through `COMPOSE_FILE` in `.env`. The user must never need to edit a compose file.
5. **Write `.env.example`** with every variable, secrets as `<FILL IN>` placeholders, optional settings commented out, hosts as container names, no `BACKUP_DATABASE_URLS`.
6. **Add `config/*.example`** for config files and git-ignore the real ones.
7. **Add `.gitignore`** (`.env` and real config files) and, if there are live database files under `../../lib/<app>`, `.rsync.exclude`.
8. **Write `README.md`** from `docs/app-readme-template.md`: what it is, required settings, database/data-directory/ownership setup, proxy target, overlays, anything unusual, usage.
9. **Check** with `docker compose config -q` (copy `.env.example` to `.env`, then delete it) for every overlay combination.
10. **Look for references**: does Prometheus scrape it (`experiments/prometheus/config/prometheus.yml.example`)? Does another app connect to it?

## Moving or deprecating an app

`git mv` the directory, then `git grep` for the app name and old path: `bin/install-*.sh`, `docs/`, other apps' READMEs and compose networks, and the Prometheus example config. Point out apps that depend on it (for example its network).

## Never

- Commit secrets, real `.env` files or real config files.
- Edit the top-level `.gitignore` or `.rsync.exclude` (generated by GitHub Actions).
- Tell the user to edit a compose file.
- Run, install or start the stack unless asked.
6 changes: 4 additions & 2 deletions .cursor/rules/docker-compose.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@
alwaysApply: true
---

Full conventions: AGENTS.md at the repo root. The person deploying an app must never have to edit a compose file; never commit secrets.

Don't use version in docker-compose files
Always use bind mounts instead of Docker volumes unless the volume is for NFS
Put NFS mounts in docker-compose.nfs.yml; enable with COMPOSE_FILE=docker-compose.yml:docker-compose.nfs.yml in .env (document in .env.example)
Put NFS mounts in docker-compose.nfs.yml; enable with COMPOSE_FILE in .env (document in .env.example)
Put LAN macvlan in docker-compose.macvlan.yml; enable with COMPOSE_FILE=docker-compose.yml:docker-compose.macvlan.yml; IP/MAC/parent NIC only via .env (see docs/macvlan.md)
Use the initial service name as the hostname and container name
restart: unless-stopped
Expand All @@ -17,7 +19,7 @@ Persistent runtime state should go in ../../lib
Log files should go in ../../log
Transient runtime state (like .pid files) should go in ../../run
Warning if no HEALTHCHECK
Warning if host mode - warn that containers should not run in host unless absolutely necessary
Warning if host mode - warn that containers should not run in host unless absolutely necessary; use a docker-compose.macvlan.yml overlay when the app must appear on the LAN
Warning if not host mode and does not declare any networks - should have a db network if it needs a database, proxy if it needs reverse proxy
Error if container has its own copy of Postgresql or Mariadb or Mysql - it should use the shared one
Whenever possible configuration files should live in a config directory in the same place as docker-compose.yml and .env - must be config not other names
Expand Down
49 changes: 43 additions & 6 deletions .github/scripts/validate_compose.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,31 @@
from typing import Optional


class ComposeLoader(yaml.SafeLoader):
"""SafeLoader that accepts Compose's merge tags (!override, !reset)."""


def _compose_tag(loader, tag_suffix, node):
if isinstance(node, yaml.MappingNode):
return loader.construct_mapping(node, deep=True)
if isinstance(node, yaml.SequenceNode):
return loader.construct_sequence(node, deep=True)
return loader.construct_scalar(node)


ComposeLoader.add_multi_constructor('!override', _compose_tag)
ComposeLoader.add_multi_constructor('!reset', _compose_tag)


def is_overlay(file_path: Path) -> bool:
"""docker-compose.<name>.yml files are overlays merged onto docker-compose.yml.

Their services are fragments, so checks for keys the base file supplies
(restart, healthcheck, networks, names) don't apply to them.
"""
return file_path.name != 'docker-compose.yml'


@dataclass
class ValidationResult:
"""Represents a validation finding."""
Expand Down Expand Up @@ -65,6 +90,11 @@ def has_errors(self):
r'mongo',
]

# Apps that bundle a database the shared services can't provide: (app dir, image pattern)
EMBEDDED_DB_EXCEPTIONS = {
('komodo', 'mongo'), # Komodo Core requires MongoDB; there is no shared MongoDB
}

# Exceptions for certain rules
RESTART_EXCEPTIONS = {'bfg-repo-cleaner', 'mailq', 'postfix-base'} # CLI tools and YAML anchors that shouldn't restart
# Published ports require `# hackstack: ports required` in the compose file (see Rule 12).
Expand Down Expand Up @@ -149,7 +179,7 @@ def validate_compose_file(file_path: Path, report: ValidationReport):

# Parse YAML
try:
data = yaml.safe_load(raw_content)
data = yaml.load(raw_content, Loader=ComposeLoader)
except yaml.YAMLError as e:
report.error("yaml-syntax", f"Invalid YAML: {e}", file_str)
return
Expand Down Expand Up @@ -229,8 +259,11 @@ def validate_service(service_name: str, config: dict, file_path: Path,

file_str = str(file_path)
service_dir = file_path.parent.name

# Rule 3: Service name should match hostname and container_name
overlay = is_overlay(file_path)

# Rule 3: Service name should match hostname and container_name.
# The app's directory name, and <app>-<service> for its other services,
# are also accepted (see AGENTS.md).
hostname = config.get('hostname')
container_name = config.get('container_name')

Expand All @@ -245,6 +278,8 @@ def name_allowed(field: str, actual: str, expected: str) -> bool:
return True
if allowed_prefix and (actual.startswith(allowed_prefix) or actual == allowed_prefix):
return True
if actual in (service_dir, f'{service_dir}-{service_name}'):
return True
return actual == expected

if hostname and not name_allowed('hostname', hostname, service_name):
Expand All @@ -259,7 +294,7 @@ def name_allowed(field: str, actual: str, expected: str) -> bool:

# Rule 4: restart: unless-stopped
restart = config.get('restart')
if service_name not in RESTART_EXCEPTIONS:
if service_name not in RESTART_EXCEPTIONS and not overlay:
if restart is None:
report.error("restart-policy",
"Missing 'restart: unless-stopped'",
Expand Down Expand Up @@ -355,7 +390,7 @@ def name_allowed(field: str, actual: str, expected: str) -> bool:
check_volume_path(host_path, file_str, service_name, report)

# Rule 17: Healthcheck warning
if 'healthcheck' not in config:
if 'healthcheck' not in config and not overlay:
report.warning("no-healthcheck",
"Consider adding a healthcheck",
file_str, service_name)
Expand All @@ -369,7 +404,7 @@ def name_allowed(field: str, actual: str, expected: str) -> bool:

# Rule 19: No networks declared (and not host mode)
networks = config.get('networks')
if network_mode != 'host' and not networks:
if network_mode != 'host' and not networks and not overlay:
report.warning("no-networks",
"No networks declared - consider adding proxy/db networks as needed",
file_str, service_name)
Expand All @@ -381,6 +416,8 @@ def name_allowed(field: str, actual: str, expected: str) -> bool:
# Skip if this IS the shared database service
if service_dir in ('postgresql', 'mariadb', 'redis'):
continue
if (service_dir, db_pattern) in EMBEDDED_DB_EXCEPTIONS:
continue
# Check if image contains forbidden DB
if re.search(rf'\b{db_pattern}\b', image_lower):
# Exception for redis used as cache within a stack (like authentik)
Expand Down
35 changes: 24 additions & 11 deletions .github/workflows/concatenate_rsync_exclude.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
name: Update .rsync.exlude
name: Update .rsync.exclude

on:
push:
paths:
- "**/.rsync.exclude"
- "apps/**/.rsync.exclude"
- "experiments/**/.rsync.exclude"
- ".rsync.exclude.initial"
- ".github/actions/concatenate_rsync_exclude.yml"
- ".github/workflows/concatenate_rsync_exclude.yml"
workflow_dispatch:

permissions:
Expand All @@ -19,16 +20,28 @@ jobs:
- name: Check out repository
uses: actions/checkout@v7

- name: Concatenate .rsync.exclude files with directory paths
- name: Concatenate .rsync.exclude files
run: |
# Start with the .rsync.exclude.initial file
cp .rsync.exclude.initial .rsync.exclude

# Find all .rsync.exclude files one level down, process each with its path prefix
find . -mindepth 2 -maxdepth 2 -name ".rsync.exclude" | while read -r rsync_excludes; do
dir_path=$(dirname "$rsync_excludes" | sed 's|^\./||')
echo -e "\n# Patterns from $dir_path/.rsync.exclude" >> .rsync.exclude
cat "$rsync_excludes" >> .rsync.exclude

# Per-app files already hold paths relative to the backup root
# (e.g. lib/<app>/app.db), so they're copied as-is.
for parent_dir in apps experiments; do
if [ ! -d "$parent_dir" ]; then
continue
fi

find "$parent_dir" -mindepth 2 -maxdepth 2 -name ".rsync.exclude" | sort | \
while read -r rsync_exclude; do
echo "" >> .rsync.exclude
echo "# [$parent_dir] $rsync_exclude" >> .rsync.exclude
cat "$rsync_exclude" >> .rsync.exclude
# Guard against a file without a trailing newline
if [ -n "$(tail -c1 "$rsync_exclude")" ]; then
echo "" >> .rsync.exclude
fi
done
done

- name: Check for changes
Expand All @@ -46,5 +59,5 @@ jobs:
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git add .rsync.exclude
git commit -m "Update top-level .rsync.exclude with directory paths"
git commit -m "Update top-level .rsync.exclude from per-app files"
git push origin HEAD:${GITHUB_REF_NAME}
24 changes: 16 additions & 8 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -123,12 +123,11 @@ experiments/headscale/config/config.yaml
experiments/headscale/config/headplane.yaml
experiments/headscale/config/acl.hujson

# [experiments] experiments/homebox/.gitignore
experiments/homebox/.env
experiments/homebox/config/config.yml
# [experiments] experiments/laser-cutterd/.gitignore
experiments/laser-cutterd/.env

# [experiments] experiments/influxdb/.gitignore
experiments/influxdb/config/influx-configs
# [experiments] experiments/library/.gitignore
experiments/library/.env

# [experiments] experiments/llama-router/.gitignore
experiments/llama-router/.env
Expand All @@ -143,20 +142,29 @@ experiments/nextcloud/.env
experiments/olah/.env
experiments/olah/config/configs.toml

# [experiments] experiments/prometheus/.gitignore
experiments/prometheus/.env
experiments/prometheus/config/prometheus.yml
experiments/prometheus/config/secrets/

# [experiments] experiments/rtlamr2mqtt/.gitignore
experiments/rtlamr2mqtt/.env
experiments/rtlamr2mqtt/config/rtlamr2mqtt.yaml

# [experiments] experiments/tailscale/.gitignore
experiments/tailscale/.env

# [experiments] experiments/telegraf/.gitignore
experiments/telegraf/config/telegraf.conf

# [experiments] experiments/tranquil/.gitignore
experiments/tranquil/.env
experiments/tranquil/config/config.toml

# [experiments] experiments/unifi-audit-daemon/.gitignore
experiments/unifi-audit-daemon/.env

# [experiments] experiments/unifi-occupancy/.gitignore
experiments/unifi-occupancy/.env
experiments/unifi-occupancy/config/config.yaml

# [experiments] experiments/unified-audio-players/.gitignore
experiments/unified-audio-players/.env

Expand Down
47 changes: 46 additions & 1 deletion .rsync.exclude
Original file line number Diff line number Diff line change
@@ -1,8 +1,53 @@
# Do not edit .rsync.exclude
# File is automatically built from .rsync.exclude.inital and */.rsync.exclude
# Built by .github/workflows/concatenate_rsync_exclude.yml from
# .rsync.exclude.initial and apps/*/.rsync.exclude, experiments/*/.rsync.exclude

lost+found/
containerd/
docker/
run/
**/*.log

# [apps] apps/calibre-web/.rsync.exclude
lib/calibre-web/metadatab.db

# [apps] apps/esphome/.rsync.exclude
lib/esphome/.esphome

# [apps] apps/home-assistant/.rsync.exclude
lib/home-assistant/backups

# [apps] apps/jellyfin/.rsync.exclude
lib/jellyfin/data/jellyfin.db
lib/jellyfin/data/jellyfin.db-shm
lib/jellyfin/data/jellyfin.db-wal
lib/jellyfin/data/library.db
lib/jellyfin/data/library.db-shm
lib/jellyfin/data/library.db-wal

# [apps] apps/mariadb/.rsync.exclude
lib/mariadb

# [apps] apps/mosquitto/.rsync.exclude
lib/mosquitto/mosquitto.db

# [apps] apps/music-assistant/.rsync.exclude
lib/music-assistant/cache.db
lib/music-assistant/library.db

# [apps] apps/nginx-proxy-manager/.rsync.exclude
lib/nginx-proxy-manager/database.sqlite

# [apps] apps/postgresql/.rsync.exclude
lib/postgresql

# [experiments] experiments/komodo/.rsync.exclude
lib/komodo/mongo

# [experiments] experiments/prometheus/.rsync.exclude
lib/prometheus

# [experiments] experiments/unifi-occupancy/.rsync.exclude
lib/unifi-occupancy/data/occupancy.sqlite3
lib/unifi-occupancy/data/occupancy.sqlite3-wal
lib/unifi-occupancy/data/occupancy.sqlite3-shm
3 changes: 2 additions & 1 deletion .rsync.exclude.initial
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Do not edit .rsync.exclude
# File is automatically built from .rsync.exclude.inital and */.rsync.exclude
# Built by .github/workflows/concatenate_rsync_exclude.yml from
# .rsync.exclude.initial and apps/*/.rsync.exclude, experiments/*/.rsync.exclude

lost+found/
containerd/
Expand Down
Loading
Loading