From 50ae7c127bb57ee2047192b81097f3529cafd6e5 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Tue, 13 Oct 2026 18:12:14 +0000 Subject: [PATCH 01/35] Back up the ClickHouse cluster to Cloudflare R2 Nothing backed up the production cluster: the weekly ooni-db-backup export reads a legacy local ClickHouse (its fastpath files have been empty since mid-2025), and scripts/dump-tables-ch.sh listed tables that no longer exist while missing obs_*, analysis_web_measurement and others. Add a clickhouse_backup role that runs ClickHouse's native BACKUP daily from a systemd timer. Each table has its own chain: a full backup every 30 days and incrementals in between, so a failure in obs_web (3.9 TB) doesn't cost the other tables their backup. The two newest complete chains are kept; chains pinned with --keep (e.g. before an upgrade step) are kept until released. A host runs the jobs set in its host_vars, one per bucket: - data3, "measurements": the replicated tables, minus the private ones and data1's. data3 isn't behind clickhouseproxy and doesn't run Airflow. - data3, "private": msmt_feedback, faulty_measurements, url_priorities, in their own bucket, since R2 tokens grant access per bucket. - data1, "data1": the plain MergeTree tables whose data only data1 holds (labeling_frames, event_detector_cusums, event_detector_v2_changepoints, changepoint_label). Remove a table from clickhouse_backup_data1_tables once it's replicated and data3 picks it up. The R2 endpoint and token come from SSM. R2 rejects multipart uploads whose parts (all but the last) differ in size, so the uploads use fixed 128 MiB parts (s3_upload_part_size_multiply_factor = 1). With R2's 10,000-part limit a file can reach 1.25 TiB; the largest file of a backup is one column of one part, and parts stop growing at 150 GB. After each table the script writes its result to a node_exporter textfile (ooni_clickhouse_backup_*), and prometheus_node_exporter gets a node_exporter_textfile_dir option for it. New alerts: a table's backup failed, no backup of a table for 3 days, no results from data1 or data3 for a day, and data3's load over half its CPUs for 15 minutes. TextfileJam skips the backup file, which changes daily rather than hourly. Things found while testing against 24.8.6.70 and MinIO: - BACKUP saves only the schema of EmbeddedRocksDB and Join tables, so their rows are exported in Native format alongside the backup. - An incremental backup stores its base_backup argument, keys included, in its .backup file. ClickHouse gets the bucket keys from endpoints in config.d instead, which it reloads without a restart. - max_backup_bandwidth is a query setting (BACKUP ... SETTINGS rejects it), and actual throughput is about a third of the value. - A 24.8 server segfaults restoring a backup taken on 26.8, so the runbook pins a backup before each upgrade step. Remove the outdated dump and restore scripts. --- ansible/deploy-clickhouse.yml | 13 + ansible/group_vars/clickhouse/vars.yml | 21 + ansible/host_vars/data1.htz-fsn.prod.ooni.nu | 7 + ansible/host_vars/data3.htz-fsn.prod.ooni.nu | 13 + .../roles/clickhouse_backup/defaults/main.yml | 67 ++ .../files/ooni_clickhouse_backup.py | 644 ++++++++++++++++++ .../roles/clickhouse_backup/tasks/main.yml | 75 ++ .../templates/clickhouse-backup.json.j2 | 19 + .../templates/ooni-clickhouse-backup.service | 14 + .../templates/ooni-clickhouse-backup.timer | 11 + .../templates/ooni_backup_s3.xml.j2 | 19 + .../roles/prometheus/files/alert_rules.yml | 39 +- .../defaults/main.yml | 3 + .../tasks/install.yml | 12 + .../templates/node_exporter.service.j2 | 2 +- docs/Runbooks.md | 155 +++++ scripts/dump-tables-ch.sh | 46 -- scripts/restore-dumps.sh | 35 - 18 files changed, 1111 insertions(+), 84 deletions(-) create mode 100644 ansible/roles/clickhouse_backup/defaults/main.yml create mode 100644 ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py create mode 100644 ansible/roles/clickhouse_backup/tasks/main.yml create mode 100644 ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 create mode 100644 ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.service create mode 100644 ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer create mode 100644 ansible/roles/clickhouse_backup/templates/ooni_backup_s3.xml.j2 delete mode 100755 scripts/dump-tables-ch.sh delete mode 100755 scripts/restore-dumps.sh diff --git a/ansible/deploy-clickhouse.yml b/ansible/deploy-clickhouse.yml index 630d1736..b40c3ec7 100644 --- a/ansible/deploy-clickhouse.yml +++ b/ansible/deploy-clickhouse.yml @@ -11,3 +11,16 @@ tags: [node_exporter] - role: oonidata_clickhouse tags: [clickhouse] + +# data3 backs up the replicated tables: they hold the same data on every +# node, and data3 isn't behind clickhouseproxy and doesn't run Airflow. +# data1 backs up the plain MergeTree tables whose data only it holds (see +# clickhouse_backup_data1_tables). The jobs are set in host_vars. +- name: Deploy ClickHouse backups + hosts: + - data1.htz-fsn.prod.ooni.nu + - data3.htz-fsn.prod.ooni.nu + become: true + roles: + - role: clickhouse_backup + tags: [clickhouse_backup] diff --git a/ansible/group_vars/clickhouse/vars.yml b/ansible/group_vars/clickhouse/vars.yml index 73275001..9b5ad668 100644 --- a/ansible/group_vars/clickhouse/vars.yml +++ b/ansible/group_vars/clickhouse/vars.yml @@ -411,6 +411,27 @@ clickhouse_prometheus: asynchronous_metrics: true status_info: true +# For textfile metrics, e.g. from clickhouse_backup +node_exporter_textfile_dir: /var/lib/node_exporter/textfile_collector + +# clickhouse_backup: Cloudflare R2. The endpoint names the account, so it's +# kept in SSM with the token's keys. +clickhouse_backup_s3_endpoint: "{{ lookup('amazon.aws.aws_ssm', '/oonidevops/secrets/clickhouse_backup_r2_endpoint', profile='oonidevops_user_prod') }}" +clickhouse_backup_s3_access_key_id: "{{ lookup('amazon.aws.aws_ssm', '/oonidevops/secrets/clickhouse_backup_r2_access_key_id', profile='oonidevops_user_prod') }}" +clickhouse_backup_s3_secret_access_key: "{{ lookup('amazon.aws.aws_ssm', '/oonidevops/secrets/clickhouse_backup_r2_secret_access_key', profile='oonidevops_user_prod') }}" +# Tables with personal or unpublished data, backed up to the private bucket +clickhouse_backup_private_tables: + - msmt_feedback # account ids and free-text comments + - faulty_measurements # until its contents are reviewed + - url_priorities # until it's decided whether it's published +# Plain MergeTree tables whose data only data1 holds (2026-10-20). Remove a +# table from here once it's replicated, and data3 picks it up. +clickhouse_backup_data1_tables: + - labeling_frames + - event_detector_cusums + - event_detector_v2_changepoints + - changepoint_label + prometheus_nginx_proxy_config: - location: /metrics/node_exporter proxy_pass: http://127.0.0.1:8100/metrics diff --git a/ansible/host_vars/data1.htz-fsn.prod.ooni.nu b/ansible/host_vars/data1.htz-fsn.prod.ooni.nu index 74f6d22f..a0b90b64 100644 --- a/ansible/host_vars/data1.htz-fsn.prod.ooni.nu +++ b/ansible/host_vars/data1.htz-fsn.prod.ooni.nu @@ -2,3 +2,10 @@ tls_cert_dir: /var/lib/dehydrated/certs ssl_domains: - "data1.htz-fsn.prod.ooni.nu" - "airflow.prod.ooni.io" + +# Only the tables whose data only data1 holds; data3 backs up the rest +clickhouse_backup_jobs: + - name: data1 + s3_bucket: ooni-clickhouse-backup + include_tables: "{{ clickhouse_backup_data1_tables }}" + exclude_tables: [] diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu index bf1b24a0..39c522cd 100644 --- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu +++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu @@ -1,2 +1,15 @@ non_admin_usernames: [] clickhouse_base_path: /data/clickhouse + +# The replicated tables, minus the ones only data1 holds data for. Private +# tables go to their own bucket: R2 tokens grant access per bucket, and +# researchers may be given read access to the measurement tables. +clickhouse_backup_jobs: + - name: measurements + s3_bucket: ooni-clickhouse-backup + include_tables: [] + exclude_tables: "{{ clickhouse_backup_private_tables + clickhouse_backup_data1_tables }}" + - name: private + s3_bucket: ooni-clickhouse-backup-private + include_tables: "{{ clickhouse_backup_private_tables }}" + exclude_tables: [] diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml new file mode 100644 index 00000000..ed9b6a9b --- /dev/null +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -0,0 +1,67 @@ +--- +# Back up the ooni ClickHouse database to S3-compatible object storage +# (Cloudflare R2) with ClickHouse's native BACKUP. Deploy where the data is: +# data3 for the replicated tables, data1 for the tables only data1 holds +# (see deploy-clickhouse.yml and host_vars). +# Restore and upgrade procedure: docs/Runbooks.md, "ClickHouse backup". + +# false stops and disables the timer, e.g. while upgrading the cluster +clickhouse_backup_enabled: true +# Daily, at a quiet time (UTC). A run backs up every job's tables, one job +# after the other; a run still going (a new full backup of obs_web takes +# more than a day) delays the next one instead of overlapping it. +clickhouse_backup_on_calendar: "*-*-* 02:30:00" + +clickhouse_backup_database: ooni +clickhouse_backup_clickhouse_url: "http://127.0.0.1:8123/" +clickhouse_backup_clickhouse_user: "{{ clickhouse_admin_user }}" +clickhouse_backup_clickhouse_password: "{{ clickhouse_admin_password }}" + +# Cloudflare R2: the endpoint names the account, +# https://.r2.cloudflarestorage.com. One API token with Object +# Read & Write on every bucket the jobs below use. +clickhouse_backup_s3_endpoint: "" +clickhouse_backup_s3_region: auto +clickhouse_backup_s3_prefix: prod +clickhouse_backup_s3_access_key_id: "" +clickhouse_backup_s3_secret_access_key: "" + +# Query settings for ClickHouse's uploads. R2 rejects a multipart upload +# whose parts (all but the last) differ in size, and ClickHouse grows the +# part size as an upload goes on unless the factor is 1. R2 also allows at +# most 10,000 parts, so with 128 MiB parts one file can reach 1.25 TiB. +clickhouse_backup_s3_settings: + s3_upload_part_size_multiply_factor: 1 + s3_min_upload_part_size: 134217728 + s3_max_upload_part_size: 134217728 + +# One job per bucket. A job backs up the tables matching include_tables +# (every table if empty) minus the excludes below and its own. +clickhouse_backup_jobs: + - name: measurements + s3_bucket: "" + include_tables: [] + exclude_tables: [] + +# A new full backup per table every this many days; incrementals in between +clickhouse_backup_full_interval_days: 30 +# Complete chains kept per table, besides chains pinned with --keep: the +# current one and the one before, so a month or two of restore points +clickhouse_backup_keep_chains: 2 +# Bytes/s passed to ClickHouse as max_backup_bandwidth. On 24.8 actual +# throughput measured at about a third of this, so 120 MB/s gives ~40 MB/s, +# which leaves room on the 1 Gbit/s link for replication. +clickhouse_backup_max_bandwidth: 120000000 + +# fnmatch patterns, for every job. Materialized views' .inner tables are +# always skipped: they're backed up with their view. +clickhouse_backup_exclude_tables: + - "*_tmp" # updater staging tables + - msmt_feedback_old + - counters_test_list # unused materialized view, to be dropped + - fastpath_tuning # experiment on data2 + +# Results for Prometheus, read by node_exporter's textfile collector +# (node_exporter_textfile_dir in the prometheus_node_exporter role) +clickhouse_backup_metrics_file: "{{ node_exporter_textfile_dir | default('/var/lib/node_exporter/textfile_collector') }}/ooni_clickhouse_backup.prom" +clickhouse_backup_state_file: /var/lib/ooni-clickhouse-backup/state.json diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py new file mode 100644 index 00000000..a96cda7c --- /dev/null +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -0,0 +1,644 @@ +#!/usr/bin/env python3 +""" +Back up a ClickHouse database to S3-compatible object storage using +ClickHouse's native BACKUP, with one incremental chain per table. + +Configured by /etc/ooni/clickhouse-backup.json (see the clickhouse_backup +Ansible role). Runs from a systemd timer. The config lists one or more jobs, +run one after the other; each job backs up a set of tables to its own bucket +(R2 tokens grant access per bucket, so private tables get a bucket of their +own). + +Layout in each bucket: + + ///// a native backup + //
///data.native.zst + data export, for + EmbeddedRocksDB and + Join tables only + //
//KEEP-
/// native backup (first one is full) +prod/ooni/
//KEEP-
/""" + out = {} + for key in keys(bucket, f"prod/ooni/{table}/"): + chain, name = key.split("/")[3:5] + backups = out.setdefault(chain, {}) + backups[name] = backups.get(name, False) or key.endswith(f"/{name}/.backup") + return out + + +def copy_prefix(bucket, src, dst): + for key in keys(bucket, src): + s3.copy_object(Bucket=bucket, Key=dst + key[len(src):], CopySource={"Bucket": bucket, "Key": key}) + + +def delete_prefix(bucket, prefix): + for key in keys(bucket, prefix): + s3.delete_object(Bucket=bucket, Key=key) + + +def wait_for(what, get, timeout=150): + """Poll get() until it returns something truthy""" + t0 = time.monotonic() + while time.monotonic() - t0 < timeout: + try: + value = get() + except OSError: + value = None + if value: + return value + time.sleep(2) + check(False, f"{what}, within {timeout}s") + + +def slack(): + """(title, text) of every message the Slack stand-in has received""" + return [(a["title"].strip(), a.get("text", "").strip()) for m in json.loads(http("http://127.0.0.1:19980/")) + for a in m.get("attachments", [])] + + +def email_attempts(): + lines = http("http://127.0.0.1:19993/metrics").splitlines() + return sum(float(l.split()[-1]) for l in lines + if l.startswith('alertmanager_notifications_total{integration="email"')) + + +def months_ago(n): + now = datetime.now(timezone.utc) + y, m = divmod(now.year * 12 + now.month - 1 - n, 12) + return f"{y}{m + 1:02}01T023000Z" + + +def setup(): + print("== waiting for the services") + wait_for("ClickHouse", lambda: ch("SELECT 1") == "1") + wait_for("MinIO", lambda: http("http://127.0.0.1:19910/minio/health/live") == "") + wait_for("Prometheus", lambda: "Ready" in http("http://127.0.0.1:19990/-/ready")) + wait_for("Alertmanager", lambda: "OK" in http("http://127.0.0.1:19993/-/ready")) + wait_for("Slack stand-in", lambda: http("http://127.0.0.1:19980/") == "[]") + for bucket in (MEASUREMENTS, PRIVATE): + s3.create_bucket(Bucket=bucket) + for sql in [ + "CREATE DATABASE ooni", + "CREATE TABLE ooni.fastpath (measurement_uid String, input String, anomaly LowCardinality(String)) " + "ENGINE = MergeTree ORDER BY measurement_uid", + "INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/' || toString(number % 100), " + "if(number % 7 = 0, 't', 'f') FROM numbers(200000)", + # ~20 MB that doesn't compress: takes ~20 s to back up with work/slow.json + "CREATE TABLE ooni.obs_web (measurement_uid String, blob String) ENGINE = MergeTree ORDER BY measurement_uid", + "INSERT INTO ooni.obs_web SELECT toString(number), randomString(4096) FROM numbers(5000)", + "CREATE TABLE ooni.fingerprints_dns (name String, pattern String) ENGINE = EmbeddedRocksDB PRIMARY KEY name", + "INSERT INTO ooni.fingerprints_dns VALUES ('a', '1.2.3.4'), ('b', '5.6.7.8')", + "CREATE TABLE ooni.msmt_feedback (measurement_uid String, comment String) ENGINE = MergeTree ORDER BY measurement_uid", + "INSERT INTO ooni.msmt_feedback VALUES ('m1', 'blocked?')", + "CREATE TABLE ooni.labeling_frames (measurement_uid String) ENGINE = MergeTree ORDER BY measurement_uid", + "INSERT INTO ooni.labeling_frames VALUES ('x')", + "CREATE TABLE ooni.asnmeta_tmp (asn UInt32) ENGINE = MergeTree ORDER BY asn", # excluded: *_tmp + ]: + ch(sql) + + +def test_first_run(): + print("== 1. first run: a full backup of every table") + r = backup("backup", "run") + check(r.returncode == 0, f"the run succeeds{'' if r.returncode == 0 else ': ' + r.stderr[-2000:]}") + for bucket, table in [(MEASUREMENTS, "fastpath"), (MEASUREMENTS, "obs_web"), + (MEASUREMENTS, "fingerprints_dns"), (PRIVATE, "msmt_feedback")]: + c = chains(bucket, table) + check(len(c) == 1 and all(all(b.values()) for b in c.values()), f"{table}: one complete chain in {bucket}") + check(any(k.endswith("/data.native.zst") for k in keys(MEASUREMENTS, "prod/ooni/fingerprints_dns/")), + "fingerprints_dns (EmbeddedRocksDB): its rows are exported too") + check(not keys(MEASUREMENTS, "prod/ooni/msmt_feedback/"), "msmt_feedback isn't in the measurements bucket") + check(not keys(MEASUREMENTS, "prod/ooni/asnmeta_tmp/"), "asnmeta_tmp (excluded) isn't backed up") + check(not keys(MEASUREMENTS, "prod/ooni/labeling_frames/"), "labeling_frames (data1's) isn't backed up") + + +def test_completed_alert(emails_before): + print("== 2. ClickHouseBackupCompleted") + for job, tables in (("measurements", 3), ("private", 1)): + title = f"ClickHouse backup {job} on {DATA3} completed" + msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0].startswith(title)]) + check(len(msgs) == 1, f"one Slack message for {job}: {msgs[0][0]!r}") + check(f"Tables backed up: {tables}, in full: {tables}" in msgs[0][1], f"it says: {msgs[0][1]!r}") + check(not any(t.startswith("[FIRING]") for t, _ in slack()), "no [FIRING] in the titles") + check(email_attempts() == emails_before, "no email sent") + + +def test_incremental_and_restore(): + print("== 3. second run: incrementals, and a restore") + ch("INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers(200000, 1000)") + r = backup("backup", "run") + check(r.returncode == 0, "the run succeeds") + (chain, backups), = chains(MEASUREMENTS, "fastpath").items() + check(len(backups) == 2 and all(backups.values()), f"fastpath: a full and an incremental in chain {chain}") + latest = f"http://minio:9000/{MEASUREMENTS}/prod/ooni/fastpath/{chain}/{max(backups)}" + ch("CREATE DATABASE restored") + ch(f"RESTORE TABLE ooni.fastpath AS restored.fastpath FROM S3('{latest}')") + digest = "SELECT count(), sum(cityHash64(measurement_uid, input, anomaly)) FROM {}" + check(ch(digest.format("restored.fastpath")) == ch(digest.format("ooni.fastpath")), + "the restored fastpath has the same rows (201000)") + + +def test_failure_alert(emails_before): + print("== 4. a run that fails: ClickHouseBackupFailed") + r = backup("broken", "run") + check(r.returncode == 1, "the run exits 1") + title = f"[FIRING] ClickHouse backup of labeling_frames (broken) failed on {DATA3}" + msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0] == title]) + check("Nothing was deleted" in msgs[0][1], f"it says what was done: {msgs[0][1]!r}") + wait_for("an email attempt for it", lambda: email_attempts() > emails_before) + check(not any(t.startswith("ClickHouse backup broken") for t, _ in slack()), "no 'completed' message for it") + + +def test_interrupted_full(): + print("== 5. a new month's full, interrupted, then completed") + (chain, backups), = chains(MEASUREMENTS, "obs_web").items() + full = f"prod/ooni/obs_web/{chain}/{min(backups)}/" + old, previous = months_ago(2), months_ago(1) + # obs_web's chains as at the start of a month: the full of two months ago and of last month + for c in (old, previous): + copy_prefix(MEASUREMENTS, full, f"prod/ooni/obs_web/{c}/{c}/") + delete_prefix(MEASUREMENTS, f"prod/ooni/obs_web/{chain}/") + + run = subprocess.Popen(COMPOSE + ["exec", "-T", "runner", "python3", "/role/ooni_clickhouse_backup.py", + "--config", "/work/slow.json", "run", "--table", "obs_web"], + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + wait_for("the new full's first files", lambda: set(chains(MEASUREMENTS, "obs_web")) - {old, previous}, 60) + subprocess.run(COMPOSE + ["restart", "clickhouse"], check=True, capture_output=True) + out, _ = run.communicate(timeout=180) + check(run.returncode == 1, "the interrupted run exits 1") + c = chains(MEASUREMENTS, "obs_web") + new = sorted(set(c) - {old, previous}) + check(new and not any(c[new[0]].values()), f"the new chain {new[0]} has no .backup") + check(all(c[x][x] for x in (old, previous)), f"both older chains, {old} and {previous}, are still there") + + wait_for("ClickHouse after the restart", lambda: ch("SELECT 1") == "1") + r = backup("backup", "run", "--table", "obs_web") + check(r.returncode == 0, "the next run succeeds") + check("removed incomplete chain" in r.stderr, "it removed the interrupted full") + c = chains(MEASUREMENTS, "obs_web") + check(previous in c and old not in c and len(c) == 2 and all(all(b.values()) for b in c.values()), + f"it took the full and pruned {old}: chains {sorted(c)}") + + +def main(): + setup() + emails_before = email_attempts() + test_first_run() + test_completed_alert(emails_before) + test_incremental_and_restore() + test_failure_alert(emails_before) + test_interrupted_full() + print("PASS") + + +if __name__ == "__main__": + main() diff --git a/ansible/roles/clickhouse_backup/tests/test_schedule.py b/ansible/roles/clickhouse_backup/tests/test_schedule.py new file mode 100644 index 00000000..b7ea3146 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/test_schedule.py @@ -0,0 +1,120 @@ +"""When ooni_clickhouse_backup.py starts a full backup and what it prunes, over months of daily or weekly runs, +some of them failing. ClickHouse and the bucket are stubbed; test_e2e.py runs the real thing. + python3 -m unittest test_schedule""" +import importlib.util +import json +import os +import unittest +from datetime import datetime, timedelta, timezone + +spec = importlib.util.spec_from_file_location( + "backup", os.path.join(os.path.dirname(__file__), "..", "files", "ooni_clickhouse_backup.py") +) +backup = importlib.util.module_from_spec(spec) +spec.loader.exec_module(backup) + +START = datetime(2026, 10, 23, 2, 30, tzinfo=timezone.utc) # a Friday + + +class FakeStore: + """The bucket: {chain: {"backups": {backup: complete}, "keep": [labels]}}""" + + def __init__(self): + self.c = {} + + def table_prefix(self, db, table): + return f"{db}/{table}/" + + def ch_s3(self, key): + return f"'{key}'" + + def chains(self, db, table): + return {k: {"backups": dict(e["backups"]), "keep": list(e["keep"])} for k, e in self.c.items()} + + def delete_prefix(self, prefix): + parts = prefix.rstrip("/").split("/") + if parts[-2] in self.c: # one backup of a chain + self.c[parts[-2]]["backups"].pop(parts[-1], None) + else: + self.c.pop(parts[-1], None) + return 1 + + def put_empty(self, key): + pass + + +class FakeClickHouse: + """Takes a backup, or fails one part-way: files uploaded, no .backup""" + + def __init__(self, store): + self.store, self.fail = store, False + + def query(self, sql, settings=None, timeout=600): + chain, name = sql.split("TO S3('")[1].split("'")[0].split("/")[-2:] + self.store.c.setdefault(chain, {"backups": {}, "keep": []})["backups"][name] = not self.fail + if self.fail: + raise RuntimeError("BACKUP_FAILED") + return json.dumps({"id": "op"}) + + +def runs(every_days, until, failing=(), jitter=True): + """Back up a table at 02:30 UTC daily or on Sundays from START. Yields (day, full, complete chains).""" + job = backup.BackupJob.__new__(backup.BackupJob) + job.db, job.name, job.keep_chains, job.settings = "ooni", "t", 2, {} + job.store = FakeStore() + job.ch = FakeClickHouse(job.store) + job.metrics = backup.Metrics() + job.wait = lambda op, what: {"num_files": 1, "uncompressed_size": 1} + t, n = START, 0 + while t < until: + # a table's backup starts a few minutes earlier or later each time + now = t + timedelta(minutes=9 if n % 2 else -7) + backup.utcnow = lambda now=now: now + job.ch.fail = now.date().isoformat() in failing + try: + full = job.backup_table("obs_web", "ReplicatedMergeTree", False, "") + except RuntimeError: + full = None + complete = sorted(c for c, e in job.store.c.items() if any(e["backups"].values())) + yield now.date().isoformat(), full, complete + n += 1 + if every_days == 7: + t += timedelta(days=(6 - t.weekday()) % 7 or 7) # the next Sunday + else: + t += timedelta(days=1) + + +class TestSchedule(unittest.TestCase): + def fulls(self, every_days): + return [day for day, full, _ in runs(every_days, datetime(2027, 4, 1, tzinfo=timezone.utc)) if full] + + def test_weekly_fulls_on_the_first_sunday_of_each_month(self): + self.assertEqual( + self.fulls(7), ["2026-10-23", "2026-11-01", "2026-12-06", "2027-01-03", "2027-02-07", "2027-03-07"] + ) + + def test_daily_fulls_on_the_first_of_each_month(self): + self.assertEqual( + self.fulls(1), ["2026-10-23", "2026-11-01", "2026-12-01", "2027-01-01", "2027-02-01", "2027-03-01"] + ) + + def test_two_complete_chains_kept(self): + *_, (day, _, complete) = runs(7, datetime(2027, 4, 1, tzinfo=timezone.utc)) + self.assertEqual([c[:8] for c in complete], ["20270207", "20270307"]) + + def test_failed_fulls_prune_nothing_and_are_retried(self): + failing = {"2026-12-06", "2026-12-13", "2027-02-07", "2027-02-14", "2027-02-21", "2027-02-28", "2027-03-07"} + before = [] + for day, full, complete in runs(7, datetime(2027, 5, 1, tzinfo=timezone.utc), failing): + month = day[:7].replace("-", "") + if not any(c.startswith(month) for c in complete): + # no complete full this month yet: every complete chain from before is still there + self.assertTrue(set(before) <= set(complete), f"{day}: deleted {set(before) - set(complete)}") + if day in failing: + self.assertIsNone(full, day) + before = complete + self.assertEqual([c[:8] for c in complete], ["20270314", "20270404"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/ansible/roles/clickhouse_backup/tests/vars.yml b/ansible/roles/clickhouse_backup/tests/vars.yml new file mode 100644 index 00000000..0db73f9f --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/vars.yml @@ -0,0 +1,34 @@ +# Role variables for the test environment in compose.yml, with data3's two +# jobs: the measurement tables and the private tables, each in its own bucket. +clickhouse_admin_user: admin +clickhouse_admin_password: admin +clickhouse_backup_clickhouse_url: http://clickhouse:8123/ +clickhouse_backup_s3_endpoint: http://minio:9000 +clickhouse_backup_s3_access_key_id: testkey +clickhouse_backup_s3_secret_access_key: testsecret123 + +test_jobs: + - name: measurements + s3_bucket: ooni-clickhouse-backup + include_tables: [] + exclude_tables: [msmt_feedback, labeling_frames] + - name: private + s3_bucket: ooni-clickhouse-backup-private + include_tables: [msmt_feedback] + exclude_tables: [] + +# One script config per entry, in work/.json +test_configs: + - name: backup # as deployed + bandwidth: 0 + jobs: "{{ test_jobs }}" + - name: slow # ~1 MB/s, so a full backup can be interrupted + bandwidth: 3000000 + jobs: "{{ test_jobs }}" + - name: broken # its bucket doesn't exist, so every backup fails + bandwidth: 0 + jobs: + - name: broken + s3_bucket: no-such-bucket + include_tables: [labeling_frames] + exclude_tables: [] From 7857e1c2490d8dbf9d62f10491954e7e748b09ce Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 10:03:31 +0000 Subject: [PATCH 07/35] clickhouse_backup tests: use Debian's python3-boto3, not pip The script now runs on Debian bookworm with its python3 and python3-boto3, the packages the role installs on the ClickHouse hosts, instead of a Python image with boto3 from pip. The workflow installs ansible-core and python3-boto3 with apt for the scripts around it. --- .github/workflows/test_clickhouse_backup.yml | 5 +---- ansible/roles/clickhouse_backup/tests/README.md | 2 +- ansible/roles/clickhouse_backup/tests/run.sh | 2 +- ansible/roles/clickhouse_backup/tests/runner.Dockerfile | 9 ++++++--- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/test_clickhouse_backup.yml b/.github/workflows/test_clickhouse_backup.yml index 04a73ed6..f660b978 100644 --- a/.github/workflows/test_clickhouse_backup.yml +++ b/.github/workflows/test_clickhouse_backup.yml @@ -22,8 +22,5 @@ jobs: timeout-minutes: 30 steps: - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 - with: - python-version: "3.11" - - run: pip install ansible-core==2.17.7 boto3==1.26.27 + - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends ansible-core python3-boto3 - run: ansible/roles/clickhouse_backup/tests/run.sh diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md index d2a97e6d..6a5c9fc1 100644 --- a/ansible/roles/clickhouse_backup/tests/README.md +++ b/ansible/roles/clickhouse_backup/tests/README.md @@ -10,7 +10,7 @@ bucket stubbed. - `test_e2e.py`: the real thing, in the containers of `compose.yml`: ClickHouse at production's version, MinIO standing in for R2, the script - in a Debian bookworm Python with bookworm's boto3, node_exporter, + on Debian bookworm with its python3-boto3, as on the hosts, node_exporter, Prometheus with the repo's alert rules, Alertmanager with the repo's routes, and a stand-in for Slack. Full and incremental backups, a restore, a failed run, an interrupted full, the month's new full and diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh index 3f15659c..f2491da8 100755 --- a/ansible/roles/clickhouse_backup/tests/run.sh +++ b/ansible/roles/clickhouse_backup/tests/run.sh @@ -1,7 +1,7 @@ #!/bin/bash # Tests the clickhouse_backup role: renders its configs from the role's templates, checks the backup alerts # (promtool) and their Alertmanager routes (amtool), runs the schedule tests, then test_e2e.py against the -# containers in compose.yml. Needs docker compose, python3 with boto3, and ansible-core. With rootless podman, +# containers in compose.yml. Needs docker compose, python3-boto3 and ansible-core. With rootless podman, # point DOCKER_HOST at its socket. KEEP=1 leaves the containers running. set -euo pipefail cd "$(dirname "$0")" diff --git a/ansible/roles/clickhouse_backup/tests/runner.Dockerfile b/ansible/roles/clickhouse_backup/tests/runner.Dockerfile index 48ba2c22..3abc14ab 100644 --- a/ansible/roles/clickhouse_backup/tests/runner.Dockerfile +++ b/ansible/roles/clickhouse_backup/tests/runner.Dockerfile @@ -1,3 +1,6 @@ -# Debian bookworm's python3 and python3-boto3, as on the ClickHouse hosts -FROM docker.io/library/python:3.11-slim-bookworm -RUN pip install --no-cache-dir boto3==1.26.27 && mkdir -p /run/lock +# Debian bookworm with its python3 and python3-boto3, which the role installs on the ClickHouse hosts +FROM docker.io/library/debian:bookworm-slim +RUN apt-get update \ + && apt-get install -y --no-install-recommends python3 python3-boto3 \ + && rm -rf /var/lib/apt/lists/* \ + && mkdir -p /run/lock From 294ef5449f3d60f1aaf93addea9af8b8f3c098cf Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 10:39:10 +0000 Subject: [PATCH 08/35] clickhouse_backup: cap backups at 50 MB/s until the real rate is measured max_backup_bandwidth was 120 MB/s, on the strength of 24.8 uploading at about a third of it in tests against MinIO. If production doesn't show the same, that would take most of the nodes' 1 Gbit/s link. Cap it at 50 MB/s, about 31 hours for a full backup of everything, and raise it if the first backups show the third: about 4 days for a full, longer than ClickHouseBackupStale's 3. The comments and the runbook say how to check the rate. --- ansible/roles/clickhouse_backup/defaults/main.yml | 10 ++++++---- .../clickhouse_backup/files/ooni_clickhouse_backup.py | 3 ++- docs/Runbooks.md | 10 +++++++--- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index c2165e59..e8a7b461 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -46,10 +46,12 @@ clickhouse_backup_jobs: # Complete chains kept per table, besides chains pinned with --keep: the # current one and the one before, so a month or two of restore points clickhouse_backup_keep_chains: 2 -# Bytes/s passed to ClickHouse as max_backup_bandwidth. On 24.8 actual -# throughput measured at about a third of this, so 120 MB/s gives ~40 MB/s, -# which leaves room on the 1 Gbit/s link for replication. -clickhouse_backup_max_bandwidth: 120000000 +# Bytes/s passed to ClickHouse as max_backup_bandwidth: at most 50 MB/s, 40% +# of the 1 Gbit/s link, leaving the rest for replication and queries. +# Against MinIO in testing, 24.8 uploaded at about a third of this value; check +# the real rate on the first backups (ooni_clickhouse_backup_last_size_bytes / +# ooni_clickhouse_backup_last_duration_seconds) and raise it if they're too slow. +clickhouse_backup_max_bandwidth: 50000000 # fnmatch patterns, for every job. Materialized views' .inner tables are # always skipped: they're backed up with their view. diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 69a5dd70..31fb628b 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -369,7 +369,8 @@ def __init__(self, conf: dict, metrics: Metrics = None): bw = int(conf.get("max_backup_bandwidth", 0)) if bw: # A query setting, not a BACKUP setting: 24.8 rejects it inside - # BACKUP ... SETTINGS. Measured throughput is ~1/3 of this value. + # BACKUP ... SETTINGS. Against MinIO in testing, the throughput + # was about a third of this value. self.settings["max_backup_bandwidth"] = str(bw) def tables(self, only: list) -> tuple: diff --git a/docs/Runbooks.md b/docs/Runbooks.md index 02b8a82f..fe04034e 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -209,9 +209,13 @@ When a data1-only table becomes replicated, remove it from so one backup file can reach 1.25 TiB. - Two complete chains are kept per table (`clickhouse_backup_keep_chains`), so a month or two of restore points. -- Bandwidth is capped with `clickhouse_backup_max_bandwidth`. On 24.8 the - actual rate is about a third of the cap: 120 MB/s gives ~40 MB/s. A new - full backup of everything (~5.6 TB) takes about 39 hours at that rate. +- Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A + new full backup of everything (~5.6 TB) takes about 31 hours at that rate. + Against MinIO in testing, 24.8 uploaded at only about a third of the cap, + which would make it about 4 days, longer than `ClickHouseBackupStale`'s 3: + check the rate of the first backups + (`ooni_clickhouse_backup_last_size_bytes / ooni_clickhouse_backup_last_duration_seconds`) + and raise the cap if they're that slow. Layout in the bucket: From caf6d7d1d04a17575d56210ef8ee37464e43e4dc Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 10:39:10 +0000 Subject: [PATCH 09/35] clickhouse_backup: stop excluding counters_test_list, which is detached counters_test_list is detached permanently, so it's no longer in system.tables, which the script takes its tables from. --- ansible/roles/clickhouse_backup/defaults/main.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index e8a7b461..7f0c2989 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -58,7 +58,6 @@ clickhouse_backup_max_bandwidth: 50000000 clickhouse_backup_exclude_tables: - "*_tmp" # updater staging tables - msmt_feedback_old - - counters_test_list # unused materialized view, to be dropped - fastpath_tuning # experiment on data2 # Results for Prometheus, read by node_exporter's textfile collector From 69bc63b043b79dda60c165818b988525176b3040 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 10:47:47 +0000 Subject: [PATCH 10/35] clickhouse_backup: say which RocksDB tables are and aren't backed up The runbook listed fingerprints_* and session_expunge among the tables whose rows each backup exports, but neither is backed up: the fingerprints tables are only on data1 and data2 and the hourly updater rebuilds them, and session_expunge is empty on both. Only test_groups, on data3, is exported. --- docs/Runbooks.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/Runbooks.md b/docs/Runbooks.md index fe04034e..c019a72b 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -200,9 +200,12 @@ When a data1-only table becomes replicated, remove it from table starts a new chain with its first backup of each calendar month (UTC), which is the first Sunday of the month if the timer runs weekly. Once a new chain is complete, older chains are deleted unless pinned. -- EmbeddedRocksDB and Join tables (`fingerprints_*`, `session_expunge`, - `test_groups`): `BACKUP` saves only their schema, so their rows are also - exported to `data.native.zst` inside each backup. +- EmbeddedRocksDB and Join tables (`test_groups`): `BACKUP` saves only their + schema, so their rows are also exported to `data.native.zst` inside each + backup. +- Not backed up: `fingerprints_dns` and `fingerprints_http`, which are only on + data1 and data2 and which the hourly updater rebuilds, and `session_expunge`, + which is empty and no longer used. - R2 rejects multipart uploads whose parts (all but the last) differ in size, so `clickhouse_backup_s3_settings` fixes ClickHouse's part size at 128 MiB (`s3_upload_part_size_multiply_factor: 1`). R2 allows 10,000 parts, From 2447141010a53a49cb016cf29a5a3c23e0175925 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 10:47:47 +0000 Subject: [PATCH 11/35] clickhouse_backup: stop excluding fastpath_tuning, which is dropped The experiment table on data2 is gone, and data2 isn't backed up anyway. --- ansible/roles/clickhouse_backup/defaults/main.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index 7f0c2989..ccd578c9 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -58,7 +58,6 @@ clickhouse_backup_max_bandwidth: 50000000 clickhouse_backup_exclude_tables: - "*_tmp" # updater staging tables - msmt_feedback_old - - fastpath_tuning # experiment on data2 # Results for Prometheus, read by node_exporter's textfile collector # (node_exporter_textfile_dir in the prometheus_node_exporter role) From 113451d19c01e87e5aafa1993971484e4f9272ba Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 11:04:35 +0000 Subject: [PATCH 12/35] clickhouse_backup: say why list prints instead of logging The listing is the command's output, for stdout and pipes; log is for what run and release report, to stderr and the journal. --- .../roles/clickhouse_backup/files/ooni_clickhouse_backup.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 31fb628b..48372969 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -559,6 +559,9 @@ def run(self, only: list, force_full: bool, keep: str) -> int: return 0 def list(self, only: list) -> int: + # The listing is the command's output, so it goes to stdout with + # print(), to be read or piped (e.g. the latest: URL into a RESTORE); + # log is for what run and release report, to stderr and the journal. tables = only or sorted( n for n in self.store.subdirs(f"{self.store.prefix}/{self.db}/") if n ) From 7cab3dc158c8bdb92bc582d7b01fda8346fe18fb Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 11:08:37 +0000 Subject: [PATCH 13/35] clickhouse_backup: back up weekly, on Sundays A full backup of everything takes more than a day, so run weekly instead of daily for now: incrementals every Sunday at 02:30 UTC, each on top of the one before, and a new full on the first Sunday of each month. The timer's first run after a deploy is then the next Sunday; the runbook says how to start the first full on deploy day. --- .../roles/clickhouse_backup/defaults/main.yml | 8 ++++---- docs/Runbooks.md | 18 +++++++++--------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index ccd578c9..4a59d13f 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -7,10 +7,10 @@ # false stops and disables the timer, e.g. while upgrading the cluster clickhouse_backup_enabled: true -# Daily, at a quiet time (UTC). A run backs up every job's tables, one job -# after the other; a run still going (a new full backup of obs_web takes -# more than a day) delays the next one instead of overlapping it. -clickhouse_backup_on_calendar: "*-*-* 02:30:00" +# Weekly, on Sundays at a quiet time (UTC): a full backup of everything takes +# more than a day. A run backs up every job's tables, one job after the other; +# a run still going delays the next one instead of overlapping it. +clickhouse_backup_on_calendar: "Sun *-*-* 02:30:00" clickhouse_backup_database: ooni clickhouse_backup_clickhouse_url: "http://127.0.0.1:8123/" diff --git a/docs/Runbooks.md b/docs/Runbooks.md index c019a72b..461528cb 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -195,10 +195,10 @@ When a data1-only table becomes replicated, remove it from `clickhouse_backup_data1_tables` and data3 picks it up. - `ooni-clickhouse-backup.timer` runs `/usr/local/bin/ooni-clickhouse-backup run` - daily at 02:30 UTC. -- Each table has its own chain: a full backup, then daily incrementals. A - table starts a new chain with its first backup of each calendar month - (UTC), which is the first Sunday of the month if the timer runs weekly. + weekly, on Sundays at 02:30 UTC (`clickhouse_backup_on_calendar`). +- Each table has its own chain: a full backup, then weekly incrementals, each + on top of the one before. A table starts a new chain with its first backup + of each calendar month (UTC): the first Sunday of the month. Once a new chain is complete, older chains are deleted unless pinned. - EmbeddedRocksDB and Join tables (`test_groups`): `BACKUP` saves only their schema, so their rows are also exported to `data.native.zst` inside each @@ -215,8 +215,7 @@ When a data1-only table becomes replicated, remove it from - Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A new full backup of everything (~5.6 TB) takes about 31 hours at that rate. Against MinIO in testing, 24.8 uploaded at only about a third of the cap, - which would make it about 4 days, longer than `ClickHouseBackupStale`'s 3: - check the rate of the first backups + which would make it about 4 days: check the rate of the first backups (`ooni_clickhouse_backup_last_size_bytes / ooni_clickhouse_backup_last_duration_seconds`) and raise the cap if they're that slow. @@ -248,9 +247,10 @@ without a restart. ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t clickhouse_backup --diff ``` and the alert rules with `deploy-monitoring.yml`. -4. The timer takes the first (full) backup at its next 02:30 UTC. A full - backup of everything takes about 39 hours, so to have it run over a - weekend, deploy on Friday after 02:30 UTC. To start one now instead: +4. The timer takes the first (full) backup at its next run, on Sunday at + 02:30 UTC. A full backup of everything takes about 31 hours, or longer if + the upload rate is below the cap. To start it on deploy day instead, e.g. a + Friday so that it runs over the weekend: ```bash sudo systemctl start --no-block ooni-clickhouse-backup.service sudo journalctl -f --identifier ooni-clickhouse-backup From f1305c615acfcf26e0d31b61a4212b23e2b857eb Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 11:08:37 +0000 Subject: [PATCH 14/35] clickhouse_backup: allow a week between backups before ClickHouseBackupStale With weekly runs a table's last backup is normally up to a week old, so ClickHouseBackupStale fired after 3 days would fire every week. It now fires after 8 days, and not while the table's job is still running: on the first Sunday of a month the fulls take more than a day, and the largest tables come last. The promtool tests cover both. --- .../clickhouse_backup/tests/alert_rules_test.yml | 11 ++++++++--- ansible/roles/prometheus/files/alert_rules.yml | 8 ++++++-- docs/Runbooks.md | 3 ++- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml index ce753a0f..85e47052 100644 --- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml +++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml @@ -10,12 +10,17 @@ tests: - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="measurements"}', values: '0x180'} - {series: 'ooni_clickhouse_backup_last_run_tables{instance="data3", backup="measurements"}', values: '14x180'} - {series: 'ooni_clickhouse_backup_last_run_full_backups{instance="data3", backup="measurements"}', values: '1x180'} - # private finished at the same time with a table failed: fastpath, last backed up 4 days ago + # private finished at the same time with a table failed: fastpath, last backed up 9 days ago. + # Test time starts at 0 (1970-01-01), so a timestamp 9 days before it is -9 * 86400. - {series: 'ooni_clickhouse_backup_last_run_finished_timestamp_seconds{instance="data3", backup="private"}', values: '0x180'} - {series: 'ooni_clickhouse_backup_last_run_duration_seconds{instance="data3", backup="private"}', values: '60x180'} - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="private"}', values: '1x180'} - {series: 'ooni_clickhouse_backup_last_attempt_failed{instance="data3", backup="private", table="fastpath"}', values: '1x180'} - - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-345600x180'} + - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-777600x180'} + - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="private"}', values: '0x180'} + # measurements' obs_web was also last backed up 9 days ago, but its job is still running + - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="measurements", table="obs_web"}', values: '-777600x180'} + - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="measurements"}', values: '1x180'} alert_rule_test: - eval_time: 10m alertname: ClickHouseBackupCompleted @@ -40,5 +45,5 @@ tests: exp_alerts: - exp_labels: {instance: data3, backup: private, table: fastpath} exp_annotations: - summary: 'No ClickHouse backup of fastpath (private) for 4d 0h 10m 0s' + summary: 'No ClickHouse backup of fastpath (private) for 9d 0h 10m 0s' description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data3.' diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml index 66eb2e3c..54ed39e3 100644 --- a/ansible/roles/prometheus/files/alert_rules.yml +++ b/ansible/roles/prometheus/files/alert_rules.yml @@ -168,9 +168,13 @@ groups: summary: 'ClickHouse backup {{ $labels.backup }} on {{ $labels.instance }} completed in {{ $value | humanizeDuration }}' description: 'Tables backed up: {{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}, in full: {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}. Nothing to do.' - # Backups run daily; a new full backup of obs_web can take a day and a half + # Backups run weekly, so a table's last one is normally up to a week old. Not + # while its job runs: on the first Sunday of a month the full backups take + # more than a day, and the largest tables come last. - alert: ClickHouseBackupStale - expr: time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 3 * 86400 + expr: | + time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 8 * 86400 + and on (instance, backup) ooni_clickhouse_backup_running == 0 annotations: summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}' description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.' diff --git a/docs/Runbooks.md b/docs/Runbooks.md index 461528cb..61e0cf4f 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -266,7 +266,8 @@ Alerts, in `alert_rules.yml`: deleted, and the next run retries it. - `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack message per run in `#ooni-bots`, no email. -- `ClickHouseBackupStale`: a table hasn't been backed up for 3 days. +- `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups + are weekly), and its job isn't running. - `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day. - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its CPUs for 15 minutes. From 83fe34e9e245c63491b60294996324be4b54922d Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 12:00:53 +0000 Subject: [PATCH 15/35] clickhouse_backup: hold back ClickHouseBackupStale only for a recent run on the host The alert was held back only while the table's own job ran. On data3 the jobs run one after the other, so on the first Sunday of a month private waited more than a day behind measurements' full backups and passed 8 days without one, while nothing was wrong. And a run killed without reaching its finally left running at 1, which held the alert back for good. The script now also exports, per job, when its current run started (ooni_clickhouse_backup_running_since_timestamp_seconds, 0 when idle). The alert is held back while any job on the host is in a run that started in the last 5 days. The promtool tests cover a 7- and a 9-day old table with nothing running, a job waiting behind another job's run, and a stuck run; test_job.py the new metric and older state files. --- .../files/ooni_clickhouse_backup.py | 8 +++ .../roles/clickhouse_backup/tests/README.md | 1 + .../tests/alert_rules_test.yml | 38 ++++++++--- ansible/roles/clickhouse_backup/tests/run.sh | 2 +- .../roles/clickhouse_backup/tests/test_job.py | 67 +++++++++++++++++++ .../roles/prometheus/files/alert_rules.yml | 8 ++- docs/Runbooks.md | 2 +- 7 files changed, 110 insertions(+), 16 deletions(-) create mode 100644 ansible/roles/clickhouse_backup/tests/test_job.py diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 48372969..3ad742e4 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -279,6 +279,7 @@ def __init__(self, metrics_file: str = "", state_file: str = ""): with open(state_file) as f: self.state = json.load(f) self.state.setdefault("runs", {}) + self.state.setdefault("running_since", {}) def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration_s: float = 0) -> None: entry = self.state["tables"].setdefault(f"{job}/{table}", {"job": job, "table": table}) @@ -294,6 +295,7 @@ def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration def job_running(self, job: str, running: bool) -> None: self.state["running"][job] = 1 if running else 0 + self.state["running_since"][job] = time.time() if running else 0 self.save() def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None: @@ -311,6 +313,12 @@ def render(self) -> str: ] for job, v in sorted(self.state["running"].items()): lines.append(f'{p}running{{backup="{label_value(job)}"}} {v}') + lines += [ + f"# HELP {p}running_since_timestamp_seconds When the job's current run started, 0 if none", + f"# TYPE {p}running_since_timestamp_seconds gauge", + ] + for job, v in sorted(self.state["running_since"].items()): + lines.append(f'{p}running_since_timestamp_seconds{{backup="{label_value(job)}"}} {v}') for name, field, help_ in self.RUN_GAUGES: lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"] for job, run in sorted(self.state["runs"].items()): diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md index 6a5c9fc1..4d548799 100644 --- a/ansible/roles/clickhouse_backup/tests/README.md +++ b/ansible/roles/clickhouse_backup/tests/README.md @@ -8,6 +8,7 @@ - `test_schedule.py`: which runs start a full backup and what gets pruned, over months of daily and weekly runs with failures, ClickHouse and the bucket stubbed. +- `test_job.py`: how a job runs and what it reports, ClickHouse stubbed. - `test_e2e.py`: the real thing, in the containers of `compose.yml`: ClickHouse at production's version, MinIO standing in for R2, the script on Debian bookworm with its python3-boto3, as on the hosts, node_exporter, diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml index 85e47052..323c12b0 100644 --- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml +++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml @@ -1,7 +1,9 @@ -# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml +# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml. Test time starts at 0 +# (1970-01-01), so a timestamp N days before the test starts is -N * 86400. rule_files: [alert_rules.yml] evaluation_interval: 1m tests: + # Completed and Failed - interval: 1m input_series: # measurements finished at t=0 after 30 min, all 14 tables backed up, 1 in full @@ -10,17 +12,11 @@ tests: - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="measurements"}', values: '0x180'} - {series: 'ooni_clickhouse_backup_last_run_tables{instance="data3", backup="measurements"}', values: '14x180'} - {series: 'ooni_clickhouse_backup_last_run_full_backups{instance="data3", backup="measurements"}', values: '1x180'} - # private finished at the same time with a table failed: fastpath, last backed up 9 days ago. - # Test time starts at 0 (1970-01-01), so a timestamp 9 days before it is -9 * 86400. + # private finished at the same time, with fastpath failed - {series: 'ooni_clickhouse_backup_last_run_finished_timestamp_seconds{instance="data3", backup="private"}', values: '0x180'} - {series: 'ooni_clickhouse_backup_last_run_duration_seconds{instance="data3", backup="private"}', values: '60x180'} - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="private"}', values: '1x180'} - {series: 'ooni_clickhouse_backup_last_attempt_failed{instance="data3", backup="private", table="fastpath"}', values: '1x180'} - - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-777600x180'} - - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="private"}', values: '0x180'} - # measurements' obs_web was also last backed up 9 days ago, but its job is still running - - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="measurements", table="obs_web"}', values: '-777600x180'} - - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="measurements"}', values: '1x180'} alert_rule_test: - eval_time: 10m alertname: ClickHouseBackupCompleted @@ -40,10 +36,30 @@ tests: exp_annotations: summary: 'ClickHouse backup of fastpath (private) failed on data3' description: 'Nothing was deleted: the older backups of fastpath are kept. The next run retries it. See `journalctl --identifier ooni-clickhouse-backup` on data3.' + + # Stale: after 8 days, unless a job on the same host is in a run started in the last 5 days + - interval: 1m + input_series: + # data1: nothing running; one table backed up 9 days ago, one 7 days ago + - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="data1", backup="data1"}', values: '0x30'} + - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data1", backup="data1", table="nine_days"}', values: '-777600x30'} + - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data1", backup="data1", table="seven_days"}', values: '-604800x30'} + # data3: measurements' monthly full started 2 days ago; private, waiting behind it, last ran 9 days ago + - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="data3", backup="measurements"}', values: '-172800x30'} + - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="data3", backup="private"}', values: '0x30'} + - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="msmt_feedback"}', values: '-777600x30'} + # stuck: a run that "started" 6 days ago, after the script was killed; its table is 9 days old + - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="stuck", backup="measurements"}', values: '-518400x30'} + - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="stuck", backup="measurements", table="fastpath"}', values: '-777600x30'} + alert_rule_test: - eval_time: 10m alertname: ClickHouseBackupStale exp_alerts: - - exp_labels: {instance: data3, backup: private, table: fastpath} + - exp_labels: {instance: data1, backup: data1, table: nine_days} + exp_annotations: + summary: 'No ClickHouse backup of nine_days (data1) for 9d 0h 10m 0s' + description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data1.' + - exp_labels: {instance: stuck, backup: measurements, table: fastpath} exp_annotations: - summary: 'No ClickHouse backup of fastpath (private) for 9d 0h 10m 0s' - description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data3.' + summary: 'No ClickHouse backup of fastpath (measurements) for 9d 0h 10m 0s' + description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on stuck.' diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh index f2491da8..51b2295a 100755 --- a/ansible/roles/clickhouse_backup/tests/run.sh +++ b/ansible/roles/clickhouse_backup/tests/run.sh @@ -26,7 +26,7 @@ route "alertname=ClickHouseBackupCompleted instance=data3 backup=measurements" s route "alertname=ClickHouseBackupFailed instance=data3 backup=measurements table=fastpath" team-all echo "== schedule" -python3 -m unittest test_schedule +python3 -m unittest test_schedule test_job echo "== end to end" [ -n "${KEEP:-}" ] || trap 'compose down -v --remove-orphans >/dev/null 2>&1' EXIT diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py new file mode 100644 index 00000000..76526d0a --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -0,0 +1,67 @@ +"""How a job runs and what it reports, with ClickHouse stubbed; test_e2e.py runs the real thing. + python3 -m unittest test_job""" +import importlib.util +import json +import os +import tempfile +import unittest +import urllib.error + +spec = importlib.util.spec_from_file_location( + "backup", os.path.join(os.path.dirname(__file__), "..", "files", "ooni_clickhouse_backup.py") +) +backup = importlib.util.module_from_spec(spec) +spec.loader.exec_module(backup) + + +class FlakyClickHouse: + """Fails the first `failures` calls with a connection error, then answers""" + + def __init__(self, failures=0, status="BACKUP_CREATED"): + self.failures, self.status, self.calls = failures, status, 0 + + def query(self, sql, settings=None, timeout=600): + self.calls += 1 + if self.calls <= self.failures: + raise urllib.error.URLError("Connection refused") + return "" + + def rows(self, sql, **kw): + self.query(sql) + if "system.backups WHERE id" in sql: + return [{"status": self.status, "error": "", "num_files": 1, "total_size": 1, "uncompressed_size": 1}] + return [] + + +def job(name="measurements", include=(), exclude=(), metrics=None, ch=None): + j = backup.BackupJob.__new__(backup.BackupJob) + j.name, j.include, j.exclude = name, list(include), list(exclude) + j.metrics = metrics or backup.Metrics() + j.ch = ch or FlakyClickHouse() + j.poll_s, j.clickhouse_wait_s = 0, 0 + return j + + +class TestRunningSince(unittest.TestCase): + def setUp(self): + d = tempfile.mkdtemp() + self.metrics = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json")) + + def test_running_since_is_set_while_a_job_runs(self): + self.metrics.job_running("measurements", True) + self.assertGreater(self.metrics.state["running_since"]["measurements"], 0) + with open(self.metrics.metrics_file) as f: + self.assertIn('ooni_clickhouse_backup_running_since_timestamp_seconds{backup="measurements"}', f.read()) + self.metrics.job_running("measurements", False) + self.assertEqual(self.metrics.state["running_since"]["measurements"], 0) + + def test_old_state_files_still_load(self): + with open(self.metrics.state_file, "w") as f: + json.dump({"tables": {}, "running": {"measurements": 0}}, f) + m = backup.Metrics(self.metrics.metrics_file, self.metrics.state_file) + m.job_running("measurements", True) + self.assertIn("measurements", m.state["running_since"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml index 54ed39e3..a0b86b2b 100644 --- a/ansible/roles/prometheus/files/alert_rules.yml +++ b/ansible/roles/prometheus/files/alert_rules.yml @@ -169,12 +169,14 @@ groups: description: 'Tables backed up: {{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}, in full: {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}. Nothing to do.' # Backups run weekly, so a table's last one is normally up to a week old. Not - # while its job runs: on the first Sunday of a month the full backups take - # more than a day, and the largest tables come last. + # while any job on the host is in a run that started in the last 5 days: on + # the first Sunday of a month the full backups take more than a day, and the + # jobs run one after the other. A run older than that is stuck (e.g. the + # script was killed and left running at 1), and no longer holds the alert back. - alert: ClickHouseBackupStale expr: | time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 8 * 86400 - and on (instance, backup) ooni_clickhouse_backup_running == 0 + unless on (instance) (time() - (ooni_clickhouse_backup_running_since_timestamp_seconds != 0) < 5 * 86400) annotations: summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}' description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.' diff --git a/docs/Runbooks.md b/docs/Runbooks.md index 61e0cf4f..307a8048 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -267,7 +267,7 @@ Alerts, in `alert_rules.yml`: - `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack message per run in `#ooni-bots`, no email. - `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups - are weekly), and its job isn't running. + are weekly), and no job on its host is in a run started in the last 5 days. - `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day. - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its CPUs for 15 minutes. From 6ea1f2f7dda39a455575df4bf794c5f37bc408e5 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 12:00:53 +0000 Subject: [PATCH 16/35] clickhouse_backup: run --table runs only the jobs that back up those tables The other jobs ran with no tables, recorded a completed run of 0 tables (ClickHouseBackupCompleted posted it to Slack) and overwrote their real weekly results. --- .../files/ooni_clickhouse_backup.py | 12 ++++++++++-- ansible/roles/clickhouse_backup/tests/test_job.py | 12 ++++++++++++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 3ad742e4..52f8b852 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -602,6 +602,13 @@ def release(self, label: str) -> int: return 0 +def jobs_for_tables(jobs: list, tables: list) -> tuple: + """(the jobs that back up one of tables, the tables no job backs up)""" + owns = lambda j, t: j.includes(t) and not j.excludes(t) + orphans = [t for t in tables if not any(owns(j, t) for j in jobs)] + return [j for j in jobs if any(owns(j, t) for t in tables)], orphans + + def job_confs(conf: dict, names: list) -> list: """Each job's config: the top-level keys, overridden by the job's own. @@ -655,8 +662,9 @@ def main() -> int: job.list(args.table) return 0 - if args.command == "run": - orphans = [t for t in args.table if not any(j.includes(t) and not j.excludes(t) for j in jobs)] + if args.command == "run" and args.table: + # only these tables' jobs: the others would run with no tables, and report that + jobs, orphans = jobs_for_tables(jobs, args.table) if orphans: ap.error(f"no job backs up: {', '.join(orphans)}") diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index 76526d0a..63424a66 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -63,5 +63,17 @@ def test_old_state_files_still_load(self): self.assertIn("measurements", m.state["running_since"]) +class TestRunTable(unittest.TestCase): + def test_only_the_jobs_that_back_up_the_tables(self): + jobs = [job("measurements", exclude=["msmt_feedback"]), job("private", include=["msmt_feedback"])] + selected, orphans = backup.jobs_for_tables(jobs, ["msmt_feedback"]) + self.assertEqual([j.name for j in selected], ["private"]) + self.assertEqual(orphans, []) + + def test_tables_no_job_backs_up(self): + jobs = [job("private", include=["msmt_feedback"])] + self.assertEqual(backup.jobs_for_tables(jobs, ["fastpath"]), ([], ["fastpath"])) + + if __name__ == "__main__": unittest.main() From 61ff892a7863a1734d0906eecf6dc7220dfbf8ff Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 12:00:54 +0000 Subject: [PATCH 17/35] clickhouse_backup: don't let a brief ClickHouse outage abort a run At boot, Persistent=true catches up a missed Sunday before ClickHouse accepts queries. The checks before the table loop then failed outside any try: the run died with a traceback, skipped the later jobs and wrote no metrics, and nothing alerted until ClickHouseBackupStale 8 days later. A run now waits up to 10 minutes for ClickHouse (clickhouse_wait_seconds). If it still can't start, the job's tables are marked failed, so ClickHouseBackupFailed says so, and the next job runs. One failed status poll also gave up on a backup that can run for days, while the BACKUP carried on server-side. Polls are now retried for up to 10 minutes. --- .../files/ooni_clickhouse_backup.py | 63 +++++++++++++++---- .../roles/clickhouse_backup/tests/test_job.py | 29 +++++++++ 2 files changed, 80 insertions(+), 12 deletions(-) diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 52f8b852..a045594a 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -298,6 +298,14 @@ def job_running(self, job: str, running: bool) -> None: self.state["running_since"][job] = time.time() if running else 0 self.save() + def job_failed(self, job: str) -> None: + """The job couldn't start: its tables' last attempts failed""" + for entry in self.state["tables"].values(): + if entry["job"] == job: + entry["last_attempt"] = time.time() + entry["last_failed"] = 1 + self.save() + def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None: self.state["runs"][job] = { "finished": time.time(), "duration": round(duration_s, 1), @@ -371,6 +379,7 @@ def __init__(self, conf: dict, metrics: Metrics = None): self.include = conf.get("include_tables", []) self.exclude = conf.get("exclude_tables", []) self.poll_s = float(conf.get("poll_interval_seconds", 30)) + self.clickhouse_wait_s = float(conf.get("clickhouse_wait_seconds", 600)) # Query settings for the uploads, e.g. R2 needs every part of a # multipart upload but the last to be the same size self.settings = {k: str(v) for k, v in conf.get("s3_settings", {}).items()} @@ -435,16 +444,41 @@ def running_backups(self) -> list: ) return [r for r in rows if self.store.ch_base in r["name"]] + def wait_for_clickhouse(self) -> None: + """At boot a missed run starts before ClickHouse accepts queries""" + t0 = time.monotonic() + while True: + try: + self.ch.query("SELECT 1", timeout=30) + return + except (OSError, RuntimeError) as e: + if time.monotonic() - t0 >= self.clickhouse_wait_s: + raise RuntimeError(f"ClickHouse not answering after {self.clickhouse_wait_s:.0f}s: {e}") + log.info(f"waiting for ClickHouse: {e}") + time.sleep(10) + def wait(self, op_id: str, what: str) -> dict: t0 = time.monotonic() last_log = t0 + failing_since = None while True: - rows = self.ch.rows( - f""" - SELECT status, error, num_files, total_size, uncompressed_size - FROM system.backups WHERE id = {sql_str(op_id)} - """ - ) + # A backup can run for days: retry a failed poll for a while + # rather than give up on a BACKUP that is still running + try: + rows = self.ch.rows( + f""" + SELECT status, error, num_files, total_size, uncompressed_size + FROM system.backups WHERE id = {sql_str(op_id)} + """ + ) + failing_since = None + except (OSError, RuntimeError) as e: + failing_since = failing_since or time.monotonic() + if time.monotonic() - failing_since > 600: + raise + log.warning(f"{what}: polling failed, retrying: {e}") + time.sleep(self.poll_s) + continue if not rows: raise RuntimeError(f"{what}: gone from system.backups (server restart?)") st = rows[0] @@ -537,14 +571,19 @@ def backup_table(self, table: str, engine: str, force_full: bool, keep: str) -> return full def run(self, only: list, force_full: bool, keep: str) -> int: - running = self.running_backups() - if running: - for r in running: - log.error(f"backup {r['id']} is still running on the server") - log.error("not starting: wait for it to finish, or for the server to drop it") + try: + self.wait_for_clickhouse() + running = self.running_backups() + if running: + for r in running: + log.error(f"backup {r['id']} is still running on the server") + raise RuntimeError("not starting: wait for it to finish, or for the server to drop it") + tables, failed = self.tables(only) + except Exception as e: + log.error(f"{self.name}: could not start: {e}") + self.metrics.job_failed(self.name) return 1 - tables, failed = self.tables(only) t0, done, fulls = time.monotonic(), 0, 0 self.metrics.job_running(self.name, True) try: diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index 63424a66..daec61c5 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -6,6 +6,7 @@ import tempfile import unittest import urllib.error +from unittest import mock spec = importlib.util.spec_from_file_location( "backup", os.path.join(os.path.dirname(__file__), "..", "files", "ooni_clickhouse_backup.py") @@ -75,5 +76,33 @@ def test_tables_no_job_backs_up(self): self.assertEqual(backup.jobs_for_tables(jobs, ["fastpath"]), ([], ["fastpath"])) +class TestClickHouseErrors(unittest.TestCase): + def setUp(self): + d = tempfile.mkdtemp() + self.metrics = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json")) + + def test_waits_for_clickhouse_at_start(self): + j = job(metrics=self.metrics, ch=FlakyClickHouse(failures=2)) + j.clickhouse_wait_s = 60 + with mock.patch.object(backup.time, "sleep"): + j.wait_for_clickhouse() + self.assertEqual(j.ch.calls, 3) + + def test_a_job_that_cant_start_marks_its_tables_failed(self): + self.metrics.table_result("measurements", "fastpath", True, {"total_size": 1, "num_files": 1}) + self.metrics.table_result("private", "msmt_feedback", True, {"total_size": 1, "num_files": 1}) + j = job(metrics=self.metrics, ch=FlakyClickHouse(failures=1000)) + self.assertEqual(j.run([], False, ""), 1) + tables = self.metrics.state["tables"] + self.assertEqual(tables["measurements/fastpath"]["last_failed"], 1) + self.assertEqual(tables["private/msmt_feedback"]["last_failed"], 0, "another job's table") + self.assertNotIn("measurements", self.metrics.state["runs"], "no completed run to report") + + def test_a_failed_poll_is_retried(self): + j = job(ch=FlakyClickHouse(failures=2)) + self.assertEqual(j.wait("op", "obs_web")["status"], "BACKUP_CREATED") + self.assertEqual(j.ch.calls, 3) + + if __name__ == "__main__": unittest.main() From 2e2fa2efe849b871d05801e2f944287e7996713d Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 12:01:06 +0000 Subject: [PATCH 18/35] clickhouse_backup: wait 8 days before ClickHouseBackupMissing A host has no backup results until its first run, which with weekly runs can be up to a week after the role is deployed: the 26 h wait alerted every 3 hours until then. --- .../clickhouse_backup/tests/alert_rules_test.yml | 13 +++++++++++++ ansible/roles/prometheus/files/alert_rules.yml | 7 ++++--- docs/Runbooks.md | 3 ++- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml index 323c12b0..08e03fe5 100644 --- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml +++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml @@ -63,3 +63,16 @@ tests: exp_annotations: summary: 'No ClickHouse backup of fastpath (measurements) for 9d 0h 10m 0s' description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on stuck.' + + # Missing: data1 has results, data3 never wrote any; it fires after 8 days + - interval: 1h + input_series: + - {series: 'ooni_clickhouse_backup_running{instance="data1.htz-fsn.prod.ooni.nu", backup="data1"}', values: '0x240'} + alert_rule_test: + - {eval_time: 7d, alertname: ClickHouseBackupMissing, exp_alerts: []} + - eval_time: 9d + alertname: ClickHouseBackupMissing + exp_alerts: + - exp_labels: {instance: data3.htz-fsn.prod.ooni.nu} + exp_annotations: + summary: 'No ClickHouse backup results from data3.htz-fsn.prod.ooni.nu for over a week' diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml index a0b86b2b..aae32954 100644 --- a/ansible/roles/prometheus/files/alert_rules.yml +++ b/ansible/roles/prometheus/files/alert_rules.yml @@ -181,12 +181,13 @@ groups: summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}' description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.' - # The job hasn't written any result since its host was set up, or the file is gone + # The job hasn't written any result since its host was set up, or the file is + # gone. Backups run weekly, so a new host has none until its first Sunday. - alert: ClickHouseBackupMissing expr: absent(ooni_clickhouse_backup_running{instance="data3.htz-fsn.prod.ooni.nu"}) or absent(ooni_clickhouse_backup_running{instance="data1.htz-fsn.prod.ooni.nu"}) - for: 26h + for: 8d annotations: - summary: 'No ClickHouse backup results from data1 or data3 for over a day' + summary: 'No ClickHouse backup results from {{ $labels.instance }} for over a week' # data3 takes the backups. Above half its CPUs (load per CPU > 0.5), the # backup or merges are crowding the node; lower clickhouse_backup_max_bandwidth diff --git a/docs/Runbooks.md b/docs/Runbooks.md index 307a8048..0fc49adc 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -268,7 +268,8 @@ Alerts, in `alert_rules.yml`: message per run in `#ooni-bots`, no email. - `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups are weekly), and no job on its host is in a run started in the last 5 days. -- `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day. +- `ClickHouseBackupMissing`: data1 or data3 hasn't written any results for 8 + days, e.g. since the role was deployed and before its first Sunday run. - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its CPUs for 15 minutes. From 79660f27a9f5b8a2648c4bde180564e67c164984 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 12:01:19 +0000 Subject: [PATCH 19/35] clickhouse_backup runbook: start the post-upgrade full as a service, restore test_groups Before upgrading, step 4 re-enabled the timer first, which with Persistent=true may start a run for a missed Sunday, then ran a multi-day run --full in a shell. Run it with systemd-run while the timer is still disabled, and re-enable the timer afterwards. Note that ClickHouse waits for running backups when it stops (shutdown_wait_backups_and_restores, TimeoutStopSec=infinity), so a restart during a full hangs until it is done. The restore example loaded fingerprints_dns, which isn't backed up; test_groups is the table it's for. --- docs/Runbooks.md | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/docs/Runbooks.md b/docs/Runbooks.md index 0fc49adc..d2735a9a 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -296,17 +296,28 @@ before that step, so: 1. Disable the timer with `clickhouse_backup_enabled: false` (or `sudo systemctl stop ooni-clickhouse-backup.timer`) and check that - `system.backups` shows nothing running. -2. Take a pinned backup: + `system.backups` shows nothing running. ClickHouse waits for running + backups when it stops (`shutdown_wait_backups_and_restores`, and the + unit's `TimeoutStopSec=infinity`), so restarting a node during a full + backup hangs until the backup is done, a day or more. +2. Take a pinned backup (incrementals, minutes): ```bash sudo ooni-clickhouse-backup run --keep pre-25.8.29.51 ``` 3. Upgrade all three nodes. Don't run backups while nodes are on different versions. -4. Re-enable the timer. Once the whole upgrade is confirmed good, start - new chains on the new version and unpin the old ones: +4. Once the whole upgrade is confirmed good, and with the timer still + disabled (once enabled, it may start a run straight away for a missed + Sunday), start new chains on the new version. That takes a day or more, + so run it as a service rather than in a shell: + ```bash + sudo systemd-run --unit=ooni-clickhouse-backup-full -p TimeoutStartSec=infinity \ + -p SyslogIdentifier=ooni-clickhouse-backup /usr/local/bin/ooni-clickhouse-backup run --full + sudo journalctl -f --identifier ooni-clickhouse-backup + ``` + When the journal shows `all tables backed up` for every job, unpin the + old chains and re-enable the timer (`clickhouse_backup_enabled: true`): ```bash - sudo ooni-clickhouse-backup run --full sudo ooni-clickhouse-backup release pre-25.8.29.51 ``` @@ -329,9 +340,9 @@ each table. Restoring the latest incremental restores the whole chain. ```sql RESTORE TABLE ooni.obs_web FROM S3('https://.r2.cloudflarestorage.com/ooni-clickhouse-backup/prod/ooni/obs_web//') ``` -3. For EmbeddedRocksDB and Join tables, load the rows too: +3. For EmbeddedRocksDB and Join tables (`test_groups`), load the rows too: ```sql - INSERT INTO ooni.fingerprints_dns + INSERT INTO ooni.test_groups SELECT * FROM s3('/data.native.zst', 'Native') ``` 4. On the other replicas, create replicated tables with the same From a16960b19ea4331e8eff47bbea671239b9ba4954 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 12:01:31 +0000 Subject: [PATCH 20/35] prometheus: make TextfileJam's exclusion match the backup file's path, test the backup host alert node_exporter's file label is the file's full path, so the anchored "ooni_clickhouse_backup.*" never matched (TextfileJam matches nothing today anyway: it uses node_exporter's pre-0.16 metric names). The promtool tests now cover the exclusion, and ClickHouseBackupHostLoadHigh, whose description no longer assumes a backup is the cause. --- .../tests/alert_rules_test.yml | 43 ++++++++++++++++++- .../roles/prometheus/files/alert_rules.yml | 4 +- 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml index 08e03fe5..3ff312dd 100644 --- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml +++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml @@ -1,5 +1,6 @@ -# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml. Test time starts at 0 -# (1970-01-01), so a timestamp N days before the test starts is -N * 86400. +# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml, and for the backup file's +# exclusion from TextfileJam. Test time starts at 0 (1970-01-01), so a timestamp N days before the test +# starts is -N * 86400. rule_files: [alert_rules.yml] evaluation_interval: 1m tests: @@ -76,3 +77,41 @@ tests: - exp_labels: {instance: data3.htz-fsn.prod.ooni.nu} exp_annotations: summary: 'No ClickHouse backup results from data3.htz-fsn.prod.ooni.nu for over a week' + + # HostLoadHigh: data3's 8 CPUs with a load of 5 + - interval: 1m + input_series: + - {series: 'node_load5{instance="data3.htz-fsn.prod.ooni.nu"}', values: '5x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="0", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="1", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="2", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="3", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="4", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="5", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="6", mode="idle"}', values: '0+1x60'} + - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="7", mode="idle"}', values: '0+1x60'} + alert_rule_test: + - {eval_time: 10m, alertname: ClickHouseBackupHostLoadHigh, exp_alerts: []} + - eval_time: 20m + alertname: ClickHouseBackupHostLoadHigh + exp_alerts: + - exp_labels: {instance: data3.htz-fsn.prod.ooni.nu} + exp_annotations: + summary: 'data3 load is 62.5% of its CPUs' + description: 'If a backup is running (ooni_clickhouse_backup_running on data3), lower clickhouse_backup_max_bandwidth; otherwise check merges and queries.' + + # TextfileJam leaves out the backup's file, which changes only once a week. node_exporter's file label + # is the file's full path. + - interval: 1m + input_series: + - {series: 'node_time{instance="data3", job="node"}', values: '0+60x240'} + - {series: 'node_textfile_mtime{instance="data3", job="node", file="/var/lib/node_exporter/textfile_collector/ooni_clickhouse_backup.prom"}', values: '0x240'} + - {series: 'node_textfile_mtime{instance="data3", job="node", file="/var/lib/node_exporter/textfile_collector/other.prom"}', values: '0x240'} + alert_rule_test: + - eval_time: 3h + alertname: TextfileJam + exp_alerts: + - exp_labels: {instance: data3, job: node, file: /var/lib/node_exporter/textfile_collector/other.prom} + exp_annotations: + summary: 'Stale `textfile` at data3' + description: '`/var/lib/node_exporter/textfile_collector/other.prom` is 3h old.' diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml index aae32954..1149f04f 100644 --- a/ansible/roles/prometheus/files/alert_rules.yml +++ b/ansible/roles/prometheus/files/alert_rules.yml @@ -102,7 +102,7 @@ groups: # all the textfile are updated at least hourly, except the ClickHouse # backup results, which change after each backup (see ClickHouseBackupStale) - alert: TextfileJam - expr: (node_textfile_mtime{file!~"ooni_clickhouse_backup.*"} - ON(instance,job) GROUP_LEFT() node_time) / -3600 > 2 + expr: (node_textfile_mtime{file!~".*/ooni_clickhouse_backup\\.prom"} - ON(instance,job) GROUP_LEFT() node_time) / -3600 > 2 annotations: summary: 'Stale `textfile` at {{ $labels.instance }}' description: '`{{ $labels.file }}` is {{ printf "%.0f" $value }}h old.' @@ -197,5 +197,5 @@ groups: for: 15m annotations: summary: 'data3 load is {{ $value | humanizePercentage }} of its CPUs' - description: 'Backup running: check ooni_clickhouse_backup_running on data3.' + description: 'If a backup is running (ooni_clickhouse_backup_running on data3), lower clickhouse_backup_max_bandwidth; otherwise check merges and queries.' ... From d45700ed4f2b77b8ed90ad7144b55f993a3ca354 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 13:59:08 +0000 Subject: [PATCH 21/35] clickhouse_backup: "*" selects every table, an empty include_tables none An empty include_tables meant every table. data1's job includes only clickhouse_backup_data1_tables, which is to be emptied as its tables become replicated; data1 would then have backed up every table, private ones included, into the measurements bucket, under the same prefixes as data3, whose in-progress full it could also have removed as incomplete. "*" now selects every table and an empty list none, with a warning; data3's measurements job and the role's default job say ["*"]. --- ansible/host_vars/data3.htz-fsn.prod.ooni.nu | 2 +- .../roles/clickhouse_backup/defaults/main.yml | 7 +++--- .../files/ooni_clickhouse_backup.py | 13 ++++++---- ansible/roles/clickhouse_backup/tests/run.sh | 4 ++- .../roles/clickhouse_backup/tests/test_e2e.py | 25 +++++++++++-------- .../roles/clickhouse_backup/tests/test_job.py | 15 ++++++++++- .../roles/clickhouse_backup/tests/vars.yml | 2 +- 7 files changed, 46 insertions(+), 22 deletions(-) diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu index 39c522cd..0e4eef14 100644 --- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu +++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu @@ -7,7 +7,7 @@ clickhouse_base_path: /data/clickhouse clickhouse_backup_jobs: - name: measurements s3_bucket: ooni-clickhouse-backup - include_tables: [] + include_tables: ["*"] exclude_tables: "{{ clickhouse_backup_private_tables + clickhouse_backup_data1_tables }}" - name: private s3_bucket: ooni-clickhouse-backup-private diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index 4a59d13f..5c3637ee 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -35,12 +35,13 @@ clickhouse_backup_s3_settings: s3_min_upload_part_size: 134217728 s3_max_upload_part_size: 134217728 -# One job per bucket. A job backs up the tables matching include_tables -# (every table if empty) minus the excludes below and its own. +# One job per bucket. A job backs up the tables matching include_tables ("*" +# for every table; an empty list backs up none) minus the excludes below and +# its own. clickhouse_backup_jobs: - name: measurements s3_bucket: "" - include_tables: [] + include_tables: ["*"] exclude_tables: [] # Complete chains kept per table, besides chains pinned with --keep: the diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index a045594a..0a5cbbb4 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -375,7 +375,8 @@ def __init__(self, conf: dict, metrics: Metrics = None): ) self.store = Store(conf) self.keep_chains = max(1, int(conf.get("keep_chains", 1))) - # fnmatch patterns; an empty include list means every table + # fnmatch patterns: "*" for every table; an empty list selects none, so + # emptying a host's list can't make it back up every table self.include = conf.get("include_tables", []) self.exclude = conf.get("exclude_tables", []) self.poll_s = float(conf.get("poll_interval_seconds", 30)) @@ -432,7 +433,7 @@ def tables(self, only: list) -> tuple: return out, missing def includes(self, table: str) -> bool: - return not self.include or any(fnmatch.fnmatchcase(table, p) for p in self.include) + return any(fnmatch.fnmatchcase(table, p) for p in self.include) def excludes(self, table: str) -> bool: return any(fnmatch.fnmatchcase(table, p) for p in self.exclude) @@ -579,6 +580,8 @@ def run(self, only: list, force_full: bool, keep: str) -> int: log.error(f"backup {r['id']} is still running on the server") raise RuntimeError("not starting: wait for it to finish, or for the server to drop it") tables, failed = self.tables(only) + if not tables and not only: + log.warning(f"{self.name}: no tables to back up (include_tables: {self.include})") except Exception as e: log.error(f"{self.name}: could not start: {e}") self.metrics.job_failed(self.name) @@ -651,11 +654,11 @@ def jobs_for_tables(jobs: list, tables: list) -> tuple: def job_confs(conf: dict, names: list) -> list: """Each job's config: the top-level keys, overridden by the job's own. - A config without "jobs" is one job named "default". A job's - exclude_tables add to the top-level ones.""" + A config without "jobs" is one job named "default", for every table. A + job's exclude_tables add to the top-level ones.""" common = {k: v for k, v in conf.items() if k != "jobs"} out = [] - for job in conf.get("jobs") or [{"name": "default"}]: + for job in conf.get("jobs") or [{"name": "default", "include_tables": ["*"]}]: merged = {**common, **job} merged["exclude_tables"] = common.get("exclude_tables", []) + job.get("exclude_tables", []) if not names or merged["name"] in names: diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh index 51b2295a..c45abe8f 100755 --- a/ansible/roles/clickhouse_backup/tests/run.sh +++ b/ansible/roles/clickhouse_backup/tests/run.sh @@ -29,6 +29,8 @@ echo "== schedule" python3 -m unittest test_schedule test_job echo "== end to end" -[ -n "${KEEP:-}" ] || trap 'compose down -v --remove-orphans >/dev/null 2>&1' EXIT +# the containers' logs if it fails, then remove them, unless KEEP is set +trap 'rc=$?; [ $rc = 0 ] || compose logs --no-color --tail=200; [ -n "${KEEP:-}" ] || compose down -v --remove-orphans >/dev/null 2>&1' EXIT +compose down -v --remove-orphans >/dev/null 2>&1 # left from a run with KEEP=1 compose up -d --build --quiet-pull python3 test_e2e.py diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py index a43750dd..fe44746f 100644 --- a/ansible/roles/clickhouse_backup/tests/test_e2e.py +++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py @@ -4,7 +4,7 @@ 2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email 3. a second run takes incrementals, and a restore from them returns the same rows 4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email - 5. a new month's full interrupted by a ClickHouse restart deletes nothing; the next run + 5. a new month's full interrupted by killing ClickHouse deletes nothing; the next run removes what it left, takes the full and prunes the oldest chain Prints each check; exits 1 at the first failure.""" import json @@ -28,9 +28,11 @@ ) -def check(ok, what): +def check(ok, what, output=""): print(f" {'ok ' if ok else 'FAIL'} {what}") if not ok: + if output: + print(" the script said:\n" + output[-3000:]) sys.exit(1) @@ -139,7 +141,7 @@ def setup(): def test_first_run(): print("== 1. first run: a full backup of every table") r = backup("backup", "run") - check(r.returncode == 0, f"the run succeeds{'' if r.returncode == 0 else ': ' + r.stderr[-2000:]}") + check(r.returncode == 0, "the run succeeds", r.stderr) for bucket, table in [(MEASUREMENTS, "fastpath"), (MEASUREMENTS, "obs_web"), (MEASUREMENTS, "fingerprints_dns"), (PRIVATE, "msmt_feedback")]: c = chains(bucket, table) @@ -166,7 +168,7 @@ def test_incremental_and_restore(): print("== 3. second run: incrementals, and a restore") ch("INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers(200000, 1000)") r = backup("backup", "run") - check(r.returncode == 0, "the run succeeds") + check(r.returncode == 0, "the run succeeds", r.stderr) (chain, backups), = chains(MEASUREMENTS, "fastpath").items() check(len(backups) == 2 and all(backups.values()), f"fastpath: a full and an incremental in chain {chain}") latest = f"http://minio:9000/{MEASUREMENTS}/prod/ooni/fastpath/{chain}/{max(backups)}" @@ -180,7 +182,7 @@ def test_incremental_and_restore(): def test_failure_alert(emails_before): print("== 4. a run that fails: ClickHouseBackupFailed") r = backup("broken", "run") - check(r.returncode == 1, "the run exits 1") + check(r.returncode == 1, "the run exits 1", r.stderr) title = f"[FIRING] ClickHouse backup of labeling_frames (broken) failed on {DATA3}" msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0] == title]) check("Nothing was deleted" in msgs[0][1], f"it says what was done: {msgs[0][1]!r}") @@ -202,17 +204,20 @@ def test_interrupted_full(): "--config", "/work/slow.json", "run", "--table", "obs_web"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) wait_for("the new full's first files", lambda: set(chains(MEASUREMENTS, "obs_web")) - {old, previous}, 60) - subprocess.run(COMPOSE + ["restart", "clickhouse"], check=True, capture_output=True) - out, _ = run.communicate(timeout=180) - check(run.returncode == 1, "the interrupted run exits 1") + # killed, not stopped: ClickHouse waits for running backups when it stops + # (shutdown_wait_backups_and_restores), so a clean stop would finish this one + subprocess.run(COMPOSE + ["kill", "clickhouse"], check=True, capture_output=True) + subprocess.run(COMPOSE + ["start", "clickhouse"], check=True, capture_output=True) + out, _ = run.communicate(timeout=900) + check(run.returncode == 1, "the interrupted run exits 1", out) c = chains(MEASUREMENTS, "obs_web") new = sorted(set(c) - {old, previous}) check(new and not any(c[new[0]].values()), f"the new chain {new[0]} has no .backup") check(all(c[x][x] for x in (old, previous)), f"both older chains, {old} and {previous}, are still there") - wait_for("ClickHouse after the restart", lambda: ch("SELECT 1") == "1") + wait_for("ClickHouse after it was killed", lambda: ch("SELECT 1") == "1") r = backup("backup", "run", "--table", "obs_web") - check(r.returncode == 0, "the next run succeeds") + check(r.returncode == 0, "the next run succeeds", r.stderr) check("removed incomplete chain" in r.stderr, "it removed the interrupted full") c = chains(MEASUREMENTS, "obs_web") check(previous in c and old not in c and len(c) == 2 and all(all(b.values()) for b in c.values()), diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index daec61c5..15c888f6 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -66,7 +66,7 @@ def test_old_state_files_still_load(self): class TestRunTable(unittest.TestCase): def test_only_the_jobs_that_back_up_the_tables(self): - jobs = [job("measurements", exclude=["msmt_feedback"]), job("private", include=["msmt_feedback"])] + jobs = [job("measurements", include=["*"], exclude=["msmt_feedback"]), job("private", include=["msmt_feedback"])] selected, orphans = backup.jobs_for_tables(jobs, ["msmt_feedback"]) self.assertEqual([j.name for j in selected], ["private"]) self.assertEqual(orphans, []) @@ -76,6 +76,19 @@ def test_tables_no_job_backs_up(self): self.assertEqual(backup.jobs_for_tables(jobs, ["fastpath"]), ([], ["fastpath"])) +class TestIncludeTables(unittest.TestCase): + def test_star_is_every_table(self): + self.assertTrue(job(include=["*"]).includes("fastpath")) + + def test_an_empty_list_is_no_table(self): + # once data1's list of its own tables is emptied, data1 mustn't back up everything + self.assertFalse(job(include=[]).includes("msmt_feedback")) + + def test_a_config_without_jobs_backs_up_every_table(self): + [conf] = backup.job_confs({"exclude_tables": ["*_tmp"]}, []) + self.assertEqual(conf["include_tables"], ["*"]) + + class TestClickHouseErrors(unittest.TestCase): def setUp(self): d = tempfile.mkdtemp() diff --git a/ansible/roles/clickhouse_backup/tests/vars.yml b/ansible/roles/clickhouse_backup/tests/vars.yml index 0db73f9f..06835714 100644 --- a/ansible/roles/clickhouse_backup/tests/vars.yml +++ b/ansible/roles/clickhouse_backup/tests/vars.yml @@ -10,7 +10,7 @@ clickhouse_backup_s3_secret_access_key: testsecret123 test_jobs: - name: measurements s3_bucket: ooni-clickhouse-backup - include_tables: [] + include_tables: ["*"] exclude_tables: [msmt_feedback, labeling_frames] - name: private s3_bucket: ooni-clickhouse-backup-private From d2a98d89e8742fc9d35ad344a49dcb3076dd2ac0 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 13:59:08 +0000 Subject: [PATCH 22/35] clickhouse_backup: forget the results of tables and jobs no longer backed up The state file kept every table and job it had ever seen, so a table dropped, or moved to another job (as data1's tables will be once replicated), kept its old results and alerted forever: ClickHouseBackupStale after 8 days, or ClickHouseBackupFailed. After a run without --table, a job now forgets the tables it no longer selects, and every run forgets the jobs no longer in the config. --- .../files/ooni_clickhouse_backup.py | 25 +++++++++++++++ .../roles/clickhouse_backup/tests/test_job.py | 32 +++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 0a5cbbb4..1106d232 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -306,6 +306,26 @@ def job_failed(self, job: str) -> None: entry["last_failed"] = 1 self.save() + def forget_tables(self, job: str, tables: set) -> None: + """Drop the job's other tables, e.g. dropped or moved to another job, + so that their old results don't alert forever""" + for key, entry in list(self.state["tables"].items()): + if entry["job"] == job and entry["table"] not in tables: + log.info(f"{job}: {entry['table']}: no longer backed up by this job, forgetting its results") + del self.state["tables"][key] + self.save() + + def forget_jobs(self, jobs: set) -> None: + """Drop the results of jobs that are no longer configured""" + for key, entry in list(self.state["tables"].items()): + if entry["job"] not in jobs: + del self.state["tables"][key] + for part in ("runs", "running", "running_since"): + for job in set(self.state[part]) - jobs: + log.info(f"{job}: no longer configured, forgetting its results") + del self.state[part][job] + self.save() + def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None: self.state["runs"][job] = { "finished": time.time(), "duration": round(duration_s, 1), @@ -601,6 +621,8 @@ def run(self, only: list, force_full: bool, keep: str) -> int: finally: self.metrics.job_running(self.name, False) self.metrics.run_result(self.name, time.monotonic() - t0, done, fulls, len(failed)) + if not only: + self.metrics.forget_tables(self.name, {table for table, _ in tables}) if failed: log.error(f"{self.name}: failed tables: {', '.join(failed)}") @@ -698,6 +720,7 @@ def main() -> int: conf = json.load(f) metrics = Metrics(conf.get("metrics_file", ""), conf.get("state_file", "")) jobs = [BackupJob(j, metrics) for j in job_confs(conf, args.job)] + configured = {j["name"] for j in job_confs(conf, [])} if args.command == "list": for job in jobs: @@ -727,6 +750,8 @@ def main() -> int: rc |= job.run(args.table, args.full, args.keep) else: rc |= job.release(args.label) + if args.command == "run": + metrics.forget_jobs(configured) return rc diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index 15c888f6..874aa1fe 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -89,6 +89,38 @@ def test_a_config_without_jobs_backs_up_every_table(self): self.assertEqual(conf["include_tables"], ["*"]) +class TestForget(unittest.TestCase): + def setUp(self): + d = tempfile.mkdtemp() + self.metrics = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json")) + for job_, table in [("data1", "labeling_frames"), ("data1", "changepoint_label"), ("private", "msmt_feedback")]: + self.metrics.table_result(job_, table, True, {"total_size": 1, "num_files": 1}) + self.metrics.job_running("old_job", False) + + def tables(self): + return sorted(self.metrics.state["tables"]) + + def test_a_table_the_job_no_longer_selects(self): + # labeling_frames was replicated, and moved from data1's job to data3's + self.metrics.forget_tables("data1", {"changepoint_label"}) + self.assertEqual(self.tables(), ["data1/changepoint_label", "private/msmt_feedback"]) + + def test_a_job_no_longer_configured(self): + self.metrics.forget_jobs({"data1"}) + self.assertEqual(self.tables(), ["data1/changepoint_label", "data1/labeling_frames"]) + self.assertNotIn("old_job", self.metrics.state["running"]) + + def test_a_run_with_table_forgets_nothing(self): + j = job("data1", include=["*"], metrics=self.metrics) + j.running_backups = lambda: [] + j.tables = lambda only: ([("changepoint_label", "MergeTree")], []) + j.backup_table = lambda *a: False + j.run(["changepoint_label"], False, "") + self.assertIn("data1/labeling_frames", self.tables()) + j.run([], False, "") + self.assertNotIn("data1/labeling_frames", self.tables()) + + class TestClickHouseErrors(unittest.TestCase): def setUp(self): d = tempfile.mkdtemp() From 862b236d0a93958033ef46001bd09cba8e4561c0 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 13:59:09 +0000 Subject: [PATCH 23/35] clickhouse_backup: match only this bucket's running backups, check deletes, fsync state - running_backups() matched the endpoint by substring, so the measurements bucket also matched the private bucket's backups; match //. - delete_objects reports keys it didn't delete in a 200 response, which were ignored; raise on them. - Fsync the state and metrics files before they replace the old ones, and start from no results if the state file is unreadable instead of failing every command. - Pass the upload settings to the EmbeddedRocksDB/Join export too: R2 needs its multipart parts to be the same size. --- .../files/ooni_clickhouse_backup.py | 22 ++++++++++--- .../roles/clickhouse_backup/tests/test_job.py | 31 +++++++++++++++++++ 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 1106d232..7f11ea7c 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -206,7 +206,12 @@ def delete_prefix(self, prefix: str) -> int: for page in pager.paginate(Bucket=self.bucket, Prefix=prefix): keys = [{"Key": o["Key"]} for o in page.get("Contents", [])] if keys: - self.s3.delete_objects(Bucket=self.bucket, Delete={"Objects": keys, "Quiet": True}) + resp = self.s3.delete_objects(Bucket=self.bucket, Delete={"Objects": keys, "Quiet": True}) + # a 200 response can still list keys it didn't delete + errors = resp.get("Errors", []) + if errors: + e = errors[0] + raise RuntimeError(f"{len(errors)} objects not deleted under {prefix}, e.g. {e.get('Key')}: {e.get('Code')} {e.get('Message')}") n += len(keys) return n @@ -233,6 +238,8 @@ def write_atomically(path: str, text: str) -> None: tmp = f"{path}.{os.getpid()}.tmp" with open(tmp, "w") as f: f.write(text) + f.flush() + os.fsync(f.fileno()) # or a power loss can leave an empty file behind os.chmod(tmp, 0o644) os.replace(tmp, path) @@ -276,8 +283,12 @@ def __init__(self, metrics_file: str = "", state_file: str = ""): self.state_file = state_file self.state = {"tables": {}, "running": {}} if state_file and os.path.exists(state_file): - with open(state_file) as f: - self.state = json.load(f) + try: + with open(state_file) as f: + self.state = json.load(f) + except ValueError as e: + # the results are rebuilt as tables are backed up again + log.warning(f"{state_file} is unreadable ({e}), starting from no results") self.state.setdefault("runs", {}) self.state.setdefault("running_since", {}) @@ -463,7 +474,9 @@ def running_backups(self) -> list: rows = self.ch.rows( "SELECT id, name FROM system.backups WHERE status = 'CREATING_BACKUP'" ) - return [r for r in rows if self.store.ch_base in r["name"]] + # this job's bucket and prefix only: one bucket's name can begin another's + ours = f"{self.store.ch_base}/{self.store.prefix}/" + return [r for r in rows if ours in r["name"]] def wait_for_clickhouse(self) -> None: """At boot a missed run starts before ClickHouse accepts queries""" @@ -564,6 +577,7 @@ def backup_table(self, table: str, engine: str, force_full: bool, keep: str) -> INSERT INTO FUNCTION s3({self.store.ch_s3(key + "/data.native.zst")}, 'Native') SELECT * FROM `{self.db}`.`{table}` """, + settings=self.settings, # R2 needs equal upload parts here too timeout=3600, ) diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index 874aa1fe..942a4607 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -4,6 +4,7 @@ import json import os import tempfile +import types import unittest import urllib.error from unittest import mock @@ -121,6 +122,36 @@ def test_a_run_with_table_forgets_nothing(self): self.assertNotIn("data1/labeling_frames", self.tables()) +class TestStoreAndState(unittest.TestCase): + def test_only_this_buckets_running_backups(self): + j = job() + j.store = types.SimpleNamespace(ch_base="https://r2.example/ooni-clickhouse-backup", prefix="prod") + j.ch = types.SimpleNamespace(rows=lambda sql: [ + {"id": "1", "name": "S3('https://r2.example/ooni-clickhouse-backup-private/prod/ooni/msmt_feedback/a/b')"}, + {"id": "2", "name": "S3('https://r2.example/ooni-clickhouse-backup/prod/ooni/fastpath/a/b')"}, + ]) + self.assertEqual([r["id"] for r in j.running_backups()], ["2"]) + + def test_objects_not_deleted_are_an_error(self): + store = backup.Store.__new__(backup.Store) + store.bucket, store.prefix = "b", "prod" + store.s3 = types.SimpleNamespace( + get_paginator=lambda name: types.SimpleNamespace(paginate=lambda **kw: [{"Contents": [{"Key": "prod/ooni/t/c/x"}]}]), + delete_objects=lambda **kw: {"Errors": [{"Key": "prod/ooni/t/c/x", "Code": "AccessDenied", "Message": "no"}]}, + ) + with self.assertRaisesRegex(RuntimeError, "1 objects not deleted"): + store.delete_prefix("prod/ooni/t/c/") + + def test_an_unreadable_state_file_starts_empty(self): + d = tempfile.mkdtemp() + state = os.path.join(d, "state.json") + with open(state, "w") as f: + f.write("") # e.g. truncated by a power loss + m = backup.Metrics(os.path.join(d, "out.prom"), state) + m.job_running("measurements", True) + self.assertEqual(m.state["running"], {"measurements": 1}) + + class TestClickHouseErrors(unittest.TestCase): def setUp(self): d = tempfile.mkdtemp() From c29e2b5124e45942402120a26fa7dfd523e361b2 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 13:59:09 +0000 Subject: [PATCH 24/35] clickhouse_backup tests: interrupt the full deterministically, show output and logs on failure ClickHouse waits for running backups when it stops (shutdown_wait_backups_and_restores), so the interrupted-full scenario only worked because compose restart killed it after 10 s; kill it instead. Failing run checks now show what the script said, run.sh prints the containers' logs when the test fails and removes containers left by KEEP=1 before starting, and the unit tests silence the script's log. --- ansible/roles/clickhouse_backup/tests/test_job.py | 1 + 1 file changed, 1 insertion(+) diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index 942a4607..1eeacbd1 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -14,6 +14,7 @@ ) backup = importlib.util.module_from_spec(spec) spec.loader.exec_module(backup) +backup.log.disabled = True # the tests check results, not log lines class FlakyClickHouse: From d5e76116c4707605b1432849251e256dbf0380a1 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 14:36:40 +0000 Subject: [PATCH 25/35] clickhouse_backup tests: restore the full and each incremental, the RocksDB rows, and the new full after pruning Only the newest incremental was restored. Two more runs now give fastpath a full and two incrementals, and each of the three restores fastpath as it was at that backup (200,000, 201,000 and 202,000 rows, by count and a checksum of every row). fingerprints_dns, an EmbeddedRocksDB table, is restored as the runbook says: RESTORE its schema, then load data.native.zst. After the interrupted full and the pruning, the new full restores obs_web. The run takes about 9 minutes. --- .../roles/clickhouse_backup/tests/README.md | 10 ++-- .../roles/clickhouse_backup/tests/test_e2e.py | 50 ++++++++++++++----- 2 files changed, 44 insertions(+), 16 deletions(-) diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md index 4d548799..77eaac1f 100644 --- a/ansible/roles/clickhouse_backup/tests/README.md +++ b/ansible/roles/clickhouse_backup/tests/README.md @@ -1,6 +1,6 @@ # Tests for the clickhouse_backup role -`./run.sh` runs them all, in about seven minutes: +`./run.sh` runs them all, in about nine minutes: - `alert_rules_test.yml`: promtool unit tests of the backup alerts in `roles/prometheus/files/alert_rules.yml`, then amtool checks that @@ -13,9 +13,11 @@ ClickHouse at production's version, MinIO standing in for R2, the script on Debian bookworm with its python3-boto3, as on the hosts, node_exporter, Prometheus with the repo's alert rules, Alertmanager with the repo's - routes, and a stand-in for Slack. Full and incremental backups, a - restore, a failed run, an interrupted full, the month's new full and - pruning, and the Slack messages and emails each one causes. + routes, and a stand-in for Slack. Full and incremental backups, and + restoring the full and each incremental of a chain, and an + EmbeddedRocksDB table with its exported rows; a failed run; an + interrupted full, the month's new full, pruning and restoring what's + left; and the Slack messages and emails each one causes. The configs are rendered from the role's templates into `work/` by `render.yml`, with the variables in `vars.yml`. diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py index fe44746f..f4c59636 100644 --- a/ansible/roles/clickhouse_backup/tests/test_e2e.py +++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py @@ -2,10 +2,11 @@ In order: 1. a first run takes a full backup of every table, each job to its own bucket 2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email - 3. a second run takes incrementals, and a restore from them returns the same rows + 3. two more runs take incrementals; restoring the full and each incremental returns the rows of + its moment, and an EmbeddedRocksDB table comes back with its exported rows 4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email 5. a new month's full interrupted by killing ClickHouse deletes nothing; the next run - removes what it left, takes the full and prunes the oldest chain + removes what it left, takes the full and prunes the oldest chain, and the full restores Prints each check; exits 1 at the first failure.""" import json import os @@ -164,19 +165,41 @@ def test_completed_alert(emails_before): check(email_attempts() == emails_before, "no email sent") +def digest(table): + """row count and a checksum of every row""" + return ch(f"SELECT count(), sum(cityHash64(*)) FROM {table}") + + +def backup_url(bucket, table, chain, name): + return f"http://minio:9000/{bucket}/prod/ooni/{table}/{chain}/{name}" + + def test_incremental_and_restore(): - print("== 3. second run: incrementals, and a restore") - ch("INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers(200000, 1000)") - r = backup("backup", "run") - check(r.returncode == 0, "the run succeeds", r.stderr) + print("== 3. incrementals, and restoring each backup of a chain") + # fastpath as at the first run's full, then after each of two more runs + states = [digest("ooni.fastpath")] + for first in (200000, 201000): + ch(f"INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers({first}, 1000)") + r = backup("backup", "run") + check(r.returncode == 0, "the run succeeds", r.stderr) + states.append(digest("ooni.fastpath")) (chain, backups), = chains(MEASUREMENTS, "fastpath").items() - check(len(backups) == 2 and all(backups.values()), f"fastpath: a full and an incremental in chain {chain}") - latest = f"http://minio:9000/{MEASUREMENTS}/prod/ooni/fastpath/{chain}/{max(backups)}" + names = sorted(backups) + check(len(names) == 3 and all(backups.values()), f"fastpath: a full and two incrementals in chain {chain}") ch("CREATE DATABASE restored") - ch(f"RESTORE TABLE ooni.fastpath AS restored.fastpath FROM S3('{latest}')") - digest = "SELECT count(), sum(cityHash64(measurement_uid, input, anomaly)) FROM {}" - check(ch(digest.format("restored.fastpath")) == ch(digest.format("ooni.fastpath")), - "the restored fastpath has the same rows (201000)") + for i, (name, state) in enumerate(zip(names, states)): + what = "the full" if i == 0 else f"incremental {i}" + ch(f"RESTORE TABLE ooni.fastpath AS restored.fastpath_{i} FROM S3('{backup_url(MEASUREMENTS, 'fastpath', chain, name)}')") + check(digest(f"restored.fastpath_{i}") == state, f"{what} restores fastpath as it was then ({state.split()[0]} rows)") + + # EmbeddedRocksDB: BACKUP keeps only the schema, the rows are in data.native.zst (the runbook's steps 2 and 3) + c = chains(MEASUREMENTS, "fingerprints_dns") + chain = max(ch_ for ch_, b in c.items() if any(b.values())) + url = backup_url(MEASUREMENTS, "fingerprints_dns", chain, max(n for n, ok in c[chain].items() if ok)) + ch(f"RESTORE TABLE ooni.fingerprints_dns AS restored.fingerprints_dns FROM S3('{url}')") + ch(f"INSERT INTO restored.fingerprints_dns SELECT * FROM s3('{url}/data.native.zst', 'Native')") + check(digest("restored.fingerprints_dns") == digest("ooni.fingerprints_dns"), + "fingerprints_dns (EmbeddedRocksDB) restores with its exported rows") def test_failure_alert(emails_before): @@ -222,6 +245,9 @@ def test_interrupted_full(): c = chains(MEASUREMENTS, "obs_web") check(previous in c and old not in c and len(c) == 2 and all(all(b.values()) for b in c.values()), f"it took the full and pruned {old}: chains {sorted(c)}") + new = max(c) + ch(f"RESTORE TABLE ooni.obs_web AS restored.obs_web FROM S3('{backup_url(MEASUREMENTS, 'obs_web', new, new)}')") + check(digest("restored.obs_web") == digest("ooni.obs_web"), f"the new full {new} restores obs_web") def main(): From 18cd22621b10e3bc56ce24bb3ded05b78d7aa1b1 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 15:08:47 +0000 Subject: [PATCH 26/35] clickhouse_backup: report the tables on the cluster that no backup covers A table created on data1 or data2 only, like data1's own tables were, was in no backup and nothing said so: data3 doesn't have it, and data1's job lists its tables. After each run, data3 now lists the tables on every node (clickhouse_backup_check_cluster: oonidata_cluster) and reports those that no job here backs up (from a table on this host), that aren't in clickhouse_backup_data1_tables (backed up by data1) and that aren't excluded, as ooni_clickhouse_backup_table_not_backed_up. ClickHouseTableNotBackedUp names the table and its nodes and says how to fix it. clickhouse_backup_exclude_tables becomes the list of tables that are never backed up, each with its reason: fingerprints_dns and fingerprints_http (the hourly updater rebuilds them) and session_expunge (empty, unused) join it. That changes no backup: they're not on data3, and data1's job lists its tables. The end-to-end test's EmbeddedRocksDB table is now test_groups, as in production. --- ansible/host_vars/data3.htz-fsn.prod.ooni.nu | 5 ++ .../roles/clickhouse_backup/defaults/main.yml | 12 ++++ .../files/ooni_clickhouse_backup.py | 55 +++++++++++++++++++ .../templates/clickhouse-backup.json.j2 | 2 + .../tests/alert_rules_test.yml | 13 +++++ .../roles/clickhouse_backup/tests/test_e2e.py | 22 ++++---- .../roles/clickhouse_backup/tests/test_job.py | 26 +++++++++ .../roles/prometheus/files/alert_rules.yml | 8 +++ docs/Runbooks.md | 13 ++++- 9 files changed, 144 insertions(+), 12 deletions(-) diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu index 0e4eef14..34959203 100644 --- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu +++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu @@ -13,3 +13,8 @@ clickhouse_backup_jobs: s3_bucket: ooni-clickhouse-backup-private include_tables: "{{ clickhouse_backup_private_tables }}" exclude_tables: [] + +# After each run, report the tables on data1-3 that no backup covers; +# data1's own tables are backed up by data1 +clickhouse_backup_check_cluster: oonidata_cluster +clickhouse_backup_backed_up_elsewhere: "{{ clickhouse_backup_data1_tables }}" diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index 5c3637ee..6ae23389 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -56,9 +56,21 @@ clickhouse_backup_max_bandwidth: 50000000 # fnmatch patterns, for every job. Materialized views' .inner tables are # always skipped: they're backed up with their view. +# Tables that are never backed up, for every job, and why clickhouse_backup_exclude_tables: - "*_tmp" # updater staging tables - msmt_feedback_old + - fingerprints_dns # the hourly fingerprints updater rebuilds them + - fingerprints_http + - session_expunge # empty, no longer used (backend #972) + +# The cluster whose tables are checked after each run: any table on any of its +# nodes that no job here backs up, that isn't excluded above and that isn't in +# clickhouse_backup_backed_up_elsewhere is reported (ClickHouseTableNotBackedUp). +# Set on one host, the one taking most backups; empty checks nothing. +clickhouse_backup_check_cluster: "" +# fnmatch patterns of the tables another host backs up +clickhouse_backup_backed_up_elsewhere: [] # Results for Prometheus, read by node_exporter's textfile collector # (node_exporter_textfile_dir in the prometheus_node_exporter role) diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index 7f11ea7c..e743fb5c 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -291,6 +291,7 @@ def __init__(self, metrics_file: str = "", state_file: str = ""): log.warning(f"{state_file} is unreadable ({e}), starting from no results") self.state.setdefault("runs", {}) self.state.setdefault("running_since", {}) + self.state.setdefault("not_backed_up", {}) def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration_s: float = 0) -> None: entry = self.state["tables"].setdefault(f"{job}/{table}", {"job": job, "table": table}) @@ -337,6 +338,11 @@ def forget_jobs(self, jobs: set) -> None: del self.state[part][job] self.save() + def not_backed_up(self, tables: dict) -> None: + """{table: [hosts]} of the cluster's tables that no backup covers""" + self.state["not_backed_up"] = tables + self.save() + def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None: self.state["runs"][job] = { "finished": time.time(), "duration": round(duration_s, 1), @@ -362,6 +368,13 @@ def render(self) -> str: lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"] for job, run in sorted(self.state["runs"].items()): lines.append(f'{p}{name}{{backup="{label_value(job)}"}} {run[field]}') + lines += [ + f"# HELP {p}table_not_backed_up 1 for a table on the cluster that no backup job covers or excludes", + f"# TYPE {p}table_not_backed_up gauge", + ] + for table, hosts in sorted(self.state["not_backed_up"].items()): + labels = f'table="{label_value(table)}",hosts="{label_value(",".join(sorted(hosts)))}"' + lines.append(f"{p}table_not_backed_up{{{labels}}} 1") for name, field, help_ in self.GAUGES: lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"] for entry in sorted(self.state["tables"].values(), key=lambda e: (e["job"], e["table"])): @@ -680,6 +693,47 @@ def release(self, label: str) -> int: return 0 +def not_backed_up(tables: dict, here: str, jobs: list, excluded: list, elsewhere: list) -> dict: + """The tables of {table: [hosts]} that no job here backs up and that + aren't excluded (never backed up) or backed up elsewhere (by another + host). A job only backs up the tables that exist on this host.""" + match = lambda t, patterns: any(fnmatch.fnmatchcase(t, p) for p in patterns) + return { + t: hosts for t, hosts in tables.items() + if not (here in hosts and any(j.includes(t) and not j.excludes(t) for j in jobs)) + and not match(t, excluded) and not match(t, elsewhere) + } + + +def check_cluster(conf: dict, metrics: "Metrics") -> None: + """Report the tables on any node of the cluster that no backup covers, + e.g. a new table created on one node only""" + cluster = conf.get("check_cluster", "") + if not cluster: + return + jobs = [BackupJob(j, metrics) for j in job_confs(conf, [])] + try: + here = jobs[0].ch.query("SELECT hostName()").strip() + rows = jobs[0].ch.rows( + f""" + SELECT name, groupUniqArray(hostName()) AS hosts + FROM clusterAllReplicas({sql_str(cluster)}, system.tables) + WHERE database = {sql_str(jobs[0].db)} AND NOT is_temporary AND NOT startsWith(name, '.inner') + GROUP BY name + """ + ) + except Exception as e: + log.error(f"could not list the tables of cluster {cluster}: {e}") + return + missing = not_backed_up( + {r["name"]: r["hosts"] for r in rows}, here, jobs, + conf.get("exclude_tables", []), conf.get("backed_up_elsewhere", []), + ) + for table, hosts in sorted(missing.items()): + log.error(f"{table} (on {', '.join(sorted(hosts))}) is in no backup job and not excluded") + metrics.not_backed_up(missing) + + def jobs_for_tables(jobs: list, tables: list) -> tuple: """(the jobs that back up one of tables, the tables no job backs up)""" owns = lambda j, t: j.includes(t) and not j.excludes(t) @@ -766,6 +820,7 @@ def main() -> int: rc |= job.release(args.label) if args.command == "run": metrics.forget_jobs(configured) + check_cluster(conf, metrics) return rc diff --git a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 index bc591a0e..01c7b3ca 100644 --- a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 +++ b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 @@ -12,6 +12,8 @@ "keep_chains": clickhouse_backup_keep_chains, "max_backup_bandwidth": clickhouse_backup_max_bandwidth, "exclude_tables": clickhouse_backup_exclude_tables, + "check_cluster": clickhouse_backup_check_cluster, + "backed_up_elsewhere": clickhouse_backup_backed_up_elsewhere, "metrics_file": clickhouse_backup_metrics_file, "state_file": clickhouse_backup_state_file, "jobs": clickhouse_backup_jobs, diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml index 3ff312dd..72d1759a 100644 --- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml +++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml @@ -115,3 +115,16 @@ tests: exp_annotations: summary: 'Stale `textfile` at data3' description: '`/var/lib/node_exporter/textfile_collector/other.prom` is 3h old.' + + # TableNotBackedUp: a table no backup job covers + - interval: 1m + input_series: + - {series: 'ooni_clickhouse_backup_table_not_backed_up{instance="data3", table="made_on_data1", hosts="data1.htz-fsn.prod.ooni.nu"}', values: '1x30'} + alert_rule_test: + - eval_time: 5m + alertname: ClickHouseTableNotBackedUp + exp_alerts: + - exp_labels: {instance: data3, table: made_on_data1, hosts: data1.htz-fsn.prod.ooni.nu} + exp_annotations: + summary: 'ClickHouse table made_on_data1 is not backed up (on data1.htz-fsn.prod.ooni.nu)' + description: 'No backup job covers it and it is not excluded. Add it to a job (clickhouse_backup_data1_tables if only data1 holds its data, clickhouse_backup_private_tables if private), or to clickhouse_backup_exclude_tables if it needs no backup. See the runbook, "ClickHouse backup".' diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py index f4c59636..f2dd11b5 100644 --- a/ansible/roles/clickhouse_backup/tests/test_e2e.py +++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py @@ -128,8 +128,8 @@ def setup(): # ~20 MB that doesn't compress: takes ~20 s to back up with work/slow.json "CREATE TABLE ooni.obs_web (measurement_uid String, blob String) ENGINE = MergeTree ORDER BY measurement_uid", "INSERT INTO ooni.obs_web SELECT toString(number), randomString(4096) FROM numbers(5000)", - "CREATE TABLE ooni.fingerprints_dns (name String, pattern String) ENGINE = EmbeddedRocksDB PRIMARY KEY name", - "INSERT INTO ooni.fingerprints_dns VALUES ('a', '1.2.3.4'), ('b', '5.6.7.8')", + "CREATE TABLE ooni.test_groups (name String, pattern String) ENGINE = EmbeddedRocksDB PRIMARY KEY name", + "INSERT INTO ooni.test_groups VALUES ('a', '1.2.3.4'), ('b', '5.6.7.8')", "CREATE TABLE ooni.msmt_feedback (measurement_uid String, comment String) ENGINE = MergeTree ORDER BY measurement_uid", "INSERT INTO ooni.msmt_feedback VALUES ('m1', 'blocked?')", "CREATE TABLE ooni.labeling_frames (measurement_uid String) ENGINE = MergeTree ORDER BY measurement_uid", @@ -144,11 +144,11 @@ def test_first_run(): r = backup("backup", "run") check(r.returncode == 0, "the run succeeds", r.stderr) for bucket, table in [(MEASUREMENTS, "fastpath"), (MEASUREMENTS, "obs_web"), - (MEASUREMENTS, "fingerprints_dns"), (PRIVATE, "msmt_feedback")]: + (MEASUREMENTS, "test_groups"), (PRIVATE, "msmt_feedback")]: c = chains(bucket, table) check(len(c) == 1 and all(all(b.values()) for b in c.values()), f"{table}: one complete chain in {bucket}") - check(any(k.endswith("/data.native.zst") for k in keys(MEASUREMENTS, "prod/ooni/fingerprints_dns/")), - "fingerprints_dns (EmbeddedRocksDB): its rows are exported too") + check(any(k.endswith("/data.native.zst") for k in keys(MEASUREMENTS, "prod/ooni/test_groups/")), + "test_groups (EmbeddedRocksDB): its rows are exported too") check(not keys(MEASUREMENTS, "prod/ooni/msmt_feedback/"), "msmt_feedback isn't in the measurements bucket") check(not keys(MEASUREMENTS, "prod/ooni/asnmeta_tmp/"), "asnmeta_tmp (excluded) isn't backed up") check(not keys(MEASUREMENTS, "prod/ooni/labeling_frames/"), "labeling_frames (data1's) isn't backed up") @@ -193,13 +193,13 @@ def test_incremental_and_restore(): check(digest(f"restored.fastpath_{i}") == state, f"{what} restores fastpath as it was then ({state.split()[0]} rows)") # EmbeddedRocksDB: BACKUP keeps only the schema, the rows are in data.native.zst (the runbook's steps 2 and 3) - c = chains(MEASUREMENTS, "fingerprints_dns") + c = chains(MEASUREMENTS, "test_groups") chain = max(ch_ for ch_, b in c.items() if any(b.values())) - url = backup_url(MEASUREMENTS, "fingerprints_dns", chain, max(n for n, ok in c[chain].items() if ok)) - ch(f"RESTORE TABLE ooni.fingerprints_dns AS restored.fingerprints_dns FROM S3('{url}')") - ch(f"INSERT INTO restored.fingerprints_dns SELECT * FROM s3('{url}/data.native.zst', 'Native')") - check(digest("restored.fingerprints_dns") == digest("ooni.fingerprints_dns"), - "fingerprints_dns (EmbeddedRocksDB) restores with its exported rows") + url = backup_url(MEASUREMENTS, "test_groups", chain, max(n for n, ok in c[chain].items() if ok)) + ch(f"RESTORE TABLE ooni.test_groups AS restored.test_groups FROM S3('{url}')") + ch(f"INSERT INTO restored.test_groups SELECT * FROM s3('{url}/data.native.zst', 'Native')") + check(digest("restored.test_groups") == digest("ooni.test_groups"), + "test_groups (EmbeddedRocksDB) restores with its exported rows") def test_failure_alert(emails_before): diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py index 1eeacbd1..51417fc1 100644 --- a/ansible/roles/clickhouse_backup/tests/test_job.py +++ b/ansible/roles/clickhouse_backup/tests/test_job.py @@ -153,6 +153,32 @@ def test_an_unreadable_state_file_starts_empty(self): self.assertEqual(m.state["running"], {"measurements": 1}) +class TestNotBackedUp(unittest.TestCase): + def test_tables_no_job_covers(self): + jobs = [job("measurements", include=["*"], exclude=["msmt_feedback", "labeling_frames", "*_tmp"]), + job("private", include=["msmt_feedback"])] + tables = { + "fastpath": ["data1", "data2", "data3"], # replicated: measurements + "msmt_feedback": ["data1", "data2", "data3"], # private + "labeling_frames": ["data1", "data2", "data3"], # data1 backs it up + "asnmeta_tmp": ["data1"], # excluded + "fingerprints_dns": ["data1", "data2"], # excluded: rebuilt hourly + "made_on_data1": ["data1"], # nothing: data3 doesn't have it + } + missing = backup.not_backed_up(tables, "data3", jobs, ["*_tmp", "fingerprints_*"], ["labeling_frames"]) + self.assertEqual(missing, {"made_on_data1": ["data1"]}) + + def test_the_metric_names_the_table_and_its_hosts(self): + d = tempfile.mkdtemp() + m = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json")) + m.not_backed_up({"made_on_data1": ["data2", "data1"]}) + with open(m.metrics_file) as f: + self.assertIn('ooni_clickhouse_backup_table_not_backed_up{table="made_on_data1",hosts="data1,data2"} 1', f.read()) + m.not_backed_up({}) + with open(m.metrics_file) as f: + self.assertNotIn("made_on_data1", f.read()) + + class TestClickHouseErrors(unittest.TestCase): def setUp(self): d = tempfile.mkdtemp() diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml index 1149f04f..a2f33105 100644 --- a/ansible/roles/prometheus/files/alert_rules.yml +++ b/ansible/roles/prometheus/files/alert_rules.yml @@ -156,6 +156,14 @@ groups: summary: 'ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) failed on {{ $labels.instance }}' description: 'Nothing was deleted: the older backups of {{ $labels.table }} are kept. The next run retries it. See `journalctl --identifier ooni-clickhouse-backup` on {{ $labels.instance }}.' + # A table on the cluster that no backup job covers, and that isn't excluded: + # e.g. a new table created on one node only (data3 checks after each run) + - alert: ClickHouseTableNotBackedUp + expr: ooni_clickhouse_backup_table_not_backed_up == 1 + annotations: + summary: 'ClickHouse table {{ $labels.table }} is not backed up (on {{ $labels.hosts }})' + description: 'No backup job covers it and it is not excluded. Add it to a job (clickhouse_backup_data1_tables if only data1 holds its data, clickhouse_backup_private_tables if private), or to clickhouse_backup_exclude_tables if it needs no backup. See the runbook, "ClickHouse backup".' + # A job's run finished with every table backed up. One Slack message per # job run, no email and no "resolved" message: Alertmanager routes it to # slack-notice. It clears an hour after the run. diff --git a/docs/Runbooks.md b/docs/Runbooks.md index d2735a9a..d4e88919 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -205,7 +205,12 @@ When a data1-only table becomes replicated, remove it from backup. - Not backed up: `fingerprints_dns` and `fingerprints_http`, which are only on data1 and data2 and which the hourly updater rebuilds, and `session_expunge`, - which is empty and no longer used. + which is empty and no longer used. They're in + `clickhouse_backup_exclude_tables`, with the other tables that are never + backed up. +- After each run, data3 lists the tables on every node of the cluster + (`clickhouse_backup_check_cluster`) and reports any that no job backs up and + that aren't excluded, e.g. a new table created on data1 only. - R2 rejects multipart uploads whose parts (all but the last) differ in size, so `clickhouse_backup_s3_settings` fixes ClickHouse's part size at 128 MiB (`s3_upload_part_size_multiply_factor: 1`). R2 allows 10,000 parts, @@ -264,6 +269,12 @@ Alerts, in `alert_rules.yml`: - `ClickHouseBackupFailed`: a table's last attempt failed. Nothing is deleted, and the next run retries it. +- `ClickHouseTableNotBackedUp`: a table on the cluster is in no backup job and + not excluded. Add it to a job: replicated tables are backed up by data3's + `measurements` job unless they're private (`clickhouse_backup_private_tables`), + and a table whose data only data1 holds goes in + `clickhouse_backup_data1_tables`. Or, if it needs no backup, add it to + `clickhouse_backup_exclude_tables`. - `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack message per run in `#ooni-bots`, no email. - `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups From 70ba6e706470ba5a2b7baa54d7e9e4f1dd6d363c Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Thu, 22 Oct 2026 15:08:47 +0000 Subject: [PATCH 27/35] clickhouse_backup tests: a second ClickHouse node, and a table on it that no backup covers The test environment gets a second ClickHouse node, data1, with data3 in one test cluster. A new step creates a table on data1 only and checks that ClickHouseTableNotBackedUp names it in Slack, and no other table (labeling_frames is data1's, asnmeta_tmp is excluded). It comes after the Completed step, whose check that nothing went to email it would otherwise break. --- .../roles/clickhouse_backup/tests/README.md | 6 ++-- .../clickhouse_backup/tests/clickhouse.xml | 9 ++++++ .../roles/clickhouse_backup/tests/compose.yml | 16 ++++++++++ .../roles/clickhouse_backup/tests/render.yml | 1 + .../roles/clickhouse_backup/tests/test_e2e.py | 29 +++++++++++++++---- .../roles/clickhouse_backup/tests/vars.yml | 6 +++- 6 files changed, 58 insertions(+), 9 deletions(-) diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md index 77eaac1f..cf2e9b50 100644 --- a/ansible/roles/clickhouse_backup/tests/README.md +++ b/ansible/roles/clickhouse_backup/tests/README.md @@ -10,12 +10,14 @@ bucket stubbed. - `test_job.py`: how a job runs and what it reports, ClickHouse stubbed. - `test_e2e.py`: the real thing, in the containers of `compose.yml`: - ClickHouse at production's version, MinIO standing in for R2, the script + two ClickHouse nodes at production's version (`data3`, where the backups + run, and `data1`, in one test cluster), MinIO standing in for R2, the script on Debian bookworm with its python3-boto3, as on the hosts, node_exporter, Prometheus with the repo's alert rules, Alertmanager with the repo's routes, and a stand-in for Slack. Full and incremental backups, and restoring the full and each incremental of a chain, and an - EmbeddedRocksDB table with its exported rows; a failed run; an + EmbeddedRocksDB table with its exported rows; a table created on `data1` + only, which ClickHouseTableNotBackedUp must name; a failed run; an interrupted full, the month's new full, pruning and restoring what's left; and the Slack messages and emails each one causes. diff --git a/ansible/roles/clickhouse_backup/tests/clickhouse.xml b/ansible/roles/clickhouse_backup/tests/clickhouse.xml index 918706bd..4410ded5 100644 --- a/ansible/roles/clickhouse_backup/tests/clickhouse.xml +++ b/ansible/roles/clickhouse_backup/tests/clickhouse.xml @@ -4,4 +4,13 @@ 0.5 + + + + + clickhouse9000adminadmin + clickhouse29000adminadmin + + + diff --git a/ansible/roles/clickhouse_backup/tests/compose.yml b/ansible/roles/clickhouse_backup/tests/compose.yml index e90a9881..0f8eea78 100644 --- a/ansible/roles/clickhouse_backup/tests/compose.yml +++ b/ansible/roles/clickhouse_backup/tests/compose.yml @@ -12,8 +12,10 @@ services: MINIO_ROOT_PASSWORD: testsecret123 ports: ["127.0.0.1:19910:9000"] + # data3, the node the backups run against clickhouse: image: docker.io/clickhouse/clickhouse-server:24.8.6.70 + hostname: data3 environment: CLICKHOUSE_USER: admin CLICKHOUSE_PASSWORD: admin @@ -25,6 +27,20 @@ services: mem_limit: 2g ports: ["127.0.0.1:19911:8123"] + # another node of the cluster, where a table nobody backs up gets created + clickhouse2: + image: docker.io/clickhouse/clickhouse-server:24.8.6.70 + hostname: data1 + environment: + CLICKHOUSE_USER: admin + CLICKHOUSE_PASSWORD: admin + volumes: + - ./clickhouse.xml:/etc/clickhouse-server/config.d/test.xml:ro,z + ulimits: + nofile: 262144 + mem_limit: 1g + ports: ["127.0.0.1:19912:8123"] + runner: build: {context: ., dockerfile: runner.Dockerfile} command: sleep infinity diff --git a/ansible/roles/clickhouse_backup/tests/render.yml b/ansible/roles/clickhouse_backup/tests/render.yml index a42ae1ef..62c84e3a 100644 --- a/ansible/roles/clickhouse_backup/tests/render.yml +++ b/ansible/roles/clickhouse_backup/tests/render.yml @@ -14,6 +14,7 @@ vars: clickhouse_backup_jobs: "{{ item.jobs }}" clickhouse_backup_max_bandwidth: "{{ item.bandwidth }}" + clickhouse_backup_check_cluster: "{{ item.check_cluster | default('') }}" loop: "{{ test_configs }}" - name: Render ClickHouse's bucket credentials, for every bucket of every config diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py index f2dd11b5..eb70e3ec 100644 --- a/ansible/roles/clickhouse_backup/tests/test_e2e.py +++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py @@ -4,8 +4,9 @@ 2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email 3. two more runs take incrementals; restoring the full and each incremental returns the rows of its moment, and an EmbeddedRocksDB table comes back with its exported rows - 4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email - 5. a new month's full interrupted by killing ClickHouse deletes nothing; the next run + 4. a table created on the cluster's other node only: ClickHouseTableNotBackedUp names it + 5. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email + 6. a new month's full interrupted by killing ClickHouse deletes nothing; the next run removes what it left, takes the full and prunes the oldest chain, and the full restores Prints each check; exits 1 at the first failure.""" import json @@ -42,8 +43,9 @@ def http(url, data=None): return r.read().decode() -def ch(sql): - return http("http://127.0.0.1:19911/?user=admin&password=admin", sql.encode()).strip() +def ch(sql, port=19911): + """a query on data3, or with port=19912 on the cluster's other node, data1""" + return http(f"http://127.0.0.1:{port}/?user=admin&password=admin", sql.encode()).strip() def backup(config, *args): @@ -113,6 +115,7 @@ def months_ago(n): def setup(): print("== waiting for the services") wait_for("ClickHouse", lambda: ch("SELECT 1") == "1") + wait_for("ClickHouse's other node", lambda: ch("SELECT 1", port=19912) == "1") wait_for("MinIO", lambda: http("http://127.0.0.1:19910/minio/health/live") == "") wait_for("Prometheus", lambda: "Ready" in http("http://127.0.0.1:19990/-/ready")) wait_for("Alertmanager", lambda: "OK" in http("http://127.0.0.1:19993/-/ready")) @@ -137,6 +140,7 @@ def setup(): "CREATE TABLE ooni.asnmeta_tmp (asn UInt32) ENGINE = MergeTree ORDER BY asn", # excluded: *_tmp ]: ch(sql) + ch("CREATE DATABASE ooni", port=19912) def test_first_run(): @@ -165,6 +169,7 @@ def test_completed_alert(emails_before): check(email_attempts() == emails_before, "no email sent") + def digest(table): """row count and a checksum of every row""" return ch(f"SELECT count(), sum(cityHash64(*)) FROM {table}") @@ -176,6 +181,8 @@ def backup_url(bucket, table, chain, name): def test_incremental_and_restore(): print("== 3. incrementals, and restoring each backup of a chain") + # for step 4: a table created on data1 alone, which no job backs up + ch("CREATE TABLE ooni.made_on_data1 (x UInt8) ENGINE = MergeTree ORDER BY x", port=19912) # fastpath as at the first run's full, then after each of two more runs states = [digest("ooni.fastpath")] for first in (200000, 201000): @@ -202,8 +209,17 @@ def test_incremental_and_restore(): "test_groups (EmbeddedRocksDB) restores with its exported rows") +def test_not_backed_up_alert(): + print("== 4. a table no backup covers: ClickHouseTableNotBackedUp") + title = "[FIRING] ClickHouse table made_on_data1 is not backed up (on data1)" + msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0] == title]) + check("Add it to a job" in msgs[0][1], f"it says what to do: {msgs[0][1][:90]!r}...") + others = [t for t, _ in slack() if "is not backed up" in t and t != title] + check(not others, f"no other table is reported (labeling_frames is data1's, asnmeta_tmp excluded): {others}") + + def test_failure_alert(emails_before): - print("== 4. a run that fails: ClickHouseBackupFailed") + print("== 5. a run that fails: ClickHouseBackupFailed") r = backup("broken", "run") check(r.returncode == 1, "the run exits 1", r.stderr) title = f"[FIRING] ClickHouse backup of labeling_frames (broken) failed on {DATA3}" @@ -214,7 +230,7 @@ def test_failure_alert(emails_before): def test_interrupted_full(): - print("== 5. a new month's full, interrupted, then completed") + print("== 6. a new month's full, interrupted, then completed") (chain, backups), = chains(MEASUREMENTS, "obs_web").items() full = f"prod/ooni/obs_web/{chain}/{min(backups)}/" old, previous = months_ago(2), months_ago(1) @@ -256,6 +272,7 @@ def main(): test_first_run() test_completed_alert(emails_before) test_incremental_and_restore() + test_not_backed_up_alert() test_failure_alert(emails_before) test_interrupted_full() print("PASS") diff --git a/ansible/roles/clickhouse_backup/tests/vars.yml b/ansible/roles/clickhouse_backup/tests/vars.yml index 06835714..33957689 100644 --- a/ansible/roles/clickhouse_backup/tests/vars.yml +++ b/ansible/roles/clickhouse_backup/tests/vars.yml @@ -17,11 +17,15 @@ test_jobs: include_tables: [msmt_feedback] exclude_tables: [] +# data1 backs up its own labeling_frames +clickhouse_backup_backed_up_elsewhere: [labeling_frames] + # One script config per entry, in work/.json test_configs: - - name: backup # as deployed + - name: backup # as deployed on data3 bandwidth: 0 jobs: "{{ test_jobs }}" + check_cluster: test_cluster - name: slow # ~1 MB/s, so a full backup can be interrupted bandwidth: 3000000 jobs: "{{ test_jobs }}" From f041162af49b3138cfee19f59c865e54d28e4287 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Fri, 23 Oct 2026 10:07:28 +0000 Subject: [PATCH 28/35] Runbook: restore a view's TO table before the view While a materialized view exists without its target table, every insert into the view's source table fails: for counters_asn_test_list, fastpath's inserts. Tested on 24.8. It matters once ooni/devops#521's follow-up gives counters_asn_test_list an explicit target table. --- docs/Runbooks.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Runbooks.md b/docs/Runbooks.md index d4e88919..aa0fa228 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -345,7 +345,9 @@ each table. Restoring the latest incremental restores the whole chain. CREATE DATABASE IF NOT EXISTS ooni ``` 2. Restore tables before the materialized views that read from them, on - one node. A replicated table that still exists makes `RESTORE` fail with + one node, and a view's `TO` table before the view: while a view exists + without its table, every insert into the view's source fails (fastpath's, + for `counters_asn_test_list`). A replicated table that still exists makes `RESTORE` fail with `REPLICA_ALREADY_EXISTS`: drop it (`DROP TABLE ... SYNC`) or restore on another server. ```sql From 0cdf727a13a54b7d3e23dae77dcd52b5758dad56 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Tue, 27 Oct 2026 18:47:31 +0000 Subject: [PATCH 29/35] Fix node_exporter.service.j2 whitespace --- .../prometheus_node_exporter/templates/node_exporter.service.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2 b/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2 index 31a822b0..29800e7d 100644 --- a/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2 +++ b/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2 @@ -4,7 +4,7 @@ Description=NodeExporter [Service] TimeoutStartSec=0 User=node_exporter -ExecStart={{ node_exporter_bin_path }} --web.listen-address={{ node_exporter_host }}:{{ node_exporter_port }} {{ node_exporter_options }}{% if node_exporter_textfile_dir %} --collector.textfile.directory={{ node_exporter_textfile_dir }}{% endif %} +ExecStart={{ node_exporter_bin_path }} --web.listen-address={{ node_exporter_host }}:{{ node_exporter_port }} {{ node_exporter_options }}{{ (' --collector.textfile.directory=' ~ node_exporter_textfile_dir) if node_exporter_textfile_dir else '' }} Restart={{ node_exporter_restart }} [Install] From 201d8b93d9836e91fd4d75f93e2b75ba282e1b23 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Wed, 28 Oct 2026 07:30:27 +0000 Subject: [PATCH 30/35] clickhouse_backup: one part in flight per file, cap at 100 MB/s With the default of 20 in-flight parts per file, disk seek on data3 caused so much degredation uploads timed out and nothing finished. Set s3_max_inflight_parts_for_one_file to 1, and raise max_backup_bandwidth to 100 MB/s. --- ansible/roles/clickhouse_backup/defaults/main.yml | 11 +++++------ .../files/ooni_clickhouse_backup.py | 3 +-- docs/Runbooks.md | 15 ++++++--------- 3 files changed, 12 insertions(+), 17 deletions(-) diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index 6ae23389..f33e38ef 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -30,10 +30,13 @@ clickhouse_backup_s3_secret_access_key: "" # whose parts (all but the last) differ in size, and ClickHouse grows the # part size as an upload goes on unless the factor is 1. R2 also allows at # most 10,000 parts, so with 128 MiB parts one file can reach 1.25 TiB. +# One part in flight per file: with the default of 20, the parts in flight +# shared the bandwidth cap so thinly that their uploads timed out. clickhouse_backup_s3_settings: s3_upload_part_size_multiply_factor: 1 s3_min_upload_part_size: 134217728 s3_max_upload_part_size: 134217728 + s3_max_inflight_parts_for_one_file: 1 # One job per bucket. A job backs up the tables matching include_tables ("*" # for every table; an empty list backs up none) minus the excludes below and @@ -47,12 +50,8 @@ clickhouse_backup_jobs: # Complete chains kept per table, besides chains pinned with --keep: the # current one and the one before, so a month or two of restore points clickhouse_backup_keep_chains: 2 -# Bytes/s passed to ClickHouse as max_backup_bandwidth: at most 50 MB/s, 40% -# of the 1 Gbit/s link, leaving the rest for replication and queries. -# Against MinIO in testing, 24.8 uploaded at about a third of this value; check -# the real rate on the first backups (ooni_clickhouse_backup_last_size_bytes / -# ooni_clickhouse_backup_last_duration_seconds) and raise it if they're too slow. -clickhouse_backup_max_bandwidth: 50000000 +# Bytes/s passed to ClickHouse as max_backup_bandwidth: 100 MB/s +clickhouse_backup_max_bandwidth: 100000000 # fnmatch patterns, for every job. Materialized views' .inner tables are # always skipped: they're backed up with their view. diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index e743fb5c..dd00dd23 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -431,8 +431,7 @@ def __init__(self, conf: dict, metrics: Metrics = None): bw = int(conf.get("max_backup_bandwidth", 0)) if bw: # A query setting, not a BACKUP setting: 24.8 rejects it inside - # BACKUP ... SETTINGS. Against MinIO in testing, the throughput - # was about a third of this value. + # BACKUP ... SETTINGS. self.settings["max_backup_bandwidth"] = str(bw) def tables(self, only: list) -> tuple: diff --git a/docs/Runbooks.md b/docs/Runbooks.md index aa0fa228..f4b44bdb 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -217,12 +217,10 @@ When a data1-only table becomes replicated, remove it from so one backup file can reach 1.25 TiB. - Two complete chains are kept per table (`clickhouse_backup_keep_chains`), so a month or two of restore points. -- Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A - new full backup of everything (~5.6 TB) takes about 31 hours at that rate. - Against MinIO in testing, 24.8 uploaded at only about a third of the cap, - which would make it about 4 days: check the rate of the first backups - (`ooni_clickhouse_backup_last_size_bytes / ooni_clickhouse_backup_last_duration_seconds`) - and raise the cap if they're that slow. +- Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 100 MB/s. +- One part uploads at a time per file (`s3_max_inflight_parts_for_one_file: 1` + in `clickhouse_backup_s3_settings`). With ClickHouse's default of 20, the + parts in flight shared the cap so thinly that their uploads timed out. Layout in the bucket: @@ -253,9 +251,8 @@ without a restart. ``` and the alert rules with `deploy-monitoring.yml`. 4. The timer takes the first (full) backup at its next run, on Sunday at - 02:30 UTC. A full backup of everything takes about 31 hours, or longer if - the upload rate is below the cap. To start it on deploy day instead, e.g. a - Friday so that it runs over the weekend: + 02:30 UTC. To start it on deploy day instead, e.g. a Friday so that it runs + over the weekend: ```bash sudo systemctl start --no-block ooni-clickhouse-backup.service sudo journalctl -f --identifier ooni-clickhouse-backup From 9b325342d0586612946170fab4eaf7249fd81329 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Wed, 28 Oct 2026 08:26:16 +0000 Subject: [PATCH 31/35] clickhouse_backup: add a restore command restore brings a node's tables back from its jobs' buckets. A table that a live replica on the cluster still has is created with that replica's definition: a replicated table then fetches its data from the other replicas, and an EmbeddedRocksDB or Join table has its rows copied from the peer. The rest are restored from the backups. It creates every table and view before it waits for any fetch, so the node can take inserts meanwhile. --- .../roles/clickhouse_backup/defaults/main.yml | 6 + .../ooni_clickhouse_backup.cpython-314.pyc | Bin 0 -> 89047 bytes .../files/ooni_clickhouse_backup.py | 254 +++++++++++++++++- .../templates/clickhouse-backup.json.j2 | 1 + .../tests/lifecycle/configs/data1.json | 36 +++ .../tests/lifecycle/configs/data2.json | 51 ++++ .../tests/lifecycle/configs/data3.json | 51 ++++ .../lifecycle/configs/researcher-private.json | 33 +++ .../tests/lifecycle/configs/researcher.json | 33 +++ 9 files changed, 462 insertions(+), 3 deletions(-) create mode 100644 ansible/roles/clickhouse_backup/files/__pycache__/ooni_clickhouse_backup.cpython-314.pyc create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/configs/data1.json create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/configs/data2.json create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/configs/data3.json create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher-private.json create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher.json diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml index f33e38ef..915406d9 100644 --- a/ansible/roles/clickhouse_backup/defaults/main.yml +++ b/ansible/roles/clickhouse_backup/defaults/main.yml @@ -63,6 +63,12 @@ clickhouse_backup_exclude_tables: - fingerprints_http - session_expunge # empty, no longer used (backend #972) +# The ClickHouse cluster this node is part of. restore creates a table with a +# live replica's definition when one of the other nodes has it, so that a +# replicated table fetches its data from them; empty restores everything +# from the backups. +clickhouse_backup_cluster: oonidata_cluster + # The cluster whose tables are checked after each run: any table on any of its # nodes that no job here backs up, that isn't excluded above and that isn't in # clickhouse_backup_backed_up_elsewhere is reported (ClickHouseTableNotBackedUp). diff --git a/ansible/roles/clickhouse_backup/files/__pycache__/ooni_clickhouse_backup.cpython-314.pyc b/ansible/roles/clickhouse_backup/files/__pycache__/ooni_clickhouse_backup.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..ca11d218dd96d4b4877582af1eaac5af86a1f7a4 GIT binary patch literal 89047 zcmeFadw3kjbuZZS9?T3d0}S2-z$V@x@do$;U&4zokRUh&K|Lr!LtsEa#ADPm5Xq6{ z@H+W6pdyE$tb|}9N8q*d3AB|JlsGYzP1cm{B+8fEy*(O`fG5nA?zi5(@!j2TNka*3 z`MdY-?^Jcq^ngQva-9FRNmTb#S5?>J)Twh$opWkSTB=RJ^}+mqeo_2OLHOV4hk6;6 zhn4+$LAWe zD)xShXEUC+u;*$|HJ)qObB(7K&vop%)>Dt?2KHR{e7&b}$J2r^m@fzg!t*ITXL$>? z;U?kvx=li}5~cR-MnFvo`>Sg5w%Y~4*5(~JFHZVJuh>31GI0Lj#ALuH4tj&$vtB#| zC&Uv?H3Ji4es6H(?5IziIQyb+ASecc6O#9wPn-;ljGwc`b5;bzaio0NC$_b=cb)91 z7B7qhhsB9;pExo;Ao<38<3aDJI56xT85jLNNyK+XeYG}Q`^5Or$hk?$Hz=OHB-Z(Y z19cM<<0EwgRH2xnZ9ef8WpF+gis1tftb$sIRJ4^h}Ni z#35;73@r>?3Iu&)gJN)G%qP`~y_BDQR~#J)1Ou#^6OuSKA^F4?C(Z_{ZPMg8i}el# zsj|p0fk4_N`n&_flt!el26zVsd_i$yh&L%fT{l8yUKkh8P7a*+1#Oj{M#P@?p+e^* z?|4x34h;AL0oD&ZNwpZ55dG4~%ibWGM_q%YDAlVJ!HOUDjYu{WTZL{r;=MF68ARXm z3h`1}Y&H?sZolLk8o9W)Znrv|cw~j*&PD)t-f4U5CIf?RNwHt3DEeA0hl1Kr zAjs;Xl>J3-RFvqu=%0{+)#A{E^sgzabL^~daL_mCnZST`w29vF!GBG0hbPeK$`DPA zk6wD>F7N8>?5Wv3>OJcl-K+G6e`LH`9QVBp5c31#e1kTdGC)*IQ5_`F3%ESl+s@!O z5cH1u17c-kedD%<`i7?7`o^aE`ub-9@&wmIBT^v9fOFDs15kREIX4o(jsPYbw zjL%*YM*(pXpQnL@sfx)KobW4vX9GyRJc8LsDQm@6-T}Y{UJ0NM$njlJzT*W@B1&5g z6-sRb-cjH9pqD1vIFLgnnpD-oCRYRM%n-()cVzUEh`tC^vlqOa6O+hxiKn>W^PNYp zqaFdO2nSXX%_C$r>cjk*7$5Lyk=n2~AYPD0fY73YzW;f~<2cN~eSuIPt}Ny&B=<9K0Y z5Um~X`q?ZelsJS|0_BMTAh^-N@roeYDotDvy%&%(hHMlls}L_tNaw3<1QBfZqN(`I zS)VBR*Z=@^QlShc>APMACrT@*G|g;kQSdYC5LBBWn0>L`)7jeFNh1nO31rP?U6QFI zYT>_fVAux?N8cA4n`?K}ZmZp1U&Wed^SFpkuu=(dnJ4WpT#`t=4%FK z_?VaaJ-(ozao#~7Q&gWLHxaAIASyQE9aZuAxVSsev{wX7`7tROHYC{NsNGh{@3n($ zK~YwT{yNvYHDbmI4suI);>?a2!7ZDAF1Bhi}OC8Up&&< z)_LT4mYWu(N3&~l@sFSmv{om6h?S$fr`3A(d!$Y%$wp8_$<9hx(gSQlv2JIns7KIG z;}e4_w2H&tmwnYDUl=iME?{{tUs$f8L zo6jp@0-&)(YVd-*D9u;Hx=XDo!)sm!&ovCt&igI_U|Jbv!9L7B#PJS3a0Mr}wT?`V=5-SlSjSK|(*{_jto2Dmd-m>B9SEd|#n~<6qnB+tUp%E`| z;TYgPMu>r28>X*FtSFXVO+&bfhLL90;3TH*2#qER35@E^runqe^b z*x)5iLBQN3uqz-@gQJ0X@L9l#cXX83lQ#u@e-soZV4f`~Eh4%G!0{r`^|*I*AUG=4 z42d;0BMjLLjnL|duwDYOKL-4* zlp?q+^a*G5JAMPq1L6vVRQc);)8X@3Da9MaL z=yc@|D0tJ#ZYziu1}IQ3Y7}eOVLV(G;*IE``mVY*38#%}RF7GGir-ZZ3C-7aR~cE8 zqlWQ`3le=IYM2ZT+!CZTJWCFWFe8HAX5t5{b`~Bk3-|2qtGi#@{g(A@#|=lsUM3sM zSf+##p^C7I*RwopkBCQqI@X*zeTh(IkW!E>svitWBr&McD|HP7B@7jKI)M)mt0(JL z7Et#Dz%Zhc)VCX{Q#Zc8<38QqGO%Cqnj#syVqIK~p9DKSDpV;6=oYVEyb~oCUeLrq zt%lgdF!!G5-+#0hRm{dKR>gAOmHJ>n+Cb${d^&>5!m>T}>aHuh-h4S?FOrQ#@rtRR z|5tV4Eu{-}0ksEn@KpuMAK!y7Q2mNn{ip{!b<}B{x;G8fhn>2&7@T(M;*~{GcEu_i z#Lx8swNC@8s=%PfqLih%owlk!dYI@){l)JdVY|{O|0uE29`SfcleYDA$??gH&sE9+-6b4+I3S zx8JARo9*mZyrM|UE+axVV_(Jt1aKCYu)kIi{JK5}Q+A-Y1`7zA_USwV6U<;1;+Npj zr$S8Dr*jC6hDrTKp@ia$N#bDhT8g} zs7}PaVJKv%Z5#?2#om!NV67@-j~cp0+M01uVpzwV!oy|ZA6%QKjLUY%RPgm3%azshhIj4L-WkL6shQT9!I?hUUHY-(@Ivdt zz{2+1n{Id9-X`xnjF@F>%2e-HQ(n_Oun6f}?ppg53hsP74js+UL5{=)bgqq>Ty?)2yS)2BoF(-ml1G15vU2()w15_oB}A9GjQ zi5IMKBzq;4nKqq%J?~oHO!I7ZB)xJey*`{?KkuIRMbdZQvF~08+_AUG#x_0-$P&A# zk#wz`LEuGE&kI2SGGxmBpdnkB0Eh>TeL@o&3i+eeV_;H9lgGy5Of1e!aihARnZ;RH zoRy`scv6DaJ|URmF`|+7sD`-?q%U%qY{CZ$o#1F#@bptC>#T=k&Tgu*aKXz`4u_V*8f z^bHJ-3lc9A8YMIdSf^R!hMIkU2m@_A^>x4W>;GD7XR4g4<1i8*c9put+`z zMx<3QY7+jeD@046QMzOctw&H7*hJvz6UJkJcS;CLWF_2cT;a+$9_s;dKW1Ii8n zG~$=`Aepq6f_(_^!blwNs^|dKvm6&k7}oxN3>tbS>Q3kg9?3dELn5LQtZRPSC>PvN_fw19r1?c>y zK2wiVO%cCiMHG67K3sLFSaKs_TB1N!U9mm^=0~3-FB^y|S>AZr8?-%{tj$Eouewg1 z@R+tM(`uKlUpJ|rH1x+dSN!Haa}O;;t1h)qdh%Aiz)$tvBoZ41#ftF9hUC95y9FxXBd`YpYdFIsDl#bZ^e z%ASp@OOCh7Z#}(9ed|f-wSxYM!J@}Jj28PX`z#~65v3hfwy17j)YxiQ&ULJEX3;yFbY)go4*0LCewQN`zfQr*Ht`1)ro-%x5cP~4f z(@k$>zg=*nU@5OIoL4t*jpXgTHqu{iWOc z<+RdBTGy0e*=oOD**w?r-l1EE7OmTs?ar%vuI`!IgMzPiU+I=N)XW9nd-c|< zi;g|_^`?~lx_jw4*Nd(dEu~k4(<|l-a(cy5`kuw~J-0KLb2ePRbnVjDURlbi3g=Ws za<)7)2pN0v8vi5ddw%g_XV%p(UHQ_Ivn=c^`%KWIv}JeZ^=;R-eQn3}eb@HQRZZ`U zxSN;UEn#;{#Jzi}<6cIQys2rvV}6^wu_cnR>vpE>=={Zf6J`B8unpb!{oR!vn}i>f z7ImbWeyF$M`G=|Yj-r$w<{Ig_$Uxzyjr;Y6zY_Lz?lJt;9uuAuRw_ckG+kGSs9t}i z(y;A(pov|(rB{|B67#OV2K^?o1Zc{!HArtdhBb)78{j_aeshrj=4(!j1V9F&n1IKv zD|hkmxHV@J<6X4tM3b&g=+n39jDp|LXXsI|6N-pkDh_iACykT(T47SZAAHM89X&)j zue#K$W#+GtGBhB3^`@-Lkd&=N~(}1GM)3QBG^Q z@xzpSgkX1xNQ)C^Rr>CqJdJ?{@&rvb?j3^^Eo$rUADbAQ9HnQ-*It_Rjw)}Q{g4xb z?xTi{PvEtszkhII0Ft-;R4jFr6hsis0?`YZ+{i#bWWUnL*~y?UfaK{KvUtEZI@;e4 ztosx)Fy`bbP9X21QYi4#3t~)u_M7OJ9~tW&8bSUt8u+1&XrrGp3h7}H!S25>2xgGW zG7lhM>{3B_Wf@Tk+{x=}aVLxqF)}`cI24z{nGm01^$>sJ3R+ki;!>b*!eYRufXji_ z31Yb3NeYXB?g?VJ_6g^Dpn-xIu7hG{X`qLK7_N!pU@;jihU=rGvKZ*2AWarRS1`?! z9dv>(a(Qwc=n?!_6B|V{SVa?9pR33e$V9-9Q@+XeKtOP#m)#!4Nc2He z(7IBOsuHsUc1A!G%gSXmjEnbqc%i&baEGQS;8E zeV2>|j(^dw2dkEZ9`=|)eJgcYrvzYE2e?(xt!foCcaj{T<`y;TG|5NVMYLF*&Xc0j zV}2tjTaxwgw;3Lr$L?`>>~;D+Bg5jR(xW{)z$7#&ur(GO;x8>NxpOV9ZH!D)L>6HSkq8tiaOMK_LJno&XX{FA5HgO z2xwYG&CQ~yN84-WyWrhi8kiO}^Zw(@p_%t!)U0%wndg@}s9FXnAVn<$!`Qe1?M>(* zYHPzzIN#ffru1I&b5b^HVyGx;MsGrx!x{x)Gh1B0LfNcpL!x#R7yDtQY*f9dVPNQ7 zfWU>*w|q&V)hU{$;v6nz;P{47DG_ z*pG%g${TmiOzItX`I5UP?4}Iv#wqKk+4+<`zg(K1;Q@ZBxeIw*2fflimfZs7;Jvpcd3Kg`r2OjshQli3nM6yI7)#A+$pEk%k(8=g!>Flu6ZuuQt56{B&| zfH0s|cqaV@BEF5kxS&jDYK8>X)O_;<1hS^)_aRfk?E^Q9f@7}IF${s*4deth#Q^gbo~3Ug2+&7jLM{huRKGGWlod}K$J#*TIEcQC z?CV>uZJ92d@y=RjLUZYJTjwg~(`9FqY;EEVpyf$fq68zJS=X22Vg0Tr-a*~3GGKmw zk&gbK@r3~Gtg&U;Vyb^2ARws~Za2}xxIMZ$6VZzbI#!5T!fusXagxf!R#&`IF7&UC z?{KKAFcB5r0%SYUR0*Y*Dro+kn^cvF6vd-DPv?o=W1dd7IZwI)fhKI)j}H(!#Lj&8 zgjUa|MN9T@Ju74xiz|C1)BhnANks<;3Sv+z(CD8k^pyloU4nKI=`;{_Q}(7~^|e7s z5EeuAfGL_>AWbf&=m4EsMJ0oyS}FsIF{H2p1&&${jCv-; zm%fg^1V6|iavsl245;zf02JcNnJIm{`bPCk(d+yy#?ZA;R2WJ;*M8$2g;2chze30yn~LT3Ox%!n^g zEb<>vr-Pa+lywwD>Lgi5Ki44-^uOH5CxIbqZetqKQF9MdS&pW3_(n&@MzH@AO(TuE ziE;h{{QI9{Ji4P+P#@6a3AO^A1$8RMYZ$#yID=sx@A|^b;mWw%<-9vUw`gh_gvq%H_R2vh1-`3 z_l67iE(9Wl?RQ-r4-7(f^QS!T%!_lyce84=Stq~#^fw3Q%D-`bc1$jWoMGz<($`{- zHtVGBzD;mtO*wugm_m9F?FJenQ8l_G{EHqg?9f>vM2a0F5sSEmEx2o_Wo@N*jpQRq!xCJ`X3u4JqaSQ_lELCmi2 z)hAQWNXj(o>teVi>3f4ZCp~7`{T&2U7GPVzWAy9y={=?rVOV+Ah}JC_kq?kq`d<7{*+RTDvEL`zN4jjhTV@E=27l@m3}+n@Lfq z@ecT)I|6fRy4_nK%ub|X`0!0k*qGF4nBeSF9r|eNPAhh zuQ#V;fH1y#;L3p|$HuT@<81JqS8l$tV2~Xf7ahBQuTsq=<1iRe0k>FtQ2u>UUJrkowai>N1WT_mS-28&;Dah!LlR$7xyhl z{y77|uT*RizFpSlFns8$Z?hP_Z`M(0F(D>_cqSr1H7lq~DK% zag4F+LlKW-?7kRA_vlob4mh1?76bb%ZZU)?-J`}KhM_u}e=-pQ9vSTvN>~O$#SvYyMqpD{nd$u18tICmkN?KO^@OjtGHn ze1+AScF_RrB03zmSJ0_q7=Tqf;6Bh80-2}Zc33b{~qamo014@ zSJcJ3KOVyx3e^BqH`f1!VQb-R?>o=kd~SX~wie#Cwye?nPsETjo=EyLd`6*kB#_!q zzz!`VReTuIOz95<1UK97)9n^;J8q8-LmF~n;#yatSadK9>!<}$Rp=T7!A{zUPd;4| z6^c|cLoM$d>HuAFVHoDdj3G-#1SMmZVvG)B$n3m!+7RSe5^N80Os|3qn5_R(VaFgN zUVV8CTDIq;N!f=)zxhobBUrEmIe(oFdx$R1#Q4o%+{}qB?Ai`eFsXdICnk4~*C2V_ zJt!6W9F{sgbR~#Xz~#5}S(37ODe>C-t$kLtHiOTw@q>}JcJ=kA#HHh>V_GVGR&}%> ziLo)Mm%T3~>DOXAsYvN~Oqx_pX=#s1!0E*q_Tf& z!v3v`ae~qU`U;4FVP|6Wq`#zq2-m3P7+bf&oE@JW@dcqL!jeVJU=?BQ7d0~qG!WXL zEbz48!}db zp1JIURtV{P6x^mD&Qqo-IEa%D0qHEg{yu^#Tb#FK`nS>#>D6CR@HYseMv@;%gY=v5 zEGLp_xm8$AXnAlQ4;4Eqe6%|O_wh)EU?v$qv<$6@U%T>aGl3;%$;Zx;IrCEG&T!?< zh0^bB`C!XOhC3}?OO;1Ht~?@JkI2p=%Z}73Fq-T0sl=pqf=R7wdhg<`iwm~fZHsjW zBJP9Xx&w2YrcXjQ*!T5=k<2pup6Za@2S0JTuLMcO+oSOOG?>_hbb01a7 zC!dj@J1Y+k%ZEnfoSI0^i_^wW^9tY2y^%XxIoB4+tD81|;?7&nEt+}h>nYQ%_wvi% zEtwno_7*U|+m6egr{&|%$j`hWd(SR;hr-?=`P^v4J0`y{E|2@=m!zeaUJk$XvV7sy z$V*?A{l5mf_$hs>9=A4qyZ#~F4(K278zK+tkLy?Hmxq$H>QB(EvX|mEoTTThllo;> z`kN(77c7BUa%*tW&@-I<-0<^;-1p=tysu3nq+ zim9vWNL^K$0~n}F%n>{T;0b?9Pa#OoH(+XMS>?nUHjaiLt4`-Lr&=`%Efk<10xQL}4E2mw2Ex zQm2QY>M`XwpydZ#WoXovIHBNEB2iXdafcJ?_XhERmC(FCAJbj~ zz9a=#OQ_`)X8PMAzEZ5%oEGDmKeoh7pI@yOi^-T2@1A3Z=7Hy5eKl%PauC+>`5D3u znV2MB%Dz)doYsGJY`u< zb3vH~lCIO&PR~}{&DuQG`H4ICN3PuEtfE={?EbknIkRr6lQz;|e>G>By)>USZ<&8d zF57uGw`JN$JQ&o*X2FGR`PjV$g1DLT-#NOJSsBi(oXdQ7V7}tpqjTeO)tUWZ=N5P3-?X$U#@9j>|1uE=Beq$WmPXQa7KcOc!Kr|xKf1=>W6#RsOpHX1MUB-fo(Xp7x&PNeImAJ^e_WLdnUktm<{RludM zB06d$rkw3qD9q~`#&Mq`LtqmEQr)+bQ4oT+fHHCWCRioh=D;f9_9%^RK{jAgaTfG!?=an%*yQh@ffVJ%qIs63RHM4ukg+y621);g?vZa| z7^_mE`iVf)3|WpJN)<+?-)o{gKC&8^3=HEIxHLWhn^+VuK0$~Nig!{mngUA`0zk=v zf}#di?2uwaq0)0MRumNMCdOdrK010y%0glWg)bv35MOHg)c)yBZ?;`O^v0pDbxrS? zHOjWK%kB44Ql|!{o30kUUNqAHdn*z%OdDTMRUThYxo5RqwZ3MBDNydU+^^*?S_>Xp z1Y7QBR>5TX&4XV#ICWyWY|)r|&uIS5!(TZ(6}W56duT_3t3{s&9QgEntGTVl^!*xp z+n$t!$!Ww(A{bWcSp}bba*{lW8EbKY#Tv5^YpvE)>j=q?bVqb=Vjj}{ttS4PPeG)O zU9q|iDRoOmyuc7kr-D?{_nL{9^t~|-zF3US(l&fd0>>ysb;YpQDvs)GxQ$CYD82-N z)@;kp7UF&={XkngAC+f~twAb?E9{P|JFo1V?w#2du@^7dH;3(;=bGkC5&O0!``)m9 z??UkQ$%ws6Hg@s0BUkLAF`;V(l&YZBkA&40x8BkRv1jSYJEjYoaL>ox?AgHd5y_Iv z>LD#jvfZ+H3RtX0M-s7ICsL~=$#o*NT9RBRQmZA&bt1J|l3XWJt0l>GBDGqQX^&`0 zLPHW7nef?#dgd^_NtY)#m=3+k4BGFgDvVl=axVxYDd<$M6+s)&d04)6D z?h}|cCA?51;0RhNXdc%x-#PGJP!X^Z>B1c_knaFdp|^_RMFCry++7XahIsuxI2PcZ zAz*S85Q~w!7#a*Dl!hu8>GnLUvP#3MhAjM^2sV2;81f&3j+Xws?2Xqyy#Kbfb$68E5KVo?@_A_8n>u|NWNelt@EVY=CLh#mHr>xNhLw+65I^G z;ucrQR_@cWRWX_M6Ro@I+N!NNMDq9-UuD5p5V~7+C9^T|nBv=s%%N#?5^1e zwZa0PxeI}X4|hz`9dgk$oE)e5%l*oPHjv*UlKepFlPefu<48dzFvfw4muRYzy&sC9 z1HNb~1WinH$T!$O>90zaXj59+h=449fC3~tMKF?p5$vh-6I?7vRI)&b+WKfw;5(>> z(TQ_W%LT7Qj#Dt#e8EBAkQaDNHB*QcE@e_NJ`@um+L74=F{X-f3Q<>l;&GixHtGH& zQewR+s9dz>E$5fM?p)3$zUk1MBxjMXr0o-X)?3AISKO$0yY@!yT>esi^T+wkFyl#m zPIoUWZ)WpSR@q`!*-KB*nb6Fryt!qeVWC9c z^R!&>%#!QbuHgKPWQNbBTe%a(hR% z;Rg*HI?@e4OgG^<(fpLCAHp{)yAiC9A({Cp^vIbJstKWiNqtGv(5e`sVkzJ}vVP(` zREC35o9a2IpL?DlMFZ*;@P4z#4NDSy6z7cmmQyc#jI>(y8Dbb0<5K_^q(^mUbdaAg35n({DqJ8S zmzi>%ruML_&vYh6_#qdQv&3aRQ1eQ%FX4ggT_qM{SQ&~_tYlghdCAH1!l`PKyeX_L z!dko)cq$v<$1WAq+-}D=q&j+Hf))q)VM36X z1&tKwD4?y80Lc%@Fp(tJT(+a=&~n@vCj___Estej{4R0=CqV#hyJnl&amQ7<gJo~%jREN$Xc+-^-ta2`ccV87v!g(m)p)nT>Vp~doXDnd2Qsk&O;s~mQ0NU;e1kB ze&eON)+y7}SlC+di9L0>v|_q8?AUZqESr6Cep5tjp6Z<59Cj4mD=wQo`_9nKp}Erw z4UzIak>b60QwPZ{&cS|JtS|0_pAwyh?+CV4(0ZDstyM?)r+dO0zxIIcs zQEl;OT_MDSxgMvE5Q`b6!wh>I21t}z>KIhV^Ji0jja4~lwbfUAj<4~R1ih0!{uQAo zCMAiNsmdiVcbJ!lpU|jCtUJr_NE^%y*juQBcOyVV9H=Q>qUyIDSi4_Y;iGPK71czt zjwe&R%fcs{8*Uz-e`(4{E8O(XJ6QAv!q(zXT3X+2esAZucD}pow)%449c#&Kd)QjJ z1{@JSWN=i9hqd8|t7#}cEjW_kz_Y%J5v?BwsM#^)Tr&X=0#XXVjg$g#<4~g=Kc#+- z6`8gEF)Ks@d?Z^T)Dgq(Ks{vnddP)VjW+qvQKai`U~>8{yOL49acCmR89^h~2DpWw zC)HjQHI1zXyd($H^$c7N#K1X~zogMNh%PU@n{#2-7!=jLLvylpS||%to2S- z)lyb{III3HsJPB$XXfDnUA?kOcRtj{N0lI1R zU{^A!E!)qBDtclWps-9@t_Sd~ou^89iS6n$ScR_VmEEMIdsCYEKvb+H=e|D2CSJw^d8-6NME)?5soDl32`0io3>?8uz&Jvc%M4DC^a8$s zCFW(8WFwk+^er?(`YD3Y&KA<$BPWn3lO!R`%_EtfA*+<0Ej8(nmb!G+HC!`BYaT)cxVEBD8lHJHd7_N+OP z34i`F>k91T;oX>1B`^&>jGkM6=01&7@uek&W-bC^O>mntvR{ubEi?&GMaRk;$o_9M zWvi4aTha%{M5|cSr_)SHnzBzao%EPaTljSHn1B$T!BS?_Fda$D=u1#XQqf7Du9{G! za-pm`+BDa=_e;W{C>!H3p%fGFBtZg+U+goX)(yPYT|i;n$=o_D32i-_L)&7=K%Kx4 zXcBzz`AxpbK_>>OSYA13$#p7qMK`I3_#A!M3EwSk=9idkQl59pxN9l=? zcZ$jBaf*n|WisZFf{ajFZDAU6vrfGv_`paUsoOf)|)uWabq>bhi zA7@srn1tMBn0;jJ{5P5Se9$;S`Ri!+`m=rjy&a$R;5iuMakGQ5A#=SNHbe`fgp}B< zZ)2T%%GHDW>dT`N|CD;wrCG9=z&f<|6^5Om_%IA-)X;lS&6DCDraAiNPz#8q6$Q3> z16aMJfK-llz@aZ^85lSzW_h$UX!X8X%4lp2NRR=t&LCn3BwBej;|=zOttD$v{thIj z<~^D0N$dIyd#X$VMH+{3*%Rjo7>iC7$7Us*j$=PlAy+v;#(kkNiPIA(u@c)bOq~!D zJDF&d5fIye+pS4<`+)!Y+YN22wlwg?`V=vcc5*SewwW`jkCGxz$ zXUcQ*beaM}SVZ|VVt*IK5bcqGm>8uI8yYE=SHVjpLg4}wI(-Bl0caO18e;_FacnS% zlLeGHvfnbko$`Arzi*#?_B&bgq1z3ShR$V2=JXKEUnb{<7tHrfh7uT$Ghp0y$C^LW z6t-?$6M4`?WiTn?VJ&Wy%v>ar|E#|sBoP!%;4g09ve8uWKtOQAz-}BskRo=`;>Q5w zMbKjeAb$7+VRW$`5=Q1-6?*`NT0OKVV)-7kgeMI96H@vWF-l*bjEiJ7F>B6Sk1P&As-n(fEbipY}_%AH)^o+;+e`Hpzsi@nrRJKCTZ={ZHDj zq7^Y+XN<2(TtVe3VVqsDaZdjw#yLa##U~tR$Ou?Ff6wFAv?Qjl4d~2_$9uAp5rG=` zv#Bt=`i^q4q3>vad~bSk`s_%RtL0d`j>m}4*sf@7#tVR(Km0d^o4kbnBmwrSOC>D$ zkzlGGKk9W7q`BkNC@q2t4-fq0DTw!EpTkqAEnV#|^gvNUs*R6HwN7s`*c4fhZaVeX z+A>x_;wj$4UGMO_r#RUN$9Jqf#nqZV0Lfnx&7pG(;0#wif=ls_&>ymthzC25^oXHq zMNld>;OqwMEr6=x5;Sb95@AJ~s35L5$wk?EpXuooL+TEyc+NWsuAYRjA(v8=qRqOe zIAkb(RSX?fqWNZPKW$b(C^zOOA2$=w)fZ3EnGwooA)^?Wlq4vhlI{fG$E8yw>d6pr z90q}*vh|ANH6qDzsB)dOv6?a~f{eF0)HW@$C=s`s-;|dmzgB* zDG)1Hm8<#@#u->+ zX9vA|qoOKE!N9>xqjvY`1a#s8d-1?iyoPK6@a0(jKk~{wC;!8+{PZ*Ov)(&TpA8=! zU=#zy6U6It>@1f~l3F%)Ir92b}6+d$T2aAD4Wh5HoCqG0POgk%T z^PKGNKhS#eK<5b(gFvU*%zsJyDRZhsl$Z3^2;fJMc@`wPE~C10@2t_F`@Ved7pKd> zf{)xv^iP~8KhPssP{xDZ(5N7NET)JRwT|JWK62f{%>*oXBwbIYfz8#jOoNj+Tahmd zOy6n}nX$qU%Gc=hGOj-AP|-6fOfV24y+l&#bza?fb>Gy!dycfLU9WY$rGMLU!}7NC zh7;c7oVT2j+#R=@?>G+La})qO_uuG$cksOzzxCom=J)bH$p5Hyv9^2BaSVmvTowG6 z?FB61Qekbluy)?~UHkj?KS{mo*b5}-C}hBieQj1*wwh6MtITlkQN;8_6xEPN$tQ_a zFhcb)B8sh+AfqUkdR7=@N2-)em3_ntRyIPD6ihG}s!3k3B6CwRL*yCpS!F^AC9DL{8qo9MJ(^@(&QQ^-sbt%TTW)$-1hv4Ql5XI`T!`qIZ2qAQsvb=31<=2%)Ib~;9+(=9i+ER zl>0CZ(z^R3vU}L3!^-$CvE`9fi17}n6uW4ZBmUt@jYeYYpGNfO8B-2Z=2~S46qL8sAaA#{(S*@r_qbTx> z8mzp!i^63Sq*?eTNwMhA@U4as9cZZkh6>P$hXx$K$XOvnR9{;k)xpCqUU5|xRRu6* z_c`A<4xp6wMvK+P!T~Pzh=$8k)BGL{TO|tl37RLTAHA7&!#Jb+PRYD?A>~g;m$n@W zZ#xv(*0t1lhVnk>ueh}HW-f@bk7!ht9jDHgb8PYTd)ZtQIGnY=4 zWGrPj29XGblH2l5(`tdi8M@l;s2zC_e zzGf@bvAKmMrTZuU0~g3S0#OZ68yN`Ucoei>66hUGMs8UM=>eA-q$bo!dK<|YOm!hC zL3uv2F}vLHiQT#E%#&dT;BZZ6eD&bvP8|Ia&g`1q|A$BCtL~KUUMg)1m$ogIc0@9} zaQ4UL1Iu>TO!4n*k?n=DkyD%lC^L4c@UuelucUTpViKD(ey*7246z_~>|pUl9{Q~w zeu5Z3Q;av>dd8BsgZ|%tB0)4TsUuR6Dx^)^jke-cSbgD+)|Ca z^jke-dFBdoOEvP+Z}pIWepirNs*#s|E3-Uv1-Ydf*`HZGxhxI0RLf&A`7DN8s%>B~ z1uVwJzF){V+K{Bs1q+2V|6S~aDo5Io{UBAOkmPJ+4C9I6)I+s`xYejp&Y(R-h6w+^sh95<;j zfYLFk6!!QsViV1^k~=2S+GlDFpnCPY;us9IXKvyx^a;Au^%ps)!`Sth@-_CINKLqt zT^jSPqz;o{X_KzG`YrJE1%;`ky@nY4982=1LgPR7+fUq+E|0ifEms3_?R2Vv_h*V;jk!MTt;ab$MLnh@rz3 z=#RNqVs(Opr3@~XwkXzQ(k&!}Sv&T3qG_599-J@`${r2|{VjEM4UOAt>+!FlWk*9} zQ(e@lO-IKkx-}88lN5<9irUG;h(hQ^(_ke>I~JY_`yhIZGsQ&B)RpCQjySd?Bh878P(>;cON0AFsC3b)!R6q}2J2=xf z`%DDxPK-EGF{fxLXN#P(<(n1XsG=hY$SHm0waS^6*_R_3bnt-%Ck2o*)q__K%2{P{ z#lDE+so3xGxs-^bG4^vaTrfH6R&YiEG)7*xd}b4}H_SAD?K%9g-P-`4Oy4Y-9eii_ z=J4FB3*JciQ<37<$cDBlld$Gl*Rh_;s`$j~Z z#X8So?ul@n=VoyBrJ3}%+GlmYzyJ605b<&D36z(5FV!`jNv;s&{CZdbr8h@Xw@&F{ z8wBOttveRR0 z9G)H&PWC*5y5OXL)VJoUEPWFL{{#HRm*5H$j+Q`hqxnAFZg#R?dNbT1DH~*VcG1SvRdAt_|ad_a`7sJ$BXqfs|)SWh-H4 z5eu*Ox8#enGJ{weVe5CC1ZDgr`>s+D^`>Mx^f+4)spwKMsgh$4jwnc~3Evk(v}z88 zap+??Vv8a#mBL4J3<5y|dB>K-WF7?Z^(1m{^r(gH@>>%$CfNODT;SMdGf%MSBcA$h-PF)z_p=_m6zxJ6))zV+1dA453}X)!y$jU}3eK zv`!l8X<@Y}3+u%`%Xxhl##32Dt$s_QI<&ehtL6Ap`ceQh=wB#Fui3VB`q!S&znggt zPVFQHRarby7c8C*Fssj1(k20{9>(EQ-dFWJ89I>I2I>tk7{?jL5Y;M@n_c2_V89bWN7=aJ6#UNKn9+)T8dU@5Edo0l&lhCfv&WK)V5n1I(ghRJ9O>MTWAlz3>J}>#;^|{*JxO4Tc215} zvTyWHjQ0bl)$?-AsPQ*uvYt>PWI*QV>A{c0XT9VgGk;hT}sG! z%+t~7p-(HlO@o$@fKyjECzOCAWUd_<2aC?lS8Oe+%En-*z6H((0+W7{`a{YM0f>a= zL+_Z$nVAOy4rPJ!BYr>Rg#!xWViSy39wy~@9S^&8F*yfxk^H(7`eZ(9;@re4L{R9f;omf zF>tl@wbn?^wx0(e?h$@Huhpo#nTg=fZ3sekGDaZ7Bl2|blKzA$8Kg>*W|#EG^z)zT z>96VOZz$MI0W&jT8XhSpTK9MMceftxjGD&0LC}%?l>S2sm>}{n(yJaozejml@e57Mg?gEXxSbrr#VCy?3JF&P7)XlHYR| zFyDMQ&9(AlcO@lHxt221JF{Od_Q?4s7G1qmC{CF@eC6I zumBRwtm12z-Y;1!+4g?Pt(G@$ zGnsGa-N>7<&o<5u&i2jilC!tpb?hMPih@s_8B<@9ts8#6k|LCDX9W9SnN#=Yq_K&K zj%OE5OakjKAgNJELZ%eND8eVO`X)l~|Hj`6G1nOJhw!v_koi|XsN?>;53*xmDfUrf zP^l!>v-j#CEoOD6V3+Q=@v!1F8e8`|$lnB#T7k>dl9NjMxQur{WC^SA3gk0CX_{QfyJag}2-mWr$-3LQ+u;w(uOm`ZpPaUMWie7gaqv+I|Db4jMk z{~fZg&*c2Slr&5M5p@Gv(EI;^7d^5D@Jfr+ZA%E&hE8=^P6D0xse~U<5JeCNyzf%f zKOjheI9d_45T_Q*#_%603mZc`&_)HjXgnE~6!Gvl?JaVG9J43EMj3I*>qE{@@r}6V zmc_J<48ynGgn1Hf@tUA4b`dv=nmAT|jEBp#vCHEV!eqpe=72FBauHOnx{`6q#34T> zwRKf?{G`<65wuT7Vo1fGeJ0M;ZX~lNEk8_@io%zIi|jMNw|%z4Md|(6_n-_g)gM-xHBx44{2UmXe3r$$&6?` zx-&)bBOJJE7Ey>>G@m9Kmt9SIxP1F}Br3rkv>KU|?ugf>H^E*S2MfjaG-3q^JN^QK zAqXb4YK9(T^0XdPDx7wcOS!l}U|6RSEyVT};^liREWJh?qN#lnq0T1ZOw5FaA7Vsj zc6I#`MiV{R0LdXzk5kt8oYNt13o)FL0 zB7&14o5pAg4&%dlz2hSTd={TWRvgZ7g1N8bM<0BEba8G8lJ!yS27nQysU6h^>ys#o zxS50>-_%%}L<{57I>~DW_}@U0*ws!=zk24iGm+H7DgAO<2H2m}jmm)r*Hf>h&b&BZ zJe|7e+WI3`_RsERZg}go?B4vDU`*YJgL|*wJ@(w|&vI;~qJ( zdc!u;cs=cnv|rp$!Dl`X051zOh1-Sirng%SA6i?twi^sTFqrTo0Y}iRFyL>6T*y8S zN6-?fJ(icbpo2v#;&pY*kqvi0XPWTSfzZYa@E|Jv0gXus1@sN+7YNp0R;|EUfkq?+ z`S?&MHbD&q0UgeV8@cPm?R^8gg^y!97(c^fFR?@%^AO#xUbNJqZH^VvH2DIU$JA@` zV}I3e`GOcouQ9fY9Yce~)et{Sz;C5;dt=A0oP(eM_#!6WWlko|anb0ou6Om3EVf>o z$D&=7dky5gh|5fR;mqxk_r@I;@V+?7m#nMPqP7&5qAj0{CRIKOI7;o^bugyi{>-0g zg{mGK!;jUh7I6`3Y-c%Etibxf{>+~yf?DzIl-9CL0IdhKxYr@l@q;@(8Z z!GWk*eW@*j>ezx5IG8P-s0M;1nWIhZwf{&W)_^_`aJ1Mv0mTc2t1uXWnW6Ct_z|-{;XL) zoT0uRY^jP~<*a3~azd&W=^z`CREPEyD`ziv6j!Tspc)5)4Ut|^Nl^uHCCEkSC)1I} zf$q?6CP+O=JVlqX+;HL#WTYg~2V?#QA=Zw$A*@D;q}m0ub}VbmGhxyqA5UVA43c;_ zD`_saHykQx5ov#xtzt|a9LWz#X{mY2$v)S=bPWPWeO~jA zn|CdA{zTSHg8!aM4+^oRc@5qDEX;be1a_=NjkE&gcHo=-iow#)Y%D zbN=%D!t-)TS0ww0>^gD}``)#0)J8Ij$-h#@SHC=)x#%p5i5_RR$(yz>xc@TucJG}X zhdwHiw|B#$Ft2CPb$oe4(X{m*cE}IEc6d5?{grF4+;x=v?6WdR0TqJ5j5dokOJb`m zwtgq`yG4Io6lrLkPSp?*V|Uz5^8<_Sov?qRZ<6=g;LMO*`qb_8+XlIyW6{;A(g)3N z@4B&T(N*!ET)DrvpNg_SXPSa-(vV-Y3r{2<%Yj5H{nMDouDwV!U_>? z>(U92IGp?z0S76_GGd_n38{foCLsP zAG?T5$8YP(VHEX6`0nJ~b6l)PyU5AdkY?r6*e(1gWw)_YLl&4;=}K!BHMP!a3F^W` z?WgM;L& zGb1!5_#uFzIBJ)u3l7Y+`cm769s;kx6rC<=&Y{C-6$o9VF8fQgg`I_@kCQ7;)FLx$ zk)GvM+Z8+E4rIYeF5&3Cd)gnn+z9NX-IPd5?G*RBAvj3Q#j!^Ih1^X%Z8NG;Y4XMW?#b?<6p!ZB! zS4YWdA1rF-rE|4nYKGa_X1d4IjH3 z7Rqj$zgP1YHHyorrOfJZX7#*f?h-lTU3f{ZX^mvIP3u&Q+BvgbF5a`?TR3&Q?eNWo8Vv-8dQA{fEqrrPMjj#v?wYQKM0@C!X>-pF6m(q*F>0~#*{pR+$&PaOWl=+^sXa)zTmrohjz%E&N za()f=dJAj9g*9?PO(d(9T+wEI_0^faC1>fPvy^skxtH%HS7q2$Id@=g=lsd}0}Bl} za4q86JJm`XvQSW*F->2XnV9pw~1!`gG4 z$xaL;DS3?4gm1|H0Noz7K^!%JgHB*U$j6bs^MqSj?Su>(ke<;>+sIKGf}4<>;C2f> zx^er^$ZmfNav8tEy-%|=ym?x|2~hzIJ^T{}^;e1>$T?aPn&4EHx?RfbxbgYbSF*j4 zby0;$OL?hO4x>2E#Vx^AdIaYu>fEM9yvJ@x3XU zxEdaJn)E=})fdwUOs19RiJ{6q>)P*koI|~AeKz&$gP?FBKqARK|4xi@IjZY5`|Ziz z_+f666dh-&rNr;+^)Zkkd0H~`IY^Z4Pu0{65L5t4@u)>XdIsiy%ji$z{>$R|pnI5x zGOUeOzq2nT*0PJNWhBRBSF-bFSV<{#@`J~w6$hc#KHE)MQp3JuXkW=E!)M>a1Cg`-{|8Q$K5?#qWxb z&ADFc+sBmk%RvG01L_E;_qjZok10J_zosYD)R5qj&V{nF5@SxU4HHh(Bs(rbh48Cb z?e`v{$yQx)p_`w2{|Vs+waU@L0&#TkS7Tf>$v=^*J!l>Z>oa&#B^9f7X-sMqh?j0PD|wf?m(>%RaL^K~TE^g4%+< ztVI807@wlkb!sh7)1t*Lq)Y0dOl>XliH$M*##Rd2Dp+;(Y+V(BpXiruNn&*{*vsn6 zZdTSllxU1$pTRX|2rUk)E_`8C4E^l+<>nG$)MDnxQFG4JI{i;);FK_&TFP?9ME0zo zpgy_!imP2B1)Tv?7HS|;jl8Fkw2~=C?h<>YNgsEE0YmZ&-hk+39u5>E+3t|B?3uDL z#sf!9Z6a|S~kV&*>R96*pkZvj|qmryNWD=!`3xQD8Bcjzq-6uLd zz2c$n-ecnalilsThoEU2XsZ0&RY$6{&|xMO-r7(g7}`=TR&=wU6_1vq$cZ;|-QX8K zH@r~$LMY>fN4_WhGwKIrr^DzasS80UjcXBzy{&CWI>i@4))z$7nXF)~Z$ zJ~*uGJib7+7)^gfa2-m)xuO@C)^hLX<~nwia>$Bfn8~hL4B1sy0K-&O#Kj148lUXQ znRTv|Ni|99e7fhDr?ASs1XGeHL{cODK=iD|*T5n3l652e;Le&Bt+fzBKaT&|P&tYn&-;rb+Qv@!2K6{x|R(ao!d57IbSAc z?O1f|Bnz>& zCwKQ9M>jSNtoEskUv*9yer&a0wO+AKcT8`a$(Peh7p-OY67>wOp1X4H>e!XBnRdB& z$DIPWbNZV-AMTkNi=-W&f*W_P2GI2O-Wz)(8@9}~-q}za!KqfpJFcx$rezfM%#~*# zgBy&bmQ3mIL7)1|uYH-^pB3M6mQES(@y`RAriP{u&je@Be*My%8(eW8c;Z+7^qAb# zar+>0z1cpamp9bj$*qfI)<>KTavHqJ-8bk`_k3o6H$Lv+*>MGczA0?oG+X*k)y=B8 zlk*)5#S4Rr*7pAbHvqKfapmxmLkv5_*^YM(-#oljx+`3|YoX(N2R=A}lV?l2Pvl3c5yK}2-Flf1tY|2B_vkUoQz~yogeh>Bl{DeN;EqRkfxV$VON|p_r&%ehDofK zWJe@Usc_!C&#cz&9ON&kU7Q!Mr{9`7CfoTClKGlLJ>eK4*(s@b^Zyn1E>KZj>AGmW zsVWMpct1oaUJ9ZpzQtFNhXRUNl%NR_OF$%|U={eNnA3E6j~h&~b_pN+udZjj#ACo5L;Ic4 z_pzJyJAd6a_S)1u%HCHM$8~R<#5E|?y;m70poeqjswI7lF`ITRsV{arb|=Po7yiNP zi1IE>9B`;wCQcf=UEoNf#;V5mr2SlsNQWmui(6FQI@3iT_gB4x%?sI=$IjkF#dXc`CiyY1hM(GR;&pi9CEyNxQE|H8i4s z9(?0D>Z2hO)2m7<`PCfWCV>wOUXAVXoIlZ4zS+RES}i$LBx7%X7{endA@#Xr_vBKo6< zh!0S>+Nnw@V3jD)9bgetndM+3q+wD-}bS6w>wIQo^23pl!?)mNP?4i}M7X z`x%wUII!R0C-YuZneGrT#26vJA6Fv@BqbOc0EAFE)Ju$hIf7ckN79gB16;Ce#!)ow zC<;3k%%*40q%WFIUldATJhJsuXZj~uZmwpnKUY5<2QcNLpT~`#5|-^0a+`0a?t%kc zcK)Bz3xvXhq4XZX)iXOk|H2dJpO{Du%`ZRGG*b6zO4g+!+05m_sq?4C4hyBb1W)UX zXWz7EpRlV#=-4mh_` zI0sPFz_I49JMO)-_u}43bJ($3ehOpsOqXmIZ3xb}>1UflW$UL7+$ekOYSHfkAZ&m8 zblCa?@e?42Svd=)9fU=3>f)&n)5FV~KTLmn&(!WWR(+WM(emc7W0%M$sr(0p!jG z7fU1mN?Nh&41Idv#& z&+&LzOF@q&tgK~d-nsSR)CN-R1Dts#b;)$y_b8Z(yp4rseo)u z%{cWttC5c5OSZ3wwtymXiiqZK;Bx8z-|@Xbs>B!rTK6B#4*Vm&Bn5UOmDm9!=TmRf z?qVVPHK#t`dUpY82%tv2N)-jUB(qUPRrp4pmLPTUpxCWhtppS=&XE(0M(l<~kO+8U zv0^s5qNrySZ%%3PqJXR}U9Fx4XC+$!t-5J32AWT6HA)K=qT|VL-71SxK|ud4vk<6N zTL=tT5U4u}gHR^aa;OCJ2+Tw4G5u)^&XkDSFIzpisud26gj*%g&gOp*%+8JVW zWz^+id`aM6dV`S_>{i7bjY)U^7u~pwl6#a) z;%?Dkm@36LPI1uGskRrU_F{7+#UcaAVTCt75OsPXpCs8b2#r}ko++ezDC}d}n4QOqm?>N}UARgpSPkG?WTsn`1px<{!+__#6g|t;U%71R|$U)$I|A3mIcL%B^ei@M z{&#wytYSkoYUxA?P203?aIQ?5IsX^VJ!oYYCss3(q%11pDfZo9amMJMbtH`FBau@6}^MN&U^V2FxcZnPfl*g$ly7%(yG2 z-HNuuT{Yw0I_=(i(_Meg!NRqSFZl6}n<=HUxdj)VI{(y6?yBkBRg+zB^}OCQH5AIN z9o;J1_m)nlmQJ)y)K7LJicTHb*$Rsq!l{k3scBaz{&mj3@fF@f;j7(F$0;kR0@-W5l`+bRhAt_PMI z%yvj?dXi=N(j?afX*tWSMZQ0n8*KvX|Azlhry!F;LMluEk+K8kep`!dZzfKEBXd4G z2|X9HJ!(3J8>s;dfIKC&+5mK8B%qtNDCQ#?g;2|CM47~;Bahi*S%bHr4O9DLcprk# zVZWmjE$JznY{?**5!$f8(_C2Cbz)kq%jGDJh^i^eu*EjRv-5O23^Wld<+hLuqAAK< zlJzcJ^KB~(ZG|+&fh5Ph>^_R&W)JR>f+?3kWl#9*y>K$;1gc{D%%zNr*)<0GP)kxo z;Hbgor1TjkLL58<*&`33!6kQWcIRlF6{czuFztz3J&D_j$@(zMbkq4kL?RN>^ns+!s08!7e~?ou)8nJ7A+My| z>Dezmb?&K|^d-~jB>n2S+%q{8O5ZrL6%e!O*SeNi@pP8TIfk74u2x>}`eV~6xUx4`7FP{U!bl(7^`OgV%{{CtJ-ZIqN`yM?3ox!98>3xg(7@$8C z$7c2YWMH5WUy^q}8`uP&6F|ZJZF+J}8`R_OB zA?ah{zWa;8-!LZUel|#oeJ8n)Hi~*9A;VQrOVQWI zqiR1S+|4J*86ZnpLTNDTA(lk)BCHg!dl^;=*eO&Qe~?Z`!bQ~4boX6$W4m--Q{0!& zgo&y=^oB8;q#cuK4U`Un7v4y9p9Xt^vLR%Yg@mebC0!8c5Z$AN=?jQ^v9+O3WVRr| z?OrIQMEF#!RX%s@YE|d?Pb_wOx(9%bg410)y8S)fUVk@mkjl%; zgC*^OMJ2q1(^Xz?RrR*g_CVQ=DsQc)zG{2@foqYDB;95V``YX=|-fP~OxQ3778a{UB-?G}z+Mc%w@Bv&MvMwL4 z`ZOzhq~h$FGiydGzFR4n3dRjc_^?oSBsz_4cW+^*8O6ge$YXXe!pUP|aV>bv4s{d{ z94wkI*JF6*`bc&!DDu)qj)5G zG6oT)W6x6I?3Y;Kswz^dXM;eDC97HLBHx_7NT5JT5h+~2n8-`0HctCr4jBe z-HrhWOI~35fLnD30DINh?LKj+n~?)_?v(owVhf@$R+mbV$9F*HUcVdz z8DReeAPaJRFvsB{EL>jr!&MVbS7M^p#LM!=;bSrHM!XCLSWBoYh3OqHcp0Hj49ppzJ41U;O$2YQg3zLvXbEP<^n0W@U*8 zYBZ#LJ!Hs?whj0Wc9&}}al|-0+H_AH&TSV>esN;YOH4l5=slFlkEBUoYgBBS&++9d z(ny1OIYiFp_iu23*fhyQkVcZRG~aM6E#+8#qMbdo z9D*Lmqp{4fC!n(O6Xg0(a5cO-Nxhn2w@A(7C-Y@a<`HoxVrMCrILPAy9!c9zL|G(B z)jT|E1Z$dQxCVh!0RrmgAs&Klbmfa1X6NON>wmoFLgo3&@xh6zpPnGRI{~)Mi1jd< z`ga)$u^_bq)tl;j*Nx<1y)8pG`oDvm5A9HY4NCnt{3*bVStPYVa&D0AlzN~nwvmtn z^>ExJwq|$^$*4qtaCUepWQs~`Gi{b!A|;~onZ#$RcAEJhjEUVOp@5X%zZ_B6G_|8z z5@HAe$bAG_Sfr3pBQ-6D+n&d=6uLu6$S`1nsLIStL31E%upsn-mPFX%v8wf#aD1e` z#0KL}kwtDL%rIN9U1Ij1E}$6YfB`Dtga;k*b%u_DhdTy*gNF*9dB)9WM`;enbl8u1 z$j_HGnvD_lkd`f+JQ8e%YD?KfM-X}rhm@3X^%sjP8q9k5bGWOBI9XKn^?AhQvc%`b z5LoO!j+u8()Osqv2~h1N3?H69?LUaPPjw$G=gq*)m^uh;?-Zfw^Ru7?%7cg%3xDx$ zKWihpCcZ}0+f!L;d?Rs*K<;QU2MP5At9#QPYZpVDX-ApS7w26sN5vm~lfWQcubP()pi zWDq8P$IwyVaiAU1Qq~awe12CTqOb5LVo-Bz(KtSJk_LtG#Str&N2sML2&Tm}r-}mp z7jevU8jxp@?qkxC-{L%%?^8QT$Q))O4g!%xd9r1{*(`Piw?f(hJO0yWPmi3Q1zu7; zzy=ZJ2A~_E)Y8}1Os;%u&FgDmcJ#}(tKJWuy!PaK9iff8NBkos2tWYz6K76v^mctx zvwpUD*LC=b1J$nx_BX`~$5&!WE1CD%y!m4=LpS6l+)lD5xNfKGGL}uI2x+TsXX#uy zcXbI2)ig&1;SQJjrZY`r6&%R@rlSOQUCHSLY|iBgQOi0_4&QL1Cq4b6C;B9ms!a3- zb0w|JmFK~$V)x)gPj$eb^{$t-s`j+qOu1q~ekmBto(JF@)d1Asab!1uy@a>$r*u9(4_KLVKkutR6^+Xkl$mLYcW1*N3+` zEU@)M1{O9AQubp>(dbABIvQotbHLCDlN6FS4;cpZGG4JB)~m+g60vuV_&WXl6nW$T zj8_MTh?$9+JlXXCgK585@eIEO)snJCfkcdSrz$t&y`~+)1a#IEM zv}XNVCtg4C*6G(z3pIi8n!%8BXnM`yWd7*FAKJ&N#*W^|S@Mx<2~JHrhXm^ouz-h{ zJoz88Dx)!gO(>5B?KefEft?4|+<*wz<-y>2^Z>}D0U+z?o#w@NbVx3z+@{?m04cEJ zgMgYt+C)9Q`-nPq+<+1!;pc67=oJXSV6Qp2hISXlW5Ql0C!ix`fN*4J1;0@)m;R2 zWtvB{=GU#Y8tB3V`!Q9WrE=Bvg&U8oFBc;*U@9RiL9Ye!5hg?NniV^}IioIEccjvL zd2Bmez}M8gS9*RljogSYL~$d~Qh0gnEd(hL?*-nD+7BPCkY%LbANkwP=L{!B< z6evQH8+vsx`j_2n;J5&BB)#!`UE?owvOzlnFa6%baJ)c7DiRrDSo@(Vk)#X|`VA89 zF=x;Tu&3KkLI|Qkaiee0PMo#?v&6X#xA|4XRiz;yHCy}46Bt@`oCo2jKEHMb4MD0EO^!R~PDlcWXf9=Ne@ zAk=zH*zFgJ0-=n-QPX`mL#I$+5_DlGx#Dc#Y~XqM*Nn3w?5q&6=7f36CznqaO&*-` zPSpun)nQi+;G||;`O~iaaX^aRbd}w8>e3c{^|_I1^yNLPF4cWcr%%X=xH8EjRz%pm zmYY;lYI@gFUQ=%PRe5~PM)R*W=#fSd!s0r}RX7imc&L!LfhpxNGsjd5kN8gg&ALBg zVF2dh0*u_fZgRnwtIbu?XE&@f|GnzSFcuN z?tx;gcdug54679~sA24;I8D;J_l;IQy=uLdsQ{8?sLsplbo5F>QR%TJ$ZI^-g#V4! zgaNb1q^#0j*`S1d21PdQpOG4kO|C#0QYyO^gp<-fV7s7xr?-0URmra;0`} zFtufGi??=%yT-G#+3l%q-dvFOA)VoC5#e3P zGvHFiDU4sOq!T`Pf{`GH3*9A%XcG4X5kahkJ8#wSWl<~@@Y1Td0VIy@hhG;j0FRay zQh+`a{f-3^lJj1Fw-{7_!&WG)7vtU!poRE%B2TysB3TNlD}P-6ZVY1XL)DFYSdcF6 zFU4&+_T#cjdK@)p)3@~p+%&AiY=L<+CW=6=DvE&MUioI*l;@2nCU;z&|GN`H+Y`Ti z=6b)7yD#kOV1Wfv@+hW2?rciQo5sniH!R_l)xRuyr~DV?A>bAO7-6U2*%PkZgUA9C zMX(X?taO7 zM)lp`$*~(#P0R7U$af{0wj~8Lz=5gEm`zb3!wy&z2ydl$hn50bx&~vfV#$sdK~v?K zM0prwxF$6Z0DeTd2p4^cQXViiqUuCOwPG$TH3#dfq%W0%MKUR&@+~n!j?piqwd0Cv z5#(J>D|2QxSmcp)N$n>V$*S&l(sC~>d1BwfC(B9~a`%K@ED;g?Gx*FZsiQYgvXK%- zBr$uU0@Ppn44Ik2Cc4h(oL}H>7T{X^3>DE?oG1#7xMb2sJi|uGK9mWKwj=Ru{qyxO zd4~_3qi#h1snhjwO4_W`Mbfd+W8Vz|z69=TjprJFuwy25;dJUkkQ`@f5ULQNJT{!$ zFm4?8{WNZ(@TUnA$3iI;&(}cOm9_{{IW7E&Z#Me%Z1j^8nLi)z`{OVsGEAWL~9 z1SWyqMW{NkdvNsA3Wyu6cY~loe0pD!2qB>wGtyy;P}#rwu;%L)m6bAmpClhDIx_*u0bFNbO8PW>vi{w0Th?TuUbUJ0Y@rHF})?^nVogQLdC zqfK>ArWh6zS0QeSq(Hi308vWNn(?A&ZW*K^|Fri_T4lRjME~qTRPlAM?P`Y2)dri%lHDfoWJ$E}q2@cxm*zckvphVB_ zjSw`#IX@~!3S>N@ZVPpAc+HP^mQ{Qf`taCrvsApQa)|n(`L1}5*oSzI)ni{46_-lg zCsOO=>B{jBP$d6zAVJmNc5f1TT(pFt^_4b^svkl26t_3PdJ4-ZC{+dSgjj-@%^Qbr zG(nMsLC1%-2TX}h8=j&i$r~NJ#-s&UsK8jDC6~lJ5uc0A(d)w3E*qUg z+U1P#D{*?9UKe}!RC;%BVl#~0V>jjLcw4kr0L-bAT)(0a*HHA+Mu zG*BO~HvGEtXA0U|V~#`VsjP3Nt5~+{-zJDE4wEOLbGu@e@AbEG+4V9#f%Oj zx?(m@rYEbJggY^tH$i!S_1IxBB@eBcC)-Q#p7T^~DaSbTI@R13a^=K&`b5bBWv@~2 zeMR&x12$EwXuS=Osv~ogjAyb?R2&8|!j2T4Nqkm(uOxv4M22|I%4Cu1=db-b&u6{j zijC~!O?cE8MeH6{HyGPPg2IoLdV&M~kJTjOud2!5gEawSAz7p#b{qwAtaaB^0ei8k zrPf_v=cZ`swR`GYye;ni5&iz)vbm&>qGtO6`6biU2DAO#{!R-9L|ZC}Z&59k#NedB z=kiAY(uwRvTMRU5s2u3@V&qgz@JvbxlMs?p`my+lVHSCe--v$D{|;VSLqM4)6Roh1 z(hx}Q1o0KXq9ghMWVB8fZW!}@Py=AOUAsT5zo7q-K2*3tGX5{rkD3bg=k0}hj`MMt zNQW2j#kuYBl>n2xE<^R`pfc2YR;Ss#Mr%Sf|=r>y9?o?2=bdCu-Iw&6#J2%h; zGW+!(prnp|ve7~~Gm!+bD%>K4OR%rez3)=ehb*PZT<=N4noFfkWccT2&;NfY*HXIj zGdf~UAKi4sEd1FJ`ZfJzNHs(Bd;+P77!UQI;A4yASPKp^`v@iBVr`C2z)CJS2 z6r8(va@XbFo2hH>8g=;_!Bhfm#i2{i3}>{yvSNJbCzX)-Y<C01_oQ zQo_QSEiAdT;o^o@HiaBpE^Zq0O`dwI{q^=2+DCScoI2Y+(tdH%v}4O`+43u!E^i8# ztq&!_ZFK#(kBmp4j`sby=B6_rXQmT3Ts2G$-bk$e0@Cjqy#xqZP&DqDNE#~|vHU4* z-dJ6@q;)2(jbb&VzVM7BXhK|<@p2)nBJ5fYpPAbRYeF%@4=R~XhnxJQWpe1hCQJ>z zbL#C=*V96qcM7}q%j^?qYuMTb|JScR2ZmGXY5kWN9)#;-ka2y4%;l36lZ8V1>ab%C zL+&x-;=g|m(1dRNSD#yS3Dx?qB8eNmyqBX(osT;cSllw(IosIov8_Vks!88ui;%r8 z?5dn~rL(}M-7~JDkgMn~x6R1>mwTz2_pP6*npz=bZw|X2)4cD`fgyaxe^a$_#|j|# zB<(0Sy_;NBgde{uS>04*dViCvsnq)Glmtpwx|_=F*9)swHmxxH)?8)Txz_Oeax?P& zenos!zVY{)GMjRYf5oUv>?oe`Z#ckSA{8Q}ShItltAEvS0bW1DC!=*cF*&|6uP@MJd zkC4K;!q6Sgw6mGd% zB|;*Da^sf6I#u(!l|4~;Uk5!)(d2k!7g+$Anogj#{U--(9y=sO$QirgN%73;*3lZL z1^cNb8)}txLby9&IZ%C2HWta)O|!M_p2WqfjCiVsPRQewctjxYfCuJVv&Vf0$PEJ?BRJ2!PJ&J`Hmh62!e<0;(7vDR{s{Fxj?A4w6f>xMZX3%WAc6MJ=|!@I zfcH~vi+87|wkBfZN5Ig>T*zPu3o$AEIiX$!`@cV!vL8tM+iR=6ZV|sfu%g5tPi+#f z!u)5@{r&_xohZ@CqM1da{|k&pm4#(8az$*N{R5{uDDZ&)09|Hu@|!eUzd=8V$c;`!xkFOS%Yz`N0{@}QZoE<9~ zJNe`Cu%q--XVz>MfQ%;UetLTRsIYR6kimkWI@4ax4LcVSo!WG+X?zuohRM}f&I!=& zq$#7YzD3CJ3NG(#8lrSfSbn-;ynNCiq^uAeD}Xt4D>+xl+Y(By797>H={aZOM~tKS zv#z{xlNyM9a?w=FJA2>W`%cH(9oJhz8+QpCJc4JhkoY)!wV?-%MK)jY!G>!a2-&>( z(a7c^Zh9Kf<#Q%FCw2>&E2b7*KRjX$TOXf|PkM1*IKEKTEVIcmbrY7?UhVv#_ge1< z{nz@1*1bae(HpHtLv^73peB7j zUwP3(e+@{IU!dXgKTyb{iG;r;B7tU4MkHjDsuKyBX<&CTz+Z30N{HaBecX=+E=}>3 z;d83Sbno_YcTD(t#(HB)T5ehpl$DgNhH95g_YthSaHy^Ibsl1?C$n1#940qhcq~FA z4zE#B9Zu0*2gwJB)=@G|5xt4vuX*%fZ_mPj-PhHHwF;)2WHrj&ddNxzS);sEnH&oy zpztj}dWu_@!t?iVfO2woTDUQl_mrawLu?esbZ+XfG-d@ z(eLk~C^QUYrN1XoF58F_kor@9J+&>s)|1Q2vK(z0a-VG+6NwULEZ=i#jA~pqK zf65voVw1~^=qte^q1d2tF(T`j_ETy^${`CGQgf=f@&m5<=pb5>S*ZpSqv?TQio0&7 z2Oje7hL)X8)H5E)iM~zCGRfqEX&_M^xNeu=ECw+Yxy>c>ePlcuqXjn6h0T-@w6x8%bWFE&gc|n?4L-r=7aReA7be+Z5=lTJV-+J^ zpCqNjG64a^+s1c|KmF5vlbtt;*Zs2ks{ZPN_ig{wbfdT-G`~^E*?zqmK}u$t+oqe_ zuAc(N38W}Od#BKO@K*Z#v91d}=X);no$m{!mkDW$C$_wHVseli5@yzHonEswv~-(L zUVl9a7LKXg^&{1{O|ZTM2=JKcLfrYd3kl~Fgpv(YEmO6Kq8rMrp2@7A&a4k*HilBS zkLpoVSJK~}oSpCfeiy_p$r<;P;fVx!2VLMENpNHt`%e>H6RpCEI${3Skdq?F0J%Bt zOx)RoGYMk}<6W0}F7}M?o#>n_p3D%|HeDaQeo#m!JTMmb7XD4Z5l01z|{yIvSP?KDAxUCA4 zn3DAumzerdl%J?NKXv#+KuD_SG9?5H5C9*|K}hZx9Oi`kIR^7`=3IQ=3QyJBzQRe4lnR!1K22afR>3%AVKbRrN{eyZ44w%^B z{&LU?|3(A@7qe3xqC_Obxq@8K-QACb$c24|wSNOGKF;OncIgl4J^FTB;TxVU5iBlmyF2)m4TQ;cD;zn0R{^9$H71Y@;=go4H*aG2GG$=5z3C?d1Vg1@5RC= z>SutXBDn=WK`Tb!$!C~H1@WUmTHqi8#Fz#MZha6=V)5QCVIA4E&DSVKk?bf1KC2Ce+e^o?#T z@07VGMmpJb!?l|JQg9(+g^- zHpU_=-_*H~%9gSssj7Dx-fp;h?4Ndo*Vcy@)Q4KyIE3|g=55kn z$lCA5|lyT;5=>b#h6sqiO-VOIT6@t#YC zmkR$;`8iXt=blk#OZpdA+Nf_VbH-J0&rCndmQ5y2*3Xn}Vn0XpBdeabUd;WR3csy0 znJr)4)!{HufLff-+>&njFR4hb+2&DN>Y%ie(s$$IkVY-N#9>n>-z6UR(MvA@)WaEs z40%OokY{lphh@v579oDPiKApZWSM+R`p9UR%qD0rBrn5)6uWVlh+{YDWhWn$EzlEi zyXo$Jq*PmHjwGgNF?2JvY0Rc&YVDCtAGOS@rB#=}Fuq3lM0~_-9HYL}VDO8)jyQri5Q9`0J5v|aEDkPvmR>l@4}QEJYZLhG2D{So{J9{ z&fM;!y~nEje&4AQqLf7F0|F~y6kgQLjL{b#p+;W`mcu3g6mZOraD#FTN$yhk(Jm@ zCH|Hk^?OQ~VuUTp#5Bqh4~z`HEdK))K?z%8p2jVZa#(bLFFADP(CD`&457rb;jN#> zCx2{Byj6z4MpMhfWt&3ro2Sb*jr(4J(auc=j7&BS*9h^OZ-X^&&-q8w^Rse=1%)(H1IRB@_X(Q@hjh(pg z%=u^j{#!RQm!5GT$S$-OwllWT-J@H_R`D20udSXeesyCgf1Tj26e{-%$-bM`129y{ zTK1VKe#VqFZOR&3c+*ty)#s%;$73-3NY^>iV7_5?iiHWO>!ub?WnC?}y6bASki0Ey zt^a3hA{lL%on-!DD-RT*k6ul!U2eK&sM=7wM)zJ^R$ZC({WbA*CB|PH=G7G%e_d$C zu{KGv0e3%|BvE(@;3}Zw`Nl{T6X*1azvq-5(GGrc7JV2GA12~t0df*?V$#z(uWl))4i^VkVP}g;iyJPDrY8I%jCZ(=$QZTg|$yiiew*bQ+9+|-T zTY^;#nUNtVvgL{9MdIse;i2V=0SGQuPpzUDKw`{GqquH#E$6!T<6tfVRIvlm;Hz|6 ze3f#C)HRF1^KxTe#zPtWY<@h3G0v@D)x2dsyE$r)C8!Q8JUF@`hrUPOv=#;~mR znrRA7_h6W+t~5+Eyl49AE*1Qp^enJ{Y@)$0wmtl%jGE$zN4^ww5AP@zUg^Pi@?S!Z z+2~3aYaLvcr?r>t=`30l9t{7i6-k`P%mDOG` zUNQbm{xwrld-mAKvGWUtw+e}MA7{)9$8SD2IGTSU@b`fmRPj*3rVD`!fgj1gX7VC@pYqS#P?w zJPql)_8dyfEJ**V zz4H|ab+LO5C=%E`xH8DHTmlWBYwD2Zd0HvJjdBSt#vP-H3YlTt30cdE53sEe{1o^C zn)|_#8U~EzTt1A7#%>DKaq_d|H*YngWhLi0Nb--3L_bY|QX=;^wY$%NO%?bKvos{=q<3SRC(dKl z;^S2=qB376t!ruxJIwiD63EsqBSARHYZx4P0Au-2OeKtu8m`h@!CT1}tp0%|eUL); zxe?0(hFHXJ`FjBbeAEq@GjV489?Y`PD_4p4GF0XMp+PsBUkiv0pv4eb84J?oE&#@| zZ>W#pGAS-DiMdf43l!n5vhyNEF}S$kxS?$(1O&z=%9TQ?fpR{5G4A9))4cy0l8Am0 z<5GgIp`(Y75*9KFMeHLnq#Tqqj8AwOZR@{A33&`e>@AErt3BB}$lL&!GXc;RluuZ8 z)JDvQ{v$^a_0~^!lc^Li(FhB@i?2Iz;&TJ^(JOoeUwg*!dHN|~AG;YhgNv~1+(j3H z=Yubw4kxTXe;RY(v58}ob(f!hp=-qVeAoHY!*$aM>u2qb;c9RMXHT9vIc5nt^Jbj8 z1m~{t)``8s+HKcYTrU=inuWq$A31lyCMl_4v`$DW80$nbVwo*0y;OOzQYhba)etJI zhPhDBwH~2qpRoTxsIK!yVdscN01IMuf9gyfvy2`8apFWV2>e-KAvT<87(IT|k&B3J zE|316UFS?5E&lFPmjj`BD=r5n^2ZKds2{7p9GITB;zr^Ma6oa1w-Pgjd1VtdSL!d< zzj<(~?T!A>;yR&htB|!VlvpoV>u2K=ha0}So2E*uQU3h-IZC0cq#?yNk3q@}X}V4e z$(B#KQZC@>>a{$+Jr3cDr~R(!o{X4pjEK43d>(7>9Cdp)icWhN6%d8SZc@1rK+9Iq&3HoWmal6^Iecr*k&cwT>Tc-K zVmEl}<|@6>?v}1=dA2{iB&9Ffi=|RvO`qp+uN29FsPmc1`4wcaM}&q+W#f-~5TFYp z?RKIL*w$Ps9VvT|BW21j>By7sDUd|r%ZXj!R^Q8$e>{w0y!nMHD+wF^f|X z8nO>}tFd{RR2OA0J!!qZLqB8~GIq%9Ja7`1OZl3mj+ahgPprIjH+%}*^iOJ4_*k!z zvhJ}o6O|ORd5S#?q}Bj&rj=u%l0(V~q4IjEptjED4bri)mvVAaN>-Ry+sUMfZEep< zc~lA$+kjOmxm86LK0uO{s1*hZNj6}I;h(YjG4Ua)ohOC-MYvGB)rL`p5kC+Q|AA^% zCh-s@)I4}rYGw4Qo0E^sNKyM~7WTB=j9k?$O&(KtJ_yCF((~GWp!sZZRXKsZ|K(PP ze0LFhg7#=E^(<uX~EY$5!fPF1c9I2*x+;fX%woOFaz4oJt!UK)`x zr3~=z?4fTrpWeJF4Xb*l~y3?Sv<^cJN>-_gd!brf?+$tY*3&?gZ*vLx?~CG7(rhX|AinW z;EbT<@#0n!ZXJwR{oTig09@C_+?xTgCrU_Fd>;b&!Nt07AZS^#{_7WL-C*33C5(IYUqibDbHvCxU_04afbRj^mT+ME zx=Udz@=KZo8LP-{#uF+0K(GvGr;NYk&u{|*u#0oKCTip=B;^Got z0F8L0yENb~E@5mwX$aXbApZsslBqOMYDB!PK_Cts0WvE|0+@9oi!t2fx1qn8cThyX zB|12WJJsVE^cjrFz~z-Ac{FdOAkx8X zMDMS{V=z99P_#x&FmW5ix!>U&Y4voAI0JqV$C30S7)lQcvgetQm$ZMwy4PB+?D^TA zN#AQvFa|JUgFX1+j)MR>4i+!Ph*(O$m-hD`?OoD|a8!p6^<&H|;ayNp*Zg)oD3~EZ zLt^9(`i^#XLuK68g)v21$Q>7-Nz61HM3SXEihU_rFWS6GH*tfIniYZ>rnLP!}TB zK=m`q9Q6A?#9MO{rvkiyA4ZliU!`*|(NRAoCG?Y-^t?gNgOqVJQy*tS{nSFOw{3y*|=!kM8nem zueg&5B{DD^hjq7XNi#P0M>h9sZNEA+Q}@(#-BV%P;o&W_Cd+d>zrAy`;kCr!ong}& zh=*qE^QP_d#>EkEY3+bgIkDig z_=K|?&uske=HZ$@k$ySjBU8pLXELePU&$b2IbrpaUle*U<*W;N=kq4guFStYe=;yt z9V&fnrgYnM>9*@B*E>U{o|&wca8}DKKp3~3**5yE@uBe^A+7wTW9c2UE_sI@3Uhnn zNY~lJXAaNUvctCQG2bl+@J>B{YAoryr$Mh@aD3mGL7saLWtXHBjNm9B^1 zgpjr|?AV0&Hpf4=>N(c~cM$}9pD4OgdbxDc`_|sq_lD9p-*9dIzJW3vK6iM$=+dH#i$dv3 zZn&0w-*6`$?R48ARb^E;-aYF`nQ`PzJMzZuVMp052gX#`Q8MdHx?jv|aK8-Qi~%6n zvi}{h6^=S+tCH4zd9Ot00tBua@M7ttZQ^MmonCh~CG~zh?w`gRSdiz>_v7#jjV0LT z|2aUc%!i2pu`>T=S*mxw>9?s#-U8Dfo%-Eb=07G?dKa36_$m{AOlNN?!Y*u!_h#F| z3k!Fr$K9|~t{Z83I!-szaaR29MYf2wql0Fkj*imhZ2j^78dH&fhLRgJ-LdKWkMuL1 zepa#zl+Xgpf78l>##^m{{c$TV^7OVg{lMLGSo+r_N8yzJHuDI}h{^ z_ICmUshm%!e$0WGmg$lWYs_{mPbhj3J^nZJP!duxvU31SGhcnkOWIE?g`dpz{6&;h zP(mUKKS>|_!<3{^A=#9$#poqEx z1U`>jHyr8j8tUubbnz^;eWJtv zgimyKb|&Q$9sZ|%s$2N4I>$fjmVIVPn6YGhWXS;JO2|?%Z2T-~#c<-REo<09-(th_ z4MIj~$gybH_E~b_c&m`KcsTJ3i(bFr3#(qAM@g;z3#(b5{e?@fuhzpEN52#HCzRe_ zdswdrx+UC+ldjiqN1p8l{pLG5O261=GU%K1U)baHx%X3#nJ7P{UmUU_TJC*U zEu5+D=qUZ7*#ak&FEU|i`$Z+0vwV>VjpY|bBuu~EV1POCNHA>7nl;5ex9!<&XT0BS Py=h9t>|@GiE%^Tet%JzW literal 0 HcmV?d00001 diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py index dd00dd23..e343ac5c 100644 --- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py +++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py @@ -41,6 +41,13 @@ run [--job J ...] [--full] [--table T ...] [--keep LABEL] list [--job J ...] [--table T ...] release [--job J ...] LABEL + restore [--job J ...] [--table T ...] + +restore brings back the tables this node should have, after it was lost, +replaced or re-imaged, or on a new server: from the cluster's live replicas +where they have a table (cluster in the config), from the backups where only +they do (see Restorer). Run it on each node; it leaves tables already there +alone. A researcher runs it on a one-node cluster with a read-only key. After each table, the result goes to a Prometheus textfile (metrics_file in the config) for node_exporter: last success and attempt per table, whether @@ -193,6 +200,13 @@ def exists(self, key: str) -> bool: return False raise + def get_text(self, key: str): + """The object's text, or None if there's no such object""" + try: + return self.s3.get_object(Bucket=self.bucket, Key=key)["Body"].read().decode() + except self.s3.exceptions.NoSuchKey: + return None + def put_empty(self, key: str) -> None: self.s3.put_object(Bucket=self.bucket, Key=key, Body=b"") @@ -503,7 +517,8 @@ def wait_for_clickhouse(self) -> None: log.info(f"waiting for ClickHouse: {e}") time.sleep(10) - def wait(self, op_id: str, what: str) -> dict: + def wait(self, op_id: str, what: str, done: str = "BACKUP_CREATED", running: str = "CREATING_BACKUP") -> dict: + """Poll a BACKUP (or, with done="RESTORED", running="RESTORING", a RESTORE) until it ends""" t0 = time.monotonic() last_log = t0 failing_since = None @@ -528,9 +543,9 @@ def wait(self, op_id: str, what: str) -> dict: if not rows: raise RuntimeError(f"{what}: gone from system.backups (server restart?)") st = rows[0] - if st["status"] == "BACKUP_CREATED": + if st["status"] == done: return st - if st["status"] != "CREATING_BACKUP": + if st["status"] != running: raise RuntimeError(f"{what}: {st['status']}: {st['error']}") now = time.monotonic() if now - last_log >= 1800: @@ -691,6 +706,235 @@ def release(self, label: str) -> int: log.info(f"released {n} chain(s) pinned as {label}; the next run prunes them") return 0 + def restorable(self, only: list) -> dict: + """{table: (key of its latest complete backup, its CREATE statement)} for this job's tables""" + out = {} + for table in self.store.subdirs(f"{self.store.prefix}/{self.db}/"): + if not table or (only and table not in only) or not self.includes(table) or self.excludes(table): + continue + chains = self.store.chains(self.db, table) + chain, backup = latest_backup(chains) + if not backup: + continue + tprefix = self.store.table_prefix(self.db, table) + # An incremental stores the definition only if it changed: the + # newest backup of the chain that has the file has the current one + create = None + for b in sorted((b for b, ok in chains[chain]["backups"].items() if ok and b <= backup), reverse=True): + create = self.store.get_text(f"{tprefix}{chain}/{b}/metadata/{self.db}/{table}.sql") + if create: + break + out[table] = (f"{tprefix}{chain}/{backup}", create or "") + return out + + +def engine_of(create: str) -> str: + """The engine of a CREATE statement: MaterializedView, ReplicatedMergeTree, ...""" + if re.match(r"\s*(CREATE|ATTACH)\s+MATERIALIZED\s+VIEW", create, re.I): + return "MaterializedView" + m = re.search(r"\bENGINE\s*=\s*(\w+)", create) + return m.group(1) if m else "" + + +class Restorer: + """Brings this node's tables back, after it was lost, replaced or re-imaged, or on a new server. + + The cluster's live replicas are the source wherever they have a table: + this node creates it with their current definition, and a replicated + table then fetches its data from them. The backups are the source for + what only they have: a replicated table no node has (the whole cluster + was lost, or a researcher's own server), and the local tables this + node's jobs back up (data1's). Join and EmbeddedRocksDB rows come from a + peer that has them, else from the backup's export. Views come last, + after the tables they read and write. Only then does it wait for the + replicated tables to fetch their data, so the node can take inserts + meanwhile. Tables already here are left alone, so running it again is + safe.""" + + def __init__(self, jobs: list, cluster: str): + self.jobs, self.cluster = jobs, cluster + self.ch, self.db = jobs[0].ch, jobs[0].db + + def peers(self) -> dict: + """{table: {"host", "engine", "create", "zk_path"}} as the cluster's other live nodes have them""" + if not self.cluster: + return {} + c, db, me = sql_str(self.cluster), sql_str(self.db), sql_str(self.hostname()) + # Each node reports its own tables and replica paths, and the other + # nodes' rows are the peers'. skip_unavailable_shards: a node that's + # down doesn't stop the restore. + rows = self.ch.rows( + f""" + SELECT host, name, engine, create, zk_path FROM clusterAllReplicas({c}, view( + SELECT hostName() AS host, t.name AS name, t.engine AS engine, + t.create_table_query AS create, r.zookeeper_path AS zk_path + FROM system.tables AS t + LEFT JOIN system.replicas AS r ON r.database = t.database AND r.table = t.name + WHERE t.database = {db} AND NOT t.is_temporary AND NOT startsWith(t.name, '.inner') + )) + WHERE host != {me} + ORDER BY host, name + """, + settings={"skip_unavailable_shards": "1"}, + ) + out = {} + for r in rows: + out.setdefault(r["name"], r) + return out + + def hostname(self) -> str: + # a literal: hostName() in a query on the cluster runs on each node + return self.ch.query("SELECT hostName()").strip() + + def macros(self) -> dict: + return {r["macro"]: r["substitution"] for r in self.ch.rows("SELECT macro, substitution FROM system.macros")} + + def drop_stale_replicas(self, paths: set) -> None: + """Remove this node's leftover replica entries from Keeper: a node that + comes back empty can't create a replicated table while Keeper still + lists it as a replica. ClickHouse refuses to drop an active replica.""" + me = self.macros().get("replica") + if not me: + return + local = {r["zookeeper_path"] for r in self.ch.rows(f"SELECT zookeeper_path FROM system.replicas WHERE database = {sql_str(self.db)}")} + for path in sorted(paths - local): + try: + rows = self.ch.rows(f"SELECT name FROM system.zookeeper WHERE path = {sql_str(path + '/replicas')}") + except RuntimeError: + continue # no such table in Keeper + if any(r["name"] == me for r in rows): + log.info(f"dropping stale replica {me} from {path}") + self.drop_replica(me, path) + + def drop_replica(self, me: str, path: str, wait_s: float = 180) -> None: + # The old server's Keeper session outlives it until it times out (30 s + # by default), and until then Keeper still says it's active. If it + # stays active, another server is running under this replica's name. + t0 = time.monotonic() + while True: + try: + self.ch.query(f"SYSTEM DROP REPLICA {sql_str(me)} FROM ZKPATH {sql_str(path)}") + return + except RuntimeError as e: + if "because it's active" not in str(e) or time.monotonic() - t0 > wait_s: + raise RuntimeError(f"{path}: replica {me} is still active: is another server running as {me}? ({e})") from None + log.info(f"{path}: replica {me} still active, waiting for the old server's Keeper session to expire") + time.sleep(5) + + def zk_path(self, create: str) -> str: + """A replicated table's Keeper path from its CREATE, with this node's macros""" + m = re.search(r"Replicated\w*MergeTree\(\s*'([^']+)'", create) + if not m: + return "" + path = m.group(1) + for k, v in self.macros().items(): + path = path.replace("{" + k + "}", v) + return path + + def run(self, only: list) -> int: + try: + self.jobs[0].wait_for_clickhouse() + self.ch.query(f"CREATE DATABASE IF NOT EXISTS `{self.db}`") + here = {r["name"] for r in self.ch.rows(f"SELECT name FROM system.tables WHERE database = {sql_str(self.db)}")} + peers = self.peers() + backups = {} + for job in self.jobs: + for table, b in job.restorable(only).items(): + backups.setdefault(table, (job, *b)) + except Exception as e: + log.error(f"restore: could not start: {e}") + return 1 + tables = sorted(t for t in set(peers) | set(backups) if not only or t in only) + engine = {t: peers[t]["engine"] if t in peers else engine_of(backups[t][2]) for t in tables} + + self.drop_stale_replicas( + {p["zk_path"] for p in peers.values() if p["zk_path"]} + | {self.zk_path(b[2]) for b in backups.values()} - {""} + ) + failed, fetching, done = [], [], 0 + # views last: they need the tables they read and write + for table in sorted(tables, key=lambda t: (engine[t] == "MaterializedView", t)): + if table in here: + log.info(f"{table}: already here") + continue + try: + how = self.restore_table(table, engine[table], peers.get(table), backups.get(table)) + log.info(f"{table}: {how}") + done += 1 + if "fetching" in how: + fetching.append(table) + except Exception as e: + log.error(f"{table}: restore failed: {e}") + failed.append(table) + # Every table exists now, so the node can take inserts while its + # replicated tables fetch their data in the background. + log.info(f"restore: tables ready, waiting for {len(fetching)} replicated tables to fetch their data") + for table in fetching: + try: + self.ch.query(f"SYSTEM SYNC REPLICA `{self.db}`.`{table}`", timeout=24 * 3600) + log.info(f"{table}: fetched") + except Exception as e: + log.error(f"{table}: fetch did not finish: {e}") + failed.append(table) + if failed: + log.error(f"restore: failed tables: {', '.join(failed)}") + return 1 + log.info(f"restore: {done} tables restored, {len(tables) - done} already here") + return 0 + + def restore_table(self, table: str, engine: str, peer: dict, backup: tuple) -> str: + replicated = engine.startswith("Replicated") + if peer and (replicated or engine == "MaterializedView" or not backup): + self.ch.query(peer["create"]) + if replicated: + return f"created like {peer['host']}'s, fetching from the other replicas" + if engine in EXPORT_ENGINES: + n = self.copy_rows(table) + return f"created like {peer['host']}'s, {n} rows copied from a peer" + return f"created like {peer['host']}'s" + if not backup: + raise RuntimeError("no live peer has it and no backup") + job, key, _ = backup + sql = f"RESTORE TABLE `{self.db}`.`{table}` FROM S3({job.store.ch_s3(key)}) ASYNC" + out = self.ch.query(sql + "\nFORMAT JSONEachRow", settings=job.settings) + op_id = json.loads(out.splitlines()[0])["id"] + job.wait(op_id, f"{table}: restoring {key}", done="RESTORED", running="RESTORING") + if engine in EXPORT_ENGINES: + self.ch.query( + f"INSERT INTO `{self.db}`.`{table}` SELECT * FROM s3({job.store.ch_s3(key + '/data.native.zst')}, 'Native')", + settings=job.settings, + timeout=3600, + ) + return f"restored from {key}" + + def copy_rows(self, table: str) -> int: + """Copy a Join or EmbeddedRocksDB table's rows from the peer that has the most. + + Each peer is asked on its own, with remote() and the cluster config's + addresses: a query on the whole cluster fails if any node lacks the + table, e.g. one not restored yet. ClickHouse hides remote()'s + password in its logs.""" + hosts = self.ch.rows( + f"SELECT host_name, port FROM system.clusters WHERE cluster = {sql_str(self.cluster)} AND NOT is_local" + ) + best, most = None, 0 + for h in hosts: + src = (f"remote({sql_str(h['host_name'] + ':' + str(h['port']))}, `{self.db}`, `{table}`, " + f"{sql_str(self.ch.user)}, {sql_str(self.ch.password)})") + try: + n = int(self.ch.query(f"SELECT count() FROM {src}").strip()) + except RuntimeError: + continue # down, or doesn't have the table + if n > most: + best, most = src, n + if not best: + return 0 + # Read them, then insert: a Join table refuses an INSERT whose SELECT + # reads it. These lookup tables are small. + rows = self.ch.query(f"SELECT * FROM {best} FORMAT JSONEachRow") + self.ch.query(f"INSERT INTO `{self.db}`.`{table}` FORMAT JSONEachRow\n{rows}") + return most + def not_backed_up(tables: dict, here: str, jobs: list, excluded: list, elsewhere: list) -> dict: """The tables of {table: [hosts]} that no job here backs up and that @@ -770,6 +1014,8 @@ def main() -> int: p.add_argument("--table", action="append", default=[]) p = sub.add_parser("release", help="unpin chains pinned under LABEL") p.add_argument("label") + p = sub.add_parser("restore", help="bring back this node's missing tables, from peers and backups") + p.add_argument("--table", action="append", default=[], help="only this table (repeatable)") for p in sub.choices.values(): p.add_argument("--job", action="append", default=[], help="only this job (repeatable)") args = ap.parse_args() @@ -810,6 +1056,8 @@ def main() -> int: except BlockingIOError: log.error(f"another instance holds {lock_path}") return 1 + if args.command == "restore": + return Restorer(jobs, conf.get("cluster", "")).run(args.table) rc = 0 for job in jobs: if args.command == "run": diff --git a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 index 01c7b3ca..083efea6 100644 --- a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 +++ b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 @@ -12,6 +12,7 @@ "keep_chains": clickhouse_backup_keep_chains, "max_backup_bandwidth": clickhouse_backup_max_bandwidth, "exclude_tables": clickhouse_backup_exclude_tables, + "cluster": clickhouse_backup_cluster, "check_cluster": clickhouse_backup_check_cluster, "backed_up_elsewhere": clickhouse_backup_backed_up_elsewhere, "metrics_file": clickhouse_backup_metrics_file, diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data1.json b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data1.json new file mode 100644 index 00000000..8084c21a --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data1.json @@ -0,0 +1,36 @@ +{ + "database": "ooni", + "cluster": "oonidata_cluster", + "clickhouse_url": "http://data1:8123/", + "clickhouse_user": "admin", + "clickhouse_password": "admin", + "s3_endpoint": "http://minio:9000", + "s3_prefix": "prod", + "s3_access_key_id": "testkey", + "s3_secret_access_key": "testsecret123", + "exclude_tables": [ + "*_tmp", + "msmt_feedback_old", + "fingerprints_dns", + "fingerprints_http", + "session_expunge" + ], + "keep_chains": 2, + "poll_interval_seconds": 1, + "lock_file": "/tmp/data1.lock", + "state_file": "/tmp/data1.state.json", + "metrics_file": "/tmp/data1.prom", + "jobs": [ + { + "name": "data1", + "s3_bucket": "ooni-clickhouse-backup", + "include_tables": [ + "labeling_frames", + "event_detector_cusums", + "event_detector_v2_changepoints", + "changepoint_label" + ], + "exclude_tables": [] + } + ] +} \ No newline at end of file diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data2.json b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data2.json new file mode 100644 index 00000000..2f5fd062 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data2.json @@ -0,0 +1,51 @@ +{ + "database": "ooni", + "cluster": "oonidata_cluster", + "clickhouse_url": "http://data2:8123/", + "clickhouse_user": "admin", + "clickhouse_password": "admin", + "s3_endpoint": "http://minio:9000", + "s3_prefix": "prod", + "s3_access_key_id": "testkey", + "s3_secret_access_key": "testsecret123", + "exclude_tables": [ + "*_tmp", + "msmt_feedback_old", + "fingerprints_dns", + "fingerprints_http", + "session_expunge" + ], + "keep_chains": 2, + "poll_interval_seconds": 1, + "lock_file": "/tmp/data2.lock", + "state_file": "/tmp/data2.state.json", + "metrics_file": "/tmp/data2.prom", + "jobs": [ + { + "name": "measurements", + "s3_bucket": "ooni-clickhouse-backup", + "include_tables": [ + "*" + ], + "exclude_tables": [ + "msmt_feedback", + "faulty_measurements", + "url_priorities", + "labeling_frames", + "event_detector_cusums", + "event_detector_v2_changepoints", + "changepoint_label" + ] + }, + { + "name": "private", + "s3_bucket": "ooni-clickhouse-backup-private", + "include_tables": [ + "msmt_feedback", + "faulty_measurements", + "url_priorities" + ], + "exclude_tables": [] + } + ] +} \ No newline at end of file diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data3.json b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data3.json new file mode 100644 index 00000000..155b1e74 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/data3.json @@ -0,0 +1,51 @@ +{ + "database": "ooni", + "cluster": "oonidata_cluster", + "clickhouse_url": "http://data3:8123/", + "clickhouse_user": "admin", + "clickhouse_password": "admin", + "s3_endpoint": "http://minio:9000", + "s3_prefix": "prod", + "s3_access_key_id": "testkey", + "s3_secret_access_key": "testsecret123", + "exclude_tables": [ + "*_tmp", + "msmt_feedback_old", + "fingerprints_dns", + "fingerprints_http", + "session_expunge" + ], + "keep_chains": 2, + "poll_interval_seconds": 1, + "lock_file": "/tmp/data3.lock", + "state_file": "/tmp/data3.state.json", + "metrics_file": "/tmp/data3.prom", + "jobs": [ + { + "name": "measurements", + "s3_bucket": "ooni-clickhouse-backup", + "include_tables": [ + "*" + ], + "exclude_tables": [ + "msmt_feedback", + "faulty_measurements", + "url_priorities", + "labeling_frames", + "event_detector_cusums", + "event_detector_v2_changepoints", + "changepoint_label" + ] + }, + { + "name": "private", + "s3_bucket": "ooni-clickhouse-backup-private", + "include_tables": [ + "msmt_feedback", + "faulty_measurements", + "url_priorities" + ], + "exclude_tables": [] + } + ] +} \ No newline at end of file diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher-private.json b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher-private.json new file mode 100644 index 00000000..638a5541 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher-private.json @@ -0,0 +1,33 @@ +{ + "database": "ooni", + "cluster": "oonidata_cluster", + "clickhouse_url": "http://researcher:8123/", + "clickhouse_user": "admin", + "clickhouse_password": "admin", + "s3_endpoint": "http://minio:9000", + "s3_prefix": "prod", + "s3_access_key_id": "researcher", + "s3_secret_access_key": "researchersecret", + "exclude_tables": [ + "*_tmp", + "msmt_feedback_old", + "fingerprints_dns", + "fingerprints_http", + "session_expunge" + ], + "keep_chains": 2, + "poll_interval_seconds": 1, + "lock_file": "/tmp/researcher.lock", + "state_file": "/tmp/researcher.state.json", + "metrics_file": "/tmp/researcher.prom", + "jobs": [ + { + "name": "private", + "s3_bucket": "ooni-clickhouse-backup-private", + "include_tables": [ + "*" + ], + "exclude_tables": [] + } + ] +} \ No newline at end of file diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher.json b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher.json new file mode 100644 index 00000000..71551170 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/configs/researcher.json @@ -0,0 +1,33 @@ +{ + "database": "ooni", + "cluster": "oonidata_cluster", + "clickhouse_url": "http://researcher:8123/", + "clickhouse_user": "admin", + "clickhouse_password": "admin", + "s3_endpoint": "http://minio:9000", + "s3_prefix": "prod", + "s3_access_key_id": "researcher", + "s3_secret_access_key": "researchersecret", + "exclude_tables": [ + "*_tmp", + "msmt_feedback_old", + "fingerprints_dns", + "fingerprints_http", + "session_expunge" + ], + "keep_chains": 2, + "poll_interval_seconds": 1, + "lock_file": "/tmp/researcher.lock", + "state_file": "/tmp/researcher.state.json", + "metrics_file": "/tmp/researcher.prom", + "jobs": [ + { + "name": "measurements", + "s3_bucket": "ooni-clickhouse-backup", + "include_tables": [ + "*" + ], + "exclude_tables": [] + } + ] +} \ No newline at end of file From a9d90f3b59b644e97528af3bffefcd71f4beefba Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Wed, 28 Oct 2026 08:11:47 +0000 Subject: [PATCH 32/35] clickhouse_backup: deploy to data2 for restores, share data3's jobs data3's jobs move to group_vars as clickhouse_backup_replicated_jobs, so that data2, which takes no backups, can restore from the same buckets. --- ansible/deploy-clickhouse.yml | 5 ++++- ansible/group_vars/clickhouse/vars.yml | 13 +++++++++++++ ansible/host_vars/data2.htz-fsn.prod.ooni.nu | 5 +++++ ansible/host_vars/data3.htz-fsn.prod.ooni.nu | 14 ++------------ 4 files changed, 24 insertions(+), 13 deletions(-) create mode 100644 ansible/host_vars/data2.htz-fsn.prod.ooni.nu diff --git a/ansible/deploy-clickhouse.yml b/ansible/deploy-clickhouse.yml index b40c3ec7..7ca65ee1 100644 --- a/ansible/deploy-clickhouse.yml +++ b/ansible/deploy-clickhouse.yml @@ -15,10 +15,13 @@ # data3 backs up the replicated tables: they hold the same data on every # node, and data3 isn't behind clickhouseproxy and doesn't run Airflow. # data1 backs up the plain MergeTree tables whose data only it holds (see -# clickhouse_backup_data1_tables). The jobs are set in host_vars. +# clickhouse_backup_data1_tables). The jobs are set in host_vars. data2 takes +# no backups, but has the role so that restore can bring it back from data3's +# buckets (docs/Runbooks.md, "ClickHouse backup"). - name: Deploy ClickHouse backups hosts: - data1.htz-fsn.prod.ooni.nu + - data2.htz-fsn.prod.ooni.nu - data3.htz-fsn.prod.ooni.nu become: true roles: diff --git a/ansible/group_vars/clickhouse/vars.yml b/ansible/group_vars/clickhouse/vars.yml index 9b5ad668..6ada2f16 100644 --- a/ansible/group_vars/clickhouse/vars.yml +++ b/ansible/group_vars/clickhouse/vars.yml @@ -424,6 +424,19 @@ clickhouse_backup_private_tables: - msmt_feedback # account ids and free-text comments - faulty_measurements # until its contents are reviewed - url_priorities # until it's decided whether it's published +# data3's jobs, which data2 also restores from: the replicated tables, minus +# the ones only data1 holds data for. Private tables go to their own bucket: +# R2 tokens grant access per bucket, and researchers may be given read access +# to the measurement tables. +clickhouse_backup_replicated_jobs: + - name: measurements + s3_bucket: ooni-clickhouse-backup + include_tables: ["*"] + exclude_tables: "{{ clickhouse_backup_private_tables + clickhouse_backup_data1_tables }}" + - name: private + s3_bucket: ooni-clickhouse-backup-private + include_tables: "{{ clickhouse_backup_private_tables }}" + exclude_tables: [] # Plain MergeTree tables whose data only data1 holds (2026-10-20). Remove a # table from here once it's replicated, and data3 picks it up. clickhouse_backup_data1_tables: diff --git a/ansible/host_vars/data2.htz-fsn.prod.ooni.nu b/ansible/host_vars/data2.htz-fsn.prod.ooni.nu new file mode 100644 index 00000000..3bd48afe --- /dev/null +++ b/ansible/host_vars/data2.htz-fsn.prod.ooni.nu @@ -0,0 +1,5 @@ +# data2 takes no backups: data3 backs up the replicated tables. The role is here +# for restore, which reads data3's buckets when data2 is lost or re-imaged +# (docs/Runbooks.md, "ClickHouse backup"). +clickhouse_backup_enabled: false +clickhouse_backup_jobs: "{{ clickhouse_backup_replicated_jobs }}" diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu index 34959203..f92846f4 100644 --- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu +++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu @@ -1,18 +1,8 @@ non_admin_usernames: [] clickhouse_base_path: /data/clickhouse -# The replicated tables, minus the ones only data1 holds data for. Private -# tables go to their own bucket: R2 tokens grant access per bucket, and -# researchers may be given read access to the measurement tables. -clickhouse_backup_jobs: - - name: measurements - s3_bucket: ooni-clickhouse-backup - include_tables: ["*"] - exclude_tables: "{{ clickhouse_backup_private_tables + clickhouse_backup_data1_tables }}" - - name: private - s3_bucket: ooni-clickhouse-backup-private - include_tables: "{{ clickhouse_backup_private_tables }}" - exclude_tables: [] +# The replicated tables (group_vars) +clickhouse_backup_jobs: "{{ clickhouse_backup_replicated_jobs }}" # After each run, report the tables on data1-3 that no backup covers; # data1's own tables are backed up by data1 From 85f175811b7197d806a012d312016f38bd7b7732 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Wed, 28 Oct 2026 08:26:16 +0000 Subject: [PATCH 33/35] clickhouse_backup tests: restore lost nodes, the cluster, and a researcher's copy With production's schema and jobs on data1-3 and Keeper: take a full and an incremental, then restore with the restore command after each of: data2 lost, data1 lost (its own tables come back from its backups), the whole cluster and Keeper lost (data2 first, from the backups, then data1 and data3 from it), and a researcher restoring the measurement tables with a read-only key, without access to the private ones. Each restored table must have the rows and definition it had at the last backup. Runs in CI. --- .../test_clickhouse_backup_lifecycle.yml | 35 ++ .../tests/lifecycle/.gitignore | 1 + .../tests/lifecycle/cluster.xml | 23 ++ .../tests/lifecycle/compose.yml | 49 +++ .../tests/lifecycle/keeper.xml | 11 + .../tests/lifecycle/macros1.xml | 1 + .../tests/lifecycle/macros2.xml | 1 + .../tests/lifecycle/macros3.xml | 1 + .../tests/lifecycle/researcher.xml | 18 + .../clickhouse_backup/tests/lifecycle/run.sh | 14 + .../tests/lifecycle/test_lifecycle.py | 319 ++++++++++++++++++ 11 files changed, 473 insertions(+) create mode 100644 .github/workflows/test_clickhouse_backup_lifecycle.yml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/.gitignore create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/cluster.xml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/compose.yml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/keeper.xml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/macros1.xml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/macros2.xml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/macros3.xml create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/researcher.xml create mode 100755 ansible/roles/clickhouse_backup/tests/lifecycle/run.sh create mode 100644 ansible/roles/clickhouse_backup/tests/lifecycle/test_lifecycle.py diff --git a/.github/workflows/test_clickhouse_backup_lifecycle.yml b/.github/workflows/test_clickhouse_backup_lifecycle.yml new file mode 100644 index 00000000..172dad40 --- /dev/null +++ b/.github/workflows/test_clickhouse_backup_lifecycle.yml @@ -0,0 +1,35 @@ +# Backs up production's schema with production's jobs, then restores after a lost replica, a lost +# data1, a lost cluster, and on a researcher's own server (ansible/roles/clickhouse_backup/tests/lifecycle), +# only when the backup script, the schema or the jobs change +name: Test ClickHouse backup lifecycle +on: + pull_request: + paths: + - "ansible/roles/clickhouse_backup/files/**" + - "ansible/roles/clickhouse_backup/defaults/main.yml" + - "ansible/roles/clickhouse_backup/tests/lifecycle/**" + - "ansible/roles/clickhouse_backup/tests/runner.Dockerfile" + - "ansible/group_vars/clickhouse/vars.yml" + - "scripts/cluster-migration/schema.sql" + - "scripts/cluster-migration/migrations/**" + - ".github/workflows/test_clickhouse_backup_lifecycle.yml" + push: + branches: [main] + paths: + - "ansible/roles/clickhouse_backup/files/**" + - "ansible/roles/clickhouse_backup/defaults/main.yml" + - "ansible/roles/clickhouse_backup/tests/lifecycle/**" + - "ansible/roles/clickhouse_backup/tests/runner.Dockerfile" + - "ansible/group_vars/clickhouse/vars.yml" + - "scripts/cluster-migration/schema.sql" + - "scripts/cluster-migration/migrations/**" + - ".github/workflows/test_clickhouse_backup_lifecycle.yml" + +jobs: + test: + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends python3-boto3 python3-yaml + - run: ansible/roles/clickhouse_backup/tests/lifecycle/run.sh diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/.gitignore b/ansible/roles/clickhouse_backup/tests/lifecycle/.gitignore new file mode 100644 index 00000000..e096425a --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/.gitignore @@ -0,0 +1 @@ +configs/ diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/cluster.xml b/ansible/roles/clickhouse_backup/tests/lifecycle/cluster.xml new file mode 100644 index 00000000..b7b723d9 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/cluster.xml @@ -0,0 +1,23 @@ + + + + + 0.5 + + + + true + data19000adminadmin + data29000adminadmin + data39000adminadmin + + + + keeper9181 + /clickhouse/task_queue/ddl + + + http://minio:9000/ooni-clickhouse-backup/testkeytestsecret123 + http://minio:9000/ooni-clickhouse-backup-private/testkeytestsecret123 + + diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/compose.yml b/ansible/roles/clickhouse_backup/tests/lifecycle/compose.yml new file mode 100644 index 00000000..70b99d00 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/compose.yml @@ -0,0 +1,49 @@ +# The lifecycle test's environment (test_lifecycle.py): production's shape, data1-3 in one shard of +# oonidata_cluster at production's ClickHouse version, Keeper on its own so a node can be wiped without +# it, MinIO in place of R2, a researcher's standalone one-node cluster, and the backup script in "runner". +name: chbackup-lifecycle +x-clickhouse: &clickhouse + image: docker.io/clickhouse/clickhouse-server:24.8.6.70 + environment: {CLICKHOUSE_USER: admin, CLICKHOUSE_PASSWORD: admin} + ulimits: {nofile: 262144} + mem_limit: 1500m +services: + keeper: + image: docker.io/clickhouse/clickhouse-keeper:24.8.6.70 + hostname: keeper + volumes: + - ./keeper.xml:/etc/clickhouse-keeper/keeper_config.xml:ro,z + minio: + image: docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Z + command: server /data + environment: {MINIO_ROOT_USER: testkey, MINIO_ROOT_PASSWORD: testsecret123} + ports: ["127.0.0.1:19920:9000"] + data1: + <<: *clickhouse + hostname: data1 + volumes: + - ./cluster.xml:/etc/clickhouse-server/config.d/cluster.xml:ro,z + - ./macros1.xml:/etc/clickhouse-server/config.d/macros.xml:ro,z + data2: + <<: *clickhouse + hostname: data2 + volumes: + - ./cluster.xml:/etc/clickhouse-server/config.d/cluster.xml:ro,z + - ./macros2.xml:/etc/clickhouse-server/config.d/macros.xml:ro,z + data3: + <<: *clickhouse + hostname: data3 + volumes: + - ./cluster.xml:/etc/clickhouse-server/config.d/cluster.xml:ro,z + - ./macros3.xml:/etc/clickhouse-server/config.d/macros.xml:ro,z + researcher: + <<: *clickhouse + hostname: researcher + volumes: + - ./researcher.xml:/etc/clickhouse-server/config.d/researcher.xml:ro,z + runner: + build: {context: .., dockerfile: runner.Dockerfile} + command: sleep infinity + volumes: + - ../../files:/role:ro,z + - ./configs:/configs:ro,z diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/keeper.xml b/ansible/roles/clickhouse_backup/tests/lifecycle/keeper.xml new file mode 100644 index 00000000..67ab93ab --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/keeper.xml @@ -0,0 +1,11 @@ + + warning1 + 0.0.0.0 + + 9181 + 1 + /var/lib/clickhouse-keeper/log + /var/lib/clickhouse-keeper/snapshots + 1keeper9234 + + diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/macros1.xml b/ansible/roles/clickhouse_backup/tests/lifecycle/macros1.xml new file mode 100644 index 00000000..9cdaa279 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/macros1.xml @@ -0,0 +1 @@ +oonidata_cluster01data1 diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/macros2.xml b/ansible/roles/clickhouse_backup/tests/lifecycle/macros2.xml new file mode 100644 index 00000000..7fa4e9da --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/macros2.xml @@ -0,0 +1 @@ +oonidata_cluster01data2 diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/macros3.xml b/ansible/roles/clickhouse_backup/tests/lifecycle/macros3.xml new file mode 100644 index 00000000..df113aa2 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/macros3.xml @@ -0,0 +1 @@ +oonidata_cluster01data3 diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/researcher.xml b/ansible/roles/clickhouse_backup/tests/lifecycle/researcher.xml new file mode 100644 index 00000000..bd2217d0 --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/researcher.xml @@ -0,0 +1,18 @@ + + + + + 0.5 + + researcher9000adminadmin + oonidata_cluster01researcher + + 91811 + /var/lib/clickhouse/coordination/log + /var/lib/clickhouse/coordination/snapshots + 1researcher9234 + + researcher9181 + + http://minio:9000/ooni-clickhouse-backup/researcherresearchersecret + diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/run.sh b/ansible/roles/clickhouse_backup/tests/lifecycle/run.sh new file mode 100755 index 00000000..c1c08f0a --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/run.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# The backup's whole lifecycle (test_lifecycle.py): backups of production's schema by production's jobs, +# then restores after a lost replica, a lost data1, a lost cluster, and on a researcher's own server. +# Needs docker compose, python3-boto3 and python3-yaml. With rootless podman, point DOCKER_HOST at its +# socket. KEEP=1 leaves the containers running. +set -euo pipefail +cd "$(dirname "$0")" +export COMPOSE=${COMPOSE:-docker compose} +export PYTHONDONTWRITEBYTECODE=1 # no __pycache__ left in the role's files/ +compose() { $COMPOSE -f compose.yml "$@"; } +trap 'rc=$?; [ $rc = 0 ] || compose logs --no-color --tail=100 runner data1 data2 data3; [ -n "${KEEP:-}" ] || compose down -v --remove-orphans >/dev/null 2>&1' EXIT +compose down -v --remove-orphans >/dev/null 2>&1 # left from a run with KEEP=1 +compose up -d --build --quiet-pull +python3 test_lifecycle.py diff --git a/ansible/roles/clickhouse_backup/tests/lifecycle/test_lifecycle.py b/ansible/roles/clickhouse_backup/tests/lifecycle/test_lifecycle.py new file mode 100644 index 00000000..729e878d --- /dev/null +++ b/ansible/roles/clickhouse_backup/tests/lifecycle/test_lifecycle.py @@ -0,0 +1,319 @@ +"""The whole life of a backup, in compose.yml's environment (run.sh starts it): production's schema +(scripts/cluster-migration/schema.sql) on data1-3, rows in every table, backups by production's jobs +(a full, then an incremental), then a restore with the script's restore command after each of: + 1. data2 is lost or re-imaged: it comes back empty and rejoins the cluster + 2. data1 is: its own tables, which only its backups hold, come back too + 3. the whole cluster is lost, Keeper too: data3, data1 and data2 are restored from the backups alone + 4. a researcher restores the measurement tables on their own one-node cluster, with a read-only key, + and can't read the private ones +After each one, every table on the restored nodes must have the rows and the definition it had at +the last backup. Prints each check; exits 1 at the first failure.""" +import fnmatch +import json +import os +import re +import shlex +import subprocess +import sys +import time + +import boto3 +import yaml + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.join(HERE, "..", "..", "..", "..", "..") +SCHEMA = os.path.join(REPO, "scripts", "cluster-migration", "schema.sql") +COMPOSE = shlex.split(os.environ.get("COMPOSE", "docker compose")) + ["-f", os.path.join(HERE, "compose.yml")] +NODES = ["data1", "data2", "data3"] +MEASUREMENTS, PRIVATE = "ooni-clickhouse-backup", "ooni-clickhouse-backup-private" +s3 = boto3.client( + "s3", endpoint_url="http://127.0.0.1:19920", region_name="us-east-1", + aws_access_key_id="testkey", aws_secret_access_key="testsecret123", +) + + +def check(ok, what, detail=""): + print(f" {'ok ' if ok else 'FAIL'} {what}") + if not ok: + if detail: + print(detail[-4000:]) + sys.exit(1) + + +def compose(*args, **kw): + return subprocess.run(COMPOSE + list(args), capture_output=True, text=True, **kw) + + +def ch(node, sql, stdin=None): + """A query on node, or with stdin, several (clickhouse-client --multiquery)""" + args = ["exec", "-T", node, "clickhouse-client", "-u", "admin", "--password", "admin"] + args += ["--database", "ooni", "--multiquery"] if stdin is not None else ["-q", sql] + r = compose(*args, input=stdin if stdin is not None else "") + if r.returncode: + raise RuntimeError(f"{node}: {r.stderr.strip()[-2000:]}") + return r.stdout.strip() + + +def tool(config, *args): + """The backup script in the runner container, with configs/.json""" + return compose("exec", "-T", "runner", "python3", "/role/ooni_clickhouse_backup.py", + "--config", f"/configs/{config}.json", *args) + + +def wait_for(what, ok, timeout=120): + t0 = time.monotonic() + while time.monotonic() - t0 < timeout: + try: + if ok(): + return + except Exception: + pass + time.sleep(1) + check(False, f"{what} within {timeout}s") + + +def wipe(*services): + """Lose these servers and their disks, and start them again empty, as after a re-image""" + compose("rm", "-s", "-f", "-v", *services) + compose("up", "-d", *services) + for s in services: + if s != "keeper": + wait_for(f"{s} answering", lambda s=s: ch(s, "SELECT 1") == "1") + + +# ---- the role's settings, as production sets them + +def role_vars(): + group = yaml.safe_load(open(os.path.join(REPO, "ansible", "group_vars", "clickhouse", "vars.yml"))) + defaults = yaml.safe_load(open(os.path.join(HERE, "..", "..", "defaults", "main.yml"))) + return group["clickhouse_backup_private_tables"], group["clickhouse_backup_data1_tables"], defaults["clickhouse_backup_exclude_tables"] + + +def write_configs(): + """Each node's config as the role renders it, with host_vars' jobs: data3 backs up the replicated + tables (measurements, private), data1 its own tables; data2 only restores, with data3's jobs""" + private, data1_tables, excluded = role_vars() + data3_jobs = [ + {"name": "measurements", "s3_bucket": MEASUREMENTS, "include_tables": ["*"], "exclude_tables": private + data1_tables}, + {"name": "private", "s3_bucket": PRIVATE, "include_tables": private, "exclude_tables": []}, + ] + jobs = {"data1": [{"name": "data1", "s3_bucket": MEASUREMENTS, "include_tables": data1_tables, "exclude_tables": []}], + "data2": data3_jobs, "data3": data3_jobs} + os.makedirs(os.path.join(HERE, "configs"), exist_ok=True) + for node in NODES + ["researcher"]: + conf = { + "database": "ooni", "cluster": "oonidata_cluster", + "clickhouse_url": f"http://{node}:8123/", "clickhouse_user": "admin", "clickhouse_password": "admin", + "s3_endpoint": "http://minio:9000", "s3_prefix": "prod", + "s3_access_key_id": "testkey", "s3_secret_access_key": "testsecret123", + "exclude_tables": excluded, "keep_chains": 2, "poll_interval_seconds": 1, + "lock_file": f"/tmp/{node}.lock", "state_file": f"/tmp/{node}.state.json", "metrics_file": f"/tmp/{node}.prom", + "jobs": jobs.get(node, []), + } + if node == "researcher": + # the published read-only key, for the measurement tables only + conf.update(s3_access_key_id="researcher", s3_secret_access_key="researchersecret", + jobs=[{"name": "measurements", "s3_bucket": MEASUREMENTS, "include_tables": ["*"], "exclude_tables": []}]) + json.dump(conf, open(os.path.join(HERE, "configs", f"{node}.json"), "w"), indent=2) + tried = dict(json.load(open(os.path.join(HERE, "configs", "researcher.json"))), + jobs=[{"name": "private", "s3_bucket": PRIVATE, "include_tables": ["*"], "exclude_tables": []}]) + json.dump(tried, open(os.path.join(HERE, "configs", "researcher-private.json"), "w"), indent=2) + return private, data1_tables, excluded + + +# ---- what a node holds + +def snapshot(node): + """{table: (engine, rows, checksum of the rows, definition)}; FINAL where merges change the rows""" + out = {} + for line in ch(node, "SELECT name, engine, create_table_query FROM system.tables WHERE database = 'ooni' " + "AND NOT startsWith(name, '.inner') ORDER BY name FORMAT TSV").splitlines(): + name, engine, create = line.split("\t", 2) + final = " FINAL" if re.search(r"(Replacing|Summing|Collapsing|Aggregating)MergeTree", engine) else "" + # a view's rows are its table's, compared there with FINAL + rows = "" if engine == "MaterializedView" else ch(node, f"SELECT count(), sum(cityHash64(*)) FROM ooni.`{name}`{final} FORMAT TSV") + out[name] = (engine, rows, re.sub(r"[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}", "", create)) + return out + + +def same(got, want, what, tables=None, empty=()): + tables = sorted(want) if tables is None else tables + bad = [t for t in tables if t not in empty and got.get(t) != want.get(t)] + bad += [t for t in empty if t not in got or not got[t][1].startswith("0\t")] + detail = "\n".join(f" {t}:\n want {want.get(t)}\n got {got.get(t)}" for t in bad) + check(not bad, what, detail) + + +def all_replicas_active(node): + rows = ch(node, "SELECT table, active_replicas, total_replicas FROM system.replicas WHERE database = 'ooni' FORMAT TSV") + bad = [r for r in rows.splitlines() if r.split("\t")[1] != "3" or r.split("\t")[2] != "3"] + check(rows and not bad, f"{node}: every replicated table has 3 active replicas", "\n".join(bad)) + + +# ---- the steps + +def setup(data1_tables): + print("== production's schema and rows in every table") + wait_for("MinIO", lambda: s3.list_buckets() is not None) + for node in NODES + ["researcher"]: + wait_for(f"{node} answering", lambda n=node: ch(n, "SELECT 1") == "1") + for bucket in (MEASUREMENTS, PRIVATE): + s3.create_bucket(Bucket=bucket) + # a key that can read the measurements bucket and nothing else, as the published one would + policy = json.dumps({"Version": "2012-10-17", "Statement": [ + {"Effect": "Allow", "Action": ["s3:GetObject"], "Resource": [f"arn:aws:s3:::{MEASUREMENTS}/*"]}, + {"Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": [f"arn:aws:s3:::{MEASUREMENTS}"]}]}) + r = compose("exec", "-T", "minio", "sh", "-c", + "mc alias set local http://127.0.0.1:9000 testkey testsecret123 >/dev/null" + " && cat > /tmp/ro.json && mc admin policy create local measurements-read /tmp/ro.json >/dev/null" + " && mc admin user add local researcher researchersecret >/dev/null" + " && mc admin policy attach local measurements-read --user researcher >/dev/null", input=policy) + check(r.returncode == 0, "a read-only key for the measurements bucket", r.stderr) + + ch("data1", "CREATE DATABASE IF NOT EXISTS ooni ON CLUSTER oonidata_cluster") + ch("data1", None, stdin=open(SCHEMA).read()) + tables = dict(line.split("\t") for line in ch("data1", "SELECT name, engine FROM system.tables WHERE database = 'ooni' " + "AND NOT startsWith(name, '.inner') FORMAT TSV").splitlines()) + check(all(ch(n, "SELECT count() FROM system.tables WHERE database = 'ooni'") == str(len(tables)) for n in NODES), + f"schema.sql's {len(tables)} tables on every node") + for t, engine in sorted(tables.items()): + # the replicated tables, and the ones only data1 holds: their rows go in on data1. The view + # fills its table from fastpath's inserts below. + if (engine.startswith("Replicated") or t in data1_tables) and t != "counters_asn_test_list_data": + ch("data1", random_rows(t, 200)) + load_lookup_tables(tables, data1_tables) + add_measurements(100) + print(f" ok rows in {len(tables) - 2} tables, and in counters_asn_test_list through fastpath") + return tables + + +def random_rows(table, n): + """An INSERT of n random rows; a CollapsingMergeTree's sign column must be 1 or -1""" + create = ch("data1", f"SELECT create_table_query FROM system.tables WHERE database = 'ooni' AND name = '{table}'") + sign = re.search(r"CollapsingMergeTree\([^)]*?(\w+)\)", create) + replace = f" REPLACE (1 AS {sign.group(1)})" if sign else "" + return (f"INSERT INTO ooni.`{table}` SELECT *{replace} FROM generateRandom() LIMIT {n} " + "SETTINGS max_partitions_per_insert_block = 0") + + +def structure(node, table): + return ch(node, f"SELECT arrayStringConcat(groupArray(concat(name, ' ', type)), ', ') FROM system.columns " + f"WHERE database = 'ooni' AND table = '{table}' AND default_kind NOT IN ('MATERIALIZED', 'ALIAS')") + + +def load_lookup_tables(tables, data1_tables): + """The same rows on every node, as the updaters write them""" + for t, engine in sorted(tables.items()): + if engine.startswith("Replicated") or engine == "MaterializedView" or t in data1_tables or t == "counters_asn_test_list_data": + continue + cols = structure("data1", t) + for node in NODES: + ch(node, f"INSERT INTO ooni.`{t}` SELECT * FROM generateRandom('{cols}', 42) LIMIT 20") + + +def add_measurements(n, tag="m"): + """Measurements of a test-list URL, through fastpath on data1 as in production: the view counts them""" + ch("data1", "INSERT INTO ooni.citizenlab SELECT 'example.org', 'https://example.org/', 'ZZ', 'NEWS' " + "WHERE (SELECT count() FROM ooni.citizenlab WHERE url = 'https://example.org/') = 0") + ch("data1", "INSERT INTO ooni.fastpath (measurement_uid, report_id, input, probe_cc, probe_asn, test_name, " + f"measurement_start_time) SELECT concat('{tag}', toString(number)), 'r', 'https://example.org/', 'IT', " + f"3269, 'web_connectivity', now() - 3600 FROM numbers({n})") + + +def backups(): + print("== production's backups: a full, more rows, an incremental") + for i, kind in enumerate(["full", "incremental"]): + if i: + ch("data1", random_rows("obs_web", 50)) + ch("data1", random_rows("labeling_frames", 50)) + add_measurements(30, tag="i") + time.sleep(1) # a new backup name: the names are timestamps to the second + for node in NODES: + ch(node, "SYSTEM SYNC REPLICA ooni.fastpath") + for node in ("data3", "data1"): + r = tool(node, "run") + check(r.returncode == 0, f"{kind} backup of {node}'s jobs", r.stderr) + want = {n: snapshot(n) for n in NODES} + counted = ch("data1", "SELECT sum(msmt_cnt) FROM ooni.counters_asn_test_list") + check(counted == "130", f"the view counted fastpath's 130 measurements ({counted})") + return want + + +def restore(node, config=None): + r = tool(config or node, "restore") + check(r.returncode == 0, f"restore on {node}", r.stderr) + return r.stderr + + +def lose_data2(want): + print("== 1. data2 lost or re-imaged, then restored") + wipe("data2") + check(ch("data2", "SELECT count() FROM system.tables WHERE database = 'ooni'") in ("0", ""), "data2 comes back empty") + log = restore("data2") + check("dropping stale replica data2" in log, "its leftover replicas were removed from Keeper", log) + same(snapshot("data2"), want["data2"], "data2 has every table, rows and definition, as before") + all_replicas_active("data3") + r = tool("data2", "restore") + check(r.returncode == 0 and "0 tables restored" in r.stderr, "a second restore finds nothing to do", r.stderr) + + +def lose_data1(want, data1_tables): + print("== 2. data1 lost or re-imaged, then restored") + wipe("data1") + log = restore("data1") + check(all(f"{t}: restored from" in log for t in data1_tables), "data1's own tables came from its backups", log) + same(snapshot("data1"), want["data1"], "data1 has every table, rows and definition, as before") + all_replicas_active("data3") + + +def lose_cluster(want, excluded_tables): + print("== 3. the whole cluster lost, Keeper too, then restored from the backups") + wipe("keeper", "data1", "data2", "data3") + # data2 first: its SSDs take the backups in and serve the other replicas fastest + log = restore("data2") + check("fetching" not in log, "data2 restored every replicated table from the backups", log) + log = restore("data1") + check(log.index("restore: tables ready") < log.index(": fetched"), "data1 had every table before its fetches finished", log) + restore("data3") + # the tables no backup holds (the fingerprints, rebuilt by their updater) come from schema.sql + ch("data1", None, stdin=open(SCHEMA).read()) + for node in NODES: + same(snapshot(node), want[node], f"{node} has every table as at the last backup, the excluded ones empty", + tables=[t for t in want[node] if t not in excluded_tables], empty=[t for t in want[node] if t in excluded_tables]) + all_replicas_active("data3") + before = int(ch("data2", "SELECT sum(msmt_cnt) FROM ooni.counters_asn_test_list")) + add_measurements(5, tag="after") + wait_for("the new rows on data2", lambda: ch("data2", "SELECT sum(msmt_cnt) FROM ooni.counters_asn_test_list") == str(before + 5), 60) + check(True, "the view counts new measurements again, and data2 has them") + + +def researcher(want, private, data1_tables, excluded_tables): + print("== 4. a researcher's own database, from the measurements bucket with a read-only key") + ch("researcher", "CREATE DATABASE IF NOT EXISTS ooni") + restore("researcher") + got = snapshot("researcher") + check(not set(private) & set(got), "no private table", ", ".join(sorted(set(private) & set(got)))) + # every table the measurements bucket holds: data3's, and data1's own tables with data1's rows + expected = [t for t in want["data3"] if t not in private and t not in excluded_tables] + check(sorted(got) == sorted(expected), f"the {len(expected)} tables in the measurements bucket", + f" missing: {sorted(set(expected) - set(got))}\n extra: {sorted(set(got) - set(expected))}") + source = {t: want["data1" if t in data1_tables else "data3"][t] for t in expected} + same(got, source, "each has the rows and definition of the node it was backed up from") + r = tool("researcher-private", "restore") + check(r.returncode != 0, "the read-only key can't restore the private tables", r.stderr) + + +def main(): + private, data1_tables, excluded = write_configs() + tables = setup(data1_tables) + excluded_tables = [t for t in tables if any(fnmatch.fnmatchcase(t, p) for p in excluded)] + want = backups() + lose_data2(want) + lose_data1(want, [t for t in data1_tables if t in tables]) + lose_cluster(want, excluded_tables) + researcher(want, private, data1_tables, excluded_tables) + print("all passed") + + +if __name__ == "__main__": + main() From 6eb011bc25f015bf5d1274251976b3cf4f9545e2 Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Wed, 28 Oct 2026 08:11:47 +0000 Subject: [PATCH 34/35] Runbook: restore a lost ClickHouse node --- docs/Runbooks.md | 85 +++++++++++++++++++++++++++++++----------------- 1 file changed, 55 insertions(+), 30 deletions(-) diff --git a/docs/Runbooks.md b/docs/Runbooks.md index f4b44bdb..385eacb7 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -244,10 +244,11 @@ without a restart. `/oonidevops/secrets/clickhouse_backup_r2_endpoint`, `/oonidevops/secrets/clickhouse_backup_r2_access_key_id` and `/oonidevops/secrets/clickhouse_backup_r2_secret_access_key`. -3. Deploy node_exporter first, for its textfile directory, then the backups: +3. Deploy node_exporter first, for its textfile directory, then the backups + (data2 gets the role for restoring, with its timer off): ```bash ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t node_exporter --diff - ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t clickhouse_backup --diff + ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data2.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t clickhouse_backup --diff ``` and the alert rules with `deploy-monitoring.yml`. 4. The timer takes the first (full) backup at its next run, on Sunday at @@ -333,37 +334,61 @@ Data written after a pinned backup isn't in it. If you roll back, that window has to be filled again by reprocessing those days. ### Restoring -The target server needs `ooni_backup_s3.xml` in `config.d` (the role -installs it). `ooni-clickhouse-backup list` prints the latest backup URL of -each table. Restoring the latest incremental restores the whole chain. - -1. The database isn't part of a table backup: - ```sql - CREATE DATABASE IF NOT EXISTS ooni - ``` -2. Restore tables before the materialized views that read from them, on - one node, and a view's `TO` table before the view: while a view exists - without its table, every insert into the view's source fails (fastpath's, - for `counters_asn_test_list`). A replicated table that still exists makes `RESTORE` fail with - `REPLICA_ALREADY_EXISTS`: drop it (`DROP TABLE ... SYNC`) or restore on - another server. - ```sql - RESTORE TABLE ooni.obs_web FROM S3('https://.r2.cloudflarestorage.com/ooni-clickhouse-backup/prod/ooni/obs_web//') - ``` -3. For EmbeddedRocksDB and Join tables (`test_groups`), load the rows too: - ```sql - INSERT INTO ooni.test_groups - SELECT * FROM s3('/data.native.zst', 'Native') - ``` -4. On the other replicas, create replicated tables with the same - `SHOW CREATE TABLE` statement; they fetch the data from the restored node. - Non-replicated tables whose data only data1 holds were backed up from - data1 (job `data1`); restore them there. -5. Compare row counts and checksums with what you expect, e.g. +`ooni-clickhouse-backup restore` creates the tables a node is missing and +skips the ones it has, so it can be rerun. A table another live node has is +created with that node's definition; replicated tables then fetch their data +from the other replicas, and Join and EmbeddedRocksDB rows are copied from a +peer. The backups are used only for what no node has: data1's own tables, and +everything after losing the whole cluster. Views come last (a view whose `TO` +table is missing makes inserts into its source fail). Then it logs `restore: +tables ready` and waits for the fetches; the node can take writes from then. + +data2 has the role too, timer off, for the config and `ooni_backup_s3.xml`. +The role's `tests/lifecycle` tests all of this. + +**A node lost or re-imaged** (same name and macros): +1. Deploy ClickHouse and the role on it (`deploy-clickhouse.yml`, tags + `clickhouse`, `clickhouse_backup`). +2. Run `sudo ooni-clickhouse-backup restore` on it. It first removes the + node's stale replicas from Keeper, waiting up to 3 minutes for the old + Keeper session to expire. If the replica is still active after that, + another server is running under its name: stop it. +3. Check the replicas: ```sql - SELECT count(), sum(cityHash64(toString(tuple(*)))) FROM ooni.obs_web + SELECT table, active_replicas, total_replicas FROM system.replicas WHERE database = 'ooni' ``` +A re-imaged node must also rejoin Keeper's ensemble (`reconfig`); `restore` +doesn't do that. + +**The whole cluster lost**, run `sudo ooni-clickhouse-backup restore` on: +1. data2: everything comes from data3's backups. Its SSDs take it in, and + serve it to the other replicas, faster than data3's HDDs. +2. data1: its own tables from its backups, the rest like data2's. Fastpath + can write to it once it logs `restore: tables ready`. +3. data3, from then on. + +Then create the tables that are never backed up +(`clickhouse_backup_exclude_tables`), empty; the updater refills the +fingerprints: +```bash +clickhouse-client --database ooni --multiquery < scripts/cluster-migration/schema.sql +``` + +**One table**: `restore --table obs_web`, from a live replica if one has it, +else from the latest backup. + +**A researcher's own database**: a one-node `oonidata_cluster` with its own +Keeper and macros, an R2 API token with Object Read on +`ooni-clickhouse-backup` in `config.d`, and a config with the `measurements` +job. Anonymous public URLs don't work: `RESTORE` lists objects, and an +incremental reads its base from R2's S3 endpoint. `--table` restores only +some tables. + +To check a table by hand, compare row counts and checksums, e.g. +```sql +SELECT count(), sum(cityHash64(*)) FROM ooni.obs_web FINAL +``` ## Grafana editing This runbook describes adding new dashboards, panels and alerts in From 66580a0ec32d46c4010f5e84050a25942067c1ab Mon Sep 17 00:00:00 2001 From: Aaron Gibson Date: Wed, 28 Oct 2026 08:41:52 +0000 Subject: [PATCH 35/35] Runbook: check a restored table from its parts, not a full scan FINAL with cityHash64(*) reads all of obs_web. Compare rows per replica from system.parts, and checksum one partition's key columns. --- docs/Runbooks.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/Runbooks.md b/docs/Runbooks.md index 385eacb7..fcb30851 100644 --- a/docs/Runbooks.md +++ b/docs/Runbooks.md @@ -385,9 +385,16 @@ job. Anonymous public URLs don't work: `RESTORE` lists objects, and an incremental reads its base from R2's S3 endpoint. `--table` restores only some tables. -To check a table by hand, compare row counts and checksums, e.g. +To check a table by hand, compare its rows on each replica (from the parts' +metadata, so it's instant): ```sql -SELECT count(), sum(cityHash64(*)) FROM ooni.obs_web FINAL +SELECT hostName(), sum(rows) FROM clusterAllReplicas('oonidata_cluster', system.parts) +WHERE database = 'ooni' AND table = 'obs_web' AND active GROUP BY 1 +``` +For a closer look, checksum one month's key columns on each node: +```sql +SELECT count(), sum(cityHash64(measurement_uid, observation_idx)) FROM ooni.obs_web +WHERE _partition_value = tuple('202610') ``` ## Grafana editing