Skip restore of non-sharded replicated tables

[mvanhorn]

Summary

Two clickhouse settings are added: skip_restore_non_sharded_replicated_tables (env CLICKHOUSE_SKIP_RESTORE_NON_SHARDED_REPLICATED_TABLES, default false) and skip_check_non_sharded_replicated_tables_cluster (env CLICKHOUSE_SKIP_CHECK_NON_SHARDED_REPLICATED_TABLES_CLUSTER, default {cluster}). getTablesForRestoreLocal now calls skipNonShardedReplicatedTables after the candidate list is built; with the bool false the list is returned and no SQL runs, and with it true a clusterAllReplicas query on system.replicas drops tables where uniqExact(zookeeper_path) = 1 and count() > 1, deletes their partition-map entries, and logs a warning, while sharded tables stay. A whitespace-only cluster name is an error, single quotes in the cluster string are backslash-escaped, and the empty-schema error names the new setting. DefaultConfig, the README, and the v2.8.2 changelog record the same defaults.

A cluster restore recreated every matched replicated table on each replica, including global tables that share a single ZooKeeper path across more than one replica. Operators had no setting to leave those tables alone on the replicas that should not own them.

Tests

• gofmt -l printed no paths for the changed Go files.
• go vet finished without diagnostics on pkg/backup and pkg/config.
• go test ./pkg/backup/... ./pkg/config/... passed.
• Ran make test in a Linux container (golang:1.24) for the clickhouse-backup CustomConfig change; it passed.
• The tests in pkg/backup/restore_test.go and pkg/config/config_test.go fail without the fix in the clickhouse-backup CustomConfig change (checked by reverting the source changes).

Fixes #1181

[Slach]

thanks a lot for your contribution

[Slach]

Review: what #1181 asks vs. what this does

The option names, defaults and the SQL all match issue #1181 literally. That is the root problem: the query in the issue is a sketch, and it shipped verbatim. The predicate has no shard dimension (P0, inline), and it is computed after the database-level drop (P0, inline).

Three findings span files and cannot be anchored to a diff line:

P1 — the owner node is not safe either. The docs' recipe is "the node that keeps the global tables leaves the bool false". But on that node the table usually already exists, so checkReplicaAlreadyExistsAndChangeReplicationPath reaches the replica-present fall-through and rebinds the engine to default_replica_path = /clickhouse/tables/{cluster}/{shard}/{database}/{table} (pkg/backup/restore.go:2196-2198) — which silently converts a global table into a sharded one. rebind_replica_path_if_exists / drop_replica_if_exists do not cover this case.

P1 — --resume is unstable. The flag is not part of the resumable-state key, and the skip set is recomputed from live system.replicas on every run. Restore round 1 (flag false) creates db.g and starts attaching data, round 2 (flag true, the table is now live on both replicas) drops db.g, so the round-1 attach is left half-applied with no way to finish or roll back.

P2 — test coverage. Only the pure helpers are covered (query string, filter, flag-off shortcut, empty cluster). Nothing exercises the live SelectContext, and the default integration topology is 1 shard x 1 replica, where the feature is a no-op. Needed: a config.d cluster with 2 replicas on one server (skip, mapping interplay, all-skipped) plus a 2x2 test for global vs sharded classification and shard-by-shard ordering.

Also worth noting: #1181 carries no error text, topology, command or version, so there is no recorded failure to reproduce. Before merge, please add one concrete end-to-end scenario (backup + topology + command + the error it used to hit) so the fix can be verified rather than argued.

[Slach]

P0 — the skip is computed after the databases have already been dropped.

restorePrologue runs restoreEmptyDatabase at :319, ~260 lines before this call. For an Atomic/Ordinary database with restore --schema --rm that path executes DROP DATABASE IF EXISTS ... SYNC (optionally ON CLUSTER) — a database-level drop that tablesForRestore cannot filter.

Use-case: backup contains db.a (local non-replicated) and db.g (global), the table already exists on >= 2 replicas so the query matches it. Run restore --rm --schema -d db:

1. DROP DATABASE db SYNC removes both tables;
2. the skip drops db.g from the restore list;
3. only db.a is recreated. db.g is gone and never comes back — before this PR the drop was paired with a recreate.

Same ordering makes any failure of the new clusterAllReplicas query (a replica is down, {cluster} undefined) abort the restore after the drop. Compute the skip set before the database loop, make the drop table-aware, or reject the --rm + flag combination. Dry-run cannot catch this either: it returns before the drop, so the dry-run verdict is computed with the table present and the real run is not.

[Slach]

P1 — enabling the flag adds an all-replica availability prerequisite.

This runs unconditionally, so:

• A backup with zero Replicated* tables still issues a clusterAllReplicas query. One replica of {cluster} down -> Code 210 NO_REMOTE -> the whole restore fails with an error that has nothing to do with what was being restored. A degraded cluster is precisely when restores happen.
• Standalone server with no {cluster} macro: Macro cluster can not be empty -> fails at the very first restore. The "{cluster}" default only works when the macro is set.
• There is no skip_unavailable_shards, and no fail-open/fail-closed decision.

Skip the query when the selection contains no Replicated* engine, and pick a side explicitly: fail closed (skip nothing, restore as today) or fail open, but do not turn a monitoring query into a restore-blocking dependency.

[Slach]

P2 — delete(partitions, title) is dead here. A table already removed from tablesForRestore has no downstream consumer of its partition list (RestoreData derives it from that list), so this line only suggests coupling that does not exist. Drop it, or keep one and comment why.

[Slach]

P2 — the name reads as the opposite of what it does. skip_check_<...>_cluster parses as "skip the check", but the value is the cluster the check queries. A user setting skip_check_non_sharded_replicated_tables_cluster: "" expects the check disabled and instead gets the error at restore.go:604.

Per the issue's env contract the name should probably stay for compatibility — in which case please say plainly in the README that it is "cluster used to detect global tables", and consider an explicit "" = "disable detection" later. Also worth noting these two fields are unrelated to use_embedded_backup_restore_cluster; the docs should not imply they share a cluster.

[Slach]

will fix comments from my side