Repository navigation
Conversation
…nmodified) Full tree of https://github.com/idealista/clickhouse_role at tag 3.5.2 (5ca82f11c0bd554c4b52045a57cb0e0bc6addb61, the last upstream release), by Idealista S.A.U. and contributors, under the Apache License 2.0, byte for byte. Not used yet: until the next commit, ansible-galaxy keeps installing idealista.clickhouse_role 3.5.1 from its GitHub repository (requirements/ansible-galaxy.yml).
Switches oonidata_clickhouse from idealista.clickhouse_role 3.5.1, which ansible-galaxy installed from GitHub, to this copy of 3.5.2, and removes the parts of it that this repo doesn't use, so that what's left can be read and reviewed: the JDBC bridge, clickhouse-copier, SQL-driven databases, copying config/users files from the playbook instead of templating them, the clickhouse_install_db toggle, the Galaxy metadata, upstream's CHANGELOG and README, and the standalone repo's scaffolding (CI, molecule tests, lint configs, Pipfiles, logo). The ansible-galaxy requirement goes. The README keeps the attribution and lists the changes. config.xml and users.xml render byte for byte as with the unmodified 3.5.2 for data1-3 and notebook1. Against 3.5.1, what production runs today, config.xml is the same, and users.xml gives the admin user the named_collection_control, show_named_collections and show_named_collections_secrets that vars already set (3.5.1 ignored them), so the first deploy restarts each node.
Upstream added the repository key with the deprecated apt_key. It is now a signed-by keyring (from upstream PR #80), copied from files/clickhouse-repo-keyring.asc instead of fetched, so rotating it takes a reviewed commit (fingerprint 3A9E A119 3A97 B548 BE14 57D4 8919 F6BD 2B48 D754, checked 2026-09-22). The entry upstream added without signed-by is removed first: apt refuses the same repository with two different keys, and apt-get update fails on the whole host (checked on Debian 12). Upstream installed clickhouse_packages at clickhouse_version but didn't pin them, so `apt upgrade` installs the newest release in the repo: 26.8.36.3 where 24.8.6.70 is meant (checked on Debian 12 on 2026-10-27). The clickhouse-* packages are now pinned to clickhouse_version in /etc/apt/preferences.d/clickhouse, and the install allows downgrades, so a host that drifted goes back to it. To upgrade or downgrade, change clickhouse_version and rerun the role.
…emplates Rewrites the SQL-driven users, quotas and grants in the format the next commit's templates read. The role as of this commit only creates users and quotas that don't exist yet, so this changes nothing on the existing nodes. - Each user carries its own settings, including readonly = 0, instead of the XML write/readonly profiles; without it a user gets the default profile's readonly = 2. The readonly profile refused log_comment for oonimeasurements on data3, which has failed every dev deploy since #1305. Its limits are CONST constraints instead, which apply on data1 and data2 for the first time (their users were created by hand with fewer limits). - The quotas were in the XML quota format, which the SQL template never read, so the role created them empty and unassigned; only data1's oonimeasurements quota was set up by hand. Its limits are now defined, but it stays unassigned (no `to`): the API's node, data2, has never had a quota, and the limits were never checked against its traffic. - The oonimeasurements role, which held nothing, is gone, and so are the keys the role never read (quota, databases) and the null user:/quota:/on: keys. - clickhouse_sql_objects_drop_unlisted is false until each node has been checked for SQL users and quotas not listed here.
…g output - Existing objects were never updated: upstream only ran CREATE ... IF NOT EXISTS, and listed existing objects by storage 'local directory' while 24.8 says 'local_directory', so it never saw them. Users and quotas are now created if missing and ALTERed if they exist; ALTER keeps a user's grants and quota assignments. Users are applied before quotas, so that a quota's TO finds its user. Those not in vars are dropped only with clickhouse_sql_objects_drop_unlisted, false by default. - Grants started with REVOKE ALL ON *.* FROM ALL EXCEPT <XML users>, so every SQL-driven user lost its privileges on every run until they were granted again (241 failed queries during two runs in a test with live traffic). Each grantee's privileges are now replaced in one GRANT ... WITH REPLACE OPTION. Roles granted by hand are left alone. - The admin password was on every command line, and all SQL task output was hidden with no_log. The admin credentials are now in a root-only clickhouse-client config file, the SQL goes in on stdin, and only the task writing that file has no_log. Each user's password is set in a statement of its own, so an error in its other settings doesn't quote the hash. - SQL roles and settings profiles, unused here, go, and so do LDAP, Kerberos, GRANTEES, DEFAULT ROLE and ON CLUSTER in CREATE USER: the templates cover the vars this repo uses (see README.md). Tested on three 24.8.6.70 nodes seeded like data1/data2/data3: users and quotas converge with their ids, grants and assignments kept, stray grants are removed, a second run changes nothing, and inserts and selects running throughout two role runs never failed.
The role restarts ClickHouse when its config changes, and the play ran on all hosts at once, so a config change restarted data1-3, which are also the Keeper ensemble, together. With serial: 1 one node restarts at a time, and the play stops at the first host that fails.
tests/run.sh, also run in CI when the role or group_vars/clickhouse/vars.yml change, checks what the role's README says it does, against ClickHouse 24.8.6.70 with the real vars (secrets replaced by test values) and a users.xml rendered from the role's template: - test_sql_objects.py runs the role's SQL tasks on an empty node and three nodes seeded like production's data1-3, and checks that every node ends up like the empty one; that each user logs in with its password from vars (an old one is refused), can do what its grants allow and nothing else (including on a database no one is granted), and has its settings, CONST ones fixed; that ids, quota assignments, roles granted by hand and objects not in vars are kept, and that users and quotas not in vars are dropped when clickhouse_sql_objects_drop_unlisted is set; that a second run changes nothing; that the role sends no REVOKE, and queries running during the role never fail; and that the output, at default verbosity, shows errors but no password or hash. - apt_pin.sh checks on Debian 12, against packages.clickhouse.com, that with the role's pin file apt installs the pinned version and not the newest release (and with an older pin too, when clickhouse_version is the newest).
…chema.sql On a cluster member (a host with a replica macro in clickhouse_macros) that has no ooni database, a new replica or a re-imaged one, the role creates the database and loads scripts/cluster-migration/schema.sql on that host only, with its ON CLUSTER removed. The replicated tables then fetch their data from the other replicas; the tables that aren't replicated start empty. A host that has the database is left alone: changes to it are migrations. Hosts that aren't replicas, like notebook1, are skipped. If loading schema.sql fails, the deploy stops with ClickHouse's error and the database is left as it is, for an operator to repair; nothing is dropped. The runbook says how to add and remove a replica: edit the cluster in vars, deploy one host at a time, and drop a removed replica from Keeper; how to finish a failed schema load by hand; and, as replicated tables need a Keeper quorum, to deploy a new cluster with --skip-tags clickhouse_schema first.
tests/run.sh, also run in CI, follows the runbook on one ClickHouse 24.8.6.70 container per production host, named like it, with the real group and host vars (secrets replaced by test values) and the configs the clickhouse role renders from them: - data1-3 form the Keeper ensemble and oonidata_cluster, and the schema step creates the ooni database on each, but not on notebook1, and leaves existing databases alone; - with data4 added to clickhouse_remote_servers and clickhouse_macros and data1-3 restarted one at a time, every node sees four replicas, data4 gets the database and catches up, and inserts and ON CLUSTER DDL reach it; - with data4 removed and dropped from Keeper, every node sees three replicas, Keeper has no trace of it, ON CLUSTER DDL doesn't wait for it, and data1-3 still replicate; - a re-imaged data4 (empty disk, same replica name) can be added again.
…back apt doesn't restart a running clickhouse-server: after a version bump the role installed the new packages and left the old version running, until something else restarted it (tested on Debian 12: 25.3.14.14 installed, 24.8.6.70 still running under the same PID). The package install now notifies Restart-clickhouse, as a changed unit file does (upstream only reloaded, which doesn't apply a changed unit), and the service step restarts right after daemon-reload instead of at the end of the play. It then fails unless the server answers SELECT version() with clickhouse_version, and waits until none of its replicated tables is read-only, i.e. until it's back in Keeper. With serial: 1 or --limit, a host is back on its new version before the next one is deployed. The admin client config the checks use is now written before the service step, in --check too; the version comparison is skipped in --check, where nothing is restarted.
tests/upgrade/run.sh installs FROM_VERSION with the role on three Debian 12 hosts with systemd, named like the production replicas, with the real vars and the apt source line the role's upstream version left, then sets clickhouse_version to TO_VERSION (by default tests/upgrade/next_version, the next hop of the plan in #477) and deploys each host with --limit in turn, while a writer inserts throughout. After each host it checks that the role succeeded, that ClickHouse is active under systemd on the new version with its packages pinned, and that every node answers and reaches Keeper, and it reports each node's version, Keeper role and the inserts that failed meanwhile. At the end every node must run TO_VERSION, take inserts, have no read-only or stuck replicated tables, and have every row whose insert succeeded. 24.8.6.70 -> 25.3.14.14 and 25.3.14.14 -> 25.8.29.51 pass; one insert failed in all, to the node that was restarting. CI runs it when a PR changes clickhouse_version (from the base's version to the PR's) or the role, other than its README and other tests, and by hand with any two versions; not for other vars changes. The runbook says how to upgrade.
A quota's TO names users, so the test's order failed once a quota in vars had one.
notebook.ooni.org isn't in [clickhouse], so the cluster deploy would have installed the role's default version there. notebook1 stays in the group for its vars (clickhouse_version, the cluster's remote_servers and Keeper), with SQL quotas off like users and grants.
The role's config listens on "::" only, as in production. Docker turns IPv6 off in containers on an IPv4-only network, so on GitHub Actions ClickHouse couldn't listen and exited (code 210).
At 700 MB, ClickHouse's server-wide limit (90% of it) failed some of the queries the test runs during the role on GitHub Actions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vendor idealista.clickhouse_role into ansible/roles. This fixes a few issues with the role and simplifies our user configuration. This depends on #522