UI extremely slow when db.ha.enabled=true after MySQL source failure (4.22.1.0)

[nabassa1323]

problem

With Database High Availability enabled (db.ha.enabled=true), after the MySQL source becomes unreachable the Management Server UI becomes extremely slow / barely usable.

Failover to the configured replica seems to happen via the MySQL connector, but each (or almost each) request appears to retry the dead source first, causing severe UI latency.

Workaround that restores normal performance:

• disable db.ha.enabled
• manually promote the replica (STOP REPLICA / RESET REPLICA ALL / read_only=OFF)
• point db.cloud.host / db.usage.host to the promoted DB in db.properties
• restart cloudstack-management

This matches the install guide "Failover" procedure better than the client-side db.ha mechanism.

Note: docs still say "Tested with MySQL 5.1 and 5.5" and suggest two-way replication.
Ref: https://docs.cloudstack.apache.org/en/4.22.1.0/adminguide/reliability.html#configuring-database-high-availability

versions

CloudStack: 4.22.1.0 (Ubuntu packages, download.cloudstack.org noble)
OS: Ubuntu 24.04
MySQL: 8.0.46 (async replication source→replica, GTID)
Hypervisor: VMware
Topology: 2 Management Servers (multi-site), reverse proxy in front of UI

The steps to reproduce the bug

1. Deploy 2 MS pointing to MySQL source; configure two-way replica (GTID)
2. Set in db.properties:
   db.ha.enabled=true
   db.cloud.replicas=replica-ip
   db.usage.replicas=replica-ip
3. Restart cloudstack-management on both nodes
4. Stop / isolate the MySQL source (simulate site/DB failure)
5. Use UI/API through the remaining Management Server

What to do about it?

Expected: MS fail over to replica promptly; UI remains usable.

Actual: UI becomes very slow after source outage (connector retries against unreachable source; defaults secondsBeforeRetrySource/queriesBeforeRetrySource/initialTimeout = 3600/5000/3600).

Suggestions:

• fix connector failover so replica is used without per-request source retries hanging the UI
• and/or document clearly that db.ha.enabled is not recommended for production on MySQL 8.x and that manual promotion (install guide Failover) is the supported path

[DaanHoogland]

@nabassa1323 I had claudecode create a PR; #14279. Can you see if this meets your requirements and test it?

I think this was never implemented seriously and I think we would need to classifyt it as a new feature, though some claims had been made.

[github-actions]

🎯 Triage report

Reporter observes that with db.ha.enabled=true on MySQL 8.x, after the configured source becomes unreachable, the management server UI becomes extremely slow because the connector keeps retrying the dead source on (nearly) every request. Manually promoting the replica and pointing db.cloud.host/db.usage.host at it (per the install guide's Failover procedure) restores normal performance. A maintainer has already commented that this DB HA mechanism was likely never fully implemented/tested, and a candidate PR (#14279) has been opened to address it.

📊 Assessment

Dimension	Value	Reasoning
Type	type:new-feature	Maintainer comment indicates the existing db.ha.enabled client-side failover was never seriously implemented and should be treated as a new feature rather than a regression/bug fix.
Component	component:database, component:management-server, component:mysql	Issue concerns MySQL connector failover behavior within the management server's DB HA configuration.
Severity	n/a	Not labeled as a bug; classified as new-feature work per maintainer.
Labels	type:new-feature, component:database, component:management-server, component:mysql, status:Has a PR	See reasoning above; status:Has a PR added since #14279 is linked.
Coding agent	Suitable	Well-documented repro steps, config, versions, and a maintainer-linked candidate PR (#14279) already exist; mainly needs validation/testing by the reporter against that PR.

🔗 Similar issues

No clearly duplicate or related open issues were found via search for "db.ha.enabled MySQL failover replica".

💡 Notes and suggestions

• PR allow loadbalanced DB #14279 was opened in response to this issue — worth asking @nabassa1323 to validate it against their reproduction steps (2 MS nodes, GTID replication, source outage simulation).
• Consider updating the admin guide (currently references MySQL 5.1/5.5 testing) once the fix/feature lands, to reflect supported MySQL 8.x behavior and clarify whether manual promotion remains the documented production path.
• Relevant connector settings called out by reporter: secondsBeforeRetrySource, queriesBeforeRetrySource, initialTimeout — may need tuned defaults or documentation if the new implementation still relies on MySQL Connector/J failover semantics.

Generated by Daily Issue Triage · sonnet50 77.8K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9