You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR improves the Indirect agent connection handling, has the following improvements.
Enhances the Host connecting logic to avoid connecting storm (where Agent opens multiple sockets against Management Server).
Implements HostConnectProcess task where Host upon connection checks whether lock is available, traces Host connecting progress, status and timeout.
Introduces AgentConnectStatusCommand, where Host checks whether lock for the Host is available (i.e. "previous" connect process is finished).
Implementes logic to check whether Management Server has lock against Host (exposed MySQL DB lock presence via API)
Removes synchronization on Host disconnect process, double-disconnect logic in clustered Management Server environment, added early removal from ping map (in case of combination ping timeout delay + synchronized disconnect process the Agent Manager submits more disconnect requests)
Introduces parameterized connection and status check timeouts
Implements backoff algorithm abstraction - can be used either constant backoff timeout or exponential with jitter to wait between connection Host attempts to Management Server
Implements ServerAttache to be used on the Agent side of communication (similar to Attache on Management Server side)
Enhances/Adds logs significantly to Host Agent and Agent Manager logic to trace Host connecting and disconnecting process, including ids, names, context UUIDs and timings (how much time took overall initialization/deinitialization)
Adds logs to communication between Management Servers (PDU requests)
Adds DB indexes to improve search performance, uses IDEMPOTENT_ADD_INDEX for safer DB schema updates
Dedupes agent connection requests. Observed scenarios where on agent with old agent code send a storm of connection requests. In such a case, to ensure other agents are not impacted, deduping the connection requests.
Remove IP address hard dependency for management server communication. It allows the management.server.address config to be a list of hostnames instead of static IPs. Rebalancing, cluster propagation, alerts, and stats collection all detect the format at runtime via ManagementServerAddressUtil and use the matching lookup (listNonUpStateMsHostnames vs listNonUpStateMsIPs) when computing the "avoid" list sent to agents. Hostnames survive MS restarts and IP reassignments where static IPs do not, which is required in deployment environments where management server instances can be rescheduled or replaced.
Propagates instance (vm, volume, etc) uuid to all operations. The uuid is propagated from MS to agent communications and to all async jobs.
Types of changes
Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Major
Minor
Bug Severity
BLOCKER
Critical
Major
Minor
Trivial
Screenshots (if appropriate):
How Has This Been Tested?
Checked the indirect agents connections (and re-connections) with KVM hosts, SSVM & CPVM.
How did you try to break this feature and the system with this change?
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
❌ Patch coverage is 24.76730% with 1859 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.96%. Comparing base (1a48a87) to head (691b7d4).
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR is a broad refactor/enhancement of indirect agent connection handling across both Management Server and Agent sides. It introduces a host-side connect/status-check process, adds configurable backoff and host-status heuristics, and expands logging/lock-check mechanisms to reduce reconnect storms and improve observability.
Changes:
Adds agent-side connection orchestration (status polling + startup submission) and a server-side AgentConnectStatus* command/answer pair to coordinate connect progress.
Introduces configurable backoff (factory + exponential-with-jitter implementation) and propagates configuration from Management Server to Agent during startup.
Refactors NIO connection lifecycle and adds supporting utilities/logging, plus DB lock availability checks and DB indexes for performance.
Reviewed changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 13 comments.
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
…ements.
- Enhances the Host connecting logic to avoid connecting storm (where Agent opens multiple sockets against Management Server).
- Implements HostConnectProcess task where Host upon connection checks whether lock is available, traces Host connecting progress, status and timeout.
- Introduces AgentConnectStatusCommand, where Host checks whether lock for the Host is available (i.e. "previous" connect process is finished).
- Implementes logic to check whether Management Server has lock against Host (exposed MySQL DB lock presence via API)
- Removes synchronization on Host disconnect process, double-disconnect logic in clustered Management Server environment, added early removal from ping map (in case of combination ping timeout delay + synchronized disconnect process the Agent Manager submits more disconnect requests)
- Introduces parameterized connection and status check timeouts
- Implements backoff algorithm abstraction - can be used either constant backoff timeout or exponential with jitter to wait between connection Host attempts to Management Server
- Implements ServerAttache to be used on the Agent side of communication (similar to Attache on Management Server side)
- Enhances/Adds logs significantly to Host Agent and Agent Manager logic to trace Host connecting and disconnecting process, including ids, names, context UUIDs and timings (how much time took overall initialization/deinitialization)
- Adds logs to communication between Management Servers (PDU requests)
- Adds DB indexes to improve search performance, uses IDEMPOTENT_ADD_INDEX for safer DB schema updates
- Bug 1 fix (AgentManagerImpl.java) — GlobalLock.isLockAvailable() now only runs when the host status is not alive. This eliminates one IS_FREE_LOCK DB query per ping per healthy host, which is the direct cause of listHosts/listNetworks degradation.
- Bug 2 fix (HostConnectProcess.java) — shutdown() → shutdownNow(). Old thread pools from prior connect cycles are now interrupted immediately instead of draining their queued tasks, preventing thread accumulation during reconnect storms.
- Bug 3 fix (AgentManagerImpl.java) — Lock timeout in handleDisconnectWithoutInvestigation now logs a warn instead of silently discarding the disconnect event.
- Bug 4 fix (ServerAttache.java) — Alarm ScheduledFuture handles are now tracked in _alarmFutures and cancelled when the corresponding listener is unregistered or all commands are cancelled on disconnect.
- Bug 5 fix (AgentAttache.java) - Fix Alarm ScheduledFuture handles in AgentAttache as well
Dedupes agent connection requests. Observed scenarios where on agent with old agent code send a storm of connection requests. In such a case, to ensure other agents are not impacted, deduping the connection requests.
Allows the management.server.address config to be a list of hostnames instead of static IPs.
Rebalancing, cluster propagation, alerts, and stats collection all detect the format at runtime via ManagementServerAddressUtil and use the matching lookup (listNonUpStateMsHostnames vs listNonUpStateMsIPs) when computing the "avoid" list sent to agents.
Hostnames survive MS restarts and IP reassignments where static IPs do not, which is required in deployment environments where management server instances can be rescheduled or replaced.
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
The method's contract says the current MDC is authoritative, but when either MDC key is absent it leaves any pre-existing value on the command untouched. A reused command can therefore carry a stale UUID or log context into a later operation. Explicitly clear/set both trace-context keys when the corresponding MDC value is blank.
This changes cluster peer URLs to always use the persisted canonical hostname. That hostname is not necessarily the configured management-server address or a resolvable alias, so existing IP-based configurations can stop reaching peers and hostname aliases are ignored. Select the matching configured address with ManagementServerAddressUtil first, then fall back to a nonblank persisted identifier.
The configured max_delay_ms is documented and named as the maximum delay, but jitter is added after the cap. Once delay reaches maxDelayMs, this returns values up to 1.5× the configured maximum, so the backoff can exceed the operational limit. Clamp the jittered result to maxDelayMs (or define the setting as a pre-jitter bound).
The reason will be displayed to describe this comment to others. Learn more.
LGTM tested manually
New config keys exist on the management server
cmk list configurations name=backoff.configuration
cmk list configurations name=agent.host.status.check.delay.sec
cmk list configurations name=alive.host.statuses
cmk list configurations name=agent.disconnect.event.broadcast.perform.full.disconnect
cmk list configurations name=indirect.agent.lb.migrate.command.wait
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=backoff.configuration
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"category": "Advanced",
"component": "AgentManager",
"defaultvalue": "backoff.implementation=com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff;backoff.min_delay_ms=5000;backoff.max_delay_ms=15000",
"description": "Backoff algorithm configuration in format key1=value1;key2=value2;...",
"displaytext": "Backoff configuration",
"group": "Miscellaneous",
"isdynamic": true,
"name": "backoff.configuration",
"subgroup": "Others",
"type": "String",
"value": "backoff.implementation=com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff;backoff.min_delay_ms=5000;backoff.max_delay_ms=15000"
}
],
"count": 1
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=agent.host.status.check.delay.sec
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"category": "Advanced",
"component": "AgentManager",
"defaultvalue": "15",
"description": "Delay in seconds between host status checks on the agent side",
"displaytext": "Agent host status check delay sec",
"group": "Management Server",
"isdynamic": true,
"name": "agent.host.status.check.delay.sec",
"subgroup": "Agent",
"type": "Number",
"value": "15"
}
],
"count": 1
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=alive.host.statuses
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"category": "Advanced",
"component": "AgentManager",
"defaultvalue": "Up,Creating,Connecting,Rebalancing",
"description": "Comma-separated list of host statuses that are considered alive and do not require agent reconnect. Valid values: Up, Alert, Connecting, Creating, Rebalancing, Down, Disconnected, Removed.",
"displaytext": "Alive host statuses",
"group": "Hypervisor",
"isdynamic": false,
"name": "alive.host.statuses",
"subgroup": "Hypervisor",
"type": "String",
"value": "Up,Creating,Connecting,Rebalancing"
}
],
"count": 1
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=agent.disconnect.event.broadcast.perform.full.disconnect
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"category": "Advanced",
"component": "AgentManager",
"defaultvalue": "false",
"description": "Performs a full host disconnect on AgentDisconnect event broadcast",
"displaytext": "Agent disconnect event broadcast perform full disconnect",
"group": "Management Server",
"isdynamic": true,
"name": "agent.disconnect.event.broadcast.perform.full.disconnect",
"subgroup": "Agent",
"type": "Boolean",
"value": "false"
}
],
"count": 1
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=indirect.agent.lb.migrate.command.wait
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"category": "Advanced",
"component": "IndirectAgentLBServiceImpl",
"defaultvalue": "60",
"description": "The time in seconds to wait for an indirect agent to answer the command asking it to migrate its connection to another management server.",
"displaytext": "Indirect agent lb migrate command wait",
"group": "Management Server",
"isdynamic": true,
"name": "indirect.agent.lb.migrate.command.wait",
"subgroup": "Agent",
"type": "Number",
"value": "60"
}
],
"count": 1
}
Check that the new DB indexes were created. The upgrade uses IDEMPOTENT_ADD_INDEX, so it should also be safe to run twice:
┌───────────────┬──────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────┐
│ Test │ How │ Expected │
├───────────────┼──────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ Agent restart │ systemctl restart cloudstack-agent on a KVM host │ Host goes Disconnected→Connecting→Up and its VMs stay running │
├───────────────┼──────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ MS restart │ Restart the MS the host is connected to │ Host reconnects to another MS from host list (or the same one once it's back) │
├───────────────┼──────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ System VMs │ Reboot the SSVM and CPVM, and restart the cloud service inside them │ Both reconnect and console/template operations work │
└───────────────┴──────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────┘
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "Lost connection to host|Reconnect info:|Reconnecting to host:|Reconnecting to|Next host to connect" $AGLOG
2026-10-01 10:57:20,574 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:f72e0b70) Reconnect info: provided link: 33668-> /10.0.35.164:8250, agent link: N/A, preferred host: null, force reconnect: false
2026-10-01 10:57:23,468 INFO [cloud.agent.Agent] (main:[]) (logid:be80a802) Reconnect info: provided link: N/A, agent link: N/A, preferred host: null, force reconnect: false
2026-10-01 10:57:23,473 INFO [cloud.agent.Agent] (main:[]) (logid:be80a802) Lost connection to host: null. Attempting reconnection while we still have 0 commands in progress.
2026-10-01 10:57:23,478 INFO [cloud.agent.Agent] (main:[]) (logid:c107a57d) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 10:57:41,863 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Reconnect info: provided link: 41914-> /10.0.35.164:8250, agent link: 41914-> /10.0.35.164:8250, preferred host: null, force reconnect: false
2026-10-01 10:57:41,870 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Lost connection to host: 10.0.35.164. Attempting reconnection while we still have 0 commands in progress.
2026-10-01 10:57:41,872 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:1a617859) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 10:57:41,883 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:1a617859) Next host to connect: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
2026-10-01 10:57:41,884 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:8afb1299) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "Connected to .*: (true|false)|SSL Handshake done for" $AGLOG
2026-10-01 10:57:23,489 INFO [utils.nio.NioClient] (main:[]) (logid:c107a57d) Connected to 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250: true
2026-10-01 10:57:24,008 INFO [utils.nio.NioClient] (main:[]) (logid:c107a57d) SSL Handshake done for 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 10:57:41,889 INFO [utils.nio.NioClient] (AgentRequest-Handler-2:[]) (logid:8afb1299) Connected to 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250: true
2026-10-01 10:57:42,081 INFO [utils.nio.NioClient] (AgentRequest-Handler-2:[]) (logid:8afb1299) SSL Handshake done for 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# grep -A8 -E "Processing Host connection (started|finished)" $MSLOG | grep -E "Processing Host|host_name|lock_name|ms_id|process_duration"
Processing Host connection started:
host_name=v-1-VM
lock_name=Host-Join-3
ms_id=32989492806711
Processing Host connection finished:
host_name=v-1-VM
lock_name=Host-Join-3
ms_id=32989492806711
process_duration=235 ms
process_duration_ms=235
Processing Host connection started:
host_name=s-2-VM
lock_name=Host-Join-4
ms_id=32989492806711
Processing Host connection finished:
host_name=s-2-VM
lock_name=Host-Join-4
ms_id=32989492806711
process_duration=2 sec, 175 ms
process_duration_ms=2175
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# grep -A10 "Failed to connect Host" $MSLOG
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# grep -E "Process connect for host|Finished process connect for host" $MSLOG
2026-10-01 11:00:21,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-132e38e8]) (logid:6eec2ba7) Process connect for host: v-1-VM (8eddb10d-b6c3-4995-a015-517bdd46e902) for rebalance: false
2026-10-01 11:00:21,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-132e38e8]) (logid:6eec2ba7) Process connect for host: v-1-VM (8eddb10d-b6c3-4995-a015-517bdd46e902) listener XcpServerDiscoverer (0 of 29) for rebalance: false
2026-10-01 11:00:21,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-132e38e8]) (logid:6eec2ba7) Finished process connect for host: v-1-VM (8eddb10d-b6c3-4995-a015-517bdd46e902) listener: XcpServerDiscoverer (0 of 29) for rebalance: false duration, s: 0
2026-10-01 11:00:21,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-132e38e8]) (logid:6eec2ba7) Process connect for host: v-1-VM (8eddb10d-b6c3-4995-a015-517bdd46e902) listener HypervServerDiscoverer (1 of 29) for rebalance: false
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# grep -E "Host .* is disconnecting with event|Attache .* is deregistering with event" $MSLOG
2026-10-01 10:57:19,579 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-1:[ctx-aa4300a8]) (logid:018405a2) Host AgentAttache {"_id":1,"_name":"ref-trl-12457-k-Mol8-kiran-chavala-kvm1","_uuid":"d0da25d5-0af1-4e31-bf92-429cc16fb3e1"} is disconnecting with event ShutdownRequested
Host-join lock and the AgentConnectStatus check
While a host is connecting (right after an agent restart), the agent should send. Watch for Checking whether global lock Host-Join- is present (MSDEBUG level) There is lock and Host status is ..., will retry later
Change agent.host.status.check.delay.sec (for example to 30) and confirm in thng interval changes. The MS pushes this value to agent.properties.
agent.host.status.check.delay.sec=30
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# cat /etc/cloudstack/agent/agent.properties
#Storage
#Thu Oct 01 11:36:13 UTC 2026
cluster=1
agent.host.status.check.delay.sec=30
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "Received agent.host.status.check.delay.sec=|Updated agent.host.status.check.delay.sec" /var/log/cloudstack/agent/agent.log
2026-10-01 11:12:18,368 INFO [cloud.agent.Agent] (Agent-Handler-5:[]) (logid:) Updated agent.host.status.check.delay.sec to 15 seconds
2026-10-01 11:36:11,000 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Received agent.host.status.check.delay.sec=30 from Management Server
2026-10-01 11:36:11,005 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Updated agent.host.status.check.delay.sec to 30 seconds
SELECT id, name, status, mgmt_server_id FROM cloud.host WHERE id IN (1,2); -- both Up
SELECT IS_FREE_LOCK('Host-Join-1'); -- 1 = free
Step 1: Start the log tails
(kvm1):
tail -F /var/log/cloudstack/agent/agent.log | grep --line-buffered -E \
"There is (no )?lock and Host status|Host is locked|Host is in operational state|Received empty agent connect status|Reconnect info|Reconnecting to|Connected to|Duplicate connection"
(mgmt1)
tail -F /var/log/cloudstack/management/management-server.log | grep --line-buffered -E \
"Host-Join-1|Processing Host connection|lock_name|host_name=.*kvm1|process_duration"
The Checking whether global lock ... lines only appear at DEBUG.
Step 2: Take the lock (T1)
SELECT GET_LOCK('Host-Join-1', 600); -- must return 1
SELECT IS_FREE_LOCK('Host-Join-1'); -- must return 0
If GET_LOCK hangs, the MS currently holds the lock. Wait for it to return 1.
Step 3: Restart the agent on kvm1 (new shell on kvm1)
systemctl restart cloudstack-agent
Step 4 : Release lock
SELECT RELEASE_LOCK('Host-Join-1');
Logs
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# tail -F /var/log/cloudstack/management/management-server.log | grep --line-buffered -E "Host-Join-1|Processing Host connection|lock_name|host_name=.*kvm1|process_duration"
2026-10-01 11:34:39,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Obtaining internal lock for Host-Join-1
2026-10-01 11:34:39,272 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Internal lock Host-Join-1 does not exist, adding
2026-10-01 11:34:39,272 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Added reference to internal lock Host-Join-1, reference count 1, lock count 0
2026-10-01 11:34:39,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Obtained internal lock for Host-Join-1
2026-10-01 11:34:39,273 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Taking ownership on global lock Host-Join-1 to acquire, owner thread: AgentTaskPool-3, reference count: 2, lock count: 0
2026-10-01 11:34:39,273 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Acquiring global lock Host-Join-1 in DB within remaining 57 seconds
2026-10-01 11:34:39,273 DEBUG [c.c.u.d.DbUtil] (AgentTaskPool-3:[ctx-4ecfe248]) (logid:7718597c) Storing connection for global lock Host-Join-1
2026-10-01 11:34:40,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-4:[ctx-381318c3]) (logid:bbf049d0) Obtaining internal lock for Host-Join-1
2026-10-01 11:34:40,398 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-4:[ctx-381318c3]) (logid:bbf049d0) Internal lock Host-Join-1 already exists with reference count 2 and lock count 0
2026-10-01 11:34:40,398 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-4:[ctx-381318c3]) (logid:bbf049d0) Added reference to internal lock Host-Join-1, reference count 3, lock count 0
2026-10-01 11:34:40,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-4:[ctx-381318c3]) (logid:bbf049d0) Obtained internal lock for Host-Join-1
2026-10-01 11:34:40,398 DEBUG [c.c.u.d.GlobalLock] (AgentTaskPool-4:[ctx-381318c3]) (logid:bbf049d0) Waiting 62 seconds to acquire global lock Host-Join-1
2026-10-01 11:34:53,831 DEBUG [c.c.u.d.GlobalLock] (AgentManager-Handler-7:[]) (logid:) Checking lock availability for Host-Join-1
2026-10-01 11:34:53,832 DEBUG [c.c.u.d.DbUtil] (AgentManager-Handler-7:[]) (logid:) IS_FREE_LOCK('Host-Join-1') returned: false
2026-10-01 11:34:53,832 DEBUG [c.c.u.d.GlobalLock] (AgentManager-Handler-7:[]) (logid:) Result of checking lock availability for Host-Join-1: false
2026-10-01 11:34:53,832 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Global lock Host-Join-1 is not present for ref-trl-12457-k-Mol8-kiran-chavala-kvm1
2026-10-01 11:34:53,832 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Checking whether global lock Host-Join-1 is present for ref-trl-12457-k-Mol8-kiran-chavala-kvm1
2026-10-01 11:34:53,832 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 1-170010885933236225: Sending: { Ans: , MgmtId: 32989492806711, via: 1, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentConnectStatusAnswer":{"lockAvailable":"false","hostStatus":"Up","result":"true","details":"Global lock Host-Join-1 is not present for ref-trl-12457-k-Mol8-kiran-chavala-kvm1","traceContextMap":{},"wait":"0","bypassHostMaintenance":"false"}}] }
2026-10-01 11:35:08,896 DEBUG [c.c.u.d.GlobalLock] (AgentManager-Handler-8:[]) (logid:) Checking lock availability for Host-Join-1
2026-10-01 11:35:08,897 DEBUG [c.c.u.d.DbUtil] (AgentManager-Handler-8:[]) (logid:) IS_FREE_LOCK('Host-Join-1') returned: false
2026-10-01 11:35:08,897 DEBUG [c.c.u.d.GlobalLock] (AgentManager-Handler-8:[]) (logid:) Result of checking lock availability for Host-Join-1: false
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# tail -F /var/log/cloudstack/agent/agent.log | grep --line-buffered -E "There is (no )?lock and Host status|Host is locked|Host is in operational state|Received empty agent connect status|Reconnect info|Reconnecting to|Connected to|Duplicate connection"
2026-10-01 11:34:43,211 INFO [cloud.agent.Agent] (main:[]) (logid:087e22c4) Reconnect info: provided link: N/A, agent link: N/A, preferred host: null, force reconnect: false
2026-10-01 11:34:43,225 INFO [cloud.agent.Agent] (main:[]) (logid:381eea0e) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 11:34:43,241 INFO [utils.nio.NioClient] (main:[]) (logid:381eea0e) Connected to 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250: true
2026-10-01 11:34:43,785 INFO [cloud.agent.Agent] (main:[]) (logid:381eea0e) Connected to the host: 10.0.35.164 (54488-> /10.0.35.164:8250)
2026-10-01 11:34:53,844 INFO [cloud.agent.HostConnectProcess] (Agent-HostStatusTask-1:[]) (logid:) There is lock and Host status is Up, will retry later
2026-10-01 11:35:08,902 INFO [cloud.agent.HostConnectProcess] (Agent-HostStatusTask-1:[]) (logid:) There is lock and Host status is Up, will retry later
2026-10-01 11:35:23,915 INFO [cloud.agent.HostConnectProcess] (Agent-HostStatusTask-1:[]) (logid:) There is lock and Host status is Up, will retry later
Backoff algorithm
The MS reads the backoff.configuration global setting and sends it to the agent in the StartupAnswer. So the agent must connect once after you change the setting before it applies.
The agent builds the algorithm from it and saves it to agent.properties (backoff.implementation, backoff.min_delay_ms, backoff.max_delay_ms or backoff.seconds). On the next agent start, it reads these values from the file.
The backoff wait only runs between failed connection attempts. To see it, you have to make the connections fail.
Setup: run the tests on kvm1, and block both MS IPs, 10.0.35.164 and 10.0.32.198. If you block only one, the agent fails over to the other MS. Use REJECT, not DROP. With DROP, every attempt also waits for the socket timeout, which hides the backoff timing.
5a. Default: exponential with jitter
Check the current setting:
cmk list configurations name=backoff.configuration filter=name,value
backoff.implementation=com.cloud.utils.backoff.impl.ExponentialWithJitterBacko00;backoff.max_delay_ms=15000
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=backoff.configuration filter=name,value
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"name": "backoff.configuration",
"value": "backoff.implementation=com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff;backoff.min_delay_ms=5000;backoff.max_delay_ms=15000"
}
],
"count": 1
}
Restart the agent on kvm1 so it gets the setting, then confirm it applied:
systemctl restart cloudstack-agent; sleep 30
grep -E "Updating backoff delay algorithm|Created .* delay algorithm implementatgent/agent.log | tail -3
grep '^backoff.' /etc/cloudstack/agent/agent.properties
Expected: Created ExponentialWithJitterBackoff delay algorithm implementationare present in agent.properties.
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep '^backoff\.' /etc/cloudstack/agent/agent.properties
backoff.max_delay_ms=15000
backoff.min_delay_ms=5000
backoff.implementation=com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff
backoff.seconds=5
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "Updating backoff delay algorithm|Created .* delay algorithm implementation" /var/log/cloudstack/agent/agent.log | tail -3
2026-10-01 11:55:11,230 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:55:25,347 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Updating backoff delay algorithm implementation, defined in StartupAnswer
2026-10-01 11:55:25,353 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
5b. Constant backoff
Change the setting:
cmk update configuration name=backoff.configuration
value="backoff.implementation=com.cloud.utils.backoff.impl.ConstantTimeBackoff
Restart the agent on kvm1 so it receives the new setting:
systemctl restart cloudstack-agent; sleep 30
grep "Created .* delay algorithm implementation" /var/log/cloudstack/agent/agent.log
grep '^backoff.' /etc/cloudstack/agent/agent.properties
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep '^backoff\.' /etc/cloudstack/agent/agent.properties
backoff.max_delay_ms=15000
backoff.min_delay_ms=5000
backoff.implementation=com.cloud.utils.backoff.impl.ConstantTimeBackoff
backoff.seconds=10
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep "Created .* delay algorithm implementation" /var/log/cloudstack/agent/agent.log
2026-10-01 10:57:21,842 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 10:57:54,142 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:03:49,216 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:03:51,358 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:03:53,497 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:03:55,628 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:03:57,799 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:04:12,199 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:09:44,798 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:10:06,542 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:10:12,612 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:10:18,819 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:12:18,366 INFO [cloud.agent.Agent] (Agent-Handler-5:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:34:41,573 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:36:10,998 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:55:11,230 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:55:25,353 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 11:58:49,893 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:05:58,271 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:07:12,354 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:09:31,012 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:11:26,700 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:14:25,521 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
5c. Negative test: invalid class name from the MS
Set a class that doesn't exist:
cmk update configuration name=backoff.configuration value="backoff.implementation=com.cloud.Bogus"
Restart the agent and check the log:
systemctl restart cloudstack-agent; sleep 30
grep -E "Failed to create backoff with provided settings|Failed to create backoff implementation for" /var/log/cloudstack/agent/agent.log | tail -3
grep '^backoff.' /etc/cloudstack/agent/agent.properties
Expected:
The agent logs the warning but still connects (Up).
It keeps the previous algorithm.
agent.properties is not overwritten with the bad class.
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep '^backoff\.' /etc/cloudstack/agent/agent.properties
backoff.max_delay_ms=15000
backoff.min_delay_ms=5000
backoff.implementation=com.cloud.utils.backoff.impl.ConstantTimeBackoff
backoff.seconds=10
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "Failed to create backoff with provided settings|Failed to create backoff implementation for" /var/log/cloudstack/agent/agent.log
2026-10-01 12:16:46,438 WARN [utils.backoff.BackoffFactory] (Agent-Handler-3:[]) (logid:) Failed to create backoff implementation for com.cloud.Bogus java.lang.ClassNotFoundException: com.cloud.Bogus
2026-10-01 12:16:46,438 WARN [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Failed to create backoff with provided settings {backoff.implementation=com.cloud.Bogus} java.lang.RuntimeException: Failed to create backoff implementation for com.cloud.Bogus
5e. Negative test: invalid class in agent.properties
systemctl stop cloudstack-agent
sed -i 's/^backoff.implementation=./backoff.implementation=com.cloud.Bogus/' /etc/cloudstack/agent/agent.properties
systemctl start cloudstack-agent; sleep 30
grep -E "Failed to create backoff algorithm from the provided properties, falling back to default|Created . delay algorithm" /var/log/cloudstack/agent/agent.log | tail -3
Expected:
The agent starts using ConstantTimeBackoff as the fallback.
After it connects, the MS setting overwrites the file again.
2026-10-01 12:11:26,700 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:14:25,521 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ExponentialWithJitterBackoff delay algorithm implementation
2026-10-01 12:14:39,722 INFO [cloud.agent.Agent] (Agent-Handler-3:[]) (logid:) Created ConstantTimeBackoff delay algorithm implementation
2026-10-01 12:16:30,041 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ConstantTimeBackoff delay algorithm implementation
2026-10-01 12:16:32,352 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Created com.cloud.utils.backoff.impl.ConstantTimeBackoff delay algorithm implementation
2026-10-01 12:19:08,424 WARN [cloud.agent.AgentShell] (main:[]) (logid:) Failed to create backoff algorithm from the provided properties, falling back to default java.lang.RuntimeException: Failed to create backoff implementation for com.cloud.Bogus
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep '^backoff\.' /etc/cloudstack/agent/agent.properties
backoff.max_delay_ms=15000
backoff.min_delay_ms=5000
backoff.implementation=com.cloud.utils.backoff.impl.ConstantTimeBackoff
backoff.seconds=10
Multi-MS, indirect agent LB and hostnames
6a: Check the current settings and update
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=indirect.agent.lb.algorithm filter=name,value
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"name": "indirect.agent.lb.algorithm",
"value": "static"
}
],
"count": 1
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk list configurations name=indirect.agent.lb.check.interval filter=name,value
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": [
{
"name": "indirect.agent.lb.check.interval",
"value": "0"
}
],
"count": 1
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk update configuration name=indirect.agent.lb.algorithm value=roundrobin
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": {
"category": "Advanced",
"component": "IndirectAgentLBServiceImpl",
"defaultvalue": "static",
"description": "The algorithm to be applied on the provided management server list in the 'host' config that that is sent to indirect agents. Allowed values are: static, roundrobin and shuffle. Note: The lb algorithm 'shuffle' disables the indirect agent lb check background task once the algorithm is applied on the agent.",
"displaytext": "Indirect agent lb algorithm",
"group": "Management Server",
"isdynamic": true,
"name": "indirect.agent.lb.algorithm",
"options": "static,roundrobin,shuffle",
"subgroup": "Agent",
"type": "Select",
"value": "roundrobin"
}
}
root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E '^host=' /etc/cloudstack/agent/agent.properties
host=10.0.35.164,10.0.32.198@roundrobin
6b. Preferred-host checker
Turn the checker on:
cmk update configuration name=indirect.agent.lb.check.interval value=60
Restart both agents. Then confirm on each:
grep "Scheduling a recurring preferred host checker task with host.lb.interval" /var/log/cloudstack/agent/agent.log | tail -1
On mgmt1, stop the service:
systemctl stop cloudstack-management
kvm1 moves to mgmt2. This is test A from the previous message.
On mgmt1, start it again:
systemctl start cloudstack-management
Then watch kvm1:
[root@ref-trl-12457-k-Mol8-kiran-chavala-mgmt1 ~]# cmk update configuration name=indirect.agent.lb.check.interval value=60
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
{
"configuration": {
"category": "Advanced",
"component": "IndirectAgentLBServiceImpl",
"defaultvalue": "0",
"description": "The interval in seconds after which indirect agent should check and try to connect to its preferred host (the first management server from the propagated list provided in the 'host' config). Set 0 to disable it.",
"displaytext": "Indirect agent lb check interval",
"group": "Management Server",
"isdynamic": true,
"name": "indirect.agent.lb.check.interval",
"subgroup": "Agent",
"type": "Number",
"value": "60"
}
}
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep "Scheduling a recurring preferred host checker task with host.lb.interval" /var/log/cloudstack/agent/agent.log | tail -1
2026-10-01 12:27:42,955 INFO [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:d53bf99a) Scheduling a recurring preferred host checker task with host.lb.interval=60000 ms
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep "Scheduling a recurring preferred host checker task with host.lb.interval" /var/log/cloudstack/agent/agent.log | tail -1
2026-10-01 12:27:42,955 INFO [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:d53bf99a) Scheduling a recurring preferred host checker task with host.lb.interval=60000 ms
[root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "Running preferred host checker task, preferred host|Already connected to the preferred host|There are still .* commands in progress|Reconnecting to" /var/log/cloudstack/agent/agent.log | tail
2026-10-01 12:11:13,705 INFO [cloud.agent.Agent] (main:[]) (logid:0b888e65) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
2026-10-01 12:11:13,706 INFO [cloud.agent.Agent] (main:[]) (logid:077c8dfb) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:11:13,707 INFO [cloud.agent.Agent] (main:[]) (logid:43f2286b) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
2026-10-01 12:11:13,709 INFO [cloud.agent.Agent] (main:[]) (logid:03e433dd) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:14:27,116 INFO [cloud.agent.Agent] (main:[]) (logid:54a4f5ed) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:16:33,875 INFO [cloud.agent.Agent] (main:[]) (logid:e264365c) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:19:09,982 INFO [cloud.agent.Agent] (main:[]) (logid:a3a54bc8) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:28:15,412 INFO [cloud.agent.Agent] (main:[]) (logid:be827df3) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:29:49,902 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:67253b63) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:29:49,910 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:23b5fcd5) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
6c. Hostname mode
cmk update configuration name=host \
value="ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com,ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com"
root@ref-trl-12457-k-Mol8-kiran-chavala-kvm1 ~]# grep -E '^host=' /etc/cloudstack/agent/agent.properties
host=ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com,ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com@roundrobin
grep -E "Reconnecting to host:|Connected to" /var/log/cloudstack/agent/agent.log
2026-10-01 12:11:13,701 INFO [cloud.agent.Agent] (main:[]) (logid:7ccb1be3) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:11:13,702 INFO [cloud.agent.Agent] (main:[]) (logid:559d5fd0) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
2026-10-01 12:11:13,703 INFO [cloud.agent.Agent] (main:[]) (logid:70ec39ac) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:11:13,705 INFO [cloud.agent.Agent] (main:[]) (logid:0b888e65) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
2026-10-01 12:11:13,706 INFO [cloud.agent.Agent] (main:[]) (logid:077c8dfb) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
2026-10-01 12:11:13,707 INFO [cloud.agent.Agent] (main:[]) (logid:43f2286b) Reconnecting to host: 10.0.32.198/ref-trl-12457-k-mol8-kiran-chavala-mgmt2.sofia.shapeblue.com:8250
2026-10-01 12:11:13,709 INFO [cloud.agent.Agent] (main:[]) (logid:03e433dd) Reconnecting to host: 10.0.35.164/ref-trl-12457-k-mol8-kiran-chavala-mgmt1.sofia.shapeblue.com:8250
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kiranchavala
left a comment
Description
Continued from #13028
This PR improves the Indirect agent connection handling, has the following improvements.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Checked the indirect agents connections (and re-connections) with KVM hosts, SSVM & CPVM.
How did you try to break this feature and the system with this change?