linstor: fix live migration with storage into Linstor when the target host has no replica by rp- · Pull Request #14344 · apache/cloudstack

rp- · 2026-10-07T14:27:49Z

Description

This PR fixes live migration with storage (migrateVirtualMachineWithVolume) from another primary
storage (e.g. NFS) into a LINSTOR primary storage, which fails whenever the LINSTOR auto-placement
does not put a copy of the new resource on the migration target host:

Failed to migrate VM [...] along with its volumes due to [...] failed in LinstorDataMotionStrategy.copyAsync.
Error message: [Exception during migrate: org.libvirt.LibvirtException: internal error:
cannot precreate storage for disk type 'block']

LinstorDataMotionStrategy.copyAsync spawns the destination resource from the resource group, so
LINSTOR picks the nodes without knowing about the target host. The PrepareForMigrationCommand
sent to the target still describes the source disks, so the target agent never connects the new
volume either. If the target host has no copy, /dev/drbd/by-res/cs-<uuid>/0 does not exist there,
and libvirt can only precreate missing file disks, not block devices.

On 3-node clusters with 2 replicas this rarely shows up, because LINSTOR adds a diskless quorum
tiebreaker on the third node. It fails with place count 1, with the tiebreaker disabled, with 4+
nodes and 2 replicas, or when the resource group's filters exclude the target host. Offline
migration (stop the VM, migrate the volume) is not affected, because starting the VM goes through
connectPhysicalDisk.

Changes:

After creating the destination resource, make it available on the target host
(resourceMakeAvailableOnNode, a diskless resource for DRBD) and take the device path from
there. No dual-primary is needed, the source disk is not this LINSTOR resource.
If the copy fails before the migrate command is sent (make-available, prepare for migration),
the already created destination volumes are now destroyed and expunged. Before, they were left
behind in LINSTOR and in the volumes table.
The error message named the source host as "storage(s)"; it now lists the destination pools.
Linstor plugin CHANGELOG entry.

The strategy was added in #12532, so 4.20.3.0 and 4.22.1.0 are affected.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

3 KVM hosts (Ubuntu 24.04), LINSTOR 1.35.2 / DRBD 9.3, a 4.22.1.1-based CloudStack build with this
patch on the management server (LinstorDataMotionStrategy is identical to the 4.22 branch). To make the failure deterministic on 3 nodes, the target LINSTOR pool uses a
resource group with place count 1 restricted to a storage pool that only exists on host 2. A
running VM with its root disk on an NFS primary storage is migrated from host 1 to host 3 with
migrateVirtualMachineWithVolume, root volume to the LINSTOR pool.

Without the patch: fails with cannot precreate storage for disk type 'block' (reproduced).
With the patch: migration succeeds, the VM keeps running (no reboot) on host 3 from
/dev/drbd/by-res/cs-<uuid>/0 (diskless, InUse; diskful copy on host 2 only), a 64 MiB random
file written in the guest before the migration has the same sha256 afterwards, the NFS volume
is expunged.

How did you try to break this feature and the system with this change?

Make-available fails: LINSTOR satellite on the target host stopped. The migration fails with the
LINSTOR error, the VM stays running on the source host on NFS, the destination volume is
expunged and its LINSTOR resource definition is deleted.
Migration fails after make-available: qemu migration/NBD ports (49152-49215) rejected on the
target host. The migration fails in blockdev-add, the VM stays on the source host, the
destination resource definition, including the new diskless resource on the target, is deleted.
Linstor plugin unit tests and checkstyle pass.

Not tested: shared (thick LVM) LINSTOR storage pools, VMs with multiple volumes.

codecov · 2026-10-07T14:35:19Z

Codecov Report

❌ Patch coverage is 79.03226% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.06%. Comparing base (2974af8) to head (97a1170).
⚠️ Report is 1 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...tack/storage/motion/LinstorDataMotionStrategy.java	79.03%	8 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #14344      +/-   ##
============================================
+ Coverage     18.02%   18.06%   +0.04%     
- Complexity    16250    16272      +22     
============================================
  Files          5936     5936              
  Lines        535823   535866      +43     
  Branches      65612    65617       +5     
============================================
+ Hits          96582    96817     +235     
+ Misses       428242   428037     -205     
- Partials      10999    11012      +13

Flag	Coverage Δ
uitests	`4.04% <ø> (ø)`
unittests	`19.14% <79.03%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Damans227 · 2026-10-08T11:32:22Z

            MigrateAnswer migrateAnswer = (MigrateAnswer) _agentManager.send(srcHost.getId(), migrateCommand);
            boolean success = migrateAnswer != null && migrateAnswer.getResult();

+            postMigrationHandled = true;


what happens if the migrate call times out while the vm is still moving? this flag is only set after it returns, so the cleanup could delete the new volume the vm ends up running on

Thanks, now the new volumes are only cleaned up if the migration itself failed.

that covers it, thanks

Damans227 · 2026-10-08T11:32:24Z


-                String devPath = LinstorUtil.createResource(
-                        destVolumeInfo, destStoragePool, _storagePoolDao, exactSize);
+                LinstorUtil.createResource(destVolumeInfo, destStoragePool, _storagePoolDao, exactSize);


if creating the linstor resource fails here, does the new volume row still get left behind? it only goes into the cleanup list a few lines later

Also thanks, it is added to the cleanup list now after the DB object was created

that covers it, thanks

…migration Live migration with storage into a Linstor pool spawned the destination resource through the resource group's auto-placement, which knows nothing about the target host. PrepareForMigrationCommand still describes the source disks, so the target agent never connected the new volume either. If the placement left the target host without a copy (place count 1, no quorum tiebreaker, 4+ nodes with 2 copies, node filters), the block device was missing there and libvirt aborted with "cannot precreate storage for disk type 'block'". Make the resource available on the target host right after creating it and take the device path from there. The source is not a Linstor resource of this pool, so no dual-primary is needed. A failure before the migration command (creating the resource, make-available, prepare for migration) now also removes the already created destination volumes; they used to be left behind. Once the migration command is sent, the destination volumes are only removed if the source host reports a failure. If the command times out, the VM may still end up on the new volumes: like StorageSystemDataMotionStrategy, check whether it runs on the target host and finish the migration in that case, otherwise keep the volumes. The error message names the destination pools instead of the source host. Unit tests cover make-available before the migration with the device path from the target host, the cleanup when creating the resource, make-available, prepare for migration or the migration itself fails, and a migration timeout with and without the VM running on the target host.

…rage LinstorDataMotionStrategy set the MigrateCommand CPU shares to the raw cpus * speed value, which is only valid on cgroup v1. On cgroup v2 hosts the source agent's updateVmSharesIfNeeded() sees that it differs from its own scaled value and writes it into the migration XML, so libvirt on the target rejects VMs above 10000 with "shares '<n>' must be in range [1, 10000]", and smaller VMs end up with an unscaled CPU weight. Use the value the target host calculated and returned in the PrepareForMigrationAnswer, as StorageSystemDataMotionStrategy and VirtualMachineManagerImpl do. A unit test checks that the target host's value is sent, not cpus * speed.

Damans227 · 2026-10-08T15:46:52Z

thanks @rp-, fix looks right. LGTM.

rp- requested review from DaanHoogland, Pearl1594 and sureshanaparti October 7, 2026 14:27

rp- self-assigned this Oct 7, 2026

boring-cyborg Bot added component:primary-storage component:storage labels Oct 7, 2026

rp- force-pushed the linstor-4.22-migrate-to-linstor-make-available branch from 0a1c6da to 70d43cd Compare October 8, 2026 10:59

rp- added the 3rd-party:linstor label Oct 8, 2026

Damans227 reviewed Oct 8, 2026

View reviewed changes

rp- added 2 commits October 8, 2026 16:16

rp- force-pushed the linstor-4.22-migrate-to-linstor-make-available branch from 70d43cd to 97a1170 Compare October 8, 2026 14:17

rp- requested a review from Damans227 October 8, 2026 14:25

Navigation Menu

Repository navigation

linstor: fix live migration with storage into Linstor when the target host has no replica - #14344

linstor: fix live migration with storage into Linstor when the target host has no replica#14344
rp- wants to merge 2 commits into
apache:4.22from
LINBIT:linstor-4.22-migrate-to-linstor-make-available

rp- commented Oct 7, 2026

Uh oh!

codecov Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

Damans227 Oct 8, 2026

Uh oh!

rp- Oct 8, 2026

Uh oh!

Damans227 Oct 8, 2026

Uh oh!

Damans227 Oct 8, 2026

Uh oh!

rp- Oct 8, 2026

Uh oh!

Damans227 Oct 8, 2026

Uh oh!

Damans227 commented Oct 8, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rp- commented Oct 7, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Damans227 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

rp- Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

rp- Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 commented Oct 8, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Oct 7, 2026 •

edited

Loading