hw/core/machine.c | 4 +- hw/virtio/virtio-mem.c | 144 ++++++++++++++++++++++++++++++++- include/hw/virtio/virtio-mem.h | 8 ++ include/migration/misc.h | 4 +- include/migration/vmstate.h | 28 ++++++- migration/migration.c | 9 +++ migration/migration.h | 4 + migration/ram.c | 8 +- migration/savevm.c | 105 +++++++++++++----------- 9 files changed, 255 insertions(+), 59 deletions(-)
While playing with migration of virtio-mem with an ordinary file backing,
I realized that migration and prealloc doesn't currently work as expected
for virtio-mem. Further, Jing Qi reported that setup issues (insufficient
huge pages on the destination) result in QEMU getting killed with SIGBUS
instead of failing gracefully.
In contrast to ordinary memory backend preallocation, virtio-mem
preallocates memory before plugging blocks to the guest. Consequently,
when migrating we are not actually preallocating on the destination but
"only" migrate pages. Fix that be migrating the bitmap early, before any
RAM content, and use that information to preallocate memory early, before
migrating any RAM.
Postcopy needs some extra care, and I realized that prealloc+postcopy is
shaky in general. Let's at least try to mimic what ordinary
prealloc+postcopy does: temporarily allocate the memory, discard it, and
cross fingers that we'll still have sufficient memory when postcopy
actually tries placing pages.
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
Cc: Juan Quintela <quintela@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Michal Privoznik <mprivozn@redhat.com>
v4 -> v5:
- "migration/savevm: Move more savevm handling into vmstate_save()"
-- Extended patch description regarding tracing
- "migration/savevm: Prepare vmdesc json writer in
qemu_savevm_state_setup()"
-- Move freeing to migrate_fd_cleanup()
- "migration/savevm: Allow immutable device state to be migrated early (i.e.,
before RAM)"
-- "immutable" -> "early_setup"
-- Extend comment
- Added some RBs (thanks!)
v3 -> v4:
- First 3 patches:
-- Minimze code changes and simplify
-- Save immutable device state during qemu_savevm_state_setup()
-- Don't use vmsd priorities, use a new flag
-- Split it logically up
- "migration/ram: Factor out check for advised postcopy"
-- Don't factor out postcopy_is_running()
- "virtio-mem: Migrate immutable properties early"
-- Adjust to changed vmsd interface
- "virtio-mem: Proper support for preallocation with migration"
-- Drop sanity check in virtio_mem_post_load_early()
v2 -> v3:
- New approach/rewrite, drop RB and TB of last patch
v1 -> v2:
- Added RBs and Tested-bys
- "virtio-mem: Fail if a memory backend with "prealloc=on" is specified"
-- Fail instead of warn
-- Adjust subject/description
David Hildenbrand (8):
migration/savevm: Move more savevm handling into vmstate_save()
migration/savevm: Prepare vmdesc json writer in
qemu_savevm_state_setup()
migration/savevm: Allow immutable device state to be migrated early
(i.e., before RAM)
migration/vmstate: Introduce VMSTATE_WITH_TMP_TEST() and
VMSTATE_BITMAP_TEST()
migration/ram: Factor out check for advised postcopy
virtio-mem: Fail if a memory backend with "prealloc=on" is specified
virtio-mem: Migrate immutable properties early
virtio-mem: Proper support for preallocation with migration
hw/core/machine.c | 4 +-
hw/virtio/virtio-mem.c | 144 ++++++++++++++++++++++++++++++++-
include/hw/virtio/virtio-mem.h | 8 ++
include/migration/misc.h | 4 +-
include/migration/vmstate.h | 28 ++++++-
migration/migration.c | 9 +++
migration/migration.h | 4 +
migration/ram.c | 8 +-
migration/savevm.c | 105 +++++++++++++-----------
9 files changed, 255 insertions(+), 59 deletions(-)
--
2.39.0
On 17.01.23 12:22, David Hildenbrand wrote: > While playing with migration of virtio-mem with an ordinary file backing, > I realized that migration and prealloc doesn't currently work as expected > for virtio-mem. Further, Jing Qi reported that setup issues (insufficient > huge pages on the destination) result in QEMU getting killed with SIGBUS > instead of failing gracefully. > > In contrast to ordinary memory backend preallocation, virtio-mem > preallocates memory before plugging blocks to the guest. Consequently, > when migrating we are not actually preallocating on the destination but > "only" migrate pages. Fix that be migrating the bitmap early, before any > RAM content, and use that information to preallocate memory early, before > migrating any RAM. > > Postcopy needs some extra care, and I realized that prealloc+postcopy is > shaky in general. Let's at least try to mimic what ordinary > prealloc+postcopy does: temporarily allocate the memory, discard it, and > cross fingers that we'll still have sufficient memory when postcopy > actually tries placing pages. > > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com> > Cc: Juan Quintela <quintela@redhat.com> > Cc: Peter Xu <peterx@redhat.com> > Cc: Michael S. Tsirkin <mst@redhat.com> > Cc: Michal Privoznik <mprivozn@redhat.com> @Juan, David: I can similarly take this via my tree as well. -- Thanks, David / dhildenb
David Hildenbrand <david@redhat.com> wrote: > On 17.01.23 12:22, David Hildenbrand wrote: >> While playing with migration of virtio-mem with an ordinary file backing, >> I realized that migration and prealloc doesn't currently work as expected >> for virtio-mem. Further, Jing Qi reported that setup issues (insufficient >> huge pages on the destination) result in QEMU getting killed with SIGBUS >> instead of failing gracefully. >> In contrast to ordinary memory backend preallocation, virtio-mem >> preallocates memory before plugging blocks to the guest. Consequently, >> when migrating we are not actually preallocating on the destination but >> "only" migrate pages. Fix that be migrating the bitmap early, before any >> RAM content, and use that information to preallocate memory early, before >> migrating any RAM. >> Postcopy needs some extra care, and I realized that >> prealloc+postcopy is >> shaky in general. Let's at least try to mimic what ordinary >> prealloc+postcopy does: temporarily allocate the memory, discard it, and >> cross fingers that we'll still have sufficient memory when postcopy >> actually tries placing pages. >> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com> >> Cc: Juan Quintela <quintela@redhat.com> >> Cc: Peter Xu <peterx@redhat.com> >> Cc: Michael S. Tsirkin <mst@redhat.com> >> Cc: Michal Privoznik <mprivozn@redhat.com> > > @Juan, David: I can similarly take this via my tree as well. Reviewing it. I can get it through migration tree, thanks. Later, Juan.
On 02.02.23 12:04, Juan Quintela wrote: > David Hildenbrand <david@redhat.com> wrote: >> On 17.01.23 12:22, David Hildenbrand wrote: >>> While playing with migration of virtio-mem with an ordinary file backing, >>> I realized that migration and prealloc doesn't currently work as expected >>> for virtio-mem. Further, Jing Qi reported that setup issues (insufficient >>> huge pages on the destination) result in QEMU getting killed with SIGBUS >>> instead of failing gracefully. >>> In contrast to ordinary memory backend preallocation, virtio-mem >>> preallocates memory before plugging blocks to the guest. Consequently, >>> when migrating we are not actually preallocating on the destination but >>> "only" migrate pages. Fix that be migrating the bitmap early, before any >>> RAM content, and use that information to preallocate memory early, before >>> migrating any RAM. >>> Postcopy needs some extra care, and I realized that >>> prealloc+postcopy is >>> shaky in general. Let's at least try to mimic what ordinary >>> prealloc+postcopy does: temporarily allocate the memory, discard it, and >>> cross fingers that we'll still have sufficient memory when postcopy >>> actually tries placing pages. >>> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com> >>> Cc: Juan Quintela <quintela@redhat.com> >>> Cc: Peter Xu <peterx@redhat.com> >>> Cc: Michael S. Tsirkin <mst@redhat.com> >>> Cc: Michal Privoznik <mprivozn@redhat.com> >> >> @Juan, David: I can similarly take this via my tree as well. > > Reviewing it. > > I can get it through migration tree, thanks. > > Later, Juan. Thanks Juan and Michael! -- Thanks, David / dhildenb
On Tue, Jan 17, 2023 at 12:22:41PM +0100, David Hildenbrand wrote: > While playing with migration of virtio-mem with an ordinary file backing, > I realized that migration and prealloc doesn't currently work as expected > for virtio-mem. Further, Jing Qi reported that setup issues (insufficient > huge pages on the destination) result in QEMU getting killed with SIGBUS > instead of failing gracefully. > > In contrast to ordinary memory backend preallocation, virtio-mem > preallocates memory before plugging blocks to the guest. Consequently, > when migrating we are not actually preallocating on the destination but > "only" migrate pages. Fix that be migrating the bitmap early, before any > RAM content, and use that information to preallocate memory early, before > migrating any RAM. > > Postcopy needs some extra care, and I realized that prealloc+postcopy is > shaky in general. Let's at least try to mimic what ordinary > prealloc+postcopy does: temporarily allocate the memory, discard it, and > cross fingers that we'll still have sufficient memory when postcopy > actually tries placing pages. > > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com> > Cc: Juan Quintela <quintela@redhat.com> > Cc: Peter Xu <peterx@redhat.com> > Cc: Michael S. Tsirkin <mst@redhat.com> > Cc: Michal Privoznik <mprivozn@redhat.com> For patch 1-5: Reviewed-by: Peter Xu <peterx@redhat.com> -- Peter Xu
On Tue, Jan 17, 2023 at 12:22:41PM +0100, David Hildenbrand wrote: > While playing with migration of virtio-mem with an ordinary file backing, > I realized that migration and prealloc doesn't currently work as expected > for virtio-mem. Further, Jing Qi reported that setup issues (insufficient > huge pages on the destination) result in QEMU getting killed with SIGBUS > instead of failing gracefully. > > In contrast to ordinary memory backend preallocation, virtio-mem > preallocates memory before plugging blocks to the guest. Consequently, > when migrating we are not actually preallocating on the destination but > "only" migrate pages. Fix that be migrating the bitmap early, before any > RAM content, and use that information to preallocate memory early, before > migrating any RAM. > > Postcopy needs some extra care, and I realized that prealloc+postcopy is > shaky in general. Let's at least try to mimic what ordinary > prealloc+postcopy does: temporarily allocate the memory, discard it, and > cross fingers that we'll still have sufficient memory when postcopy > actually tries placing pages. > > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com> > Cc: Juan Quintela <quintela@redhat.com> > Cc: Peter Xu <peterx@redhat.com> > Cc: Michael S. Tsirkin <mst@redhat.com> > Cc: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> > v4 -> v5: > - "migration/savevm: Move more savevm handling into vmstate_save()" > -- Extended patch description regarding tracing > - "migration/savevm: Prepare vmdesc json writer in > qemu_savevm_state_setup()" > -- Move freeing to migrate_fd_cleanup() > - "migration/savevm: Allow immutable device state to be migrated early (i.e., > before RAM)" > -- "immutable" -> "early_setup" > -- Extend comment > - Added some RBs (thanks!) > > v3 -> v4: > - First 3 patches: > -- Minimze code changes and simplify > -- Save immutable device state during qemu_savevm_state_setup() > -- Don't use vmsd priorities, use a new flag > -- Split it logically up > - "migration/ram: Factor out check for advised postcopy" > -- Don't factor out postcopy_is_running() > - "virtio-mem: Migrate immutable properties early" > -- Adjust to changed vmsd interface > - "virtio-mem: Proper support for preallocation with migration" > -- Drop sanity check in virtio_mem_post_load_early() > > v2 -> v3: > - New approach/rewrite, drop RB and TB of last patch > > v1 -> v2: > - Added RBs and Tested-bys > - "virtio-mem: Fail if a memory backend with "prealloc=on" is specified" > -- Fail instead of warn > -- Adjust subject/description > > > David Hildenbrand (8): > migration/savevm: Move more savevm handling into vmstate_save() > migration/savevm: Prepare vmdesc json writer in > qemu_savevm_state_setup() > migration/savevm: Allow immutable device state to be migrated early > (i.e., before RAM) > migration/vmstate: Introduce VMSTATE_WITH_TMP_TEST() and > VMSTATE_BITMAP_TEST() > migration/ram: Factor out check for advised postcopy > virtio-mem: Fail if a memory backend with "prealloc=on" is specified > virtio-mem: Migrate immutable properties early > virtio-mem: Proper support for preallocation with migration > > hw/core/machine.c | 4 +- > hw/virtio/virtio-mem.c | 144 ++++++++++++++++++++++++++++++++- > include/hw/virtio/virtio-mem.h | 8 ++ > include/migration/misc.h | 4 +- > include/migration/vmstate.h | 28 ++++++- > migration/migration.c | 9 +++ > migration/migration.h | 4 + > migration/ram.c | 8 +- > migration/savevm.c | 105 +++++++++++++----------- > 9 files changed, 255 insertions(+), 59 deletions(-) > > -- > 2.39.0
© 2016 - 2023 Red Hat, Inc.