docs/devel/vfio-migration-plugin.rst | 165 +++++++ hw/vfio/meson.build | 2 + hw/vfio/migration-local.c | 456 +++++++++++++++++++ hw/vfio/migration-plugin.c | 266 +++++++++++ hw/vfio/migration.c | 577 ++++++------------------ hw/vfio/pci.c | 2 + hw/vfio/trace-events | 9 +- include/hw/vfio/vfio-common.h | 37 +- include/hw/vfio/vfio-migration-plugin.h | 21 + 9 files changed, 1096 insertions(+), 439 deletions(-) create mode 100644 docs/devel/vfio-migration-plugin.rst create mode 100644 hw/vfio/migration-local.c create mode 100644 hw/vfio/migration-plugin.c create mode 100644 include/hw/vfio/vfio-migration-plugin.h
Migration of a VFIO passthrough device can be supported by using a device
specific kernel driver to save/restore the device state thru device specific
interfaces. But this approach doesn't work for devices that lack a state
migration interface, e.g. NVMe.
On the other hand, Infrastructure Process Unit (IPU) or Data Processing Unit
(DPU) vendors may choose to implement an out-of-band interface from the SoC to
help manage the state of such non-migratable devices e.g. via gRPC or JSON-RPC
protocols.
This RFC attempts to support such out-of-band migration interface by introducing
the concept of migration backends in vfio. The existing logic around vfio
migration uAPI is now called the 'local' backend while a new 'out-of-band'
backend is further introduced allowing vfio to redirect VMState ops to an
external plugin.
Currently, the backend migration Ops is defined close to SaveVMHandlers. We also
considered whether there is value of abstracting it in a lower level e.g. close
to vfio migration uAPI but no clear conclusion. Hence this is one part which
we'd like to hear suggestions.
This proposal adopts a plugin mechanism (an example can be found in [1]) given
that IPU/DPU vendors usually implement proprietary migration interfaces without
a standard. But we are also open if an alternative option makes better sense,
e.g. via loadable modules (with Qemu supporting gRPC or JSON-RPC support) or an
IPC mechanism similar to vhost-user.
The following graph describes the overall component relationship:
+----------------------------------------------------+
| QEMU |
| +------------------------------------------------+ |
| | VFIO Live Migration Framework | |
| | +--------------------------------------+ | |
| | | VFIOMigrationOps | | |
| | +-------^---------------------^--------+ | |
| | | | | |
| | +-------v-------+ +-------v--------+ | |
| | | LM Backend Via| | LM Backend Via | | |
| | | Device Fd | | Plugins | | |
| | +-------^-------+ | +----------+ | |
| | | | |Plugin Ops+----+-+------------+
| | | +-----+----------+ | | |
| | | | | +---------v----------+
| +------------+-----------------------------------+ | | Vendor Specific |
| | | | Plugins(.so) |
+--------------+-------------------------------------+ +----------+---------+
UserSpace | |
----------------+--------------------------------------------- |
Kernel | |
| |
+----------v----------------------+ |
| Kernel VFIO Driver | |
| +-------------------------+ | |
| | | | | Network
| | Vendor-Specific Driver | | |
| | | | |
| +----------^--------------+ | |
| | | |
+---------------+-----------------+ |
| |
| |
---------------------+----------------------------------------- |
Hardware | |
| +-----+-----+-----+----+-----+ |
+----------v------+ | VF0 | VF1 | VF2 | ...| VFn | |
| Traditional | +-----+-----+-----+----+-----+ |
| PCIe Devices | | | |
+-----------------+ | +--------+------------+ | |
| | | Agent |<-+----+
| | +------------+ |
| | | |
| | SOC | |
| +---------------------+ |
| IPU |
+----------------------------+
Two command-line parameters (x-plugin-path and x-plugin-arg) are introduced to
enable the out-of-band backend. If specified, vfio will attempt to use the
out-of-band backend.
The following is an example of VFIO command-line parameters for OOB-Approach:
-device vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_path,x-plugin-arg=$plugin_arg
[1] https://github.com/raolei-intel/vfio-lm-plugin-example.git
Lei Rao (13):
vfio/migration: put together checks of migration initialization
conditions
vfio/migration: move migration struct allocation out of
vfio_migration_init
vfio/migration: move vfio_get_dev_region_info out of
vfio_migration_probe
vfio/migration: Separated functions that relate to the In-Band
approach
vfio/migration: rename functions that relate to the In-Band approach
vfio/migration: introduce VFIOMigrationOps layer in VFIO live
migration framework
vfio/migration: move the statistics of bytes_transferred to generic
VFIO migration layer
vfio/migration: split migration handler registering from
vfio_migration_init
vfio/migration: move the functions of In-Band approach to a new file
vfio/pci: introduce command-line parameters to specify migration
method
vfio/migration: add a plugin layer to support out-of-band live
migration
vfio/migration: add some trace-events for vfio migration plugin
vfio/migration: make the region and plugin member of struct
VFIOMigration to be a union
docs/devel/vfio-migration-plugin.rst | 165 +++++++
hw/vfio/meson.build | 2 +
hw/vfio/migration-local.c | 456 +++++++++++++++++++
hw/vfio/migration-plugin.c | 266 +++++++++++
hw/vfio/migration.c | 577 ++++++------------------
hw/vfio/pci.c | 2 +
hw/vfio/trace-events | 9 +-
include/hw/vfio/vfio-common.h | 37 +-
include/hw/vfio/vfio-migration-plugin.h | 21 +
9 files changed, 1096 insertions(+), 439 deletions(-)
create mode 100644 docs/devel/vfio-migration-plugin.rst
create mode 100644 hw/vfio/migration-local.c
create mode 100644 hw/vfio/migration-plugin.c
create mode 100644 include/hw/vfio/vfio-migration-plugin.h
--
2.32.0
On Tue, 24 May 2022 14:18:35 +0800 Lei Rao <lei.rao@intel.com> wrote: > Migration of a VFIO passthrough device can be supported by using a device > specific kernel driver to save/restore the device state thru device specific > interfaces. But this approach doesn't work for devices that lack a state > migration interface, e.g. NVMe. > > On the other hand, Infrastructure Process Unit (IPU) or Data Processing Unit > (DPU) vendors may choose to implement an out-of-band interface from the SoC to > help manage the state of such non-migratable devices e.g. via gRPC or JSON-RPC > protocols. > > This RFC attempts to support such out-of-band migration interface by introducing > the concept of migration backends in vfio. The existing logic around vfio > migration uAPI is now called the 'local' backend while a new 'out-of-band' > backend is further introduced allowing vfio to redirect VMState ops to an > external plugin. > > Currently, the backend migration Ops is defined close to SaveVMHandlers. We also > considered whether there is value of abstracting it in a lower level e.g. close > to vfio migration uAPI but no clear conclusion. Hence this is one part which > we'd like to hear suggestions. > > This proposal adopts a plugin mechanism (an example can be found in [1]) given > that IPU/DPU vendors usually implement proprietary migration interfaces without > a standard. But we are also open if an alternative option makes better sense, > e.g. via loadable modules (with Qemu supporting gRPC or JSON-RPC support) or an > IPC mechanism similar to vhost-user. AFAIU, QEMU is not interested in supporting plugin modules, especially proprietary ones. I don't see that a case has really been made that this cannot be done in-band, through a vfio-pci variant driver, possibly making use of proprietary interfaces to a userspace agent if necessary (though please don't ask such to be accepted in-tree for the kernel either). A vfio-user device server might also host such proprietary, device specific agents while supporting the standard, in-band migration interface. Thanks, Alex > > The following graph describes the overall component relationship: > > +----------------------------------------------------+ > | QEMU | > | +------------------------------------------------+ | > | | VFIO Live Migration Framework | | > | | +--------------------------------------+ | | > | | | VFIOMigrationOps | | | > | | +-------^---------------------^--------+ | | > | | | | | | > | | +-------v-------+ +-------v--------+ | | > | | | LM Backend Via| | LM Backend Via | | | > | | | Device Fd | | Plugins | | | > | | +-------^-------+ | +----------+ | | > | | | | |Plugin Ops+----+-+------------+ > | | | +-----+----------+ | | | > | | | | | +---------v----------+ > | +------------+-----------------------------------+ | | Vendor Specific | > | | | | Plugins(.so) | > +--------------+-------------------------------------+ +----------+---------+ > UserSpace | | > ----------------+--------------------------------------------- | > Kernel | | > | | > +----------v----------------------+ | > | Kernel VFIO Driver | | > | +-------------------------+ | | > | | | | | Network > | | Vendor-Specific Driver | | | > | | | | | > | +----------^--------------+ | | > | | | | > +---------------+-----------------+ | > | | > | | > ---------------------+----------------------------------------- | > Hardware | | > | +-----+-----+-----+----+-----+ | > +----------v------+ | VF0 | VF1 | VF2 | ...| VFn | | > | Traditional | +-----+-----+-----+----+-----+ | > | PCIe Devices | | | | > +-----------------+ | +--------+------------+ | | > | | | Agent |<-+----+ > | | +------------+ | > | | | | > | | SOC | | > | +---------------------+ | > | IPU | > +----------------------------+ > > Two command-line parameters (x-plugin-path and x-plugin-arg) are introduced to > enable the out-of-band backend. If specified, vfio will attempt to use the > out-of-band backend. > > The following is an example of VFIO command-line parameters for OOB-Approach: > > -device vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_path,x-plugin-arg=$plugin_arg > > [1] https://github.com/raolei-intel/vfio-lm-plugin-example.git > > Lei Rao (13): > vfio/migration: put together checks of migration initialization > conditions > vfio/migration: move migration struct allocation out of > vfio_migration_init > vfio/migration: move vfio_get_dev_region_info out of > vfio_migration_probe > vfio/migration: Separated functions that relate to the In-Band > approach > vfio/migration: rename functions that relate to the In-Band approach > vfio/migration: introduce VFIOMigrationOps layer in VFIO live > migration framework > vfio/migration: move the statistics of bytes_transferred to generic > VFIO migration layer > vfio/migration: split migration handler registering from > vfio_migration_init > vfio/migration: move the functions of In-Band approach to a new file > vfio/pci: introduce command-line parameters to specify migration > method > vfio/migration: add a plugin layer to support out-of-band live > migration > vfio/migration: add some trace-events for vfio migration plugin > vfio/migration: make the region and plugin member of struct > VFIOMigration to be a union > > docs/devel/vfio-migration-plugin.rst | 165 +++++++ > hw/vfio/meson.build | 2 + > hw/vfio/migration-local.c | 456 +++++++++++++++++++ > hw/vfio/migration-plugin.c | 266 +++++++++++ > hw/vfio/migration.c | 577 ++++++------------------ > hw/vfio/pci.c | 2 + > hw/vfio/trace-events | 9 +- > include/hw/vfio/vfio-common.h | 37 +- > include/hw/vfio/vfio-migration-plugin.h | 21 + > 9 files changed, 1096 insertions(+), 439 deletions(-) > create mode 100644 docs/devel/vfio-migration-plugin.rst > create mode 100644 hw/vfio/migration-local.c > create mode 100644 hw/vfio/migration-plugin.c > create mode 100644 include/hw/vfio/vfio-migration-plugin.h >
> -----Original Message----- > From: Qemu-devel <qemu-devel- > bounces+eddie.dong=intel.com@nongnu.org> On Behalf Of Alex Williamson > Sent: Thursday, May 26, 2022 11:44 AM > To: Rao, Lei <Lei.Rao@intel.com> > Cc: Tian, Kevin <kevin.tian@intel.com>; Dong, Eddie > <eddie.dong@intel.com>; Zeng, Jason <jason.zeng@intel.com>; > quintela@redhat.com; dgilbert@redhat.com; Li, Yadong > <yadong.li@intel.com>; Liu, Yi L <yi.l.liu@intel.com>; qemu- > devel@nongnu.org > Subject: Re: [RFC PATCH 00/13] Add a plugin to support out-of-band live > migration for VFIO pass-through device > > On Tue, 24 May 2022 14:18:35 +0800 > Lei Rao <lei.rao@intel.com> wrote: > > > Migration of a VFIO passthrough device can be supported by using a > > device specific kernel driver to save/restore the device state thru > > device specific interfaces. But this approach doesn't work for devices > > that lack a state migration interface, e.g. NVMe. > > > > On the other hand, Infrastructure Process Unit (IPU) or Data > > Processing Unit > > (DPU) vendors may choose to implement an out-of-band interface from > > the SoC to help manage the state of such non-migratable devices e.g. > > via gRPC or JSON-RPC protocols. > > > > This RFC attempts to support such out-of-band migration interface by > > introducing the concept of migration backends in vfio. The existing > > logic around vfio migration uAPI is now called the 'local' backend while a > new 'out-of-band' > > backend is further introduced allowing vfio to redirect VMState ops to > > an external plugin. > > > > Currently, the backend migration Ops is defined close to > > SaveVMHandlers. We also considered whether there is value of > > abstracting it in a lower level e.g. close to vfio migration uAPI but > > no clear conclusion. Hence this is one part which we'd like to hear > suggestions. > > > > This proposal adopts a plugin mechanism (an example can be found in > > [1]) given that IPU/DPU vendors usually implement proprietary > > migration interfaces without a standard. But we are also open if an > > alternative option makes better sense, e.g. via loadable modules (with > > Qemu supporting gRPC or JSON-RPC support) or an IPC mechanism similar > to vhost-user. > > AFAIU, QEMU is not interested in supporting plugin modules, especially > proprietary ones. I don't see that a case has really been made that this > cannot be done in-band, through a vfio-pci variant driver, possibly making > use of proprietary interfaces to a userspace agent if necessary (though > please don't ask such to be accepted in-tree for the kernel either). A vfio- > user device server might also host such proprietary, device specific agents > while supporting the standard, in-band migration interface. Thanks, > Thanks Alex. Removing plug-in module is not a problem. Do you mean to implement the migration and protocol handling inside Qemu-client (probably vfio-client after the VFIO-user is merged)? Or to build as part of libvfio-user? We can also build it as a separate process of Qemu-server as part of Multi-Process Qemu. In here, the protocol between host CPU and SoC is based on gRPC, which support Rust code in client (Host CPU side here) more than C/C++. Do you have any suggestion to better support Rust code with Qemu C/C++ code? Thx Eddie > Alex > > > > > The following graph describes the overall component relationship: > > > > +----------------------------------------------------+ > > | QEMU | > > | +------------------------------------------------+ | > > | | VFIO Live Migration Framework | | > > | | +--------------------------------------+ | | > > | | | VFIOMigrationOps | | | > > | | +-------^---------------------^--------+ | | > > | | | | | | > > | | +-------v-------+ +-------v--------+ | | > > | | | LM Backend Via| | LM Backend Via | | | > > | | | Device Fd | | Plugins | | | > > | | +-------^-------+ | +----------+ | | > > | | | | |Plugin Ops+----+-+------------+ > > | | | +-----+----------+ | | | > > | | | | | +---------v----------+ > > | +------------+-----------------------------------+ | | Vendor Specific | > > | | | | Plugins(.so) | > > +--------------+-------------------------------------+ +----------+---------+ > > UserSpace | | > > ----------------+--------------------------------------------- | > > Kernel | | > > | | > > +----------v----------------------+ | > > | Kernel VFIO Driver | | > > | +-------------------------+ | | > > | | | | | Network > > | | Vendor-Specific Driver | | | > > | | | | | > > | +----------^--------------+ | | > > | | | | > > +---------------+-----------------+ | > > | | > > | | > > ---------------------+----------------------------------------- | > > Hardware | | > > | +-----+-----+-----+----+-----+ | > > +----------v------+ | VF0 | VF1 | VF2 | ...| VFn | | > > | Traditional | +-----+-----+-----+----+-----+ | > > | PCIe Devices | | | | > > +-----------------+ | +--------+------------+ | | > > | | | Agent |<-+----+ > > | | +------------+ | > > | | | | > > | | SOC | | > > | +---------------------+ | > > | IPU | > > +----------------------------+ > > > > Two command-line parameters (x-plugin-path and x-plugin-arg) are > > introduced to enable the out-of-band backend. If specified, vfio will > > attempt to use the out-of-band backend. > > > > The following is an example of VFIO command-line parameters for OOB- > Approach: > > > > -device > > vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_pat > > h,x-plugin-arg=$plugin_arg > > > > [1] https://github.com/raolei-intel/vfio-lm-plugin-example.git > > > > Lei Rao (13): > > vfio/migration: put together checks of migration initialization > > conditions > > vfio/migration: move migration struct allocation out of > > vfio_migration_init > > vfio/migration: move vfio_get_dev_region_info out of > > vfio_migration_probe > > vfio/migration: Separated functions that relate to the In-Band > > approach > > vfio/migration: rename functions that relate to the In-Band approach > > vfio/migration: introduce VFIOMigrationOps layer in VFIO live > > migration framework > > vfio/migration: move the statistics of bytes_transferred to generic > > VFIO migration layer > > vfio/migration: split migration handler registering from > > vfio_migration_init > > vfio/migration: move the functions of In-Band approach to a new file > > vfio/pci: introduce command-line parameters to specify migration > > method > > vfio/migration: add a plugin layer to support out-of-band live > > migration > > vfio/migration: add some trace-events for vfio migration plugin > > vfio/migration: make the region and plugin member of struct > > VFIOMigration to be a union > > > > docs/devel/vfio-migration-plugin.rst | 165 +++++++ > > hw/vfio/meson.build | 2 + > > hw/vfio/migration-local.c | 456 +++++++++++++++++++ > > hw/vfio/migration-plugin.c | 266 +++++++++++ > > hw/vfio/migration.c | 577 ++++++------------------ > > hw/vfio/pci.c | 2 + > > hw/vfio/trace-events | 9 +- > > include/hw/vfio/vfio-common.h | 37 +- > > include/hw/vfio/vfio-migration-plugin.h | 21 + > > 9 files changed, 1096 insertions(+), 439 deletions(-) create mode > > 100644 docs/devel/vfio-migration-plugin.rst > > create mode 100644 hw/vfio/migration-local.c create mode 100644 > > hw/vfio/migration-plugin.c create mode 100644 > > include/hw/vfio/vfio-migration-plugin.h > > >
On Wed, 1 Jun 2022 17:09:25 +0000 "Dong, Eddie" <eddie.dong@intel.com> wrote: > > -----Original Message----- > > From: Qemu-devel <qemu-devel- > > bounces+eddie.dong=intel.com@nongnu.org> On Behalf Of Alex Williamson > > On Tue, 24 May 2022 14:18:35 +0800 > > Lei Rao <lei.rao@intel.com> wrote: > > > This proposal adopts a plugin mechanism (an example can be found in > > > [1]) given that IPU/DPU vendors usually implement proprietary > > > migration interfaces without a standard. But we are also open if an > > > alternative option makes better sense, e.g. via loadable modules (with > > > Qemu supporting gRPC or JSON-RPC support) or an IPC mechanism similar > > to vhost-user. > > > > AFAIU, QEMU is not interested in supporting plugin modules, especially > > proprietary ones. I don't see that a case has really been made that this > > cannot be done in-band, through a vfio-pci variant driver, possibly making > > use of proprietary interfaces to a userspace agent if necessary (though > > please don't ask such to be accepted in-tree for the kernel either). A vfio- > > user device server might also host such proprietary, device specific agents > > while supporting the standard, in-band migration interface. Thanks, > > > > Thanks Alex. Removing plug-in module is not a problem. > > Do you mean to implement the migration and protocol handling inside > Qemu-client (probably vfio-client after the VFIO-user is merged)? Or > to build as part of libvfio-user? We can also build it as a separate > process of Qemu-server as part of Multi-Process Qemu. AIUI, the QEMU "client" in a vfio-user configuration is simply QEMU itself. The vfio-user "server" provides the actual device implementation, which could support different license models, depending on what libraries or existing code is incorporated to implement that server. The QEMU remote machine type is simply a QEMU-based implementation of a vfio-user server. The vfio-user server is analogous to a vfio-pci variant driver in the kernel/ioctl interface model. The vfio-user client should be device agnostic, possibly with similar exceptions we have today via device specific quirk handling for the vfio kernel interface. > In here, the protocol between host CPU and SoC is based on gRPC, > which support Rust code in client (Host CPU side here) more than > C/C++. Do you have any suggestion to better support Rust code with > Qemu C/C++ code? I'm not qualified to provide suggestions regarding Rust code integration with QEMU, but I think that's only required if the device specific migration support is on the "client". As above, I don't think that's the correct model, the vfio migration protocol is meant to support any device specific requirements on the device end of the connection, ie. the "server" end for vfio-user, which can be an entirely separate, non-QEMU based process. I think there are also ways to write kernel drivers in Rust, so possibly a kernel interface vfio-pci variant driver could also work. Thanks, Alex
> -----Original Message----- > From: Alex Williamson <alex.williamson@redhat.com> > Sent: Wednesday, June 1, 2022 11:01 AM > To: Dong, Eddie <eddie.dong@intel.com> > Cc: Rao, Lei <lei.rao@intel.com>; Tian, Kevin <kevin.tian@intel.com>; Zeng, > Jason <jason.zeng@intel.com>; quintela@redhat.com; dgilbert@redhat.com; > Li, Yadong <yadong.li@intel.com>; Liu, Yi L <yi.l.liu@intel.com>; qemu- > devel@nongnu.org > Subject: Re: [RFC PATCH 00/13] Add a plugin to support out-of-band live > migration for VFIO pass-through device > > On Wed, 1 Jun 2022 17:09:25 +0000 > "Dong, Eddie" <eddie.dong@intel.com> wrote: > > > > -----Original Message----- > > > From: Qemu-devel <qemu-devel- > > > bounces+eddie.dong=intel.com@nongnu.org> On Behalf Of Alex > > > bounces+Williamson > > > On Tue, 24 May 2022 14:18:35 +0800 > > > Lei Rao <lei.rao@intel.com> wrote: > > > > This proposal adopts a plugin mechanism (an example can be found > > > > in > > > > [1]) given that IPU/DPU vendors usually implement proprietary > > > > migration interfaces without a standard. But we are also open if > > > > an alternative option makes better sense, e.g. via loadable > > > > modules (with Qemu supporting gRPC or JSON-RPC support) or an IPC > > > > mechanism similar > > > to vhost-user. > > > > > > AFAIU, QEMU is not interested in supporting plugin modules, > > > especially proprietary ones. I don't see that a case has really > > > been made that this cannot be done in-band, through a vfio-pci > > > variant driver, possibly making use of proprietary interfaces to a > > > userspace agent if necessary (though please don't ask such to be > > > accepted in-tree for the kernel either). A vfio- user device server > > > might also host such proprietary, device specific agents while > > > supporting the standard, in-band migration interface. Thanks, > > > > > > > Thanks Alex. Removing plug-in module is not a problem. > > > > Do you mean to implement the migration and protocol handling inside > > Qemu-client (probably vfio-client after the VFIO-user is merged)? Or > > to build as part of libvfio-user? We can also build it as a separate > > process of Qemu-server as part of Multi-Process Qemu. > > AIUI, the QEMU "client" in a vfio-user configuration is simply QEMU itself. > The vfio-user "server" provides the actual device implementation, which > could support different license models, depending on what libraries or > existing code is incorporated to implement that server. The QEMU remote > machine type is simply a QEMU-based implementation of a vfio-user server. > The vfio-user server is analogous to a vfio-pci variant driver in the > kernel/ioctl interface model. The vfio-user client should be device agnostic, > possibly with similar exceptions we have today via device specific quirk > handling for the vfio kernel interface. > > > In here, the protocol between host CPU and SoC is based on gRPC, which > > support Rust code in client (Host CPU side here) more than C/C++. Do > > you have any suggestion to better support Rust code with Qemu C/C++ > > code? > > I'm not qualified to provide suggestions regarding Rust code integration with > QEMU, but I think that's only required if the device specific migration > support is on the "client". As above, I don't think that's the correct model, > the vfio migration protocol is meant to support any device specific > requirements on the device end of the connection, ie. the "server" end for > vfio-user, which can be an entirely separate, non-QEMU based process. I > think there are also ways to write kernel drivers in Rust, so possibly a kernel > interface vfio-pci variant driver could also work. Thanks, > Alex: Thanks for your suggestion. Yes, agree Qemu (client) is, by nature, neutral to physical device knowledge. With more thinking, it seems that: 1: Solution to have a separate kernel driver: The way the host CPU talking with the SoC chip of the device is going thru TCP/IP network, plus high level protocol (gRPC or Json..). This is going to be very complicated and might be hard to be accepted by community. 2: Implement as a full qemu-server device model. This way works if we implement a full device model in vfio-user, but given that the device (NVME for now) works in VFIO passthru mode for performance, the issue Kevin Tian raised in another email is a real concern too. 3: Implement partial (or supplemental) feature in Qemu-server device model. This solution defines a Qemu/VFIO migration interface between the client and server for migration. Client migration-proxy uses hardware transparent interface to talk with the remote-migration server. The remote server may manage device-specific (protocol specific to be more precisely) components to talk with different hardware backend. In this case, we rely on today's VFIO module to manage the device and manipulate the device thru kernel driver. During migration, it will use the migration-proxy to get/set state etc... Users can configure the additional Qemu command line parameter to choose a remote migration-proxy for a VFIO device. Can you suggest? Thx Eddie
Hi, Alex,
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Thursday, June 2, 2022 2:01 AM
>
> On Wed, 1 Jun 2022 17:09:25 +0000
> "Dong, Eddie" <eddie.dong@intel.com> wrote:
>
> > > -----Original Message-----
> > > From: Qemu-devel <qemu-devel-
> > > bounces+eddie.dong=intel.com@nongnu.org> On Behalf Of Alex
> Williamson
> > > On Tue, 24 May 2022 14:18:35 +0800
> > > Lei Rao <lei.rao@intel.com> wrote:
> > > > This proposal adopts a plugin mechanism (an example can be found in
> > > > [1]) given that IPU/DPU vendors usually implement proprietary
> > > > migration interfaces without a standard. But we are also open if an
> > > > alternative option makes better sense, e.g. via loadable modules (with
> > > > Qemu supporting gRPC or JSON-RPC support) or an IPC mechanism
> similar
> > > to vhost-user.
> > >
> > > AFAIU, QEMU is not interested in supporting plugin modules, especially
> > > proprietary ones. I don't see that a case has really been made that this
> > > cannot be done in-band, through a vfio-pci variant driver, possibly
> making
> > > use of proprietary interfaces to a userspace agent if necessary (though
> > > please don't ask such to be accepted in-tree for the kernel either). A vfio-
> > > user device server might also host such proprietary, device specific agents
> > > while supporting the standard, in-band migration interface. Thanks,
> > >
> >
> > Thanks Alex. Removing plug-in module is not a problem.
> >
> > Do you mean to implement the migration and protocol handling inside
> > Qemu-client (probably vfio-client after the VFIO-user is merged)? Or
> > to build as part of libvfio-user? We can also build it as a separate
> > process of Qemu-server as part of Multi-Process Qemu.
>
> AIUI, the QEMU "client" in a vfio-user configuration is simply QEMU
> itself. The vfio-user "server" provides the actual device
> implementation, which could support different license models, depending
> on what libraries or existing code is incorporated to implement that
> server. The QEMU remote machine type is simply a QEMU-based
> implementation of a vfio-user server. The vfio-user server is analogous
> to a vfio-pci variant driver in the kernel/ioctl interface model. The
> vfio-user client should be device agnostic, possibly with similar
> exceptions we have today via device specific quirk handling for the
> vfio kernel interface.
>
Sounds like vfio-user is currently defined around virtual device
oriented usages, e.g.:
- Client accesses virtual pci regions via VFIO_USER_REGION_READ/WRITE
or mmap if the server passes a shmem fd for a given region in
VFIO_USER_DEVICE_GET_REGION_INFO;
- Client maps DMA memory to server via VFIO_USER_DMA_MAP/UNMAP;
- Server access client memory via VFIO_USER_DMA_READ/WRITE or
mmap if client passes a fd for the DMA region when doing DMA_MAP;
- Server delivers virtual interrupt to client via IPC e.g. eventfd;
But in this usage it is still expected to allow Qemu directly access the
physical device for performance. Turning to vfio-user suggests that it
may need to support a model where from kernel p.o.v the physical
device is assigned to vfio-user server but in the end vfio-user client
actually operates the device (or, 'partially').
'partially' comes from that mmap must be done by Qemu otherwise
we lose all the performance merit of passthrough while DMA map
and interrupt delivery may be still routed via server (for DMA map
this requires that the server can mmap the client's memory). Such
indirect routes does bring some overhead but it's less obvious than
losing mmap.
I'm not sure how it works w/o additional kernel-side cooperation
though. Currently there is only one fd to represent the entire
kernel vfio device then how could the server delegate pci regions
to another process (client) for mmap?
If the server just passes the vfio device fd to the client then it's
not vfio-user any more.
Did I misunderstand your idea?
Thanks
Kevin
© 2016 - 2024 Red Hat, Inc.