[v6] Support exporting BDSs via VDUSE

[PATCH v6 0/8] Support exporting BDSs via VDUSE

Posted by Xie Yongji 1 year, 11 months ago

Hi all,

Last few months ago, VDUSE (vDPA Device in Userspace) [1] has
been merged into Linux kernel as a framework that make it
possible to emulate a vDPA device in userspace. This series
aimed at implementing a VDUSE block backend based on the
qemu-storage-daemon infrastructure.

To support that, we firstly introduce a VDUSE library as a
subproject (like what libvhost-user does) to help implementing
VDUSE backends in QEMU. Then a VDUSE block export is implemented
based on this library. At last, we add resize and reconnect support
to the VDUSE block export and VDUSE library.

Now this series is based on Stefan's patch [2]. And since we don't
support vdpa-blk in QEMU currently, the VM case is tested with my
previous patchset [3].

[1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html
[2] https://lore.kernel.org/all/20220518130945.2657905-1-stefanha@redhat.com/
[3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg797569.html

Please review, thanks!

V5 to V6:
- Remove blk_get_guest_block_size() [Stefan]
- A patch is splited to fix incorrect length for
  vhost-user-blk-server
- Define a VirtioBlkHandler structure for virtio-blk
  I/O process [Stefan]
- Add documentation for block export id [Stefan]
- Remove some assert from libvduse library [Stefan]
- Remove unused VIRTIO_BLK_F_SIZE_MAX for vduse block export

V4 to V5:
- Abstract out the logic for virito-blk I/O process from
  vhost-user-blk-server and reuse it [Kevin]
- Fix missing VIRTIO_BLK_F_FLUSH [Kevin]
- Support discard and write_zeroes [Kevin]
- Rebase to the newest tree

V3 to V4:
- Fix some comments on QAPI [Eric]

V2 to V3:
- Introduce vduse_get_virtio_features() [Stefan]
- Update MAINTAINERS file [Stefan]
- Fix handler of VIRTIO_BLK_T_GET_ID request [Stefan]
- Add barrier for vduse_queue_inflight_get() [Stefan]

V1 to V2:
- Move vduse header to linux-headers [Stefan]
- Add two new API to support creating device from /dev/vduse/$NAME or
  file descriptor [Stefan]
- Check VIRTIO_F_VERSION_1 during intialization [Stefan]
- Replace malloc() + memset to calloc() [Stefan]
- Increase default queue size to 256 for vduse-blk [Stefan]
- Zero-initialize virtio-blk config space [Stefan]
- Add a patch to support reset blk->dev_ops
- Validate vq->log->inflight fields [Stefan]
- Add vduse_set_reconnect_log_file() API to support specifing the
  reconnect log file
- Fix some bugs [Stefan]

Xie Yongji (8):
  block: Support passing NULL ops to blk_set_dev_ops()
  block/export: Fix incorrect length passed to vu_queue_push()
  block/export: Abstract out the logic of virtio-blk I/O process
  linux-headers: Add vduse.h
  libvduse: Add VDUSE (vDPA Device in Userspace) library
  vduse-blk: Implement vduse-blk export
  vduse-blk: Add vduse-blk resize support
  libvduse: Add support for reconnecting

 MAINTAINERS                                 |    9 +
 block/block-backend.c                       |    2 +-
 block/export/export.c                       |    6 +
 block/export/meson.build                    |    7 +-
 block/export/vduse-blk.c                    |  341 +++++
 block/export/vduse-blk.h                    |   20 +
 block/export/vhost-user-blk-server.c        |  260 +---
 block/export/virtio-blk-handler.c           |  240 ++++
 block/export/virtio-blk-handler.h           |   37 +
 linux-headers/linux/vduse.h                 |  306 ++++
 meson.build                                 |   28 +
 meson_options.txt                           |    4 +
 qapi/block-export.json                      |   28 +-
 scripts/meson-buildoptions.sh               |    7 +
 scripts/update-linux-headers.sh             |    2 +-
 subprojects/libvduse/include/atomic.h       |    1 +
 subprojects/libvduse/include/compiler.h     |    1 +
 subprojects/libvduse/libvduse.c             | 1392 +++++++++++++++++++
 subprojects/libvduse/libvduse.h             |  247 ++++
 subprojects/libvduse/linux-headers/linux    |    1 +
 subprojects/libvduse/meson.build            |   10 +
 subprojects/libvduse/standard-headers/linux |    1 +
 22 files changed, 2705 insertions(+), 245 deletions(-)
 create mode 100644 block/export/vduse-blk.c
 create mode 100644 block/export/vduse-blk.h
 create mode 100644 block/export/virtio-blk-handler.c
 create mode 100644 block/export/virtio-blk-handler.h
 create mode 100644 linux-headers/linux/vduse.h
 create mode 120000 subprojects/libvduse/include/atomic.h
 create mode 120000 subprojects/libvduse/include/compiler.h
 create mode 100644 subprojects/libvduse/libvduse.c
 create mode 100644 subprojects/libvduse/libvduse.h
 create mode 120000 subprojects/libvduse/linux-headers/linux
 create mode 100644 subprojects/libvduse/meson.build
 create mode 120000 subprojects/libvduse/standard-headers/linux

-- 
2.20.1

Re: [PATCH v6 0/8] Support exporting BDSs via VDUSE

Posted by Kevin Wolf 1 year, 11 months ago

Am 23.05.2022 um 10:46 hat Xie Yongji geschrieben:
> Hi all,
> 
> Last few months ago, VDUSE (vDPA Device in Userspace) [1] has
> been merged into Linux kernel as a framework that make it
> possible to emulate a vDPA device in userspace. This series
> aimed at implementing a VDUSE block backend based on the
> qemu-storage-daemon infrastructure.
> 
> To support that, we firstly introduce a VDUSE library as a
> subproject (like what libvhost-user does) to help implementing
> VDUSE backends in QEMU. Then a VDUSE block export is implemented
> based on this library. At last, we add resize and reconnect support
> to the VDUSE block export and VDUSE library.
> 
> Now this series is based on Stefan's patch [2]. And since we don't
> support vdpa-blk in QEMU currently, the VM case is tested with my
> previous patchset [3].
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html
> [2] https://lore.kernel.org/all/20220518130945.2657905-1-stefanha@redhat.com/
> [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg797569.html

Thanks, applied to the block branch.

Kevin

Re: [PATCH v6 0/8] Support exporting BDSs via VDUSE

Posted by Stefan Hajnoczi 1 year, 11 months ago

On Mon, May 23, 2022 at 04:46:03PM +0800, Xie Yongji wrote:
> Hi all,
> 
> Last few months ago, VDUSE (vDPA Device in Userspace) [1] has
> been merged into Linux kernel as a framework that make it
> possible to emulate a vDPA device in userspace. This series
> aimed at implementing a VDUSE block backend based on the
> qemu-storage-daemon infrastructure.
> 
> To support that, we firstly introduce a VDUSE library as a
> subproject (like what libvhost-user does) to help implementing
> VDUSE backends in QEMU. Then a VDUSE block export is implemented
> based on this library. At last, we add resize and reconnect support
> to the VDUSE block export and VDUSE library.
> 
> Now this series is based on Stefan's patch [2]. And since we don't
> support vdpa-blk in QEMU currently, the VM case is tested with my
> previous patchset [3].
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html
> [2] https://lore.kernel.org/all/20220518130945.2657905-1-stefanha@redhat.com/
> [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg797569.html
> 
> Please review, thanks!

I talked to Kevin about who should merge this. He will do a final
review and it can go through his tree.

I will drop it from my 'block' branch for now.

Stefan

Re: [PATCH v6 0/8] Support exporting BDSs via VDUSE

Posted by Yongji Xie 1 year, 11 months ago

On Wed, May 25, 2022 at 8:48 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Mon, May 23, 2022 at 04:46:03PM +0800, Xie Yongji wrote:
> > Hi all,
> >
> > Last few months ago, VDUSE (vDPA Device in Userspace) [1] has
> > been merged into Linux kernel as a framework that make it
> > possible to emulate a vDPA device in userspace. This series
> > aimed at implementing a VDUSE block backend based on the
> > qemu-storage-daemon infrastructure.
> >
> > To support that, we firstly introduce a VDUSE library as a
> > subproject (like what libvhost-user does) to help implementing
> > VDUSE backends in QEMU. Then a VDUSE block export is implemented
> > based on this library. At last, we add resize and reconnect support
> > to the VDUSE block export and VDUSE library.
> >
> > Now this series is based on Stefan's patch [2]. And since we don't
> > support vdpa-blk in QEMU currently, the VM case is tested with my
> > previous patchset [3].
> >
> > [1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html
> > [2] https://lore.kernel.org/all/20220518130945.2657905-1-stefanha@redhat.com/
> > [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg797569.html
> >
> > Please review, thanks!
>
> I talked to Kevin about who should merge this. He will do a final
> review and it can go through his tree.
>
> I will drop it from my 'block' branch for now.
>

OK, thank you!

Thanks,
Yongji

Re: [PATCH v6 0/8] Support exporting BDSs via VDUSE

Posted by Stefan Hajnoczi 1 year, 11 months ago

On Mon, May 23, 2022 at 04:46:03PM +0800, Xie Yongji wrote:
> Hi all,
> 
> Last few months ago, VDUSE (vDPA Device in Userspace) [1] has
> been merged into Linux kernel as a framework that make it
> possible to emulate a vDPA device in userspace. This series
> aimed at implementing a VDUSE block backend based on the
> qemu-storage-daemon infrastructure.
> 
> To support that, we firstly introduce a VDUSE library as a
> subproject (like what libvhost-user does) to help implementing
> VDUSE backends in QEMU. Then a VDUSE block export is implemented
> based on this library. At last, we add resize and reconnect support
> to the VDUSE block export and VDUSE library.
> 
> Now this series is based on Stefan's patch [2]. And since we don't
> support vdpa-blk in QEMU currently, the VM case is tested with my
> previous patchset [3].
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html
> [2] https://lore.kernel.org/all/20220518130945.2657905-1-stefanha@redhat.com/
> [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg797569.html
> 
> Please review, thanks!

Thanks, applied to my block tree:
https://gitlab.com/stefanha/qemu/commits/block

Successfully manually tested on Linux 5.17.6.

Please send a follow-up patch that adds a tests/qemu-iotests/ test case
that launches qemu-storage-daemon with a vduse-blk export, writes a
pattern to the disk, and reads the pattern back to verify it. An
automated test will prevent bitrot.

I am sending a follow-up patch with documentation so users can discover
and learn how to use this new feature.

Stefan

Re: [PATCH v6 0/8] Support exporting BDSs via VDUSE

Posted by Yongji Xie 1 year, 11 months ago

On Wed, May 25, 2022 at 7:02 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Mon, May 23, 2022 at 04:46:03PM +0800, Xie Yongji wrote:
> > Hi all,
> >
> > Last few months ago, VDUSE (vDPA Device in Userspace) [1] has
> > been merged into Linux kernel as a framework that make it
> > possible to emulate a vDPA device in userspace. This series
> > aimed at implementing a VDUSE block backend based on the
> > qemu-storage-daemon infrastructure.
> >
> > To support that, we firstly introduce a VDUSE library as a
> > subproject (like what libvhost-user does) to help implementing
> > VDUSE backends in QEMU. Then a VDUSE block export is implemented
> > based on this library. At last, we add resize and reconnect support
> > to the VDUSE block export and VDUSE library.
> >
> > Now this series is based on Stefan's patch [2]. And since we don't
> > support vdpa-blk in QEMU currently, the VM case is tested with my
> > previous patchset [3].
> >
> > [1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html
> > [2] https://lore.kernel.org/all/20220518130945.2657905-1-stefanha@redhat.com/
> > [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg797569.html
> >
> > Please review, thanks!
>
> Thanks, applied to my block tree:
> https://gitlab.com/stefanha/qemu/commits/block
>
> Successfully manually tested on Linux 5.17.6.
>
> Please send a follow-up patch that adds a tests/qemu-iotests/ test case
> that launches qemu-storage-daemon with a vduse-blk export, writes a
> pattern to the disk, and reads the pattern back to verify it. An
> automated test will prevent bitrot.
>

Will do it.

> I am sending a follow-up patch with documentation so users can discover
> and learn how to use this new feature.
>

OK.

Thanks,
Yongji