[libvirt] [PATCH RFC 00/39] qemu: Add support for -blockdev

Peter Krempa posted 39 patches 5 years, 8 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/cover.1532533586.git.pkrempa@redhat.com
Test syntax-check passed
There is a newer version of this series
docs/formatdomain.html.in                          |   7 +-
docs/schemas/domaincommon.rng                      |  19 ++
src/conf/domain_conf.c                             |  89 +++++-
src/conf/domain_conf.h                             |   7 +
src/qemu/qemu_alias.c                              |  86 +++--
src/qemu/qemu_alias.h                              |   3 +-
src/qemu/qemu_block.c                              |  22 ++
src/qemu/qemu_block.h                              |   2 +
src/qemu/qemu_capabilities.c                       |   2 +
src/qemu/qemu_capabilities.h                       |   1 +
src/qemu/qemu_command.c                            |  94 +++++-
src/qemu/qemu_command.h                            |   3 +
src/qemu/qemu_domain.c                             | 173 ++++++++++-
src/qemu/qemu_domain.h                             |   9 +
src/qemu/qemu_driver.c                             | 244 ++++++++++-----
src/qemu/qemu_hotplug.c                            | 175 +++++++++--
src/qemu/qemu_monitor.c                            | 108 ++++++-
src/qemu/qemu_monitor.h                            |  36 ++-
src/qemu/qemu_monitor_json.c                       | 346 ++++++++++++++++++---
src/qemu/qemu_monitor_json.h                       |  33 +-
src/qemu/qemu_process.c                            |  84 ++++-
src/util/virqemu.c                                 |   5 +-
tests/qemucapabilitiesdata/caps_3.0.0.ppc64.xml    |   1 +
tests/qemucapabilitiesdata/caps_3.0.0.x86_64.xml   |   1 +
tests/qemumonitorjsontest.c                        |  14 +-
tests/qemustatusxml2xmldata/modern-in.xml          |   6 +
tests/qemuxml2argvdata/disk-aio.x86_64-2.12.0.args |  37 +++
tests/qemuxml2argvdata/disk-aio.x86_64-latest.args |  19 +-
.../disk-backing-chains-index.x86_64-2.12.0.args   |   1 +
.../disk-backing-chains-index.x86_64-latest.args   |   1 +
.../qemuxml2argvdata/disk-backing-chains-index.xml | 145 +++++++++
.../disk-backing-chains-noindex.x86_64-2.12.0.args |  58 ++++
.../disk-backing-chains-noindex.x86_64-latest.args | 163 ++++++++++
.../disk-backing-chains-noindex.xml                | 145 +++++++++
.../qemuxml2argvdata/disk-cache.x86_64-2.12.0.args |  50 +++
.../qemuxml2argvdata/disk-cache.x86_64-latest.args |  50 ++-
.../disk-cdrom-network.x86_64-2.12.0.args          |  41 +++
.../disk-cdrom-network.x86_64-latest.args          |  32 +-
.../disk-cdrom-tray.x86_64-2.12.0.args             |  39 +++
.../disk-cdrom-tray.x86_64-latest.args             |  24 +-
.../qemuxml2argvdata/disk-cdrom.x86_64-2.12.0.args |  35 +++
.../qemuxml2argvdata/disk-cdrom.x86_64-latest.args |  17 +-
.../disk-copy_on_read.x86_64-2.12.0.args           |  41 +++
.../disk-copy_on_read.x86_64-latest.args           |  19 +-
.../disk-detect-zeroes.x86_64-2.12.0.args          |  37 +++
.../disk-detect-zeroes.x86_64-latest.args          |  17 +-
.../disk-error-policy.x86_64-2.12.0.args           |  41 +++
.../disk-error-policy.x86_64-latest.args           |  30 +-
.../disk-floppy.x86_64-2.12.0.args                 |  35 +++
.../disk-network-gluster.x86_64-2.12.0.args        |  44 +++
.../disk-network-gluster.x86_64-latest.args        |  32 +-
.../disk-network-iscsi.x86_64-2.12.0.args          |  63 ++++
.../disk-network-iscsi.x86_64-latest.args          |  58 ++--
.../disk-network-nbd.x86_64-2.12.0.args            |  46 +++
.../disk-network-nbd.x86_64-latest.args            |  41 ++-
.../disk-network-rbd.x86_64-2.12.0.args            |  61 ++++
.../disk-network-rbd.x86_64-latest.args            |  67 ++--
.../disk-network-sheepdog.x86_64-2.12.0.args       |  35 +++
.../disk-network-sheepdog.x86_64-latest.args       |  16 +-
.../disk-network-source-auth.x86_64-2.12.0.args    |  47 +++
.../disk-network-source-auth.x86_64-latest.args    |  30 +-
.../disk-network-tlsx509.x86_64-2.12.0.args        |  59 ++++
.../disk-network-tlsx509.x86_64-latest.args        |  61 ++--
.../disk-readonly-disk.x86_64-2.12.0.args          |  34 ++
.../disk-readonly-disk.x86_64-latest.args          |  14 +-
.../disk-shared.x86_64-2.12.0.args                 |  37 +++
.../disk-shared.x86_64-latest.args                 |  18 +-
...isk-virtio-scsi-reservations.x86_64-2.12.0.args |  43 +++
...isk-virtio-scsi-reservations.x86_64-latest.args |  20 +-
.../floppy-drive-fat.x86_64-2.12.0.args            |  33 ++
.../hostdev-mdev-display-missing-graphics.xml      |   6 -
...v-display-spice-egl-headless.x86_64-latest.args |   2 -
.../hostdev-mdev-display-spice-egl-headless.xml    |   6 -
...ev-mdev-display-spice-opengl.x86_64-latest.args |   2 -
.../hostdev-mdev-display-spice-opengl.xml          |   6 -
...dev-display-vnc-egl-headless.x86_64-latest.args |   2 -
.../hostdev-mdev-display-vnc-egl-headless.xml      |   6 -
.../hostdev-mdev-display-vnc.x86_64-latest.args    |   2 -
.../qemuxml2argvdata/hostdev-mdev-display-vnc.xml  |   6 -
tests/qemuxml2argvdata/hostdev-mdev-display.xml    |   6 -
.../hostdev-mdev-invalid-target-address.xml        |   5 -
.../qemuxml2argvdata/hostdev-mdev-precreated.args  |   2 -
tests/qemuxml2argvdata/hostdev-mdev-precreated.xml |   6 -
.../hostdev-mdev-src-address-invalid.xml           |   6 -
tests/qemuxml2argvtest.c                           |  24 ++
.../disk-backing-chains-inactive.xml               |  35 +++
.../disk-backing-chains-index-active.xml           | 156 ++++++++++
.../disk-backing-chains-index-inactive.xml         | 156 ++++++++++
.../disk-backing-chains-noindex-active.xml         | 156 ++++++++++
.../disk-backing-chains-noindex-inactive.xml       | 156 ++++++++++
tests/qemuxml2xmloutdata/disk-mirror-inactive.xml  |   4 +
.../disk-mirror-old-inactive.xml                   |   4 +
tests/qemuxml2xmloutdata/hostdev-mdev-display.xml  |   6 -
.../qemuxml2xmloutdata/hostdev-mdev-precreated.xml |   6 -
tests/qemuxml2xmltest.c                            |   2 +
95 files changed, 3781 insertions(+), 493 deletions(-)
create mode 100644 tests/qemuxml2argvdata/disk-aio.x86_64-2.12.0.args
create mode 120000 tests/qemuxml2argvdata/disk-backing-chains-index.x86_64-2.12.0.args
create mode 120000 tests/qemuxml2argvdata/disk-backing-chains-index.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-index.xml
create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-noindex.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-noindex.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-noindex.xml
create mode 100644 tests/qemuxml2argvdata/disk-cache.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-cdrom-tray.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-cdrom.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-copy_on_read.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-detect-zeroes.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-error-policy.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-floppy.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-gluster.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-iscsi.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-nbd.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-rbd.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-sheepdog.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-source-auth.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-network-tlsx509.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-readonly-disk.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-shared.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/disk-virtio-scsi-reservations.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2argvdata/floppy-drive-fat.x86_64-2.12.0.args
create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-index-active.xml
create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-index-inactive.xml
create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-noindex-active.xml
create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-noindex-inactive.xml
[libvirt] [PATCH RFC 00/39] qemu: Add support for -blockdev
Posted by Peter Krempa 5 years, 8 months ago
This series adds support for starting and hotplug of disks with
-blockdev/blockdev-add.

Blockjobs are not supported and thus the last patch should not be
applied yet as some refactoring of the jobs is required.

At the beginning of the series there are a few cleanup patches which may
be pushed even at this point.

The main reason this is in RFC state is that block stats reporting does
not work.

The following command:

{"execute":"query-blockstats","arguments":{"query-nodes":true}}

Returns no reasonable data:

    {
      "stats": {
        "flush_total_time_ns": 0,
        "wr_highest_offset": 0,
        "wr_total_time_ns": 0,
        "failed_wr_operations": 0,
        "failed_rd_operations": 0,
        "wr_merged": 0,
        "wr_bytes": 0,
        "timed_stats": [

        ],
        "failed_flush_operations": 0,
        "account_invalid": false,
        "rd_total_time_ns": 0,
        "flush_operations": 0,
        "wr_operations": 0,
        "rd_merged": 0,
        "rd_bytes": 0,
        "invalid_flush_operations": 0,
        "account_failed": false,
        "rd_operations": 0,
        "invalid_wr_operations": 0,
        "invalid_rd_operations": 0
      },
      "node-name": "libvirt-7-storage"
    },
    {
      "parent": {
        "stats": {
          "flush_total_time_ns": 0,
          "wr_highest_offset": 0,
          "wr_total_time_ns": 0,
          "failed_wr_operations": 0,
          "failed_rd_operations": 0,
          "wr_merged": 0,
          "wr_bytes": 0,
          "timed_stats": [

          ],
          "failed_flush_operations": 0,
          "account_invalid": false,
          "rd_total_time_ns": 0,
          "flush_operations": 0,
          "wr_operations": 0,
          "rd_merged": 0,
          "rd_bytes": 0,
          "invalid_flush_operations": 0,
          "account_failed": false,
          "rd_operations": 0,
          "invalid_wr_operations": 0,
          "invalid_rd_operations": 0
        },
        "node-name": "libvirt-7-storage"
      },
      "stats": {
        "flush_total_time_ns": 0,
        "wr_highest_offset": 0,
        "wr_total_time_ns": 0,
        "failed_wr_operations": 0,
        "failed_rd_operations": 0,
        "wr_merged": 0,
        "wr_bytes": 0,
        "timed_stats": [

        ],
        "failed_flush_operations": 0,
        "account_invalid": false,
        "rd_total_time_ns": 0,
        "flush_operations": 0,
        "wr_operations": 0,
        "rd_merged": 0,
        "rd_bytes": 0,
        "invalid_flush_operations": 0,
        "account_failed": false,
        "rd_operations": 0,
        "invalid_wr_operations": 0,
        "invalid_rd_operations": 0
      },
      "node-name": "libvirt-7-format"
    },

the 'libvirt-7-storage' and 'libvirt-7-format' nodes represent the ISO
 backing the CDROM used to boot the VM so reads were executed.

In the old approach when we use -drive and query-nodes is false the
 output looks like this:

    {
      "device": "drive-ide0-0-0",
      "parent": {
        "stats": {
          "flush_total_time_ns": 0,
          "wr_highest_offset": 0,
          "wr_total_time_ns": 0,
          "failed_wr_operations": 0,
          "failed_rd_operations": 0,
          "wr_merged": 0,
          "wr_bytes": 0,
          "timed_stats": [

          ],
          "failed_flush_operations": 0,
          "account_invalid": false,
          "rd_total_time_ns": 0,
          "flush_operations": 0,
          "wr_operations": 0,
          "rd_merged": 0,
          "rd_bytes": 0,
          "invalid_flush_operations": 0,
          "account_failed": false,
          "rd_operations": 0,
          "invalid_wr_operations": 0,
          "invalid_rd_operations": 0
        },
        "node-name": "#block080"
      },
      "stats": {
        "flush_total_time_ns": 0,
        "wr_highest_offset": 0,
        "wr_total_time_ns": 0,
        "failed_wr_operations": 0,
        "failed_rd_operations": 0,
        "wr_merged": 0,
        "wr_bytes": 0,
        "timed_stats": [

        ],
        "failed_flush_operations": 0,
        "account_invalid": true,
        "rd_total_time_ns": 204236271,
        "flush_operations": 0,
        "wr_operations": 0,
        "rd_merged": 0,
        "rd_bytes": 30046628,
        "invalid_flush_operations": 0,
        "account_failed": true,
        "idle_time_ns": 18766797619,
        "rd_operations": 14680,
        "invalid_wr_operations": 0,
        "invalid_rd_operations": 0
      },
      "node-name": "#block152"
    },

I also get all zeroes when I use 'query-nodes' true on a machine started
 with -drive.

Without the stats we'd not achieve feature parity unfortunately.

Kevin, could you please have a look?

Peter Krempa (39):
  qemu: monitor: Reuse qemuMonitorJSONQueryBlock in
    qemuMonitorJSONBlockIoThrottleInfo
  qemu: monitor: Allow using 'id' instead of 'device' for
    'block_set_io_throttle'
  qemu: monitor: Allow using 'qdev' instead of 'device' for getting disk
    throttling
  tests: qemu: Drop disk from hostdev-mdev tests
  tests: qemuxml2argv: Fork CAPS_LATEST test cases for 'blockdev'
  tests: qemu: Add test data for backing chains and indexes
  qemu: hotplug: Don't generate alias when detaching disk
  util: virqemu: Simplify debugging if building QOM object with missing
    args
  qemu: caps: Add capability for using the blockdev infrastructure
  qemu: process: clear QEMU_CAPS_BLOCKDEV for VMs where we can't support
    it
  qemu: domain: Don't redetect backing chain when using -blockdev
  qemu: process: Don't detect nodenames when we support -blockdev
  conf: domain: Format out user provided backing chains in XML
  qemu: domain: Add infrastructure to generate block node names
  conf: Implement private data formatting and parsing for disks
  conf: Allow formatting and parsing of 'index' for disk source image
  qemu: Use proper backingIndex when reporting stats for backing chain
  qemu: Add field to store QDEV path of a disk in private data
  qemu: alias: Generate QDEV name of the block backend for disks
  qemu: domain: Add field for storing node name for copy-on-read
  qemu: proces: assign node names for user defined backing chains
  qemu: block: Add generator for the 'copy-on-read' blockdev driver
  qemu: domain: Prepare qemuDomainDiskGetBackendAlias for -blockdev
  qemu: command: format disk source commandline for -blockdev
  qemu: command: Add helper to check if disk throttling is enabled
  qemu: process: Setup disk io throttling for -blockdev
  qemu: driver: Use QOM backend name for disk IO throttling APIs
  qemu: hotplug: Prepare for blockdev-add/blockdev-del with backing
    chains
  qemu: monitor: Add APIs for cdrom tray handling for -blockdev
  qemu: hotplug: Implement removable media change for -blockdev
  qemu: monitor: Prepare query-block calls for dropping of -drive
  qemu: Use QOM path with query-block when using -blockdev
  qemu: monitor: Add API to retrieve blockstats by nodenames
  qemu: monitor: Add APIs for refreshing disk capacity when using
    -blockdev
  qemu: driver: Don't pass 'virDomainDiskDefPtr' to
    qemuDomainGetStatsOneBlock
  qemu: driver: Allow using blockdev with qemuDomainBlocksStatsGather
  qemu: monitor: Extract 'write-threshold' automatically for -blockdev
  qemu: driver: Prepare qemuDomainGetStatsBlock (bulk disk stats) for
    -blockdev
  DO NOT APPLY: Enable QEMU_CAPS_BLOCKDEV if 'copy-on-read' is supported

 docs/formatdomain.html.in                          |   7 +-
 docs/schemas/domaincommon.rng                      |  19 ++
 src/conf/domain_conf.c                             |  89 +++++-
 src/conf/domain_conf.h                             |   7 +
 src/qemu/qemu_alias.c                              |  86 +++--
 src/qemu/qemu_alias.h                              |   3 +-
 src/qemu/qemu_block.c                              |  22 ++
 src/qemu/qemu_block.h                              |   2 +
 src/qemu/qemu_capabilities.c                       |   2 +
 src/qemu/qemu_capabilities.h                       |   1 +
 src/qemu/qemu_command.c                            |  94 +++++-
 src/qemu/qemu_command.h                            |   3 +
 src/qemu/qemu_domain.c                             | 173 ++++++++++-
 src/qemu/qemu_domain.h                             |   9 +
 src/qemu/qemu_driver.c                             | 244 ++++++++++-----
 src/qemu/qemu_hotplug.c                            | 175 +++++++++--
 src/qemu/qemu_monitor.c                            | 108 ++++++-
 src/qemu/qemu_monitor.h                            |  36 ++-
 src/qemu/qemu_monitor_json.c                       | 346 ++++++++++++++++++---
 src/qemu/qemu_monitor_json.h                       |  33 +-
 src/qemu/qemu_process.c                            |  84 ++++-
 src/util/virqemu.c                                 |   5 +-
 tests/qemucapabilitiesdata/caps_3.0.0.ppc64.xml    |   1 +
 tests/qemucapabilitiesdata/caps_3.0.0.x86_64.xml   |   1 +
 tests/qemumonitorjsontest.c                        |  14 +-
 tests/qemustatusxml2xmldata/modern-in.xml          |   6 +
 tests/qemuxml2argvdata/disk-aio.x86_64-2.12.0.args |  37 +++
 tests/qemuxml2argvdata/disk-aio.x86_64-latest.args |  19 +-
 .../disk-backing-chains-index.x86_64-2.12.0.args   |   1 +
 .../disk-backing-chains-index.x86_64-latest.args   |   1 +
 .../qemuxml2argvdata/disk-backing-chains-index.xml | 145 +++++++++
 .../disk-backing-chains-noindex.x86_64-2.12.0.args |  58 ++++
 .../disk-backing-chains-noindex.x86_64-latest.args | 163 ++++++++++
 .../disk-backing-chains-noindex.xml                | 145 +++++++++
 .../qemuxml2argvdata/disk-cache.x86_64-2.12.0.args |  50 +++
 .../qemuxml2argvdata/disk-cache.x86_64-latest.args |  50 ++-
 .../disk-cdrom-network.x86_64-2.12.0.args          |  41 +++
 .../disk-cdrom-network.x86_64-latest.args          |  32 +-
 .../disk-cdrom-tray.x86_64-2.12.0.args             |  39 +++
 .../disk-cdrom-tray.x86_64-latest.args             |  24 +-
 .../qemuxml2argvdata/disk-cdrom.x86_64-2.12.0.args |  35 +++
 .../qemuxml2argvdata/disk-cdrom.x86_64-latest.args |  17 +-
 .../disk-copy_on_read.x86_64-2.12.0.args           |  41 +++
 .../disk-copy_on_read.x86_64-latest.args           |  19 +-
 .../disk-detect-zeroes.x86_64-2.12.0.args          |  37 +++
 .../disk-detect-zeroes.x86_64-latest.args          |  17 +-
 .../disk-error-policy.x86_64-2.12.0.args           |  41 +++
 .../disk-error-policy.x86_64-latest.args           |  30 +-
 .../disk-floppy.x86_64-2.12.0.args                 |  35 +++
 .../disk-network-gluster.x86_64-2.12.0.args        |  44 +++
 .../disk-network-gluster.x86_64-latest.args        |  32 +-
 .../disk-network-iscsi.x86_64-2.12.0.args          |  63 ++++
 .../disk-network-iscsi.x86_64-latest.args          |  58 ++--
 .../disk-network-nbd.x86_64-2.12.0.args            |  46 +++
 .../disk-network-nbd.x86_64-latest.args            |  41 ++-
 .../disk-network-rbd.x86_64-2.12.0.args            |  61 ++++
 .../disk-network-rbd.x86_64-latest.args            |  67 ++--
 .../disk-network-sheepdog.x86_64-2.12.0.args       |  35 +++
 .../disk-network-sheepdog.x86_64-latest.args       |  16 +-
 .../disk-network-source-auth.x86_64-2.12.0.args    |  47 +++
 .../disk-network-source-auth.x86_64-latest.args    |  30 +-
 .../disk-network-tlsx509.x86_64-2.12.0.args        |  59 ++++
 .../disk-network-tlsx509.x86_64-latest.args        |  61 ++--
 .../disk-readonly-disk.x86_64-2.12.0.args          |  34 ++
 .../disk-readonly-disk.x86_64-latest.args          |  14 +-
 .../disk-shared.x86_64-2.12.0.args                 |  37 +++
 .../disk-shared.x86_64-latest.args                 |  18 +-
 ...isk-virtio-scsi-reservations.x86_64-2.12.0.args |  43 +++
 ...isk-virtio-scsi-reservations.x86_64-latest.args |  20 +-
 .../floppy-drive-fat.x86_64-2.12.0.args            |  33 ++
 .../hostdev-mdev-display-missing-graphics.xml      |   6 -
 ...v-display-spice-egl-headless.x86_64-latest.args |   2 -
 .../hostdev-mdev-display-spice-egl-headless.xml    |   6 -
 ...ev-mdev-display-spice-opengl.x86_64-latest.args |   2 -
 .../hostdev-mdev-display-spice-opengl.xml          |   6 -
 ...dev-display-vnc-egl-headless.x86_64-latest.args |   2 -
 .../hostdev-mdev-display-vnc-egl-headless.xml      |   6 -
 .../hostdev-mdev-display-vnc.x86_64-latest.args    |   2 -
 .../qemuxml2argvdata/hostdev-mdev-display-vnc.xml  |   6 -
 tests/qemuxml2argvdata/hostdev-mdev-display.xml    |   6 -
 .../hostdev-mdev-invalid-target-address.xml        |   5 -
 .../qemuxml2argvdata/hostdev-mdev-precreated.args  |   2 -
 tests/qemuxml2argvdata/hostdev-mdev-precreated.xml |   6 -
 .../hostdev-mdev-src-address-invalid.xml           |   6 -
 tests/qemuxml2argvtest.c                           |  24 ++
 .../disk-backing-chains-inactive.xml               |  35 +++
 .../disk-backing-chains-index-active.xml           | 156 ++++++++++
 .../disk-backing-chains-index-inactive.xml         | 156 ++++++++++
 .../disk-backing-chains-noindex-active.xml         | 156 ++++++++++
 .../disk-backing-chains-noindex-inactive.xml       | 156 ++++++++++
 tests/qemuxml2xmloutdata/disk-mirror-inactive.xml  |   4 +
 .../disk-mirror-old-inactive.xml                   |   4 +
 tests/qemuxml2xmloutdata/hostdev-mdev-display.xml  |   6 -
 .../qemuxml2xmloutdata/hostdev-mdev-precreated.xml |   6 -
 tests/qemuxml2xmltest.c                            |   2 +
 95 files changed, 3781 insertions(+), 493 deletions(-)
 create mode 100644 tests/qemuxml2argvdata/disk-aio.x86_64-2.12.0.args
 create mode 120000 tests/qemuxml2argvdata/disk-backing-chains-index.x86_64-2.12.0.args
 create mode 120000 tests/qemuxml2argvdata/disk-backing-chains-index.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-index.xml
 create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-noindex.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-noindex.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-backing-chains-noindex.xml
 create mode 100644 tests/qemuxml2argvdata/disk-cache.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-cdrom-tray.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-cdrom.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-copy_on_read.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-detect-zeroes.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-error-policy.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-floppy.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-gluster.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-iscsi.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-nbd.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-rbd.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-sheepdog.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-source-auth.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-tlsx509.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-readonly-disk.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-shared.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/disk-virtio-scsi-reservations.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2argvdata/floppy-drive-fat.x86_64-2.12.0.args
 create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-index-active.xml
 create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-index-inactive.xml
 create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-noindex-active.xml
 create mode 100644 tests/qemuxml2xmloutdata/disk-backing-chains-noindex-inactive.xml

-- 
2.16.2

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH RFC 00/39] qemu: Add support for -blockdev
Posted by Kevin Wolf 5 years, 8 months ago
Am 25.07.2018 um 17:57 hat Peter Krempa geschrieben:
> This series adds support for starting and hotplug of disks with
> -blockdev/blockdev-add.
> 
> Blockjobs are not supported and thus the last patch should not be
> applied yet as some refactoring of the jobs is required.
> 
> At the beginning of the series there are a few cleanup patches which may
> be pushed even at this point.
> 
> The main reason this is in RFC state is that block stats reporting does
> not work.
> 
> The following command:
> 
> {"execute":"query-blockstats","arguments":{"query-nodes":true}}

query-nodes was added in commit f71eaa74c0b by Fam and Max, CCed. I'm
not sure what it was needed for at all and the commit message doesn't
help with that, but I suppose the addition was related to
wr_highest_offset (see below).

> Returns no reasonable data:
> 
>     {
>       "stats": {
>         "flush_total_time_ns": 0,
>         "wr_highest_offset": 0,
>         "wr_total_time_ns": 0,
>         "failed_wr_operations": 0,
>         "failed_rd_operations": 0,
>         "wr_merged": 0,
>         "wr_bytes": 0,
>         "timed_stats": [
> 
>         ],
>         "failed_flush_operations": 0,
>         "account_invalid": false,
>         "rd_total_time_ns": 0,
>         "flush_operations": 0,
>         "wr_operations": 0,
>         "rd_merged": 0,
>         "rd_bytes": 0,
>         "invalid_flush_operations": 0,
>         "account_failed": false,
>         "rd_operations": 0,
>         "invalid_wr_operations": 0,
>         "invalid_rd_operations": 0
>       },
>       "node-name": "libvirt-7-storage"
>     },
>     {
>       "parent": {
>         "stats": {
>           "flush_total_time_ns": 0,
>           "wr_highest_offset": 0,
>           "wr_total_time_ns": 0,
>           "failed_wr_operations": 0,
>           "failed_rd_operations": 0,
>           "wr_merged": 0,
>           "wr_bytes": 0,
>           "timed_stats": [
> 
>           ],
>           "failed_flush_operations": 0,
>           "account_invalid": false,
>           "rd_total_time_ns": 0,
>           "flush_operations": 0,
>           "wr_operations": 0,
>           "rd_merged": 0,
>           "rd_bytes": 0,
>           "invalid_flush_operations": 0,
>           "account_failed": false,
>           "rd_operations": 0,
>           "invalid_wr_operations": 0,
>           "invalid_rd_operations": 0
>         },
>         "node-name": "libvirt-7-storage"
>       },
>       "stats": {
>         "flush_total_time_ns": 0,
>         "wr_highest_offset": 0,
>         "wr_total_time_ns": 0,
>         "failed_wr_operations": 0,
>         "failed_rd_operations": 0,
>         "wr_merged": 0,
>         "wr_bytes": 0,
>         "timed_stats": [
> 
>         ],
>         "failed_flush_operations": 0,
>         "account_invalid": false,
>         "rd_total_time_ns": 0,
>         "flush_operations": 0,
>         "wr_operations": 0,
>         "rd_merged": 0,
>         "rd_bytes": 0,
>         "invalid_flush_operations": 0,
>         "account_failed": false,
>         "rd_operations": 0,
>         "invalid_wr_operations": 0,
>         "invalid_rd_operations": 0
>       },
>       "node-name": "libvirt-7-format"
>     },
> 
> the 'libvirt-7-storage' and 'libvirt-7-format' nodes represent the ISO
>  backing the CDROM used to boot the VM so reads were executed.

The only value that even exists that the node level is
wr_highest_offset. All the other stats are only accounted at the device
level, so we can't return anything meaningful here.

> In the old approach when we use -drive and query-nodes is false the
>  output looks like this:
> 
>     {
>       "device": "drive-ide0-0-0",
>       "parent": {
>         "stats": {
>           "flush_total_time_ns": 0,
>           "wr_highest_offset": 0,
>           "wr_total_time_ns": 0,
>           "failed_wr_operations": 0,
>           "failed_rd_operations": 0,
>           "wr_merged": 0,
>           "wr_bytes": 0,
>           "timed_stats": [
> 
>           ],
>           "failed_flush_operations": 0,
>           "account_invalid": false,
>           "rd_total_time_ns": 0,
>           "flush_operations": 0,
>           "wr_operations": 0,
>           "rd_merged": 0,
>           "rd_bytes": 0,
>           "invalid_flush_operations": 0,
>           "account_failed": false,
>           "rd_operations": 0,
>           "invalid_wr_operations": 0,
>           "invalid_rd_operations": 0
>         },
>         "node-name": "#block080"
>       },
>       "stats": {
>         "flush_total_time_ns": 0,
>         "wr_highest_offset": 0,
>         "wr_total_time_ns": 0,
>         "failed_wr_operations": 0,
>         "failed_rd_operations": 0,
>         "wr_merged": 0,
>         "wr_bytes": 0,
>         "timed_stats": [
> 
>         ],
>         "failed_flush_operations": 0,
>         "account_invalid": true,
>         "rd_total_time_ns": 204236271,
>         "flush_operations": 0,
>         "wr_operations": 0,
>         "rd_merged": 0,
>         "rd_bytes": 30046628,
>         "invalid_flush_operations": 0,
>         "account_failed": true,
>         "idle_time_ns": 18766797619,
>         "rd_operations": 14680,
>         "invalid_wr_operations": 0,
>         "invalid_rd_operations": 0
>       },
>       "node-name": "#block152"
>     },
> 
> I also get all zeroes when I use 'query-nodes' true on a machine started
>  with -drive.
> 
> Without the stats we'd not achieve feature parity unfortunately.
> 
> Kevin, could you please have a look?

As the information you're interested in is device-level information,
keep using query-nodes=false and identify which device it is for with
the 'device' field (the result contains a 'node-name', but this is not
suitable to identify the device when there are two users of the same
node).

Kevin

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH RFC 00/39] qemu: Add support for -blockdev
Posted by Fam Zheng 5 years, 8 months ago
On Thu, 07/26 10:44, Kevin Wolf wrote:
> Am 25.07.2018 um 17:57 hat Peter Krempa geschrieben:
> > This series adds support for starting and hotplug of disks with
> > -blockdev/blockdev-add.
> > 
> > Blockjobs are not supported and thus the last patch should not be
> > applied yet as some refactoring of the jobs is required.
> > 
> > At the beginning of the series there are a few cleanup patches which may
> > be pushed even at this point.
> > 
> > The main reason this is in RFC state is that block stats reporting does
> > not work.
> > 
> > The following command:
> > 
> > {"execute":"query-blockstats","arguments":{"query-nodes":true}}
> 
> query-nodes was added in commit f71eaa74c0b by Fam and Max, CCed. I'm
> not sure what it was needed for at all and the commit message doesn't
> help with that, but I suppose the addition was related to
> wr_highest_offset (see below).

Yes, this was part of RHBZ 1158094. Sorry about the poor commit message.

Fam

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH RFC 00/39] qemu: Add support for -blockdev
Posted by Peter Krempa 5 years, 8 months ago
On Thu, Jul 26, 2018 at 17:04:19 +0800, Fam Zheng wrote:
> On Thu, 07/26 10:44, Kevin Wolf wrote:
> > Am 25.07.2018 um 17:57 hat Peter Krempa geschrieben:
> > > This series adds support for starting and hotplug of disks with
> > > -blockdev/blockdev-add.
> > > 
> > > Blockjobs are not supported and thus the last patch should not be
> > > applied yet as some refactoring of the jobs is required.
> > > 
> > > At the beginning of the series there are a few cleanup patches which may
> > > be pushed even at this point.
> > > 
> > > The main reason this is in RFC state is that block stats reporting does
> > > not work.
> > > 
> > > The following command:
> > > 
> > > {"execute":"query-blockstats","arguments":{"query-nodes":true}}
> > 
> > query-nodes was added in commit f71eaa74c0b by Fam and Max, CCed. I'm
> > not sure what it was needed for at all and the commit message doesn't
> > help with that, but I suppose the addition was related to
> > wr_highest_offset (see below).
> 
> Yes, this was part of RHBZ 1158094. Sorry about the poor commit message.

Well the problem is that when using -blockdev with 'query-nodes' false/not present you get a
even more useless output:

virsh qemu-monitor-command --pretty upstream '{"execute":"query-blockstats","arguments":{"query-nodes":false}}'
{
  "return": [

  ],
  "id": "libvirt-20"
}

'query-named-block-nodes' don't provide the 'stats' section either:
(other drives snipped)

    {
      "iops_rd": 0,
      "detect_zeroes": "off",
      "image": {
        "virtual-size": 520032256,
        "filename": "/var/lib/libvirt/images/systemrescuecd-x86-4.9.5.iso",
        "format": "raw",
        "actual-size": 520036352,
        "dirty-flag": false
      },
      "iops_wr": 0,
      "ro": true,
      "node-name": "libvirt-7-format",
      "backing_file_depth": 0,
      "drv": "raw",
      "iops": 0,
      "bps_wr": 0,
      "write_threshold": 0,
      "encrypted": false,
      "bps": 0,
      "bps_rd": 0,
      "cache": {
        "no-flush": false,
        "direct": false,
        "writeback": true
      },
      "file": "/var/lib/libvirt/images/systemrescuecd-x86-4.9.5.iso",
      "encryption_key_missing": false
    },
    {
      "iops_rd": 0,
      "detect_zeroes": "off",
      "image": {
        "virtual-size": 520032256,
        "filename": "/var/lib/libvirt/images/systemrescuecd-x86-4.9.5.iso",
        "format": "file",
        "actual-size": 520036352,
        "dirty-flag": false
      },
      "iops_wr": 0,
      "ro": true,
      "node-name": "libvirt-7-storage",
      "backing_file_depth": 0,
      "drv": "file",
      "iops": 0,
      "bps_wr": 0,
      "write_threshold": 0,
      "encrypted": false,
      "bps": 0,
      "bps_rd": 0,
      "cache": {
        "no-flush": false,
        "direct": false,
        "writeback": true
      },
      "file": "/var/lib/libvirt/images/systemrescuecd-x86-4.9.5.iso",
      "encryption_key_missing": false
    }
  ],
  "id": "libvirt-19"
}

Thus libvirt can't provide the stats it used to with -drive. Since we
need to special-case the code for gathering stats for -blockdev it
should not be a problem if e.g. query-named-block-nodes reported the
stats section along. (perhaps with a boolean flag to do so which also
could disable the nesting)

In either case we need the stats in the same extent as we had before
with -drive and query-blockstats.
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH RFC 00/39] qemu: Add support for -blockdev
Posted by Kevin Wolf 5 years, 8 months ago
Am 27.07.2018 um 10:01 hat Peter Krempa geschrieben:
> On Thu, Jul 26, 2018 at 17:04:19 +0800, Fam Zheng wrote:
> > On Thu, 07/26 10:44, Kevin Wolf wrote:
> > > Am 25.07.2018 um 17:57 hat Peter Krempa geschrieben:
> > > > This series adds support for starting and hotplug of disks with
> > > > -blockdev/blockdev-add.
> > > > 
> > > > Blockjobs are not supported and thus the last patch should not be
> > > > applied yet as some refactoring of the jobs is required.
> > > > 
> > > > At the beginning of the series there are a few cleanup patches which may
> > > > be pushed even at this point.
> > > > 
> > > > The main reason this is in RFC state is that block stats reporting does
> > > > not work.
> > > > 
> > > > The following command:
> > > > 
> > > > {"execute":"query-blockstats","arguments":{"query-nodes":true}}
> > > 
> > > query-nodes was added in commit f71eaa74c0b by Fam and Max, CCed. I'm
> > > not sure what it was needed for at all and the commit message doesn't
> > > help with that, but I suppose the addition was related to
> > > wr_highest_offset (see below).
> > 
> > Yes, this was part of RHBZ 1158094. Sorry about the poor commit message.
> 
> Well the problem is that when using -blockdev with 'query-nodes' false/not present you get a
> even more useless output:
> 
> virsh qemu-monitor-command --pretty upstream '{"execute":"query-blockstats","arguments":{"query-nodes":false}}'
> {
>   "return": [
> 
>   ],
>   "id": "libvirt-20"
> }

That's a problem. Apparently it ignores anonymous BlockBackends, which
it shouldn't do. I'll look into that.

> 'query-named-block-nodes' don't provide the 'stats' section either:
> (other drives snipped)
[...]
> Thus libvirt can't provide the stats it used to with -drive. Since we
> need to special-case the code for gathering stats for -blockdev it
> should not be a problem if e.g. query-named-block-nodes reported the
> stats section along. (perhaps with a boolean flag to do so which also
> could disable the nesting)

The only one it could theoretically provide is wr_highest_offset, the
other ones simply don't exist at the node level. And this is probably
not the only one you're interested in.

> In either case we need the stats in the same extent as we had before
> with -drive and query-blockstats.

Yes, that's clear.

Kevin
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list