From nobody Wed Feb 11 04:39:47 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1498230356073891.1385282427433; Fri, 23 Jun 2017 08:05:56 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E362AC060201; Fri, 23 Jun 2017 15:05:51 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BE90C6F9DD; Fri, 23 Jun 2017 15:05:51 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 5D9E626C; Fri, 23 Jun 2017 15:05:51 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v5NF5kZa004574 for ; Fri, 23 Jun 2017 11:05:46 -0400 Received: by smtp.corp.redhat.com (Postfix) id C454D6E8D3; Fri, 23 Jun 2017 15:05:46 +0000 (UTC) Received: from inaba.usersys.redhat.com (ovpn-12-57.pek2.redhat.com [10.72.12.57]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E7CB46F9C5 for ; Fri, 23 Jun 2017 15:05:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com E362AC060201 Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com E362AC060201 From: Andrea Bolognani To: libvir-list@redhat.com Date: Fri, 23 Jun 2017 23:03:27 +0800 Message-Id: <1498230208-14144-26-git-send-email-abologna@redhat.com> In-Reply-To: <1498230208-14144-1-git-send-email-abologna@redhat.com> References: <1498230208-14144-1-git-send-email-abologna@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v3 25/26] qemu: Isolate hostdevs on pSeries guests X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 23 Jun 2017 15:05:52 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" All the pieces are now in place, so we can finally start using isolation groups to achieve our initial goal, which is separating hostdevs from emulated PCI devices while keeping hostdevs that belong to the same host IOMMU group together. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=3D1280542 Signed-off-by: Andrea Bolognani --- src/qemu/qemu_domain_address.c | 118 +++++++++++++++++= ++++ src/qemu/qemu_hotplug.c | 10 ++ .../qemuxml2argv-pseries-hostdevs-1.args | 8 +- .../qemuxml2argv-pseries-hostdevs-2.args | 3 +- .../qemuxml2argv-pseries-hostdevs-3.args | 2 +- .../qemuxml2xmlout-pseries-hostdevs-1.xml | 14 ++- .../qemuxml2xmlout-pseries-hostdevs-2.xml | 6 +- .../qemuxml2xmlout-pseries-hostdevs-3.xml | 2 +- 8 files changed, 153 insertions(+), 10 deletions(-) diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index 3d56900..b884d06 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -906,6 +906,121 @@ qemuDomainFillAllPCIConnectFlags(virDomainDefPtr def, =20 =20 /** + * qemuDomainFillDeviceIsolationGroupIter: + * @def: domain definition + * @dev: device definition + * @info: device information + * @opaque: user data + * + * Fill isolation group information for a single device. + * + * Used by qemuDomainFillAllIsolationGroups(), don't call directly. + * + * Return: 0 on success, <0 on failure + * */ +static int +qemuDomainFillDeviceIsolationGroupIter(virDomainDefPtr def ATTRIBUTE_UNUSE= D, + virDomainDeviceDefPtr dev, + virDomainDeviceInfoPtr info, + void *opaque ATTRIBUTE_UNUSED) +{ + virDomainHostdevDefPtr hostdev; + virPCIDeviceAddressPtr hostAddr; + + /* Only hostdevs... */ + if (dev->type !=3D VIR_DOMAIN_DEVICE_HOSTDEV) + return 0; + + hostdev =3D dev->data.hostdev; + + /* ... of the PCI kind need this extra information */ + if (hostdev->mode !=3D VIR_DOMAIN_HOSTDEV_MODE_SUBSYS || + hostdev->source.subsys.type !=3D VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PC= I) { + return 0; + } + + hostAddr =3D &hostdev->source.subsys.u.pci.addr; + + /* The isolation group is simply the IOMMU group assigned by the host = */ + info->isolationGroup =3D virPCIDeviceAddressGetIOMMUGroupNum(hostAddr); + + if (info->isolationGroup < 0) { + VIR_WARN("Can't look up isolation group for device %04x:%02x:%02x.= %x", + hostAddr->domain, hostAddr->bus, + hostAddr->slot, hostAddr->function); + } else { + VIR_DEBUG("Isolation group for device %04x:%02x:%02x.%x is %d", + hostAddr->domain, hostAddr->bus, + hostAddr->slot, hostAddr->function, + info->isolationGroup); + } + + return info->isolationGroup; +} + + +/** + * qemuDomainFillAllIsolationGroups: + * @def: domain definition + * + * Fill isolation group information for all devices in @def. + * + * Used by qemuDomainSetupIsolationGroups(), don't call directly. + * + * Return: 0 on success, <0 on failure + */ +static int +qemuDomainFillAllIsolationGroups(virDomainDefPtr def) +{ + return virDomainDeviceInfoIterate(def, + qemuDomainFillDeviceIsolationGroupIt= er, + NULL); +} + + +/** + * qemuDomainSetupIsolationGroups: + * @def: domain definition + * + * High-level function to set up isolation groups for all devices + * and controllers in @def. Isolation groups will only be set up if + * the guest architecture and machine type require it, so this + * function can and should be called unconditionally before attempting + * to assign any PCI address. + * + * Return: 0 on success, <0 on failure + */ +static int +qemuDomainSetupIsolationGroups(virDomainDefPtr def) +{ + virDomainControllerDefPtr defaultPHB; + int idx; + int ret =3D -1; + + /* Only pSeries guests care about isolation groups at the moment */ + if (!qemuDomainIsPSeries(def)) + return 0; + + if (qemuDomainFillAllIsolationGroups(def) < 0) + goto out; + + idx =3D virDomainControllerFind(def, VIR_DOMAIN_CONTROLLER_TYPE_PCI, 0= ); + if (idx < 0) + goto out; + + defaultPHB =3D def->controllers[idx]; + + /* We want to prevent hostdevs from being plugged into the default + * PHB, so we lock its isolation group */ + defaultPHB->info.isolationGroupLocked =3D true; + + ret =3D 0; + + out: + return ret; +} + +/** * qemuDomainFillDevicePCIConnectFlags: * * @def: the entire DomainDef @@ -2076,6 +2191,9 @@ qemuDomainAssignPCIAddresses(virDomainDefPtr def, if (qemuDomainFillAllPCIConnectFlags(def, qemuCaps, driver) < 0) goto cleanup; =20 + if (qemuDomainSetupIsolationGroups(def) < 0) + goto cleanup; + if (nbuses > 0) { /* 1st pass to figure out how many PCI bridges we need */ if (!(addrs =3D qemuDomainPCIAddressSetCreate(def, nbuses, true))) diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 8ca5054..85db7af 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -1468,6 +1468,16 @@ qemuDomainAttachHostPCIDevice(virQEMUDriverPtr drive= r, =20 if (qemuAssignDeviceHostdevAlias(vm->def, &info->alias, -1) < 0) goto error; + + /* When attaching a hostdev to a pSeries guest, we need to set up + * its isolation group if we want the assigned address to respect + * isolation constraints */ + if (qemuDomainIsPSeries(vm->def)) { + virPCIDeviceAddressPtr hostAddr =3D &hostdev->source.subsys.u.pci.= addr; + + info->isolationGroup =3D virPCIDeviceAddressGetIOMMUGroupNum(hostA= ddr); + } + if (qemuDomainEnsurePCIAddress(vm, &dev, driver) < 0) goto error; releaseaddr =3D true; diff --git a/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-1.args b/= tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-1.args index 88eb081..36f2def 100644 --- a/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-1.args +++ b/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-1.args @@ -18,6 +18,8 @@ QEMU_AUDIO_DRV=3Dnone \ server,nowait \ -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dreadline \ -boot c \ --device vfio-pci,host=3D0001:01:00.0,id=3Dhostdev0,bus=3Dpci.0,addr=3D0x1 \ --device vfio-pci,host=3D0005:90:01.0,id=3Dhostdev1,bus=3Dpci.0,addr=3D0x2 \ --device vfio-pci,host=3D0001:01:00.1,id=3Dhostdev2,bus=3Dpci.0,addr=3D0x3 +-device spapr-pci-host-bridge,index=3D1,id=3Dpci.1 \ +-device spapr-pci-host-bridge,index=3D2,id=3Dpci.2 \ +-device vfio-pci,host=3D0001:01:00.0,id=3Dhostdev0,bus=3Dpci.1.0,addr=3D0x= 1 \ +-device vfio-pci,host=3D0005:90:01.0,id=3Dhostdev1,bus=3Dpci.2.0,addr=3D0x= 1 \ +-device vfio-pci,host=3D0001:01:00.1,id=3Dhostdev2,bus=3Dpci.1.0,addr=3D0x2 diff --git a/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-2.args b/= tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-2.args index 83d4306..cd5b664 100644 --- a/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-2.args +++ b/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-2.args @@ -19,6 +19,7 @@ server,nowait \ -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dreadline \ -boot c \ -device spapr-pci-host-bridge,index=3D1,id=3Dpci.1 \ +-device spapr-pci-host-bridge,index=3D2,id=3Dpci.2 \ -device virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.1.0,addr=3D0x1 \ -device vfio-pci,host=3D0001:01:00.0,id=3Dhostdev0,bus=3Dpci.1.0,addr=3D0x= 2 \ --device vfio-pci,host=3D0005:90:01.0,id=3Dhostdev1,bus=3Dpci.0,addr=3D0x1 +-device vfio-pci,host=3D0005:90:01.0,id=3Dhostdev1,bus=3Dpci.2.0,addr=3D0x1 diff --git a/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-3.args b/= tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-3.args index eda6cc7..66a31ba 100644 --- a/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-3.args +++ b/tests/qemuxml2argvdata/qemuxml2argv-pseries-hostdevs-3.args @@ -21,4 +21,4 @@ server,nowait \ -device spapr-pci-host-bridge,index=3D1,id=3Dpci.1 \ -device spapr-pci-host-bridge,index=3D2,id=3Dpci.2 \ -device vfio-pci,host=3D0001:01:00.0,id=3Dhostdev0,bus=3Dpci.2.0,addr=3D0x= 1 \ --device vfio-pci,host=3D0001:01:00.1,id=3Dhostdev1,bus=3Dpci.0,addr=3D0x1 +-device vfio-pci,host=3D0001:01:00.1,id=3Dhostdev1,bus=3Dpci.2.0,addr=3D0x2 diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-1.xml= b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-1.xml index 34defea..494c414 100644 --- a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-1.xml +++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-1.xml @@ -19,26 +19,34 @@ + + + + + + + +
-
+
-
+
-
+
diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-2.xml= b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-2.xml index 17ff4c8..cfa395b 100644 --- a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-2.xml +++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-2.xml @@ -26,6 +26,10 @@ + + + + @@ -38,7 +42,7 @@
-
+
diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-3.xml= b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-3.xml index 58023ec..f91959b 100644 --- a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-3.xml +++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pseries-hostdevs-3.xml @@ -39,7 +39,7 @@
-
+
--=20 2.7.5 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list