From nobody Sun Dec 29 19:37:03 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1720047161; cv=none; d=zohomail.com; s=zohoarc; b=FsCmME6tVqFzB1VndDZfe4K3Hx3l0cxsB6bwIAOJkLORlECAs5pOIl2miAFM1Qb3ZPPK2BNQUA3vjG15fZCC/aVpIGrCCngoB4rI+sQM3GKtOVr/I+7Ibdk+MZTnGZkjFmHFKuU2auhTWdJV3ReFKrcaSfI1Rg9VmzQq2ZmtM9Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1720047161; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=L4UFSErF8Ci7X9ojGTRrl+PVA8jTE90RlPCAhdXfrho=; b=dakTvqiN9+uhKc8QALzVnTy7bbo/OKiXB2l3TthLWSRi0jvILTbgL3SC1lNtyoo9aQtQo607R8yfYg0i02HF3q6rgTlG87XtD1aJKDXonOjJO6zbSbziC/xypMUWbNKM55n2yPd26n1v4axJ1MrmTkn+Zd5PSxne/XDzA6ZC+bY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1720047161087181.32948997051153; Wed, 3 Jul 2024 15:52:41 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sP8oD-0006F4-9F; Wed, 03 Jul 2024 18:50:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sP8oB-0005u5-2b for qemu-devel@nongnu.org; Wed, 03 Jul 2024 18:50:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sP8o6-0007ZB-8T for qemu-devel@nongnu.org; Wed, 03 Jul 2024 18:50:38 -0400 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-288-87OYWLRMMzilwtm5H_q-uw-1; Wed, 03 Jul 2024 18:50:32 -0400 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4257f95ffc6so11285e9.1 for ; Wed, 03 Jul 2024 15:50:32 -0700 (PDT) Received: from redhat.com ([2a0d:6fc7:441:91a8:a47d:5a9:c02f:92f2]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36795d1fc9csm2270500f8f.83.2024.07.03.15.50.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 15:50:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720047033; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=L4UFSErF8Ci7X9ojGTRrl+PVA8jTE90RlPCAhdXfrho=; b=a7JOllFoIDmINg3ETAnXUKDhEjokLtAk3Y3NcRMIQ7m2sTU5aMcAJZoQoqbhIswmqj0rkW tXQjN84Qtl6f9mdX0izGzOQj9fYoWEaua6CWWWC+hsNIgU9aEOthwasGp7aGJsi6BQYAG0 pjMOFiMS5gu2WUHZNIKVRBLGfmfRe/w= X-MC-Unique: 87OYWLRMMzilwtm5H_q-uw-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720047030; x=1720651830; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=L4UFSErF8Ci7X9ojGTRrl+PVA8jTE90RlPCAhdXfrho=; b=D3t4kidlJvMF5uB4ci0EOKz6iwfiHBKH4zl7ccRTUyCrlNrTe+tHAzrycHYmhfEn8/ DBiFLJCseC/Cipxd4KSFblvbOxJ9x3Wy9AiEiFrSfvuawQZ30//oDQ3U9Xs9tJgwWksb E25tzjDhnLS4BJegxKDLAuWAEzt0o7oyK3rPIF6tP2YkWgjB7a3YR1By5KZe/mNt1Uvo vRW+0Fb0PjEUu/ByMeBWSJ0Hh0NY7cxzwP49KBfirbg2bmpVLGysNPf6yYa0oeFBiZpr sCxhwiqxn/e4OAcEHDIausnxddA7NC+03a+V0atM1iAXNoBtAXsrq/pse4cZroRGfntH ztFQ== X-Gm-Message-State: AOJu0Yyc4sWyVfXKadLF74lCUZFbQ8qITN4f7+kxjLak3jj9TaYOk2ys 7GsghXpzIosjsbk784m6h+G/X5ykgwmoERqfY/oZQlzdbW5bFiWULJYusDCkkzwpaBTUXsx+9eM 0FPRXtVLy4sCYgTLtV3RsicXbOOv/SFicAhU5Y065mS7NxSdSGteIYMtn4OxmO7NtzMuJ+BvuXs CRYbXqw0jKXjYFQRsawRNaI9GVHrz1QQ== X-Received: by 2002:a05:600c:28a:b0:424:9f28:22ff with SMTP id 5b1f17b1804b1-4264a3cf814mr353095e9.5.1720047030747; Wed, 03 Jul 2024 15:50:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGdl8KWDXUE7OjJnPO3OwkYv4TJIdiTCSdwNerRIowPNWdhLXr7NOXnyMhAO43/5fT9oFl+3g== X-Received: by 2002:a05:600c:28a:b0:424:9f28:22ff with SMTP id 5b1f17b1804b1-4264a3cf814mr352945e9.5.1720047030069; Wed, 03 Jul 2024 15:50:30 -0700 (PDT) Date: Wed, 3 Jul 2024 18:50:26 -0400 From: "Michael S. Tsirkin" To: qemu-devel@nongnu.org Cc: Peter Maydell , Akihiko Odaki , Marcel Apfelbaum Subject: [PULL v3 81/85] pcie_sriov: Reuse SR-IOV VF device instances Message-ID: <139610ae67f6ecf92127bb7bf53ac6265b459ec8.1720046570.git.mst@redhat.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Mailer: git-send-email 2.27.0.106.g8ac3dc51b1 X-Mutt-Fcc: =sent Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=mst@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1720047161627100001 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Akihiko Odaki Disable SR-IOV VF devices by reusing code to power down PCI devices instead of removing them when the guest requests to disable VFs. This allows to realize devices and report VF realization errors at PF realization time. Signed-off-by: Akihiko Odaki Message-Id: <20240627-reuse-v10-6-7ca0b8ed3d9f@daynix.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin --- include/hw/pci/pci.h | 5 -- include/hw/pci/pci_device.h | 15 ++++++ include/hw/pci/pcie_sriov.h | 1 - hw/pci/pci.c | 2 +- hw/pci/pcie_sriov.c | 95 ++++++++++++++++--------------------- 5 files changed, 56 insertions(+), 62 deletions(-) diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h index fe04b4fafd..14a869eeaa 100644 --- a/include/hw/pci/pci.h +++ b/include/hw/pci/pci.h @@ -680,9 +680,4 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev) MSIMessage pci_get_msi_message(PCIDevice *dev, int vector); void pci_set_enabled(PCIDevice *pci_dev, bool state); =20 -static inline void pci_set_power(PCIDevice *pci_dev, bool state) -{ - pci_set_enabled(pci_dev, state); -} - #endif diff --git a/include/hw/pci/pci_device.h b/include/hw/pci/pci_device.h index d57f9ce838..ca15132508 100644 --- a/include/hw/pci/pci_device.h +++ b/include/hw/pci/pci_device.h @@ -205,6 +205,21 @@ static inline uint16_t pci_get_bdf(PCIDevice *dev) return PCI_BUILD_BDF(pci_bus_num(pci_get_bus(dev)), dev->devfn); } =20 +static inline void pci_set_power(PCIDevice *pci_dev, bool state) +{ + /* + * Don't change the enabled state of VFs when powering on/off the devi= ce. + * + * When powering on, VFs must not be enabled immediately but they must + * wait until the guest configures SR-IOV. + * When powering off, their corresponding PFs will be reset and disable + * VFs. + */ + if (!pci_is_vf(pci_dev)) { + pci_set_enabled(pci_dev, state); + } +} + uint16_t pci_requester_id(PCIDevice *dev); =20 /* DMA access functions */ diff --git a/include/hw/pci/pcie_sriov.h b/include/hw/pci/pcie_sriov.h index aa704e8f9d..70649236c1 100644 --- a/include/hw/pci/pcie_sriov.h +++ b/include/hw/pci/pcie_sriov.h @@ -18,7 +18,6 @@ typedef struct PCIESriovPF { uint16_t num_vfs; /* Number of virtual functions created */ uint8_t vf_bar_type[PCI_NUM_REGIONS]; /* Store type for each VF bar = */ - const char *vfname; /* Reference to the device type used for the VFs */ PCIDevice **vf; /* Pointer to an array of num_vfs VF devices */ } PCIESriovPF; =20 diff --git a/hw/pci/pci.c b/hw/pci/pci.c index 68d30feb86..e32a69f3fa 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -2895,7 +2895,7 @@ void pci_set_enabled(PCIDevice *d, bool state) memory_region_set_enabled(&d->bus_master_enable_region, (pci_get_word(d->config + PCI_COMMAND) & PCI_COMMAND_MASTER) && d->enabled); - if (!d->enabled) { + if (d->qdev.realized) { pci_device_reset(d); } } diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c index f0bde0d3fc..faadb0d2ea 100644 --- a/hw/pci/pcie_sriov.c +++ b/hw/pci/pcie_sriov.c @@ -20,9 +20,16 @@ #include "qapi/error.h" #include "trace.h" =20 -static PCIDevice *register_vf(PCIDevice *pf, int devfn, - const char *name, uint16_t vf_num); -static void unregister_vfs(PCIDevice *dev); +static void unparent_vfs(PCIDevice *dev, uint16_t total_vfs) +{ + for (uint16_t i =3D 0; i < total_vfs; i++) { + PCIDevice *vf =3D dev->exp.sriov_pf.vf[i]; + object_unparent(OBJECT(vf)); + object_unref(OBJECT(vf)); + } + g_free(dev->exp.sriov_pf.vf); + dev->exp.sriov_pf.vf =3D NULL; +} =20 bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, const char *vfname, uint16_t vf_dev_id, @@ -30,6 +37,8 @@ bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, uint16_t vf_offset, uint16_t vf_stride, Error **errp) { + BusState *bus =3D qdev_get_parent_bus(&dev->qdev); + int32_t devfn =3D dev->devfn + vf_offset; uint8_t *cfg =3D dev->config + offset; uint8_t *wmask; =20 @@ -49,7 +58,6 @@ bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, offset, PCI_EXT_CAP_SRIOV_SIZEOF); dev->exp.sriov_cap =3D offset; dev->exp.sriov_pf.num_vfs =3D 0; - dev->exp.sriov_pf.vfname =3D g_strdup(vfname); dev->exp.sriov_pf.vf =3D NULL; =20 pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset); @@ -83,14 +91,34 @@ bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, =20 qdev_prop_set_bit(&dev->qdev, "multifunction", true); =20 + dev->exp.sriov_pf.vf =3D g_new(PCIDevice *, total_vfs); + + for (uint16_t i =3D 0; i < total_vfs; i++) { + PCIDevice *vf =3D pci_new(devfn, vfname); + vf->exp.sriov_vf.pf =3D dev; + vf->exp.sriov_vf.vf_number =3D i; + + if (!qdev_realize(&vf->qdev, bus, errp)) { + unparent_vfs(dev, i); + return false; + } + + /* set vid/did according to sr/iov spec - they are not used */ + pci_config_set_vendor_id(vf->config, 0xffff); + pci_config_set_device_id(vf->config, 0xffff); + + dev->exp.sriov_pf.vf[i] =3D vf; + devfn +=3D vf_stride; + } + return true; } =20 void pcie_sriov_pf_exit(PCIDevice *dev) { - unregister_vfs(dev); - g_free((char *)dev->exp.sriov_pf.vfname); - dev->exp.sriov_pf.vfname =3D NULL; + uint8_t *cfg =3D dev->config + dev->exp.sriov_cap; + + unparent_vfs(dev, pci_get_word(cfg + PCI_SRIOV_TOTAL_VF)); } =20 void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num, @@ -156,38 +184,11 @@ void pcie_sriov_vf_register_bar(PCIDevice *dev, int r= egion_num, } } =20 -static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name, - uint16_t vf_num) -{ - PCIDevice *dev =3D pci_new(devfn, name); - dev->exp.sriov_vf.pf =3D pf; - dev->exp.sriov_vf.vf_number =3D vf_num; - PCIBus *bus =3D pci_get_bus(pf); - Error *local_err =3D NULL; - - qdev_realize(&dev->qdev, &bus->qbus, &local_err); - if (local_err) { - error_report_err(local_err); - return NULL; - } - - /* set vid/did according to sr/iov spec - they are not used */ - pci_config_set_vendor_id(dev->config, 0xffff); - pci_config_set_device_id(dev->config, 0xffff); - - return dev; -} - static void register_vfs(PCIDevice *dev) { uint16_t num_vfs; uint16_t i; uint16_t sriov_cap =3D dev->exp.sriov_cap; - uint16_t vf_offset =3D - pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_OFFSET); - uint16_t vf_stride =3D - pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_STRIDE); - int32_t devfn =3D dev->devfn + vf_offset; =20 assert(sriov_cap > 0); num_vfs =3D pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF); @@ -195,18 +196,10 @@ static void register_vfs(PCIDevice *dev) return; } =20 - dev->exp.sriov_pf.vf =3D g_new(PCIDevice *, num_vfs); - trace_sriov_register_vfs(dev->name, PCI_SLOT(dev->devfn), PCI_FUNC(dev->devfn), num_vfs); for (i =3D 0; i < num_vfs; i++) { - dev->exp.sriov_pf.vf[i] =3D register_vf(dev, devfn, - dev->exp.sriov_pf.vfname, i); - if (!dev->exp.sriov_pf.vf[i]) { - num_vfs =3D i; - break; - } - devfn +=3D vf_stride; + pci_set_enabled(dev->exp.sriov_pf.vf[i], true); } dev->exp.sriov_pf.num_vfs =3D num_vfs; } @@ -219,12 +212,8 @@ static void unregister_vfs(PCIDevice *dev) trace_sriov_unregister_vfs(dev->name, PCI_SLOT(dev->devfn), PCI_FUNC(dev->devfn), num_vfs); for (i =3D 0; i < num_vfs; i++) { - PCIDevice *vf =3D dev->exp.sriov_pf.vf[i]; - object_unparent(OBJECT(vf)); - object_unref(OBJECT(vf)); + pci_set_enabled(dev->exp.sriov_pf.vf[i], false); } - g_free(dev->exp.sriov_pf.vf); - dev->exp.sriov_pf.vf =3D NULL; dev->exp.sriov_pf.num_vfs =3D 0; } =20 @@ -246,14 +235,10 @@ void pcie_sriov_config_write(PCIDevice *dev, uint32_t= address, PCI_FUNC(dev->devfn), off, val, len); =20 if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) { - if (dev->exp.sriov_pf.num_vfs) { - if (!(val & PCI_SRIOV_CTRL_VFE)) { - unregister_vfs(dev); - } + if (val & PCI_SRIOV_CTRL_VFE) { + register_vfs(dev); } else { - if (val & PCI_SRIOV_CTRL_VFE) { - register_vfs(dev); - } + unregister_vfs(dev); } } } --=20 MST