From nobody Sun Apr 28 01:49:44 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1677511971; cv=none; d=zohomail.com; s=zohoarc; b=gm6qv6arjqNmySckCC7w0Cmf+UZiru+bRvixJx6cvaEYnoymoFZcoHofnq/HQmzGPxCJ8D5dQUsN9Iw36cgDApTFIX5zH2WspIGzJGuwgFJSi64mP98zsXE7kAS+7lxQ/nY4vJx2AcYEqXZQqlVHuD5C6ep8Pv0qeIvzDUOemOk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1677511971; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=grB/qU4ojXSTp7LF998QAe7w94s0ScsryIWfXQBfClE=; b=eNcghpg1n2yUlh4syPpTaBvrWgTcSxcqtp4SH6EaW9XryRL1cbzE5ICfN+BywvrC7gsqBSqs5OU+CStk5i2f/HwQox+SzcWLzQ0GhDJRTE5IWoxdmzNyNJj8C9bKzdi4XlIkG6fpybeedgEr3vytS8gwWA7pLMPO19PX0l67q2w= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16775119716991001.153423034817; Mon, 27 Feb 2023 07:32:51 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pWfU4-0003NO-1k; Mon, 27 Feb 2023 10:32:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pWfTu-0003EQ-Fm for qemu-devel@nongnu.org; Mon, 27 Feb 2023 10:32:06 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pWfTs-00015d-SD for qemu-devel@nongnu.org; Mon, 27 Feb 2023 10:32:02 -0500 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PQPVw5bBbz686qC; Mon, 27 Feb 2023 23:27:04 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Mon, 27 Feb 2023 15:31:57 +0000 To: , Michael Tsirkin , Fan Ni CC: Ben Widawsky , , , Ira Weiny , Gregory Price , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Mike Maslenkin , Dave Jiang Subject: [PATCH v2 1/2] hw/pci: Add pcie_count_ds_port() and pcie_find_port_first() helpers Date: Mon, 27 Feb 2023 15:31:27 +0000 Message-ID: <20230227153128.8164-2-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230227153128.8164-1-Jonathan.Cameron@huawei.com> References: <20230227153128.8164-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100005.china.huawei.com (7.191.160.25) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron From: Jonathan Cameron via Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZM-MESSAGEID: 1677511973425100003 Content-Type: text/plain; charset="utf-8" These two helpers enable host bridges to operate differently depending on the number of downstream ports, in particular if there is only a single port. Useful for CXL where HDM address decoders are allowed to be implicit in the host bridge if there is only a single root port. Signed-off-by: Jonathan Cameron --- hw/pci/pcie_port.c | 38 ++++++++++++++++++++++++++++++++++++++ include/hw/pci/pcie_port.h | 2 ++ 2 files changed, 40 insertions(+) diff --git a/hw/pci/pcie_port.c b/hw/pci/pcie_port.c index 65a397ad23..1e9c485e5e 100644 --- a/hw/pci/pcie_port.c +++ b/hw/pci/pcie_port.c @@ -161,6 +161,44 @@ PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t p= n) return NULL; } =20 +/* Find first port in devfn number order */ +PCIDevice *pcie_find_port_first(PCIBus *bus) +{ + int devfn; + + for (devfn =3D 0; devfn < ARRAY_SIZE(bus->devices); devfn++) { + PCIDevice *d =3D bus->devices[devfn]; + + if (!d || !pci_is_express(d) || !d->exp.exp_cap) { + continue; + } + + if (object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) { + return d; + } + } + + return NULL; +} + +int pcie_count_ds_ports(PCIBus *bus) +{ + int dsp_count =3D 0; + int devfn; + + for (devfn =3D 0; devfn < ARRAY_SIZE(bus->devices); devfn++) { + PCIDevice *d =3D bus->devices[devfn]; + + if (!d || !pci_is_express(d) || !d->exp.exp_cap) { + continue; + } + if (object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) { + dsp_count++; + } + } + return dsp_count; +} + static const TypeInfo pcie_port_type_info =3D { .name =3D TYPE_PCIE_PORT, .parent =3D TYPE_PCI_BRIDGE, diff --git a/include/hw/pci/pcie_port.h b/include/hw/pci/pcie_port.h index 6c40e3733f..90e6cf45b8 100644 --- a/include/hw/pci/pcie_port.h +++ b/include/hw/pci/pcie_port.h @@ -41,6 +41,8 @@ struct PCIEPort { void pcie_port_init_reg(PCIDevice *d); =20 PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t pn); +PCIDevice *pcie_find_port_first(PCIBus *bus); +int pcie_count_ds_ports(PCIBus *bus); =20 #define TYPE_PCIE_SLOT "pcie-slot" OBJECT_DECLARE_SIMPLE_TYPE(PCIESlot, PCIE_SLOT) --=20 2.37.2 From nobody Sun Apr 28 01:49:44 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1677511993; cv=none; d=zohomail.com; s=zohoarc; b=bCBGuWdS6EtMbYmku1xra/w/ooAsR7w++rwiBCqM9x6+6/72HkDC6iRDTGuOCFhR1qSUxXrWhljk2UJr24vHyi5D7pBm/wumxWjA573CMabfc2X5vg6ti3sMRWF3QTBA/eBMyEj13iS9ECtgXXvtzlrBmQqFTrhnvexJ/LTX0eY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1677511993; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=x3pc/VUD8yVxWMX7DckURDiYd27J7DEpp2HAzZcpIGI=; b=ccmC90kXFLJWAAwpMSTRBZJW9mXGu+iNewKpuDEl17YA541SU9XzGSRBMasuKwzWLc8RJVLEfjzHht6AMWNM7As9UpkRzsjBwG75pUdgUIerafr7YViPEFyIFbeNAxLBrnj86T6PWafVVtYFTBxj6Bl8VnySYbquuOcWgudKT/4= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1677511993046484.6674905245029; Mon, 27 Feb 2023 07:33:13 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pWfUW-0004YT-3V; Mon, 27 Feb 2023 10:32:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pWfUR-0004IJ-6J for qemu-devel@nongnu.org; Mon, 27 Feb 2023 10:32:36 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pWfUP-000173-6K for qemu-devel@nongnu.org; Mon, 27 Feb 2023 10:32:34 -0500 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PQPWW4hWcz6J79W; Mon, 27 Feb 2023 23:27:35 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Mon, 27 Feb 2023 15:32:28 +0000 To: , Michael Tsirkin , Fan Ni CC: Ben Widawsky , , , Ira Weiny , Gregory Price , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Mike Maslenkin , Dave Jiang Subject: [PATCH v2 2/2] hw/pxb-cxl: Support passthrough HDM Decoders unless overridden Date: Mon, 27 Feb 2023 15:31:28 +0000 Message-ID: <20230227153128.8164-3-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230227153128.8164-1-Jonathan.Cameron@huawei.com> References: <20230227153128.8164-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100005.china.huawei.com (7.191.160.25) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron From: Jonathan Cameron via Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZM-MESSAGEID: 1677511993476100001 Content-Type: text/plain; charset="utf-8" The CXL r3.0 specification allows for there to be no HDM decoders on CXL Host Bridges if they have only a single root port. Instead, all accesses directed to the host bridge (as specified in CXL Fixed Memory Windows) are assumed to be routed to the single root port. Linux currently assumes this implementation choice. So to simplify testing, make QEMU emulation also default to no HDM decoders under these particular circumstances, but provide a hdm_for_passthrough boolean option to have HDM decoders as previously. Technically this is breaking backwards compatibility, but given the only known software stack used with the QEMU emulation is the Linux kernel and this configuration did not work before this change, there are unlikely to be any complaints that it now works. The option is retained to allow testing of software that does allow for these HDM decoders to exis= t, once someone writes it. Reported-by: Fan Ni Reviewed-by: Fan Ni Tested-by: Fan Ni Signed-off-by: Jonathan Cameron -- v2: Pick up and fix typo in tag from Fan Ni --- hw/cxl/cxl-host.c | 31 ++++++++++++-------- hw/pci-bridge/pci_expander_bridge.c | 44 +++++++++++++++++++++++++---- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_component.h | 1 + include/hw/pci/pci_bridge.h | 1 + 5 files changed, 61 insertions(+), 17 deletions(-) diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c index 3c1ec8732a..6e923ceeaf 100644 --- a/hw/cxl/cxl-host.c +++ b/hw/cxl/cxl-host.c @@ -146,21 +146,28 @@ static PCIDevice *cxl_cfmws_find_device(CXLFixedWindo= w *fw, hwaddr addr) return NULL; } =20 - hb_cstate =3D cxl_get_hb_cstate(hb); - if (!hb_cstate) { - return NULL; - } + if (cxl_get_hb_passthrough(hb)) { + rp =3D pcie_find_port_first(hb->bus); + if (!rp) { + return NULL; + } + } else { + hb_cstate =3D cxl_get_hb_cstate(hb); + if (!hb_cstate) { + return NULL; + } =20 - cache_mem =3D hb_cstate->crb.cache_mem_registers; + cache_mem =3D hb_cstate->crb.cache_mem_registers; =20 - target_found =3D cxl_hdm_find_target(cache_mem, addr, &target); - if (!target_found) { - return NULL; - } + target_found =3D cxl_hdm_find_target(cache_mem, addr, &target); + if (!target_found) { + return NULL; + } =20 - rp =3D pcie_find_port_by_pn(hb->bus, target); - if (!rp) { - return NULL; + rp =3D pcie_find_port_by_pn(hb->bus, target); + if (!rp) { + return NULL; + } } =20 d =3D pci_bridge_get_sec_bus(PCI_BRIDGE(rp))->devices[0]; diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expand= er_bridge.c index e752a21292..ead33f0c05 100644 --- a/hw/pci-bridge/pci_expander_bridge.c +++ b/hw/pci-bridge/pci_expander_bridge.c @@ -15,6 +15,7 @@ #include "hw/pci/pci.h" #include "hw/pci/pci_bus.h" #include "hw/pci/pci_host.h" +#include "hw/pci/pcie_port.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" #include "hw/pci-bridge/pci_expander_bridge.h" @@ -79,6 +80,13 @@ CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb) return &host->cxl_cstate; } =20 +bool cxl_get_hb_passthrough(PCIHostState *hb) +{ + CXLHost *host =3D PXB_CXL_HOST(hb); + + return host->passthrough; +} + static int pxb_bus_num(PCIBus *bus) { PXBDev *pxb =3D convert_to_pxb(bus->parent_dev); @@ -289,15 +297,32 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin) return pin - PCI_SLOT(pxb->devfn); } =20 -static void pxb_dev_reset(DeviceState *dev) +static void pxb_cxl_dev_reset(DeviceState *dev) { CXLHost *cxl =3D PXB_CXL_DEV(dev)->cxl.cxl_host_bridge; CXLComponentState *cxl_cstate =3D &cxl->cxl_cstate; + PCIHostState *hb =3D PCI_HOST_BRIDGE(cxl); uint32_t *reg_state =3D cxl_cstate->crb.cache_mem_registers; uint32_t *write_msk =3D cxl_cstate->crb.cache_mem_regs_write_mask; + int dsp_count =3D 0; =20 cxl_component_register_init_common(reg_state, write_msk, CXL2_ROOT_POR= T); - ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, TARGET_COUNT, = 8); + /* + * The CXL specification allows for host bridges with no HDM decoders + * if they only have a single root port. + */ + if (!PXB_DEV(dev)->hdm_for_passthrough) { + dsp_count =3D pcie_count_ds_ports(hb->bus); + } + /* Initial reset will have 0 dsp so wait until > 0 */ + if (dsp_count =3D=3D 1) { + cxl->passthrough =3D true; + /* Set Capability ID in header to NONE */ + ARRAY_FIELD_DP32(reg_state, CXL_HDM_CAPABILITY_HEADER, ID, 0); + } else { + ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, TARGET_COU= NT, + 8); + } } =20 static gint pxb_compare(gconstpointer a, gconstpointer b) @@ -481,9 +506,18 @@ static void pxb_cxl_dev_realize(PCIDevice *dev, Error = **errp) } =20 pxb_dev_realize_common(dev, CXL, errp); - pxb_dev_reset(DEVICE(dev)); + pxb_cxl_dev_reset(DEVICE(dev)); } =20 +static Property pxb_cxl_dev_properties[] =3D { + /* Note: 0 is not a legal PXB bus number. */ + DEFINE_PROP_UINT8("bus_nr", PXBDev, bus_nr, 0), + DEFINE_PROP_UINT16("numa_node", PXBDev, numa_node, NUMA_NODE_UNASSIGNE= D), + DEFINE_PROP_BOOL("bypass_iommu", PXBDev, bypass_iommu, false), + DEFINE_PROP_BOOL("hdm_for_passthrough", PXBDev, hdm_for_passthrough, f= alse), + DEFINE_PROP_END_OF_LIST(), +}; + static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc =3D DEVICE_CLASS(klass); @@ -497,12 +531,12 @@ static void pxb_cxl_dev_class_init(ObjectClass *klass= , void *data) */ =20 dc->desc =3D "CXL Host Bridge"; - device_class_set_props(dc, pxb_dev_properties); + device_class_set_props(dc, pxb_cxl_dev_properties); set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories); =20 /* Host bridges aren't hotpluggable. FIXME: spec reference */ dc->hotpluggable =3D false; - dc->reset =3D pxb_dev_reset; + dc->reset =3D pxb_cxl_dev_reset; } =20 static const TypeInfo pxb_cxl_dev_info =3D { diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h index b161be59b7..b2cffbb364 100644 --- a/include/hw/cxl/cxl.h +++ b/include/hw/cxl/cxl.h @@ -49,6 +49,7 @@ struct CXLHost { PCIHostState parent_obj; =20 CXLComponentState cxl_cstate; + bool passthrough; }; =20 #define TYPE_PXB_CXL_HOST "pxb-cxl-host" diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index ec4203b83f..42c7e581a7 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -247,6 +247,7 @@ static inline hwaddr cxl_decode_ig(int ig) } =20 CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb); +bool cxl_get_hb_passthrough(PCIHostState *hb); =20 void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp); void cxl_doe_cdat_release(CXLComponentState *cxl_cstate); diff --git a/include/hw/pci/pci_bridge.h b/include/hw/pci/pci_bridge.h index 63a7521567..81a058bb2c 100644 --- a/include/hw/pci/pci_bridge.h +++ b/include/hw/pci/pci_bridge.h @@ -92,6 +92,7 @@ struct PXBDev { uint8_t bus_nr; uint16_t numa_node; bool bypass_iommu; + bool hdm_for_passthrough; struct cxl_dev { CXLHost *cxl_host_bridge; /* Pointer to a CXLHost */ } cxl; --=20 2.37.2