From nobody Sun Jul 13 04:24:40 2025 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1665158499; cv=none; d=zohomail.com; s=zohoarc; b=YV0t+0FKY7gX4O+7Vbc9PeczYSQvmov2WfRIQOdTuh/MhpPW3cYuvwKkDUIWFvsAhhsPyjqWD/SruuY6z8tVPuZht/RkjxsoEJcY2FthI5vaTOlEVpf4dCQmD5q2gCK0GqSkGYwPceq06ItJL5J9rc2xNzMkNNP2btWN4XVknUw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1665158499; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=EOubzT3xT6Ri97lGwCK2J2wRNpyFnWtuuSDPiKwru4M=; b=Dp7ExlFYaO27P4MjjvgtheoP0BDrNKWxXZBtUM3lu5OPE/moqfxXMkQTzDZS8fPPl77ZYdl4TmIpUkZ3Aef4NXAUQ8qSY40xLfBZ/yGz9sbKwzy4elZ6cDmNdIlOPmoQm4VYYiFNKrLfE5TjR9dMRYpzX5yjU4hMgq3A8W+7+es= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 166515849939353.16497520987741; Fri, 7 Oct 2022 09:01:39 -0700 (PDT) Received: from localhost ([::1]:52690 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ogpn7-0003xi-UT for importer2@patchew.org; Fri, 07 Oct 2022 12:01:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57848) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogpDC-0004zM-By for qemu-devel@nongnu.org; Fri, 07 Oct 2022 11:24:30 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:2707) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogpDA-0005yK-EA for qemu-devel@nongnu.org; Fri, 07 Oct 2022 11:24:30 -0400 Received: from fraeml741-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4MkXCJ0bn1z6872P; Fri, 7 Oct 2022 23:23:56 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml741-chm.china.huawei.com (10.206.15.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 7 Oct 2022 17:24:26 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 7 Oct 2022 16:24:26 +0100 To: , Michael Tsirkin , Ben Widawsky , , Huai-Cheng Kuo , Chris Browy CC: , Subject: [PATCH v7 5/5] hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE Date: Fri, 7 Oct 2022 16:21:56 +0100 Message-ID: <20221007152156.24883-6-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221007152156.24883-1-Jonathan.Cameron@huawei.com> References: <20221007152156.24883-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml500002.china.huawei.com (7.191.160.78) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" Reply-to: Jonathan Cameron From: Jonathan Cameron via X-ZM-MESSAGEID: 1665158501642100001 Content-Type: text/plain; charset="utf-8" This Data Object Exchange Mailbox allows software to query the latency and bandwidth between ports on the switch. For now only provide information on routes between the upstream port and each downstream port (not p2p). Signed-off-by: Jonathan Cameron --- hw/pci-bridge/cxl_upstream.c | 182 ++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_cdat.h | 1 + 2 files changed, 182 insertions(+), 1 deletion(-) diff --git a/hw/pci-bridge/cxl_upstream.c b/hw/pci-bridge/cxl_upstream.c index a83a3e81e4..9209c704ae 100644 --- a/hw/pci-bridge/cxl_upstream.c +++ b/hw/pci-bridge/cxl_upstream.c @@ -10,11 +10,12 @@ =20 #include "qemu/osdep.h" #include "qemu/log.h" +#include "hw/qdev-properties.h" #include "hw/pci/msi.h" #include "hw/pci/pcie.h" #include "hw/pci/pcie_port.h" =20 -#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 1 +#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 2 =20 #define CXL_UPSTREAM_PORT_MSI_OFFSET 0x70 #define CXL_UPSTREAM_PORT_PCIE_CAP_OFFSET 0x90 @@ -28,6 +29,7 @@ typedef struct CXLUpstreamPort { =20 /*< public >*/ CXLComponentState cxl_cstate; + DOECap doe_cdat; } CXLUpstreamPort; =20 CXLComponentState *cxl_usp_to_cstate(CXLUpstreamPort *usp) @@ -60,6 +62,9 @@ static void cxl_usp_dvsec_write_config(PCIDevice *dev, ui= nt32_t addr, static void cxl_usp_write_config(PCIDevice *d, uint32_t address, uint32_t val, int len) { + CXLUpstreamPort *usp =3D CXL_USP(d); + + pcie_doe_write_config(&usp->doe_cdat, address, val, len); pci_bridge_write_config(d, address, val, len); pcie_cap_flr_write_config(d, address, val, len); pcie_aer_write_config(d, address, val, len); @@ -67,6 +72,18 @@ static void cxl_usp_write_config(PCIDevice *d, uint32_t = address, cxl_usp_dvsec_write_config(d, address, val, len); } =20 +static uint32_t cxl_usp_read_config(PCIDevice *d, uint32_t address, int le= n) +{ + CXLUpstreamPort *usp =3D CXL_USP(d); + uint32_t val; + =20 + if (pcie_doe_read_config(&usp->doe_cdat, address, len, &val)) { + return val; + } + + return pci_default_read_config(d, address, len); +} + static void latch_registers(CXLUpstreamPort *usp) { uint32_t *reg_state =3D usp->cxl_cstate.crb.cache_mem_registers; @@ -119,6 +136,155 @@ static void build_dvsecs(CXLComponentState *cxl) REG_LOC_DVSEC_REVID, dvsec); } =20 +static bool cxl_doe_cdat_rsp(DOECap *doe_cap) +{ + CDATObject *cdat =3D &CXL_USP(doe_cap->pdev)->cxl_cstate.cdat; + uint16_t ent; + void *base; + uint32_t len; + CDATReq *req =3D pcie_doe_get_write_mbox_ptr(doe_cap); + CDATRsp rsp; + + cxl_doe_cdat_update(&CXL_USP(doe_cap->pdev)->cxl_cstate, &error_fatal); + assert(cdat->entry_len); + + /* Discard if request length mismatched */ + if (pcie_doe_get_obj_len(req) < + DIV_ROUND_UP(sizeof(CDATReq), sizeof(uint32_t))) { + return false; + } + + ent =3D req->entry_handle; + base =3D cdat->entry[ent].base; + len =3D cdat->entry[ent].length; + + rsp =3D (CDATRsp) { + .header =3D { + .vendor_id =3D CXL_VENDOR_ID, + .data_obj_type =3D CXL_DOE_TABLE_ACCESS, + .reserved =3D 0x0, + .length =3D DIV_ROUND_UP((sizeof(rsp) + len), sizeof(uint32_t)= ), + }, + .rsp_code =3D CXL_DOE_TAB_RSP, + .table_type =3D CXL_DOE_TAB_TYPE_CDAT, + .entry_handle =3D (ent < cdat->entry_len - 1) ? + ent + 1 : CXL_DOE_TAB_ENT_MAX, + }; + + memcpy(doe_cap->read_mbox, &rsp, sizeof(rsp)); + memcpy(doe_cap->read_mbox + DIV_ROUND_UP(sizeof(rsp), sizeof(uint3= 2_t)), + base, len); + + doe_cap->read_mbox_len +=3D rsp.header.length; + + return true; +} + +static DOEProtocol doe_cdat_prot[] =3D { + { CXL_VENDOR_ID, CXL_DOE_TABLE_ACCESS, cxl_doe_cdat_rsp }, + { } +}; + +static int build_cdat_table(CDATSubHeader ***cdat_table, void *priv) +{ + g_autofree CDATSslbis *sslbis_latency =3D NULL; + g_autofree CDATSslbis *sslbis_bandwidth =3D NULL; + CXLUpstreamPort *us =3D CXL_USP(priv); + PCIBus *bus =3D &PCI_BRIDGE(us)->sec_bus; + int devfn, sslbis_size; + int len =3D 0; + int i =3D 0; + int count =3D 0; + uint16_t port_ids[256]; + + for (devfn =3D 0; devfn < ARRAY_SIZE(bus->devices); devfn++) { + PCIDevice *d =3D bus->devices[devfn]; + PCIEPort *port; + =20 + if (!d || !pci_is_express(d) || !d->exp.exp_cap) { + continue; + } + + /* + * Whilst the PCI express spec doesn't allow anything other than + * downstream ports on this bus, let us be a little paranoid + */ + if (!object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) { + continue; + } + + port =3D PCIE_PORT(d); + port_ids[count] =3D port->port; + count++; + } + + /* May not yet have any ports - try again later */ + if (count =3D=3D 0) { + return 0; + } + + sslbis_size =3D sizeof(CDATSslbis) + sizeof(*sslbis_latency->sslbe) * = count; + sslbis_latency =3D g_malloc(sslbis_size); + *sslbis_latency =3D (CDATSslbis) { + .sslbis_header =3D { + .header =3D { + .type =3D CDAT_TYPE_SSLBIS, + .length =3D sslbis_size, + }, + .data_type =3D HMATLB_DATA_TYPE_ACCESS_LATENCY, + .entry_base_unit =3D 10000, + }, + }; + =20 + for (i =3D 0; i < count; i++) { + sslbis_latency->sslbe[i] =3D (CDATSslbe) { + .port_x_id =3D CDAT_PORT_ID_USP, + .port_y_id =3D port_ids[i], + .latency_bandwidth =3D 15, /* 150ns */ + }; + } + len++; + =20 + sslbis_bandwidth =3D g_malloc(sslbis_size); + *sslbis_bandwidth =3D (CDATSslbis) { + .sslbis_header =3D { + .header =3D { + .type =3D CDAT_TYPE_SSLBIS, + .length =3D sslbis_size, + }, + .data_type =3D HMATLB_DATA_TYPE_ACCESS_BANDWIDTH, + .entry_base_unit =3D 1000, + }, + }; + =20 + for (i =3D 0; i < count; i++) { + sslbis_bandwidth->sslbe[i] =3D (CDATSslbe) { + .port_x_id =3D CDAT_PORT_ID_USP, + .port_y_id =3D port_ids[i], + .latency_bandwidth =3D 16, /* 16 GB/s */ + }; + } + len++; + *cdat_table =3D g_malloc0(len * sizeof(*cdat_table)); + /* Header always at start of structure */ + i =3D 0; + (*cdat_table)[i++] =3D g_steal_pointer(&sslbis_latency); + (*cdat_table)[i++] =3D g_steal_pointer(&sslbis_bandwidth); + =20 + return len; +} + +static void free_default_cdat_table(CDATSubHeader **cdat_table, int num, + void *priv) +{ + int i; + + for (i =3D 0; i < num; i++) { + g_free(cdat_table[i]); + } + g_free(cdat_table); +} + static void cxl_usp_realize(PCIDevice *d, Error **errp) { PCIEPort *p =3D PCIE_PORT(d); @@ -161,6 +327,13 @@ static void cxl_usp_realize(PCIDevice *d, Error **errp) PCI_BASE_ADDRESS_MEM_TYPE_64, component_bar); =20 + pcie_doe_init(d, &usp->doe_cdat, cxl_cstate->dvsec_offset, doe_cdat_pr= ot, true, 1); + + cxl_cstate->cdat.build_cdat_table =3D build_cdat_table; + cxl_cstate->cdat.free_cdat_table =3D free_default_cdat_table; + cxl_cstate->cdat.private =3D d; + cxl_doe_cdat_init(cxl_cstate, errp); + return; =20 err_cap: @@ -179,6 +352,11 @@ static void cxl_usp_exitfn(PCIDevice *d) pci_bridge_exitfn(d); } =20 +static Property cxl_upstream_props[] =3D { + DEFINE_PROP_STRING("cdat", CXLUpstreamPort, cxl_cstate.cdat.filename), + DEFINE_PROP_END_OF_LIST() +}; + static void cxl_upstream_class_init(ObjectClass *oc, void *data) { DeviceClass *dc =3D DEVICE_CLASS(oc); @@ -186,6 +364,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, vo= id *data) =20 k->is_bridge =3D true; k->config_write =3D cxl_usp_write_config; + k->config_read =3D cxl_usp_read_config; k->realize =3D cxl_usp_realize; k->exit =3D cxl_usp_exitfn; k->vendor_id =3D 0x19e5; /* Huawei */ @@ -194,6 +373,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, vo= id *data) set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories); dc->desc =3D "CXL Switch Upstream Port"; dc->reset =3D cxl_usp_reset; + device_class_set_props(dc, cxl_upstream_props); } =20 static const TypeInfo cxl_usp_info =3D { diff --git a/include/hw/cxl/cxl_cdat.h b/include/hw/cxl/cxl_cdat.h index fdb1fa98f4..6d251dc0fb 100644 --- a/include/hw/cxl/cxl_cdat.h +++ b/include/hw/cxl/cxl_cdat.h @@ -131,6 +131,7 @@ typedef struct CDATSslbisHeader { uint64_t entry_base_unit; } QEMU_PACKED CDATSslbisHeader; =20 +#define CDAT_PORT_ID_USP 0x100 /* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ typedef struct CDATSslbe { uint16_t port_x_id; --=20 2.37.2