From nobody Thu May 15 11:52:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1507837004079197.42518601637983; Thu, 12 Oct 2017 12:36:44 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B2C8163E12; Thu, 12 Oct 2017 19:36:42 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8AD145C89F; Thu, 12 Oct 2017 19:36:42 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 4F75318355DF; Thu, 12 Oct 2017 19:36:42 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v9CJWGsT006071 for ; Thu, 12 Oct 2017 15:32:16 -0400 Received: by smtp.corp.redhat.com (Postfix) id 251E817B8A; Thu, 12 Oct 2017 19:32:16 +0000 (UTC) Received: from mx1.redhat.com (ext-mx02.extmail.prod.ext.phx2.redhat.com [10.5.110.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1EB3E6957D for ; Thu, 12 Oct 2017 19:32:13 +0000 (UTC) Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BDD38820F7 for ; Thu, 12 Oct 2017 19:32:11 +0000 (UTC) Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v9CJWAoX019023 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Oct 2017 19:32:10 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id v9CJWAMM007120 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Oct 2017 19:32:10 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id v9CJW9b5011987; Thu, 12 Oct 2017 19:32:09 GMT Received: from nina.dynamic.ziggo.nl (/10.175.236.97) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 12 Oct 2017 12:32:09 -0700 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B2C8163E12 Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com B2C8163E12 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com BDD38820F7 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=wim.ten.have@oracle.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com BDD38820F7 From: Wim Ten Have To: Libvirt Development List Date: Thu, 12 Oct 2017 21:31:54 +0200 Message-Id: <20171012193157.2711-2-wim.ten.have@oracle.com> In-Reply-To: <20171012193157.2711-1-wim.ten.have@oracle.com> References: <20171012193157.2711-1-wim.ten.have@oracle.com> X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 205 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 12 Oct 2017 19:32:12 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 12 Oct 2017 19:32:12 +0000 (UTC) for IP:'156.151.31.81' DOMAIN:'userp1040.oracle.com' HELO:'userp1040.oracle.com' FROM:'wim.ten.have@oracle.com' RCPT:'' X-RedHat-Spam-Score: -105.101 (RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H2, RP_MATCHES_RCVD, SPF_PASS, UNPARSEABLE_RELAY, USER_IN_WHITELIST) 156.151.31.81 userp1040.oracle.com 156.151.31.81 userp1040.oracle.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.26 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: libvir-list@redhat.com Cc: Wim ten Have Subject: [libvirt] [PATCH v5 1/4] numa: describe siblings distances within cells X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 12 Oct 2017 19:36:43 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Wim ten Have Add support for describing NUMA distances in a domain's XML description. Below is an example of a 4 node setup: A defines a NUMA node. describes the NUMA distance from the to the other NUMA nodes (the s). For example, in above XML description, the distance between NUMA node0 and NUMA node2 is 31. Valid distance value are '10 <=3D value <=3D 255'. A distance value of 10 represents the distance to the node itself. A distance value of 20 represents the default value for remote nodes but other values are possible depending on the physical topology of the system. When distances are not fully described, any missing sibling distance values will default to 10 for local nodes and 20 for remote nodes. Signed-off-by: Wim ten Have Reviewed-by: Daniel P. Berrange --- Changes on v1: - Add changes to docs/formatdomain.html.in describing schema update. Changes on v2: - Automatically apply distance symmetry maintaining cell <-> sibling. - Check for maximum '255' on numaDistanceValue. - Automatically complete empty distance ranges. - Check that sibling_id's are in range with cell identifiers. - Allow non-contiguous ranges, starting from any node id. - Respect parameters as ATTRIBUTE_NONNULL fix functions and callers. - Add and apply topology for LOCAL_DISTANCE=3D10 and REMOTE_DISTANCE=3D20. Changes on v3 - Add UNREACHABLE if one locality is unreachable from another. - Add code cleanup aligning function naming in a separated patch. - Add numa related driver code in a separated patch. - Remove from numaDistanceValue schema/basictypes.rng - Correct doc changes. Changes on v4 - Fix symmetry error under virDomainNumaDefNodeDistanceParseXML() --- docs/formatdomain.html.in | 63 ++++++++++++- docs/schemas/basictypes.rng | 7 ++ docs/schemas/cputypes.rng | 18 ++++ src/conf/numa_conf.c | 221 ++++++++++++++++++++++++++++++++++++++++= +++- 4 files changed, 305 insertions(+), 4 deletions(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index c0e3c2221..ab2a89c6a 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -1529,7 +1529,68 @@

=20

- This guest NUMA specification is currently available only for QEMU/K= VM. + This guest NUMA specification is currently available only for + QEMU/KVM and Xen. Whereas Xen driver also allows for a distinct + description of NUMA arranged sibling cell + distances Since 3.8.0. +

+ +

+ Under NUMA h/w architecture, distinct resources such as memory + create a designated distance between cell and + siblings that now can be described with the help of + distances. A detailed description can be found within + the ACPI (Advanced Configuration and Power Interface Specification) + within the chapter explaining the system's SLIT (System Locality + Distance Information Table). +

+ +
+...
+<cpu>
+  ...
+  <numa>
+    <cell id=3D'0' cpus=3D'0,4-7' memory=3D'512000' unit=3D'KiB'>
+      <distances>
+        <sibling id=3D'0' value=3D'10'/>
+        <sibling id=3D'1' value=3D'21'/>
+        <sibling id=3D'2' value=3D'31'/>
+        <sibling id=3D'3' value=3D'41'/>
+      </distances>
+    </cell>
+    <cell id=3D'1' cpus=3D'1,8-10,12-15' memory=3D'512000' unit=3D'KiB'=
 memAccess=3D'shared'>
+      <distances>
+        <sibling id=3D'0' value=3D'21'/>
+        <sibling id=3D'1' value=3D'10'/>
+        <sibling id=3D'2' value=3D'21'/>
+        <sibling id=3D'3' value=3D'31'/>
+      </distances>
+    </cell>
+    <cell id=3D'2' cpus=3D'2,11' memory=3D'512000' unit=3D'KiB' memAcce=
ss=3D'shared'>
+      <distances>
+        <sibling id=3D'0' value=3D'31'/>
+        <sibling id=3D'1' value=3D'21'/>
+        <sibling id=3D'2' value=3D'10'/>
+        <sibling id=3D'3' value=3D'21'/>
+      </distances>
+    </cell>
+    <cell id=3D'3' cpus=3D'3' memory=3D'512000' unit=3D'KiB'>
+      <distances>
+        <sibling id=3D'0' value=3D'41'/>
+        <sibling id=3D'1' value=3D'31'/>
+        <sibling id=3D'2' value=3D'21'/>
+        <sibling id=3D'3' value=3D'10'/>
+      </distances>
+    </cell>
+  </numa>
+  ...
+</cpu>
+...
+ +

+ Under Xen driver, if no distances are given to describe + the SLIT data between different cells, it will default to a scheme + using 10 for local and 20 for remote distances.

=20

Events configuration

diff --git a/docs/schemas/basictypes.rng b/docs/schemas/basictypes.rng index 1ea667cdf..1a18cd31b 100644 --- a/docs/schemas/basictypes.rng +++ b/docs/schemas/basictypes.rng @@ -77,6 +77,13 @@ =20 + + + 10 + 255 + + + diff --git a/docs/schemas/cputypes.rng b/docs/schemas/cputypes.rng index 3eef16abc..c45b6dfb2 100644 --- a/docs/schemas/cputypes.rng +++ b/docs/schemas/cputypes.rng @@ -129,6 +129,24 @@ + + + + + + + + + + + + + + + + + + =20 diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c index b71dc012c..00a3343fb 100644 --- a/src/conf/numa_conf.c +++ b/src/conf/numa_conf.c @@ -29,6 +29,15 @@ #include "virnuma.h" #include "virstring.h" =20 +/* + * Distance definitions defined Conform ACPI 2.0 SLIT. + * See include/linux/topology.h + */ +#define LOCAL_DISTANCE 10 +#define REMOTE_DISTANCE 20 +/* SLIT entry value is a one-byte unsigned integer. */ +#define UNREACHABLE 255 + #define VIR_FROM_THIS VIR_FROM_DOMAIN =20 VIR_ENUM_IMPL(virDomainNumatuneMemMode, @@ -48,6 +57,8 @@ VIR_ENUM_IMPL(virDomainMemoryAccess, VIR_DOMAIN_MEMORY_AC= CESS_LAST, "shared", "private") =20 +typedef struct _virDomainNumaDistance virDomainNumaDistance; +typedef virDomainNumaDistance *virDomainNumaDistancePtr; =20 typedef struct _virDomainNumaNode virDomainNumaNode; typedef virDomainNumaNode *virDomainNumaNodePtr; @@ -66,6 +77,12 @@ struct _virDomainNuma { virBitmapPtr nodeset; /* host memory nodes where this guest node= resides */ virDomainNumatuneMemMode mode; /* memory mode selection */ virDomainMemoryAccess memAccess; /* shared memory access configura= tion */ + + struct _virDomainNumaDistance { + unsigned int value; /* locality value for node i->j or j->i */ + unsigned int cellid; + } *distances; /* remote node distances */ + size_t ndistances; } *mem_nodes; /* guest node configuration */ size_t nmem_nodes; =20 @@ -686,6 +703,174 @@ virDomainNumatuneNodesetIsAvailable(virDomainNumaPtr = numatune, } =20 =20 +static int +virDomainNumaDefNodeDistanceParseXML(virDomainNumaPtr def, + xmlXPathContextPtr ctxt, + unsigned int cur_cell) +{ + int ret =3D -1; + int sibling; + char *tmp =3D NULL; + xmlNodePtr *nodes =3D NULL; + size_t i, ndistances =3D def->nmem_nodes; + + if (!ndistances) + return 0; + + /* check if NUMA distances definition is present */ + if (!virXPathNode("./distances[1]", ctxt)) + return 0; + + if ((sibling =3D virXPathNodeSet("./distances[1]/sibling", ctxt, &node= s)) <=3D 0) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("NUMA distances defined without siblings")); + goto cleanup; + } + + for (i =3D 0; i < sibling; i++) { + virDomainNumaDistancePtr ldist, rdist; + unsigned int sibling_id, sibling_value; + + /* siblings are in order of parsing or explicitly numbered */ + if (!(tmp =3D virXMLPropString(nodes[i], "id"))) { + virReportError(VIR_ERR_XML_ERROR, + _("Missing 'id' attribute in NUMA " + "distances under 'cell id %d'"), + cur_cell); + goto cleanup; + } + + /* The "id" needs to be applicable */ + if (virStrToLong_uip(tmp, NULL, 10, &sibling_id) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'id' attribute in NUMA " + "distances for sibling: '%s'"), + tmp); + goto cleanup; + } + VIR_FREE(tmp); + + /* The "id" needs to be within numa/cell range */ + if (sibling_id >=3D ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("There is no cell administrated matching " + "'sibling_id %d' under NUMA 'cell id %d' "), + sibling_id, cur_cell); + goto cleanup; + } + + /* We need a locality value. Check and correct + * distance to local and distance to remote node. + */ + if (!(tmp =3D virXMLPropString(nodes[i], "value"))) { + virReportError(VIR_ERR_XML_ERROR, + _("Missing 'value' attribute in NUMA distances " + "under 'cell id %d' for 'sibling id %d'"), + cur_cell, sibling_id); + goto cleanup; + } + + /* The "value" needs to be applicable */ + if (virStrToLong_uip(tmp, NULL, 10, &sibling_value) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'value' attribute in NUMA " + "distances for value: '%s'"), + tmp); + goto cleanup; + } + VIR_FREE(tmp); + + /* LOCAL_DISTANCE <=3D "value" <=3D UNREACHABLE */ + if (sibling_value < LOCAL_DISTANCE || + sibling_value > UNREACHABLE) { + virReportError(VIR_ERR_XML_ERROR, + _("Out of range value '%d' set for " + "'sibling id %d' under NUMA 'cell id %d' "), + sibling_value, sibling_id, cur_cell); + goto cleanup; + } + + /* Assure correct LOCAL_DISTANCE setting if given */ + if (sibling_id =3D=3D cur_cell && + sibling_value !=3D LOCAL_DISTANCE) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid value '%d !LOCAL_DISTANCE' set for " + "'sibling id %d' under NUMA 'cell id %d' "), + sibling_value, sibling_id, cur_cell); + goto cleanup; + } + + /* Assure correct LOCAL_DISTANCE setting if given */ + if (sibling_id !=3D cur_cell && + sibling_value =3D=3D LOCAL_DISTANCE) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid value '%d LOCAL_DISTANCE' set for " + "'sibling id %d' under NUMA 'cell id %d' "), + sibling_value, sibling_id, cur_cell); + goto cleanup; + } + + /* Apply the local / remote distance */ + ldist =3D def->mem_nodes[cur_cell].distances; + if (!ldist) { + if (def->mem_nodes[cur_cell].ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'ndistances' set in NUMA " + "distances for sibling id: '%d'"), + cur_cell); + goto cleanup; + } + + if (VIR_ALLOC_N(ldist, ndistances) < 0) + goto cleanup; + + ldist[cur_cell].value =3D LOCAL_DISTANCE; + ldist[cur_cell].cellid =3D cur_cell; + def->mem_nodes[cur_cell].ndistances =3D ndistances; + } + + ldist[sibling_id].cellid =3D sibling_id; + ldist[sibling_id].value =3D sibling_value; + def->mem_nodes[cur_cell].distances =3D ldist; + + /* Apply symmetry if none given */ + rdist =3D def->mem_nodes[sibling_id].distances; + if (!rdist) { + if (def->mem_nodes[sibling_id].ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'ndistances' set in NUMA " + "distances for sibling id: '%d'"), + sibling_id); + goto cleanup; + } + + if (VIR_ALLOC_N(rdist, ndistances) < 0) + goto cleanup; + + rdist[sibling_id].value =3D LOCAL_DISTANCE; + rdist[sibling_id].cellid =3D sibling_id; + def->mem_nodes[sibling_id].ndistances =3D ndistances; + } + + rdist[cur_cell].cellid =3D cur_cell; + if (!rdist[cur_cell].value) + rdist[cur_cell].value =3D sibling_value; + def->mem_nodes[sibling_id].distances =3D rdist; + } + + ret =3D 0; + + cleanup: + if (ret) { + for (i =3D 0; i < ndistances; i++) + VIR_FREE(def->mem_nodes[i].distances); + } + VIR_FREE(nodes); + VIR_FREE(tmp); + + return ret; +} + int virDomainNumaDefCPUParseXML(virDomainNumaPtr def, xmlXPathContextPtr ctxt) @@ -694,7 +879,7 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, xmlNodePtr oldNode =3D ctxt->node; char *tmp =3D NULL; int n; - size_t i; + size_t i, j; int ret =3D -1; =20 /* check if NUMA definition is present */ @@ -712,7 +897,6 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, def->nmem_nodes =3D n; =20 for (i =3D 0; i < n; i++) { - size_t j; int rc; unsigned int cur_cell =3D i; =20 @@ -788,6 +972,10 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, def->mem_nodes[cur_cell].memAccess =3D rc; VIR_FREE(tmp); } + + /* Parse NUMA distances info */ + if (virDomainNumaDefNodeDistanceParseXML(def, ctxt, cur_cell) < 0) + goto cleanup; } =20 ret =3D 0; @@ -815,6 +1003,8 @@ virDomainNumaDefCPUFormatXML(virBufferPtr buf, virBufferAddLit(buf, "\n"); virBufferAdjustIndent(buf, 2); for (i =3D 0; i < ncells; i++) { + int ndistances; + memAccess =3D virDomainNumaGetNodeMemoryAccessMode(def, i); =20 if (!(cpustr =3D virBitmapFormat(virDomainNumaGetNodeCpumask(def, = i)))) @@ -829,7 +1019,32 @@ virDomainNumaDefCPUFormatXML(virBufferPtr buf, if (memAccess) virBufferAsprintf(buf, " memAccess=3D'%s'", virDomainMemoryAccessTypeToString(memAccess)= ); - virBufferAddLit(buf, "/>\n"); + + ndistances =3D def->mem_nodes[i].ndistances; + if (!ndistances) { + virBufferAddLit(buf, "/>\n"); + } else { + size_t j; + virDomainNumaDistancePtr distances =3D def->mem_nodes[i].dista= nces; + + virBufferAddLit(buf, ">\n"); + virBufferAdjustIndent(buf, 2); + virBufferAddLit(buf, "\n"); + virBufferAdjustIndent(buf, 2); + for (j =3D 0; j < ndistances; j++) { + if (distances[j].value) { + virBufferAddLit(buf, "\n"); + } + } + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + } + VIR_FREE(cpustr); } virBufferAdjustIndent(buf, -2); --=20 2.13.6 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list