From nobody Thu May 15 08:59:01 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 150963766968292.39108325928714; Thu, 2 Nov 2017 08:47:49 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 485E980469; Thu, 2 Nov 2017 15:47:48 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E0E1960CA0; Thu, 2 Nov 2017 15:47:47 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 659631800BCF; Thu, 2 Nov 2017 15:47:47 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id vA2FlkLO025010 for ; Thu, 2 Nov 2017 11:47:46 -0400 Received: by smtp.corp.redhat.com (Postfix) id 727A55EE02; Thu, 2 Nov 2017 15:47:46 +0000 (UTC) Received: from mx1.redhat.com (ext-mx03.extmail.prod.ext.phx2.redhat.com [10.5.110.27]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6D48A5D758 for ; Thu, 2 Nov 2017 15:47:44 +0000 (UTC) Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 40A017E44E for ; Thu, 2 Nov 2017 15:47:43 +0000 (UTC) Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA2Flgq2002233 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 15:47:42 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id vA2Flg6u029041 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Nov 2017 15:47:42 GMT Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id vA2Flg24007345; Thu, 2 Nov 2017 15:47:42 GMT Received: from nina.dynamic.ziggo.nl (/10.175.172.23) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 02 Nov 2017 08:47:41 -0700 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 485E980469 Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 485E980469 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 40A017E44E Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=wim.ten.have@oracle.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 40A017E44E From: Wim Ten Have To: Libvirt Development List Date: Thu, 2 Nov 2017 16:47:20 +0100 Message-Id: <20171102154723.26914-2-wim.ten.have@oracle.com> In-Reply-To: <20171102154723.26914-1-wim.ten.have@oracle.com> References: <20171102154723.26914-1-wim.ten.have@oracle.com> X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 205 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 02 Nov 2017 15:47:43 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 02 Nov 2017 15:47:43 +0000 (UTC) for IP:'141.146.126.69' DOMAIN:'aserp1040.oracle.com' HELO:'aserp1040.oracle.com' FROM:'wim.ten.have@oracle.com' RCPT:'' X-RedHat-Spam-Score: -102.321 (RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, RP_MATCHES_RCVD, SPF_PASS, UNPARSEABLE_RELAY, USER_IN_WHITELIST) 141.146.126.69 aserp1040.oracle.com 141.146.126.69 aserp1040.oracle.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.27 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Cc: Wim ten Have Subject: [libvirt] [PATCH v6 1/4] numa: describe siblings distances within cells X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Thu, 02 Nov 2017 15:47:48 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Wim ten Have Add support for describing NUMA distances in a domain's XML description. Below is an example of a 4 node setup: A defines a NUMA node. describes the NUMA distance from the to the other NUMA nodes (the s). For example, in above XML description, the distance between NUMA node0 and NUMA node2 is 31. Valid distance value are '10 <=3D value <=3D 255'. A distance value of 10 represents the distance to the node itself. A distance value of 20 represents the default value for remote nodes but other values are possible depending on the physical topology of the system. When distances are not fully described, any missing sibling distance values will default to 10 for local nodes and 20 for remote nodes. If distance is given for A -> B, then we default B -> A to the same value instead of 20. Signed-off-by: Wim ten Have Reviewed-by: Daniel P. Berrange --- Changes on v1: - Add changes to docs/formatdomain.html.in describing schema update. Changes on v2: - Automatically apply distance symmetry maintaining cell <-> sibling. - Check for maximum '255' on numaDistanceValue. - Automatically complete empty distance ranges. - Check that sibling_id's are in range with cell identifiers. - Allow non-contiguous ranges, starting from any node id. - Respect parameters as ATTRIBUTE_NONNULL fix functions and callers. - Add and apply topology for LOCAL_DISTANCE=3D10 and REMOTE_DISTANCE=3D20. Changes on v3: - Add UNREACHABLE if one locality is unreachable from another. - Add code cleanup aligning function naming in a separated patch. - Add numa related driver code in a separated patch. - Remove from numaDistanceValue schema/basictypes.rng - Correct doc changes. Changes on v4: - Fix symmetry error under virDomainNumaDefNodeDistanceParseXML() Changes on v5: - Apply doc suggestions. - Apply virReportError() message suggestions. - Remove redundant sanity checks from virDomainNumaDefNodeDistanceParseXML(= ). --- docs/formatdomain.html.in | 63 ++++++++++++++- docs/schemas/basictypes.rng | 7 ++ docs/schemas/cputypes.rng | 18 +++++ src/conf/numa_conf.c | 191 ++++++++++++++++++++++++++++++++++++++++= +++- 4 files changed, 275 insertions(+), 4 deletions(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 4f28dce35..df2f17f5d 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -1529,7 +1529,68 @@

=20

- This guest NUMA specification is currently available only for QEMU/K= VM. + This guest NUMA specification is currently available only for + QEMU/KVM and Xen. Whereas Xen driver also allows for a distinct + description of NUMA arranged sibling cell + distances Since 3.9.0. +

+ +

+ Under NUMA hardware architecture, distinct resources such as memory + create a designated distance between cell and + siblings that now can be described with the help of + distances. A detailed description can be found within + the ACPI (Advanced Configuration and Power Interface Specification) + within the chapter explaining the system's SLIT (System Locality + Distance Information Table). +

+ +
+...
+<cpu>
+  ...
+  <numa>
+    <cell id=3D'0' cpus=3D'0,4-7' memory=3D'512000' unit=3D'KiB'>
+      <distances>
+        <sibling id=3D'0' value=3D'10'/>
+        <sibling id=3D'1' value=3D'21'/>
+        <sibling id=3D'2' value=3D'31'/>
+        <sibling id=3D'3' value=3D'41'/>
+      </distances>
+    </cell>
+    <cell id=3D'1' cpus=3D'1,8-10,12-15' memory=3D'512000' unit=3D'KiB'=
 memAccess=3D'shared'>
+      <distances>
+        <sibling id=3D'0' value=3D'21'/>
+        <sibling id=3D'1' value=3D'10'/>
+        <sibling id=3D'2' value=3D'21'/>
+        <sibling id=3D'3' value=3D'31'/>
+      </distances>
+    </cell>
+    <cell id=3D'2' cpus=3D'2,11' memory=3D'512000' unit=3D'KiB' memAcce=
ss=3D'shared'>
+      <distances>
+        <sibling id=3D'0' value=3D'31'/>
+        <sibling id=3D'1' value=3D'21'/>
+        <sibling id=3D'2' value=3D'10'/>
+        <sibling id=3D'3' value=3D'21'/>
+      </distances>
+    </cell>
+    <cell id=3D'3' cpus=3D'3' memory=3D'512000' unit=3D'KiB'>
+      <distances>
+        <sibling id=3D'0' value=3D'41'/>
+        <sibling id=3D'1' value=3D'31'/>
+        <sibling id=3D'2' value=3D'21'/>
+        <sibling id=3D'3' value=3D'10'/>
+      </distances>
+    </cell>
+  </numa>
+  ...
+</cpu>
+...
+ +

+ Under Xen driver, if no distances are given to describe + the SLIT data between different cells, it will default to a scheme + using 10 for local and 20 for remote distances.

=20

Events configuration

diff --git a/docs/schemas/basictypes.rng b/docs/schemas/basictypes.rng index 1ea667cdf..1a18cd31b 100644 --- a/docs/schemas/basictypes.rng +++ b/docs/schemas/basictypes.rng @@ -77,6 +77,13 @@ =20 + + + 10 + 255 + + + diff --git a/docs/schemas/cputypes.rng b/docs/schemas/cputypes.rng index 3eef16abc..c45b6dfb2 100644 --- a/docs/schemas/cputypes.rng +++ b/docs/schemas/cputypes.rng @@ -129,6 +129,24 @@ + + + + + + + + + + + + + + + + + + =20 diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c index b71dc012c..5fbcc7204 100644 --- a/src/conf/numa_conf.c +++ b/src/conf/numa_conf.c @@ -29,6 +29,15 @@ #include "virnuma.h" #include "virstring.h" =20 +/* + * Distance definitions defined Conform ACPI 2.0 SLIT. + * See include/linux/topology.h + */ +#define LOCAL_DISTANCE 10 +#define REMOTE_DISTANCE 20 +/* SLIT entry value is a one-byte unsigned integer. */ +#define UNREACHABLE 255 + #define VIR_FROM_THIS VIR_FROM_DOMAIN =20 VIR_ENUM_IMPL(virDomainNumatuneMemMode, @@ -48,6 +57,8 @@ VIR_ENUM_IMPL(virDomainMemoryAccess, VIR_DOMAIN_MEMORY_AC= CESS_LAST, "shared", "private") =20 +typedef struct _virDomainNumaDistance virDomainNumaDistance; +typedef virDomainNumaDistance *virDomainNumaDistancePtr; =20 typedef struct _virDomainNumaNode virDomainNumaNode; typedef virDomainNumaNode *virDomainNumaNodePtr; @@ -66,6 +77,12 @@ struct _virDomainNuma { virBitmapPtr nodeset; /* host memory nodes where this guest node= resides */ virDomainNumatuneMemMode mode; /* memory mode selection */ virDomainMemoryAccess memAccess; /* shared memory access configura= tion */ + + struct _virDomainNumaDistance { + unsigned int value; /* locality value for node i->j or j->i */ + unsigned int cellid; + } *distances; /* remote node distances */ + size_t ndistances; } *mem_nodes; /* guest node configuration */ size_t nmem_nodes; =20 @@ -686,6 +703,144 @@ virDomainNumatuneNodesetIsAvailable(virDomainNumaPtr = numatune, } =20 =20 +static int +virDomainNumaDefNodeDistanceParseXML(virDomainNumaPtr def, + xmlXPathContextPtr ctxt, + unsigned int cur_cell) +{ + int ret =3D -1; + int sibling; + char *tmp =3D NULL; + xmlNodePtr *nodes =3D NULL; + size_t i, ndistances =3D def->nmem_nodes; + + if (!ndistances) + return 0; + + /* check if NUMA distances definition is present */ + if (!virXPathNode("./distances[1]", ctxt)) + return 0; + + if ((sibling =3D virXPathNodeSet("./distances[1]/sibling", ctxt, &node= s)) <=3D 0) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("NUMA distances defined without siblings")); + goto cleanup; + } + + for (i =3D 0; i < sibling; i++) { + virDomainNumaDistancePtr ldist, rdist; + unsigned int sibling_id, sibling_value; + + /* siblings are in order of parsing or explicitly numbered */ + if (!(tmp =3D virXMLPropString(nodes[i], "id"))) { + virReportError(VIR_ERR_XML_ERROR, + _("Missing 'id' attribute in NUMA " + "distances under 'cell id %d'"), + cur_cell); + goto cleanup; + } + + /* The "id" needs to be applicable */ + if (virStrToLong_uip(tmp, NULL, 10, &sibling_id) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'id' attribute in NUMA " + "distances for sibling: '%s'"), + tmp); + goto cleanup; + } + VIR_FREE(tmp); + + /* The "id" needs to be within numa/cell range */ + if (sibling_id >=3D ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("'sibling_id %d' does not refer to a " + "valid cell within NUMA 'cell id %d'"), + sibling_id, cur_cell); + goto cleanup; + } + + /* We need a locality value. Check and correct + * distance to local and distance to remote node. + */ + if (!(tmp =3D virXMLPropString(nodes[i], "value"))) { + virReportError(VIR_ERR_XML_ERROR, + _("Missing 'value' attribute in NUMA distances " + "under 'cell id %d' for 'sibling id %d'"), + cur_cell, sibling_id); + goto cleanup; + } + + /* The "value" needs to be applicable */ + if (virStrToLong_uip(tmp, NULL, 10, &sibling_value) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("'value %s' is invalid for " + "'sibling id %d' under NUMA 'cell id %d'"), + tmp, sibling_id, cur_cell); + goto cleanup; + } + VIR_FREE(tmp); + + /* Assure LOCAL_DISTANCE <=3D "value" <=3D UNREACHABLE + * and correct LOCAL_DISTANCE setting if such applies. + */ + if ((sibling_value < LOCAL_DISTANCE || + sibling_value > UNREACHABLE) || + (sibling_id =3D=3D cur_cell && + sibling_value !=3D LOCAL_DISTANCE) || + (sibling_id !=3D cur_cell && + sibling_value =3D=3D LOCAL_DISTANCE)) { + virReportError(VIR_ERR_XML_ERROR, + _("'value %d' is invalid for " + "'sibling id %d' under NUMA 'cell id %d'"), + sibling_value, sibling_id, cur_cell); + goto cleanup; + } + + /* Apply the local / remote distance */ + ldist =3D def->mem_nodes[cur_cell].distances; + if (!ldist) { + if (VIR_ALLOC_N(ldist, ndistances) < 0) + goto cleanup; + + ldist[cur_cell].value =3D LOCAL_DISTANCE; + ldist[cur_cell].cellid =3D cur_cell; + def->mem_nodes[cur_cell].ndistances =3D ndistances; + } + + ldist[sibling_id].cellid =3D sibling_id; + ldist[sibling_id].value =3D sibling_value; + def->mem_nodes[cur_cell].distances =3D ldist; + + /* Apply symmetry if none given */ + rdist =3D def->mem_nodes[sibling_id].distances; + if (!rdist) { + if (VIR_ALLOC_N(rdist, ndistances) < 0) + goto cleanup; + + rdist[sibling_id].value =3D LOCAL_DISTANCE; + rdist[sibling_id].cellid =3D sibling_id; + def->mem_nodes[sibling_id].ndistances =3D ndistances; + } + + rdist[cur_cell].cellid =3D cur_cell; + if (!rdist[cur_cell].value) + rdist[cur_cell].value =3D sibling_value; + def->mem_nodes[sibling_id].distances =3D rdist; + } + + ret =3D 0; + + cleanup: + if (ret) { + for (i =3D 0; i < ndistances; i++) + VIR_FREE(def->mem_nodes[i].distances); + } + VIR_FREE(nodes); + VIR_FREE(tmp); + + return ret; +} + int virDomainNumaDefCPUParseXML(virDomainNumaPtr def, xmlXPathContextPtr ctxt) @@ -694,7 +849,7 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, xmlNodePtr oldNode =3D ctxt->node; char *tmp =3D NULL; int n; - size_t i; + size_t i, j; int ret =3D -1; =20 /* check if NUMA definition is present */ @@ -712,7 +867,6 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, def->nmem_nodes =3D n; =20 for (i =3D 0; i < n; i++) { - size_t j; int rc; unsigned int cur_cell =3D i; =20 @@ -788,6 +942,10 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, def->mem_nodes[cur_cell].memAccess =3D rc; VIR_FREE(tmp); } + + /* Parse NUMA distances info */ + if (virDomainNumaDefNodeDistanceParseXML(def, ctxt, cur_cell) < 0) + goto cleanup; } =20 ret =3D 0; @@ -815,6 +973,8 @@ virDomainNumaDefCPUFormatXML(virBufferPtr buf, virBufferAddLit(buf, "\n"); virBufferAdjustIndent(buf, 2); for (i =3D 0; i < ncells; i++) { + int ndistances; + memAccess =3D virDomainNumaGetNodeMemoryAccessMode(def, i); =20 if (!(cpustr =3D virBitmapFormat(virDomainNumaGetNodeCpumask(def, = i)))) @@ -829,7 +989,32 @@ virDomainNumaDefCPUFormatXML(virBufferPtr buf, if (memAccess) virBufferAsprintf(buf, " memAccess=3D'%s'", virDomainMemoryAccessTypeToString(memAccess)= ); - virBufferAddLit(buf, "/>\n"); + + ndistances =3D def->mem_nodes[i].ndistances; + if (!ndistances) { + virBufferAddLit(buf, "/>\n"); + } else { + size_t j; + virDomainNumaDistancePtr distances =3D def->mem_nodes[i].dista= nces; + + virBufferAddLit(buf, ">\n"); + virBufferAdjustIndent(buf, 2); + virBufferAddLit(buf, "\n"); + virBufferAdjustIndent(buf, 2); + for (j =3D 0; j < ndistances; j++) { + if (distances[j].value) { + virBufferAddLit(buf, "\n"); + } + } + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + } + VIR_FREE(cpustr); } virBufferAdjustIndent(buf, -2); --=20 2.13.6 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list