From nobody Fri May 16 00:26:03 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1504883577249310.02225963199714; Fri, 8 Sep 2017 08:12:57 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B7EA2356F0; Fri, 8 Sep 2017 15:12:55 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 998BE61289; Fri, 8 Sep 2017 15:12:55 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 643F33FAD1; Fri, 8 Sep 2017 15:12:55 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v88Elbh5031392 for ; Fri, 8 Sep 2017 10:47:37 -0400 Received: by smtp.corp.redhat.com (Postfix) id 2D83860E3B; Fri, 8 Sep 2017 14:47:37 +0000 (UTC) Received: from mx1.redhat.com (ext-mx08.extmail.prod.ext.phx2.redhat.com [10.5.110.32]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 26973619C3 for ; Fri, 8 Sep 2017 14:47:34 +0000 (UTC) Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 42FC0C057FA6 for ; Fri, 8 Sep 2017 14:47:33 +0000 (UTC) Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v88ElWS6012134 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 8 Sep 2017 14:47:32 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v88ElWY0022873 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 8 Sep 2017 14:47:32 GMT Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id v88ElWek017113; Fri, 8 Sep 2017 14:47:32 GMT Received: from nina.dynamic.ziggo.nl (/10.175.252.161) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 08 Sep 2017 07:47:31 -0700 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B7EA2356F0 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com B7EA2356F0 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 42FC0C057FA6 Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=wim.ten.have@oracle.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 42FC0C057FA6 From: Wim Ten Have To: Libvirt Development List Date: Fri, 8 Sep 2017 16:47:15 +0200 Message-Id: <20170908144718.27983-3-wim.ten.have@oracle.com> In-Reply-To: <20170908144718.27983-1-wim.ten.have@oracle.com> References: <20170908144718.27983-1-wim.ten.have@oracle.com> X-Source-IP: userv0022.oracle.com [156.151.31.74] X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 205 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 08 Sep 2017 14:47:33 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 08 Sep 2017 14:47:33 +0000 (UTC) for IP:'156.151.31.81' DOMAIN:'userp1040.oracle.com' HELO:'userp1040.oracle.com' FROM:'wim.ten.have@oracle.com' RCPT:'' X-RedHat-Spam-Score: -105.101 (RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H2, RP_MATCHES_RCVD, SPF_PASS, UNPARSEABLE_RELAY, USER_IN_WHITELIST) 156.151.31.81 userp1040.oracle.com 156.151.31.81 userp1040.oracle.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.32 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: libvir-list@redhat.com Cc: Wim ten Have Subject: [libvirt] [PATCH v4 2/5] numa: describe siblings distances within cells X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 08 Sep 2017 15:12:56 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Wim ten Have Add libvirtd NUMA cell domain administration functionality to describe underlying cell id sibling distances in full fashion when configuring HVM guests. Schema updates are made to docs/schemas/cputypes.rng enforcing domain administration to follow the syntax below the numa cell id and docs/schemas/basictypes.rng to add "numaDistanceValue". A minimum value of 10 representing the LOCAL_DISTANCE as 0-9 are reserved values and can not be used as System Locality Distance Information. A value of 20 represents the default setting of REMOTE_DISTANCE where a maximum value of 255 represents UNREACHABLE. Effectively any cell sibling can be assigned a distance value where practically 'LOCAL_DISTANCE <=3D value <=3D UNREACHABLE'. [below is an example of a 4 node setup] Whenever a sibling id the cell LOCAL_DISTANCE does apply and for any sibling id not being covered a default of REMOTE_DISTANCE is used for internal computations. Signed-off-by: Wim ten Have --- Changes on v1: - Add changes to docs/formatdomain.html.in describing schema update. Changes on v2: - Automatically apply distance symmetry maintaining cell <-> sibling. - Check for maximum '255' on numaDistanceValue. - Automatically complete empty distance ranges. - Check that sibling_id's are in range with cell identifiers. - Allow non-contiguous ranges, starting from any node id. - Respect parameters as ATTRIBUTE_NONNULL fix functions and callers. - Add and apply topology for LOCAL_DISTANCE=3D10 and REMOTE_DISTANCE=3D20. Changes on v3 - Add UNREACHABLE if one locality is unreachable from another. - Add code cleanup aligning function naming in a separated patch. - Add numa related driver code in a separated patch. - Remove from numaDistanceValue schema/basictypes.rng - Correct doc changes. --- docs/formatdomain.html.in | 63 +++++++++++++- docs/schemas/basictypes.rng | 7 ++ docs/schemas/cputypes.rng | 18 ++++ src/conf/numa_conf.c | 200 ++++++++++++++++++++++++++++++++++++++++= +++- 4 files changed, 284 insertions(+), 4 deletions(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 8ca7637..c453d44 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -1529,7 +1529,68 @@

=20

- This guest NUMA specification is currently available only for QEMU/K= VM. + This guest NUMA specification is currently available only for + QEMU/KVM and Xen. Whereas Xen driver also allows for a distinct + description of NUMA arranged sibling cell + distances Since 3.6.0. +

+ +

+ Under NUMA h/w architecture, distinct resources such as memory + create a designated distance between cell and + siblings that now can be described with the help of + distances. A detailed description can be found within + the ACPI (Advanced Configuration and Power Interface Specification) + within the chapter explaining the system's SLIT (System Locality + Distance Information Table). +

+ +
+...
+<cpu>
+  ...
+  <numa>
+    <cell id=3D'0' cpus=3D'0,4-7' memory=3D'512000' unit=3D'KiB'>
+      <distances>
+        <sibling id=3D'0' value=3D'10'/>
+        <sibling id=3D'1' value=3D'21'/>
+        <sibling id=3D'2' value=3D'31'/>
+        <sibling id=3D'3' value=3D'41'/>
+      </distances>
+    </cell>
+    <cell id=3D'1' cpus=3D'1,8-10,12-15' memory=3D'512000' unit=3D'KiB'=
 memAccess=3D'shared'>
+      <distances>
+        <sibling id=3D'0' value=3D'21'/>
+        <sibling id=3D'1' value=3D'10'/>
+        <sibling id=3D'2' value=3D'21'/>
+        <sibling id=3D'3' value=3D'31'/>
+      </distances>
+    </cell>
+    <cell id=3D'2' cpus=3D'2,11' memory=3D'512000' unit=3D'KiB' memAcce=
ss=3D'shared'>
+      <distances>
+        <sibling id=3D'0' value=3D'31'/>
+        <sibling id=3D'1' value=3D'21'/>
+        <sibling id=3D'2' value=3D'10'/>
+        <sibling id=3D'3' value=3D'21'/>
+      </distances>
+    </cell>
+    <cell id=3D'3' cpus=3D'3' memory=3D'512000' unit=3D'KiB'>
+      <distances>
+        <sibling id=3D'0' value=3D'41'/>
+        <sibling id=3D'1' value=3D'31'/>
+        <sibling id=3D'2' value=3D'21'/>
+        <sibling id=3D'3' value=3D'10'/>
+      </distances>
+    </cell>
+  </numa>
+  ...
+</cpu>
+...
+ +

+ Under Xen driver, if no distances are given to describe + the SLIT data between different cells, it will default to a scheme + using 10 for local and 20 for remote distances.

=20

Events configuration

diff --git a/docs/schemas/basictypes.rng b/docs/schemas/basictypes.rng index 1ea667c..1a18cd3 100644 --- a/docs/schemas/basictypes.rng +++ b/docs/schemas/basictypes.rng @@ -77,6 +77,13 @@ =20 + + + 10 + 255 + + + diff --git a/docs/schemas/cputypes.rng b/docs/schemas/cputypes.rng index 3eef16a..c45b6df 100644 --- a/docs/schemas/cputypes.rng +++ b/docs/schemas/cputypes.rng @@ -129,6 +129,24 @@ + + + + + + + + + + + + + + + + + + =20 diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c index b71dc01..5db4311 100644 --- a/src/conf/numa_conf.c +++ b/src/conf/numa_conf.c @@ -29,6 +29,15 @@ #include "virnuma.h" #include "virstring.h" =20 +/* + * Distance definitions defined Conform ACPI 2.0 SLIT. + * See include/linux/topology.h + */ +#define LOCAL_DISTANCE 10 +#define REMOTE_DISTANCE 20 +/* SLIT entry value is a one-byte unsigned integer. */ +#define UNREACHABLE 255 + #define VIR_FROM_THIS VIR_FROM_DOMAIN =20 VIR_ENUM_IMPL(virDomainNumatuneMemMode, @@ -48,6 +57,8 @@ VIR_ENUM_IMPL(virDomainMemoryAccess, VIR_DOMAIN_MEMORY_AC= CESS_LAST, "shared", "private") =20 +typedef struct _virDomainNumaDistance virDomainNumaDistance; +typedef virDomainNumaDistance *virDomainNumaDistancePtr; =20 typedef struct _virDomainNumaNode virDomainNumaNode; typedef virDomainNumaNode *virDomainNumaNodePtr; @@ -66,6 +77,12 @@ struct _virDomainNuma { virBitmapPtr nodeset; /* host memory nodes where this guest node= resides */ virDomainNumatuneMemMode mode; /* memory mode selection */ virDomainMemoryAccess memAccess; /* shared memory access configura= tion */ + + struct _virDomainNumaDistance { + unsigned int value; /* locality value for node i->j or j->i */ + unsigned int cellid; + } *distances; /* remote node distances */ + size_t ndistances; } *mem_nodes; /* guest node configuration */ size_t nmem_nodes; =20 @@ -686,6 +703,153 @@ virDomainNumatuneNodesetIsAvailable(virDomainNumaPtr = numatune, } =20 =20 +static int +virDomainNumaDefNodeDistanceParseXML(virDomainNumaPtr def, + xmlXPathContextPtr ctxt, + unsigned int cur_cell) +{ + int ret =3D -1; + int sibling; + char *tmp =3D NULL; + xmlNodePtr *nodes =3D NULL; + size_t i, ndistances =3D def->nmem_nodes; + + if (!ndistances) + return 0; + + /* check if NUMA distances definition is present */ + if (!virXPathNode("./distances[1]", ctxt)) + return 0; + + if ((sibling =3D virXPathNodeSet("./distances[1]/sibling", ctxt, &node= s)) <=3D 0) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("NUMA distances defined without siblings")); + goto cleanup; + } + + for (i =3D 0; i < sibling; i++) { + virDomainNumaDistancePtr ldist, rdist; + unsigned int sibling_id, sibling_value; + + /* siblings are in order of parsing or explicitly numbered */ + if (!(tmp =3D virXMLPropString(nodes[i], "id"))) { + virReportError(VIR_ERR_XML_ERROR, + _("Missing 'id' attribute in NUMA " + "distances under 'cell id %d'"), + cur_cell); + goto cleanup; + } + + /* The "id" needs to be applicable */ + if (virStrToLong_uip(tmp, NULL, 10, &sibling_id) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'id' attribute in NUMA " + "distances for sibling: '%s'"), + tmp); + goto cleanup; + } + VIR_FREE(tmp); + + /* The "id" needs to be within numa/cell range */ + if (sibling_id >=3D ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("There is no cell administrated matching " + "'sibling_id %d' under NUMA 'cell id %d' "), + sibling_id, cur_cell); + goto cleanup; + } + + /* We need a locality value. Check and correct + * distance to local and distance to remote node. + */ + if (!(tmp =3D virXMLPropString(nodes[i], "value"))) { + virReportError(VIR_ERR_XML_ERROR, + _("Missing 'value' attribute in NUMA distances " + "under 'cell id %d' for 'sibling id %d'"), + cur_cell, sibling_id); + goto cleanup; + } + + /* The "value" needs to be applicable */ + if (virStrToLong_uip(tmp, NULL, 10, &sibling_value) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'value' attribute in NUMA " + "distances for value: '%s'"), + tmp); + goto cleanup; + } + VIR_FREE(tmp); + + /* LOCAL_DISTANCE <=3D "value" <=3D UNREACHABLE */ + if (sibling_value < LOCAL_DISTANCE || + sibling_value > UNREACHABLE) { + virReportError(VIR_ERR_XML_ERROR, + _("Out of range value '%d' set for " + "'sibling id %d' under NUMA 'cell id %d' "), + sibling_value, sibling_id, cur_cell); + goto cleanup; + } + + ldist =3D def->mem_nodes[cur_cell].distances; + if (!ldist) { + if (def->mem_nodes[cur_cell].ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'ndistances' set in NUMA " + "distances for sibling id: '%d'"), + cur_cell); + goto cleanup; + } + + if (VIR_ALLOC_N(ldist, ndistances) < 0) + goto cleanup; + + if (!ldist[cur_cell].value) + ldist[cur_cell].value =3D LOCAL_DISTANCE; + ldist[cur_cell].cellid =3D cur_cell; + def->mem_nodes[cur_cell].ndistances =3D ndistances; + } + + ldist[sibling_id].cellid =3D sibling_id; + ldist[sibling_id].value =3D sibling_value; + def->mem_nodes[cur_cell].distances =3D ldist; + + rdist =3D def->mem_nodes[sibling_id].distances; + if (!rdist) { + if (def->mem_nodes[sibling_id].ndistances) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid 'ndistances' set in NUMA " + "distances for sibling id: '%d'"), + sibling_id); + goto cleanup; + } + + if (VIR_ALLOC_N(rdist, ndistances) < 0) + goto cleanup; + + if (!rdist[sibling_id].value) + rdist[sibling_id].value =3D LOCAL_DISTANCE; + rdist[sibling_id].cellid =3D sibling_id; + def->mem_nodes[sibling_id].ndistances =3D ndistances; + } + + rdist[cur_cell].cellid =3D cur_cell; + rdist[cur_cell].value =3D sibling_value; + def->mem_nodes[sibling_id].distances =3D rdist; + } + + ret =3D 0; + + cleanup: + if (ret) { + for (i =3D 0; i < ndistances; i++) + VIR_FREE(def->mem_nodes[i].distances); + } + VIR_FREE(nodes); + VIR_FREE(tmp); + + return ret; +} + int virDomainNumaDefCPUParseXML(virDomainNumaPtr def, xmlXPathContextPtr ctxt) @@ -694,7 +858,7 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, xmlNodePtr oldNode =3D ctxt->node; char *tmp =3D NULL; int n; - size_t i; + size_t i, j; int ret =3D -1; =20 /* check if NUMA definition is present */ @@ -712,7 +876,6 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, def->nmem_nodes =3D n; =20 for (i =3D 0; i < n; i++) { - size_t j; int rc; unsigned int cur_cell =3D i; =20 @@ -788,6 +951,10 @@ virDomainNumaDefCPUParseXML(virDomainNumaPtr def, def->mem_nodes[cur_cell].memAccess =3D rc; VIR_FREE(tmp); } + + /* Parse NUMA distances info */ + if (virDomainNumaDefNodeDistanceParseXML(def, ctxt, cur_cell) < 0) + goto cleanup; } =20 ret =3D 0; @@ -815,6 +982,8 @@ virDomainNumaDefCPUFormatXML(virBufferPtr buf, virBufferAddLit(buf, "\n"); virBufferAdjustIndent(buf, 2); for (i =3D 0; i < ncells; i++) { + int ndistances; + memAccess =3D virDomainNumaGetNodeMemoryAccessMode(def, i); =20 if (!(cpustr =3D virBitmapFormat(virDomainNumaGetNodeCpumask(def, = i)))) @@ -829,7 +998,32 @@ virDomainNumaDefCPUFormatXML(virBufferPtr buf, if (memAccess) virBufferAsprintf(buf, " memAccess=3D'%s'", virDomainMemoryAccessTypeToString(memAccess)= ); - virBufferAddLit(buf, "/>\n"); + + ndistances =3D def->mem_nodes[i].ndistances; + if (!ndistances) { + virBufferAddLit(buf, "/>\n"); + } else { + size_t j; + virDomainNumaDistancePtr distances =3D def->mem_nodes[i].dista= nces; + + virBufferAddLit(buf, ">\n"); + virBufferAdjustIndent(buf, 2); + virBufferAddLit(buf, "\n"); + virBufferAdjustIndent(buf, 2); + for (j =3D 0; j < ndistances; j++) { + if (distances[j].value) { + virBufferAddLit(buf, "\n"); + } + } + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + } + VIR_FREE(cpustr); } virBufferAdjustIndent(buf, -2); --=20 2.9.5 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list