From nobody Sun Jul 13 01:58:28 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=intel.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1535373580627497.15347127336884; Mon, 27 Aug 2018 05:39:40 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BE17930832D2; Mon, 27 Aug 2018 12:39:38 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8627F100190E; Mon, 27 Aug 2018 12:39:38 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3F97F181A13C; Mon, 27 Aug 2018 12:39:38 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w7RBO2Mv024658 for ; Mon, 27 Aug 2018 07:24:02 -0400 Received: by smtp.corp.redhat.com (Postfix) id 7832460A9A; Mon, 27 Aug 2018 11:24:02 +0000 (UTC) Received: from mx1.redhat.com (ext-mx15.extmail.prod.ext.phx2.redhat.com [10.5.110.44]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 684F9608FA for ; Mon, 27 Aug 2018 11:24:00 +0000 (UTC) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2B9C83083391 for ; Mon, 27 Aug 2018 11:23:59 +0000 (UTC) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Aug 2018 04:23:56 -0700 Received: from david-unc.bj.intel.com ([10.238.145.63]) by fmsmga001.fm.intel.com with ESMTP; 27 Aug 2018 04:23:47 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,295,1531810800"; d="scan'208";a="84804970" From: Wang Huaqiang To: libvir-list@redhat.com Date: Mon, 27 Aug 2018 19:23:09 +0800 Message-Id: <1535368993-24901-7-git-send-email-huaqiang.wang@intel.com> In-Reply-To: <1535368993-24901-1-git-send-email-huaqiang.wang@intel.com> References: <1535368993-24901-1-git-send-email-huaqiang.wang@intel.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 212 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Mon, 27 Aug 2018 11:23:59 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Mon, 27 Aug 2018 11:23:59 +0000 (UTC) for IP:'134.134.136.20' DOMAIN:'mga02.intel.com' HELO:'mga02.intel.com' FROM:'huaqiang.wang@intel.com' RCPT:'' X-RedHat-Spam-Score: -2.301 (RCVD_IN_DNSWL_MED, SPF_PASS) 134.134.136.20 mga02.intel.com 134.134.136.20 mga02.intel.com X-Scanned-By: MIMEDefang 2.84 on 10.5.110.44 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com X-Mailman-Approved-At: Mon, 27 Aug 2018 08:39:21 -0400 Cc: shaohe.feng@intel.com, Wang Huaqiang , bing.niu@intel.com, jian-feng.ding@intel.com, rui.zang@intel.com Subject: [libvirt] [PATCH 06/10] util: Introduce resctrl monitor for CMT X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Mon, 27 Aug 2018 12:39:39 +0000 (UTC) X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" 'virResctrlAllocMon' denotes a resctrl monitor reporting the resource consumption information. This patch introduced the interfaces for resctrl monitor. Relationship of 'resctrl allocation' and 'resctrl monitor': 1. resctrl monitor monitors resources (cache or memory bandwidth) of particular allocation. 2. resctrl allocation may refer to the 'default' allocation if no dedicated resource 'control' applied to it. The 'default' allocation enjoys remaining resource that not allocated. 3. resctrl monitor belongs to 'default' allocation if no 'cachetune' specified in XML file. 4. one resctrl allocation may have several monitors. It is also permitted that there is no resctrl monitor associated with an allocation. Key data structures: + struct _virResctrlAllocMon { + char *id; + char *path; + }; struct _virResctrlAlloc { virObject parent; @@ -276,6 +289,12 @@ struct _virResctrlAlloc { virResctrlAllocMemBWPtr mem_bw; + virResctrlAllocMonPtr *monitors; + size_t nmonitors; } Signed-off-by: Wang Huaqiang --- src/libvirt_private.syms | 6 + src/util/virresctrl.c | 361 +++++++++++++++++++++++++++++++++++++++++++= +++- src/util/virresctrl.h | 31 ++++ 3 files changed, 394 insertions(+), 4 deletions(-) diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 47ea35f..1439327 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -2645,12 +2645,17 @@ virCacheKernelTypeFromString; virCacheKernelTypeToString; virCacheTypeFromString; virCacheTypeToString; +virResctrlAllocAddMonitorPID; virResctrlAllocAddPID; virResctrlAllocCreate; +virResctrlAllocCreateMonitor; +virResctrlAllocDeleteMonitor; +virResctrlAllocDetermineMonitorPath; virResctrlAllocDeterminePath; virResctrlAllocForeachCache; virResctrlAllocForeachMemory; virResctrlAllocFormat; +virResctrlAllocGetCacheOccupancy; virResctrlAllocGetID; virResctrlAllocGetUnused; virResctrlAllocNew; @@ -2658,6 +2663,7 @@ virResctrlAllocRemove; virResctrlAllocSetCacheSize; virResctrlAllocSetID; virResctrlAllocSetMemoryBandwidth; +virResctrlAllocSetMonitor; virResctrlInfoGetCache; virResctrlInfoNew; =20 diff --git a/src/util/virresctrl.c b/src/util/virresctrl.c index b3bae6e..7215a47 100644 --- a/src/util/virresctrl.c +++ b/src/util/virresctrl.c @@ -257,6 +257,19 @@ struct _virResctrlAllocMemBW { size_t nbandwidths; }; =20 + +typedef struct _virResctrlAllocMon virResctrlAllocMon; +typedef virResctrlAllocMon *virResctrlAllocMonPtr; +/* virResctrlAllocMon denotes a resctrl monitoring group reporting the res= ource + * consumption information for resource of either cache or memory + * bandwidth. */ +struct _virResctrlAllocMon { + /* monitoring group identifier, should be unique in scope of allocatio= n */ + char *id; + /* directory path under /sys/fs/resctrl*/ + char *path; +}; + struct _virResctrlAlloc { virObject parent; =20 @@ -265,11 +278,21 @@ struct _virResctrlAlloc { =20 virResctrlAllocMemBWPtr mem_bw; =20 + /* monintoring groups associated with current resource allocation + * it might report resource consumption information at a finer + * granularity */ + virResctrlAllocMonPtr *monitors; + size_t nmonitors; + /* The identifier (any unique string for now) */ char *id; /* libvirt-generated path in /sys/fs/resctrl for this particular * allocation */ char *path; + /* is this a default resctrl group? + * true : default group, directory path equals '/sys/fs/resctrl' + * false: non-default group */ + bool default_group; }; =20 =20 @@ -315,6 +338,13 @@ virResctrlAllocDispose(void *obj) VIR_FREE(alloc->mem_bw); } =20 + for (i =3D 0; i < alloc->nmonitors; i++) { + virResctrlAllocMonPtr monitor =3D alloc->monitors[i]; + VIR_FREE(monitor->id); + VIR_FREE(monitor->path); + VIR_FREE(monitor); + } + VIR_FREE(alloc->monitors); VIR_FREE(alloc->id); VIR_FREE(alloc->path); VIR_FREE(alloc->levels); @@ -805,6 +835,7 @@ virResctrlInfoGetCache(virResctrlInfoPtr resctrl, memcpy((*controls)[*ncontrols - 1], &i_type->control, sizeof(i_typ= e->control)); } =20 + cachemon->nfeatures =3D 0; cachemon->max_allocation =3D 0; =20 if (resctrl->monitor_info) { @@ -817,7 +848,7 @@ virResctrlInfoGetCache(virResctrlInfoPtr resctrl, if (STREQLEN(info->features[i], "llc_", strlen("llc_"))) { if (virStringListAdd(&cachemon->features, info->features[i]) < 0) - goto error; + goto error; cachemon->nfeatures++; } } @@ -841,10 +872,19 @@ virResctrlInfoGetCache(virResctrlInfoPtr resctrl, virResctrlAllocPtr virResctrlAllocNew(void) { + virResctrlAllocPtr ret =3D NULL; + if (virResctrlInitialize() < 0) return NULL; =20 - return virObjectNew(virResctrlAllocClass); + ret =3D virObjectNew(virResctrlAllocClass); + if (!ret) + return NULL; + + /* By default, a resource group is a default group */ + ret->default_group =3D true; + + return ret; } =20 =20 @@ -861,6 +901,9 @@ virResctrlAllocIsEmpty(virResctrlAllocPtr alloc) if (alloc->mem_bw) return false; =20 + if (alloc->nmonitors) + return false; + for (i =3D 0; i < alloc->nlevels; i++) { virResctrlAllocPerLevelPtr a_level =3D alloc->levels[i]; =20 @@ -2124,10 +2167,18 @@ int virResctrlAllocDeterminePath(virResctrlAllocPtr alloc, const char *machinename) { - return virResctrlDeterminePath(alloc->id, NULL, NULL, - machinename, &alloc->path); + if (alloc->default_group) { + if (VIR_STRDUP(alloc->path, SYSFS_RESCTRL_PATH) < 0) + return -1; + return 0; + } else { + + return virResctrlDeterminePath(alloc->id, NULL, NULL, + machinename, &alloc->path); + } } =20 + static int virResctrlCreateGroup(virResctrlInfoPtr resctrl, char *path) @@ -2169,6 +2220,27 @@ virResctrlCreateGroup(virResctrlInfoPtr resctrl, return ret; } =20 + /* In case of no explicit requirement for allocating cache and memory + * bandwidth, set 'alloc->default' to 'true', then the monitoring + * group will be created under '/sys/fs/resctrl/mon_groups' in later + * invocation of virResctrlAllocCreate. + * Otherwise, set 'alloc->default' to false, create a new directory + * under '/sys/fs/resctrl/'. This is will cost a hardware 'COSID'.*/ +static int +virResctrlAllocCheckDefault(virResctrlAllocPtr alloc) +{ + bool default_group =3D true; + if (!alloc) + return -1; + + if (alloc->nlevels) + default_group =3D false; + if (alloc->mem_bw && alloc->mem_bw->nbandwidths) + default_group =3D false; + + alloc->default_group =3D default_group; + return 0; +} =20 /* This checks if the directory for the alloc exists. If not it tries to = create * it and apply appropriate alloc settings. */ @@ -2185,6 +2257,8 @@ virResctrlAllocCreate(virResctrlInfoPtr resctrl, if (!alloc) return 0; =20 + virResctrlAllocCheckDefault(alloc); + if (virResctrlAllocDeterminePath(alloc, machinename) < 0) return -1; =20 @@ -2275,10 +2349,289 @@ virResctrlAllocRemove(virResctrlAllocPtr alloc) return 0; =20 VIR_DEBUG("Removing resctrl allocation %s", alloc->path); + + while (alloc->nmonitors > 0) { + ret =3D virResctrlAllocDeleteMonitor(alloc, alloc->monitors[0]->id= ); + if (ret < 0) + goto cleanup; + } + if (rmdir(alloc->path) !=3D 0 && errno !=3D ENOENT) { ret =3D -errno; VIR_ERROR(_("Unable to remove %s (%d)"), alloc->path, errno); } =20 + ret =3D 0; + cleanup: + return ret; +} + + +static int +virResctrlAllocGetMonitor(virResctrlAllocPtr alloc, + const char *id, + virResctrlAllocMonPtr *monitor, + size_t *pos) +{ + size_t i =3D 0; + + if (!alloc || !id) + return -1; + + for (i =3D 0; i < alloc->nmonitors; i++) { + if (alloc->monitors[i]->id && + STREQ(id, (alloc->monitors[i])->id)) { + if (monitor) + *monitor =3D alloc->monitors[i]; + if (pos) + *pos =3D i; + return 0; + } + } + + return -1; +} + + +int +virResctrlAllocDetermineMonitorPath(virResctrlAllocPtr alloc, + const char *id, + const char *machinename) +{ + virResctrlAllocMonPtr monitor =3D NULL; + + if (virResctrlAllocGetMonitor(alloc, id, &monitor, NULL) < 0) + return -1; + + + return virResctrlDeterminePath(monitor->id, + alloc->path, + "mon_groups", + machinename, + &monitor->path); +} + + +int +virResctrlAllocAddMonitorPID(virResctrlAllocPtr alloc, + const char *id, + pid_t pid) +{ + virResctrlAllocMonPtr monitor =3D NULL; + + if (virResctrlAllocGetMonitor(alloc, id, &monitor, NULL) < 0) + return -1; + + return virResctrlAddPID(monitor->path, pid); +} + + +int +virResctrlAllocSetMonitor(virResctrlAllocPtr alloc, + const char *id) +{ + virResctrlAllocMonPtr monitor =3D NULL; + + if (!alloc || !id) + return - 1; + + if (virResctrlAllocGetMonitor(alloc, id, &monitor, NULL) < 0) { + if (VIR_ALLOC(monitor) < 0) + return -1; + } + + if (VIR_STRDUP(monitor->id, (char*)id) < 0) + return -1; + + if (VIR_APPEND_ELEMENT(alloc->monitors, alloc->nmonitors, monitor) < 0) + return -1; + + return 0; +} + +int +virResctrlAllocCreateMonitor(virResctrlInfoPtr resctrl, + virResctrlAllocPtr alloc, + const char *machinename, + const char *id) +{ + virResctrlAllocMonPtr monitor =3D NULL; + + if (virResctrlAllocGetMonitor(alloc, id, &monitor, NULL) < 0) + return - 1; + + if (virResctrlAllocDetermineMonitorPath(alloc, id, machinename) < 0) + return -1; + + VIR_DEBUG("Creating resctrl monitor %s", monitor->path); + if (virResctrlCreateGroup(resctrl, monitor->path) < 0) + return -1; + + return 0; +} + + +int +virResctrlAllocDeleteMonitor(virResctrlAllocPtr alloc, + const char *id) +{ + int ret =3D 0; + + virResctrlAllocMonPtr monitor =3D NULL; + size_t pos =3D 0; + + if (virResctrlAllocGetMonitor(alloc, id, &monitor, &pos) < 0) + return -1; + + VIR_DELETE_ELEMENT(alloc->monitors, pos, alloc->nmonitors); + + VIR_DEBUG("Deleting resctrl monitor %s ", monitor->path); + if (rmdir(monitor->path) !=3D 0 && errno !=3D ENOENT) { + ret =3D -errno; + VIR_ERROR(_("Unable to remove %s (%d)"), monitor->path, errno); + } + + VIR_FREE(monitor->id); + VIR_FREE(monitor->path); + VIR_FREE(monitor); + return ret; +} + + +static int +virResctrlAllocGetStatistic(virResctrlAllocPtr alloc, + const char *id, + const char *resfile, + unsigned int *nnodes, + unsigned int **nodeids, + unsigned int **nodevals) +{ + DIR *dirp =3D NULL; + int ret =3D -1; + int rv =3D -1; + struct dirent *ent =3D NULL; + virBuffer buf =3D VIR_BUFFER_INITIALIZER; + char *mondatapath =3D NULL; + size_t ntmpid =3D 0; + size_t ntmpval =3D 0; + virResctrlAllocMonPtr monitor =3D NULL; + + if (!nnodes || !nodeids || !nodevals) + return -1; + + if (virResctrlAllocGetMonitor(alloc, id, &monitor, NULL) < 0) + goto cleanup; + + if (!monitor || !monitor->path) + goto cleanup; + + rv =3D virDirOpenIfExists(&dirp, monitor->path); + if (rv <=3D 0) + goto cleanup; + + *nnodes =3D 0; + + virBufferAsprintf(&buf, "%s/mon_data", monitor->path); + + mondatapath =3D virBufferContentAndReset(&buf); + if (!mondatapath) + goto cleanup; + + if (virDirOpen(&dirp, mondatapath) < 0) + goto cleanup; + + while ((rv =3D virDirRead(dirp, &ent, mondatapath)) > 0) { + char *pstrid =3D NULL; + size_t i =3D 0; + unsigned int len =3D 0; + unsigned int counter =3D 0; + unsigned int cacheid =3D 0; + unsigned int cur_cacheid =3D 0; + unsigned int val =3D 0; + int tmpnodeid =3D 0; + int tmpnodeval =3D 0; + + if (ent->d_type !=3D DT_DIR) + continue; + + /* mon_L3(|CODE|DATA)_xx, xx is cache id */ + if (STRNEQLEN(ent->d_name, "mon_L", 5)) + continue; + + len =3D strlen(ent->d_name); + pstrid =3D ent->d_name; + /* locating the cache id string: 'xx' */ + for (i =3D 0; i < len; i++) { + if (*(pstrid + i) =3D=3D '_') + counter ++; + if (counter =3D=3D 2) + break; + } + i++; + + if (i >=3D len) + goto cleanup; + + if (virStrToLong_uip(pstrid + i, NULL, 0, &cacheid) < 0) { + VIR_DEBUG("Cannot parse id from folder '%s'", ent->d_name); + goto cleanup; + } + + rv =3D virFileReadValueUint(&val, + "%s/%s/%s", + mondatapath, ent->d_name, resfile); + + if (rv =3D=3D -2) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("file %s/%s/%s does not exist"), + mondatapath, ent->d_name, resfile); + goto cleanup; + } else { + if (rv < 0) + goto cleanup; + } + + /* The ultimate caller will be responiblefor free memory of + * 'nodeids' an 'nodevals' */ + if (VIR_APPEND_ELEMENT(*nodeids, ntmpid, cacheid) < 0) + goto cleanup; + if (VIR_APPEND_ELEMENT(*nodevals, ntmpval, val) < 0) + goto cleanup; + + cur_cacheid =3D ntmpval - 1; + /* sort the cache information in caach bank id's ascending order */ + for (i =3D 0; i < cur_cacheid; i++) { + if ((*nodeids)[cur_cacheid] < (*nodeids)[i]) { + tmpnodeid =3D (*nodeids)[cur_cacheid]; + tmpnodeval =3D (*nodevals)[cur_cacheid]; + (*nodeids)[cur_cacheid] =3D (*nodeids)[i]; + (*nodevals)[cur_cacheid] =3D (*nodevals)[i]; + (*nodeids)[i] =3D tmpnodeid; + (*nodevals)[i] =3D tmpnodeval; + } + } + } + + (*nnodes) =3D ntmpval; + ret =3D 0; + cleanup: + VIR_FREE(mondatapath); + VIR_DIR_CLOSE(dirp); + return ret; +} + + +int +virResctrlAllocGetCacheOccupancy(virResctrlAllocPtr alloc, + const char *id, + unsigned int *nbank, + unsigned int **bankids, + unsigned int **bankcaches) +{ + int ret =3D - 1; + + ret =3D virResctrlAllocGetStatistic(alloc, id, "llc_occupancy", + nbank, bankids, bankcaches); + return ret; } diff --git a/src/util/virresctrl.h b/src/util/virresctrl.h index 51bb68b..0f63997 100644 --- a/src/util/virresctrl.h +++ b/src/util/virresctrl.h @@ -160,4 +160,35 @@ virResctrlAllocAddPID(virResctrlAllocPtr alloc, int virResctrlAllocRemove(virResctrlAllocPtr alloc); =20 +int +virResctrlAllocDetermineMonitorPath(virResctrlAllocPtr alloc, + const char *id, + const char *machinename); + +int +virResctrlAllocAddMonitorPID(virResctrlAllocPtr alloc, + const char *id, + pid_t pid); + +int +virResctrlAllocSetMonitor(virResctrlAllocPtr alloc, + const char *id); + +int +virResctrlAllocCreateMonitor(virResctrlInfoPtr resctrl, + virResctrlAllocPtr alloc, + const char *machinename, + const char *id); + +int +virResctrlAllocDeleteMonitor(virResctrlAllocPtr alloc, + const char *id); + +int +virResctrlAllocGetCacheOccupancy(virResctrlAllocPtr alloc, + const char *id, + unsigned int *nbank, + unsigned int **bankids, + unsigned int **bankcaches); + #endif /* __VIR_RESCTRL_H__ */ --=20 2.7.4 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list