From nobody Thu May 15 01:41:31 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1517821858140935.4092911827397; Mon, 5 Feb 2018 01:10:58 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AAD5968563; Mon, 5 Feb 2018 09:10:56 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 801F2605FB; Mon, 5 Feb 2018 09:10:56 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3A98318033DA; Mon, 5 Feb 2018 09:10:56 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w1598U0e000821 for ; Mon, 5 Feb 2018 04:08:30 -0500 Received: by smtp.corp.redhat.com (Postfix) id 71A1D5D6A2; Mon, 5 Feb 2018 09:08:30 +0000 (UTC) Received: from mx1.redhat.com (ext-mx08.extmail.prod.ext.phx2.redhat.com [10.5.110.32]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 66CEB5EDEA for ; Mon, 5 Feb 2018 09:08:28 +0000 (UTC) Received: from prv3-mh.provo.novell.com (prv3-mh.provo.novell.com [137.65.250.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5689FC0587D1 for ; Mon, 5 Feb 2018 09:08:25 +0000 (UTC) Received: from localhost.localdomain (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (NOT encrypted); Mon, 05 Feb 2018 02:08:14 -0700 From: river To: libvir-list@redhat.com Date: Mon, 5 Feb 2018 17:07:54 +0800 Message-Id: <1517821676-14399-2-git-send-email-lfu@suse.com> In-Reply-To: <1517821676-14399-1-git-send-email-lfu@suse.com> References: <1517821676-14399-1-git-send-email-lfu@suse.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 207 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) for IP:'137.65.250.26' DOMAIN:'prv3-mh.provo.novell.com' HELO:'prv3-mh.provo.novell.com' FROM:'lfu@suse.com' RCPT:'' X-RedHat-Spam-Score: -2.301 (RCVD_IN_DNSWL_MED, SPF_PASS) 137.65.250.26 prv3-mh.provo.novell.com 137.65.250.26 prv3-mh.provo.novell.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.32 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 1/3] Add a plugin dlm for lock manager X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 05 Feb 2018 09:10:57 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" dlm is implemented by linux kernel, provides userspace API by 'libdlm' to lock/unlock resource, cooperated with cluster communication software, such as corosync, could satisfy the demands of libvirt lock manager. Signed-off-by: river --- configure.ac | 6 + m4/virt-cpg.m4 | 37 ++ m4/virt-dlm.m4 | 36 ++ src/Makefile.am | 15 + src/locking/lock_driver_dlm.c | 1056 +++++++++++++++++++++++++++++++++++++= ++++ src/util/virlist.h | 110 +++++ 6 files changed, 1260 insertions(+) create mode 100644 m4/virt-cpg.m4 create mode 100644 m4/virt-dlm.m4 create mode 100644 src/locking/lock_driver_dlm.c create mode 100644 src/util/virlist.h diff --git a/configure.ac b/configure.ac index 4cccf7f4d..4ad90470d 100644 --- a/configure.ac +++ b/configure.ac @@ -265,6 +265,8 @@ LIBVIRT_ARG_PM_UTILS LIBVIRT_ARG_POLKIT LIBVIRT_ARG_READLINE LIBVIRT_ARG_SANLOCK +LIBVIRT_ARG_CPG +LIBVIRT_ARG_DLM LIBVIRT_ARG_SASL LIBVIRT_ARG_SELINUX LIBVIRT_ARG_SSH2 @@ -307,6 +309,8 @@ LIBVIRT_CHECK_POLKIT LIBVIRT_CHECK_PTHREAD LIBVIRT_CHECK_READLINE LIBVIRT_CHECK_SANLOCK +LIBVIRT_CHECK_CPG +LIBVIRT_CHECK_DLM LIBVIRT_CHECK_SASL LIBVIRT_CHECK_SELINUX LIBVIRT_CHECK_SSH2 @@ -1005,6 +1009,8 @@ LIBVIRT_RESULT_POLKIT LIBVIRT_RESULT_RBD LIBVIRT_RESULT_READLINE LIBVIRT_RESULT_SANLOCK +LIBVIRT_RESULT_CPG +LIBVIRT_RESULT_DLM LIBVIRT_RESULT_SASL LIBVIRT_RESULT_SELINUX LIBVIRT_RESULT_SSH2 diff --git a/m4/virt-cpg.m4 b/m4/virt-cpg.m4 new file mode 100644 index 000000000..27acda665 --- /dev/null +++ b/m4/virt-cpg.m4 @@ -0,0 +1,37 @@ +dnl The libcpg.so library +dnl +dnl Copyright (C) 2018 SUSE LINUX Products, Beijing, China. +dnl +dnl This library is free software; you can redistribute it and/or +dnl modify it under the terms of the GNU Lesser General Public +dnl License as published by the Free Software Foundation; either +dnl version 2.1 of the License, or (at your option) any later version. +dnl +dnl This library is distributed in the hope that it will be useful, +dnl but WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl Lesser General Public License for more details. +dnl +dnl You should have received a copy of the GNU Lesser General Public +dnl License along with this library. If not, see +dnl . +dnl + +AC_DEFUN([LIBVIRT_ARG_CPG],[ + LIBVIRT_ARG_WITH_FEATURE([CPG], [cluster engine CPG library], [check]) +]) + +AC_DEFUN([LIBVIRT_CHECK_CPG],[ + dnl in some distribution, Version is `UNKNOW` in libcpg.pc + if test "x$with_cpg" !=3D "xno"; then + PKG_CHECK_MODULES([CPG], [libcpg], [ + with_cpg=3Dyes + ],[ + with_cpg=3Dno + ]) + fi +]) + +AC_DEFUN([LIBVIRT_RESULT_CPG],[ + LIBVIRT_RESULT_LIB([CPG]) +]) diff --git a/m4/virt-dlm.m4 b/m4/virt-dlm.m4 new file mode 100644 index 000000000..dc5f5f152 --- /dev/null +++ b/m4/virt-dlm.m4 @@ -0,0 +1,36 @@ +dnl The libdlm.so library +dnl +dnl Copyright (C) 2018 SUSE LINUX Products, Beijing, China. +dnl +dnl This library is free software; you can redistribute it and/or +dnl modify it under the terms of the GNU Lesser General Public +dnl License as published by the Free Software Foundation; either +dnl version 2.1 of the License, or (at your option) any later version. +dnl +dnl This library is distributed in the hope that it will be useful, +dnl but WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl Lesser General Public License for more details. +dnl +dnl You should have received a copy of the GNU Lesser General Public +dnl License along with this library. If not, see +dnl . +dnl + +AC_DEFUN([LIBVIRT_ARG_DLM],[ + LIBVIRT_ARG_WITH_FEATURE([DLM], [Distributed Lock Manager library], [che= ck]) +]) + +AC_DEFUN([LIBVIRT_CHECK_DLM],[ + AC_REQUIRE([LIBVIRT_CHECK_CPG]) + LIBVIRT_CHECK_PKG([DLM], [libdlm], [4.0.0]) + + if test "x$with_dlm" =3D=3D "xyes" && test "x$with_cpg" !=3D "xyes"; then + AC_MSG_ERROR([You must install libcpg to build dlm lock]) + fi +]) + +AC_DEFUN([LIBVIRT_RESULT_DLM],[ + AC_REQUIRE([LIBVIRT_RESULT_CPG]) + LIBVIRT_RESULT_LIB([DLM]) +]) diff --git a/src/Makefile.am b/src/Makefile.am index 79adc9ba5..8742921fa 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -305,6 +305,10 @@ DRIVER_SOURCES =3D \ logging/log_manager.c logging/log_manager.h \ $(NULL) =20 +LOCK_DRIVER_DLM_SOURCES =3D \ + locking/lock_driver_dlm.c \ + $(NULL) + LOCK_DRIVER_SANLOCK_SOURCES =3D \ locking/lock_driver_sanlock.c =20 @@ -2879,6 +2883,17 @@ virtlogd.socket: logging/virtlogd.socket.in $(top_bu= ilddir)/config.status < $< > $@-t && \ mv $@-t $@ =20 +if WITH_DLM +lockdriver_LTLIBRARIES +=3D dlm.la +dlm_la_SOURCES =3D $(LOCK_DRIVER_DLM_SOURCES) +dlm_la_CFLAGS =3D -I$(srcdir)/conf $(AM_CFLAGS) +dlm_la_LDFLAGS =3D -module -avoid-version $(AM_LDFLAGS) +dlm_la_LIBADD =3D ../gnulib/lib/libgnu.la \ + $(CPG_LIBS) \ + $(DLM_LIBS) +else ! WITH_DLM +EXTRA_DIST +=3D $(LOCK_DRIVER_DLM_SOURCES) +endif =20 if WITH_SANLOCK lockdriver_LTLIBRARIES +=3D sanlock.la diff --git a/src/locking/lock_driver_dlm.c b/src/locking/lock_driver_dlm.c new file mode 100644 index 000000000..e197c0bdf --- /dev/null +++ b/src/locking/lock_driver_dlm.c @@ -0,0 +1,1056 @@ +/* + * lock_driver_dlm.c: a lock driver for dlm + * + * Copyright (C) 2018 SUSE LINUX Products, Beijing, China. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + * + */ + +#include + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "lock_driver.h" +#include "viralloc.h" +#include "virconf.h" +#include "vircrypto.h" +#include "virerror.h" +#include "virfile.h" +#include "virlist.h" +#include "virlog.h" +#include "virstring.h" +#include "virthread.h" +#include "viruuid.h" + +#define VIR_FROM_THIS VIR_FROM_LOCKING + +#define DLM_LOCKSPACE_MODE 0600 +#define DLM_LOCKSPACE_NAME "libvirt" + +#define LOCK_RECORD_FILE_MODE 0644 +#define LOCK_RECORD_FILE_PATH "/tmp/libvirtd-dlm-file" + +#define PRMODE "PRMODE" +#define EXMODE "EXMODE" + +#define STATUS "STATUS" +#define RESOURCE_NAME "RESOURCE_NAME" +#define LOCK_ID "LOCK_ID" +#define LOCK_MODE "LOCK_MODE" +#define VM_PID "VM_PID" + +#define BUFFERLEN 128 + +/* This will be set after dlm_controld is started. */ +#define DLM_CLUSTER_NAME_PATH "/sys/kernel/config/dlm/cluster/cluster_name" + +VIR_LOG_INIT("locking.lock_driver_dlm"); + +typedef struct _virLockInformation virLockInformation; +typedef virLockInformation *virLockInformationPtr; + +typedef struct _virLockManagerDlmResource virLockManagerDlmResource; +typedef virLockManagerDlmResource *virLockManagerDlmResourcePtr; + +typedef struct _virLockManagerDlmPrivate virLockManagerDlmPrivate; +typedef virLockManagerDlmPrivate *virLockManagerDlmPrivatePtr; + +typedef struct _virLockManagerDlmDriver virLockManagerDlmDriver; +typedef virLockManagerDlmDriver *virLockManagerDlmDriverPtr; + +typedef struct _virListWait virListWait; +typedef virListWait *virListWaitPtr; + +struct _virLockInformation { + virListHead entry; + char *name; + uint32_t mode; + uint32_t lkid; + pid_t vm_pid; +}; + +struct _virLockManagerDlmResource { + char *name; + uint32_t mode; +}; + +struct _virLockManagerDlmPrivate { + unsigned char vm_uuid[VIR_UUID_BUFLEN]; + char *vm_name; + pid_t vm_pid; + int vm_id; + + size_t nresources; + virLockManagerDlmResourcePtr resources; + + bool hasRWDisks; +}; + +struct _virLockManagerDlmDriver { + bool autoDiskLease; + bool requireLeaseForDisks; + + bool purgeLockspace; + char *lockspaceName; + char *lockRecordFilePath; +}; + +struct _virListWait { + virMutex listMutex; + virMutex fileMutex; + virListHead list; +}; + +static virLockManagerDlmDriverPtr driver; +static dlm_lshandle_t lockspace; +static virListWait lockListWait; + +static int virLockManagerDlmLoadConfig(const char *configFile) +{ + virConfPtr conf =3D NULL; + int ret =3D -1; + + if (access(configFile, R_OK) =3D=3D -1) { + if (errno !=3D ENOENT) { + virReportSystemError(errno, + _("Unable to access config file %s"), + configFile); + return -1; + } + return 0; + } + + if (!(conf =3D virConfReadFile(configFile, 0))) + return -1; + + if (virConfGetValueBool(conf, "auto_disk_leases", &driver->autoDiskLea= se) < 0) + goto cleanup; + + driver->requireLeaseForDisks =3D !driver->autoDiskLease; + if (virConfGetValueBool(conf, "require_lease_for_disks", &driver->requ= ireLeaseForDisks) < 0) + goto cleanup; + + if (virConfGetValueBool(conf, "purge_lockspace", &driver->purgeLockspa= ce) < 0) + goto cleanup; + + if (virConfGetValueString(conf, "lockspace_name", &driver->lockspaceNa= me) < 0) + goto cleanup; + + if (virConfGetValueString(conf, "lock_record_file_path", &driver->lock= RecordFilePath) < 0) + goto cleanup; + + ret =3D 0; + + cleanup: + virConfFree(conf); + return ret; +} + +static int virLockManagerDlmToModeUint(const char *token) +{ + if (STREQ(token, PRMODE)) + return LKM_PRMODE; + if (STREQ(token, EXMODE)) + return LKM_EXMODE; + + return 0; +} + +static const char *virLockManagerDlmToModeText(const uint32_t mode) +{ + switch (mode) { + case LKM_PRMODE: + return PRMODE; + case LKM_EXMODE: + return EXMODE; + default: + return NULL; + } +} + +static virLockInformationPtr virLockManagerDlmRecordLock(const char *name, + const uint32_t mo= de, + const uint32_t lk= id, + const pid_t vm_pi= d) +{ + virLockInformationPtr lock =3D NULL; + + if (VIR_ALLOC(lock) < 0) + goto error; + + if (VIR_STRDUP(lock->name, name) < 0) + goto error; + + lock->mode =3D mode; + lock->lkid =3D lkid; + lock->vm_pid =3D vm_pid; + + virMutexLock(&(lockListWait.listMutex)); + virListAddTail(&lock->entry, &(lockListWait.list)); + virMutexUnlock(&(lockListWait.listMutex)); + + VIR_DEBUG("record lock sucessfully, lockName=3D%s lockMode=3D%s lockId= =3D%d", + NULLSTR(name), NULLSTR(virLockManagerDlmToModeText(mode)), l= kid); + + return lock; + + error: + if (lock) + VIR_FREE(lock->name); + VIR_FREE(lock); + return NULL; +} + +static void virLockManagerDlmWriteLock(virLockInformationPtr lock, int fd,= bool status) +{ + char buffer[BUFFERLEN] =3D {0}; + off_t offset =3D 0, rv =3D 0; + + if (!lock) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("lock is NULL")); + return; + } + + /* + * STATUS RESOURCE_NAME LOCK_MODE VM_PID\n + * 6 64 9 10 + * 93 =3D 6 + 1 + 64 + 1 + 9 + 1 + 10 + 1 + */ + offset =3D 93 * lock->lkid; + rv =3D lseek(fd, offset, SEEK_SET); + if (rv < 0) { + virReportSystemError(errno, + _("unable to lseek fd '%d'"), + fd); + return; + } + + snprintf(buffer, sizeof(buffer), "%6d %64s %9s %10jd\n", \ + status, lock->name, + NULLSTR(virLockManagerDlmToModeText(lock->mode)), + (intmax_t)lock->vm_pid); + + if (safewrite(fd, buffer, strlen(buffer)) !=3D strlen(buffer)) { + virReportSystemError(errno, + _("unable to write lock information '%s' to f= ile '%s'"), + buffer, NULLSTR(driver->lockRecordFilePath)); + return; + } + + VIR_DEBUG("write '%s' to fd=3D%d", buffer, fd); + + fdatasync(fd); + + return; +} + +static void virLockManagerDlmAdoptLock(char *raw) { + char *str =3D NULL, *subtoken =3D NULL, *saveptr =3D NULL, *endptr =3D= NULL; + int i =3D 0, status =3D 0; + char *name =3D NULL; + uint32_t mode =3D 0; + pid_t vm_pid =3D 0; + struct dlm_lksb lksb =3D {0}; + + /* every line is the following format: + * STATUS RESOURCE_NAME LOCK_MODE VM_PID + */ + for (i =3D 0, str =3D raw, status =3D 0; ; str =3D NULL, i++) { + subtoken =3D strtok_r(str, " \n", &saveptr); + if (subtoken =3D=3D NULL) + break; + + switch(i) { + case 0: + if (virStrToLong_i(subtoken, &endptr, 10, &status) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("cannot extract lock status '%s'"), subto= ken); + goto cleanup; + } + break; + case 1: + if (VIR_STRDUP(name, subtoken) !=3D 1) + goto cleanup; + break; + case 2: + mode =3D virLockManagerDlmToModeUint(subtoken); + if (!mode) + goto cleanup; + break; + case 3: + if ((virStrToLong_i(subtoken, &endptr, 10, &vm_pid) < 0) || !v= m_pid) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("cannot extract lock vm_pid '%s'"), subto= ken); + goto cleanup; + } + break; + default: + goto cleanup; + break; + } + + if (status !=3D 1) + goto cleanup; + } + + if (i !=3D 4) + goto cleanup; + + /* copy from `lm_adopt_dlm` in daemons/lvmlockd/lvmlockd-dlm.c of lvm2: + * dlm returns 0 for success, -EAGAIN if an orphan is + * found with another mode, and -ENOENT if no orphan. + * + * cast/bast/param are (void *)1 because the kernel + * returns errors if some are null. + */ + + status =3D dlm_ls_lockx(lockspace, mode, &lksb, LKF_PERSISTENT|LKF_ORP= HAN, + name, strlen(name), 0, + (void *)1, (void *)1, (void *)1, + NULL, NULL); + if (status) { + virReportSystemError(errno, + _("unable to adopt lock, rv=3D%d lockName=3D%= s lockMode=3D%s"), + status, name, NULLSTR(virLockManagerDlmToMode= Text(mode))); + goto cleanup; + } + + if (!virLockManagerDlmRecordLock(name, mode, lksb.sb_lkid, vm_pid)) { + virReportSystemError(errno, + _("unable to record lock information, " + "lockName=3D%s lockMode=3D%s lockId=3D%d = vm_pid=3D%jd"), + NULLSTR(name), NULLSTR(virLockManagerDlmToMod= eText(mode)), + lksb.sb_lkid, (intmax_t)vm_pid); + } + + + cleanup: + if (name) + VIR_FREE(name); + + return; +} + +static int virLockManagerDlmPrepareLockList(const char *lockRecordFilePath) +{ + FILE *fp =3D NULL; + int line =3D 0; + size_t n =3D 0; + ssize_t count =3D 0; + char *buffer =3D NULL; + + fp =3D fopen(lockRecordFilePath, "r"); + if (!fp) { + virReportSystemError(errno, + _("unable to open '%s'"), lockRecordFilePath); + return -1; + } + + /* lock information is from the second line */ + for (line =3D 0; !feof(fp); line++) { + count =3D getline(&buffer, &n, fp); + if (count <=3D 0) + break; + + switch (line) { + case 0: + break; + default: + virLockManagerDlmAdoptLock(buffer); + break; + } + } + + VIR_FORCE_FCLOSE(fp); + VIR_FREE(buffer); + + return 0; +} + +static int virLockManagerDlmGetLocalNodeId(uint32_t *nodeId) +{ + cpg_handle_t handle =3D 0; + int rv =3D -1; + + if (cpg_model_initialize(&handle, CPG_MODEL_V1, NULL, NULL) !=3D CS_OK= ) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to create a new connection to the CPG ser= vice")); + return -1; + } + + if( cpg_local_get(handle, nodeId) !=3D CS_OK) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to get the local node id by the CPG servi= ce")); + goto cleanup; + } + + VIR_DEBUG("the local nodeid=3D%u", *nodeId); + + rv =3D 0; + + cleanup: + if (cpg_finalize(handle) !=3D CS_OK) + VIR_WARN("unable to finalize the CPG service"); + + return rv; +} + +static int virLockManagerDlmDumpLockList(const char *lockRecordFilePath) +{ + virLockInformationPtr theLock =3D NULL; + char buffer[BUFFERLEN] =3D {0}; + int fd =3D -1, rv =3D -1; + + /* not need mutex because of only one instance would be initialized */ + fd =3D open(lockRecordFilePath, O_WRONLY|O_CREAT|O_TRUNC, LOCK_RECORD_= FILE_MODE); + if (fd < 0) { + virReportSystemError(errno, + _("unable to open '%s'"), + lockRecordFilePath); + return -1; + } + + snprintf(buffer, sizeof(buffer), "%6s %64s %9s %10s\n", \ + STATUS, RESOURCE_NAME, LOCK_MODE, VM_PID); + if (safewrite(fd, buffer, strlen(buffer)) !=3D strlen(buffer)) { + virReportSystemError(errno, + _("unable to write '%s' to '%s'"), + buffer, lockRecordFilePath); + goto cleanup; + } + + virListForEachEntry(theLock, &(lockListWait.list), entry) { + virLockManagerDlmWriteLock(theLock, fd, 1); + } + + if (VIR_CLOSE(fd) < 0) { + virReportSystemError(errno, + _("unable to close file '%s'"), + lockRecordFilePath); + goto cleanup; + } + + rv =3D 0; + + cleanup: + if (rv) + VIR_FORCE_CLOSE(fd); + return rv; +} + +static int virLockManagerDlmSetupLockRecordFile(const char *lockRecordFile= Path, + const bool newLockspace, + const bool purgeLockspace) +{ + uint32_t nodeId =3D 0; + + /* there maybe some orphan locks recorded in the lock record file which + * should be adopted if lockspace is opened instead of created, we ado= pt + * them then add them in the list. + */ + if (!newLockspace && + virLockManagerDlmPrepareLockList(lockRecordFilePath)) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unable to adopt locks from '%s'"), + NULLSTR(lockRecordFilePath)); + return -1; + } + + /* purgeLockspace flag means purging orphan locks belong to any process + * in this lockspace. + */ + if (purgeLockspace && !virLockManagerDlmGetLocalNodeId(&nodeId)) { + if (dlm_ls_purge(lockspace, nodeId, 0)) { + VIR_WARN("node=3D%u purge DLM locks failed in lockspace=3D%s", + nodeId, NULLSTR(driver->lockspaceName)); + } + else + VIR_DEBUG("node=3D%u purge DLM locks success in lockspace=3D%s= ", + nodeId, NULLSTR(driver->lockspaceName)); + } + + /* initialize the lock record file */ + if (virLockManagerDlmDumpLockList(lockRecordFilePath)) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unable to initialize the lock record file '%s'"), + lockRecordFilePath); + return -1; + } + + return 0; +} + +static int virLockManagerDlmSetup(void) +{ + bool newLockspace =3D false; + + virListHeadInit(&(lockListWait.list)); + if ((virMutexInit(&(lockListWait.listMutex)) < 0) || + (virMutexInit(&(lockListWait.fileMutex)) < 0)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to initialize mutex")); + return -1; + } + + + /* check whether dlm is running or not */ + if (access(DLM_CLUSTER_NAME_PATH, F_OK)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("check dlm_controld, ensure it has setuped")); + return -1; + } + + /* open lockspace, create it if it doesn't exist */ + lockspace =3D dlm_open_lockspace(driver->lockspaceName); + if (!lockspace) { + lockspace =3D dlm_create_lockspace(driver->lockspaceName, DLM_LOCK= SPACE_MODE); + if (!lockspace) { + virReportSystemError(errno, "%s", + _("unable to open and create DLM lockspac= e")); + return -1; + } + newLockspace =3D true; + } + + /* create thread to receive notification from kernel */ + if (dlm_ls_pthread_init(lockspace)) { + if (errno !=3D EEXIST) { + virReportSystemError(errno, "%s", + _("unable to initialize lockspace")); + return -1; + } + } + + /* we need file to record lock information used by rebooted libvirtd */ + if (virLockManagerDlmSetupLockRecordFile(driver->lockRecordFilePath, + newLockspace, + driver->purgeLockspace)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to initialize DLM lock file")); + return -1; + } + + return 0; +} + +static int virLockManagerDlmDeinit(void); + +static int virLockManagerDlmInit(unsigned int version, + const char *configFile, + unsigned int flags) +{ + VIR_DEBUG("version=3D%u configFile=3D%s flags=3D0x%x", version, NULLST= R(configFile), flags); + + virCheckFlags(0, -1); + + if (driver) + return 0; + + if (geteuid() !=3D 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("dlm lock requires root privileges")); + return -1; + } + + if (VIR_ALLOC(driver) < 0) + return -1; + + driver->autoDiskLease =3D true; + driver->requireLeaseForDisks =3D !driver->autoDiskLease; + driver->purgeLockspace =3D true; + + if (virAsprintf(&driver->lockspaceName, + "%s", DLM_LOCKSPACE_NAME) < 0) + goto error; + + if (virAsprintf(&driver->lockRecordFilePath, + "%s", LOCK_RECORD_FILE_PATH) < 0) + goto error; + + if (virLockManagerDlmLoadConfig(configFile) < 0) + goto error; + + if (virLockManagerDlmSetup() < 0) + goto error; + + return 0; + + error: + virLockManagerDlmDeinit(); + return -1; +} + +static int virLockManagerDlmDeinit(void) +{ + virLockInformationPtr theLock =3D NULL; + + if (!driver) + return 0; + + if(lockspace) + dlm_close_lockspace(lockspace); + + /* not care about whether adopting lock or not, + * just release those to prevent memory leak + */ + virListForEachEntry(theLock, &(lockListWait.list), entry) { + virListDelete(&(theLock->entry)); + VIR_FREE(theLock->name); + VIR_FREE(theLock); + } + + VIR_FREE(driver->lockspaceName); + VIR_FREE(driver->lockRecordFilePath); + VIR_FREE(driver); + + return 0; +} + +static int virLockManagerDlmNew(virLockManagerPtr lock, + unsigned int type, + size_t nparams, + virLockManagerParamPtr params, + unsigned int flags) +{ + virLockManagerDlmPrivatePtr priv =3D NULL; + size_t i; + + virCheckFlags(VIR_LOCK_MANAGER_NEW_STARTED, -1); + + if (!driver) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("dlm plugin is not initialized")); + return -1; + } + + if (type !=3D VIR_LOCK_MANAGER_OBJECT_TYPE_DOMAIN) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unsupported object type %d"), type); + return -1; + } + + if (VIR_ALLOC(priv) < 0) + return -1; + + for (i =3D 0; i< nparams; i++) { + if (STREQ(params[i].key, "uuid")) { + memcpy(priv->vm_uuid, params[i].value.uuid, VIR_UUID_BUFLEN); + } else if (STREQ(params[i].key, "name")) { + if (VIR_STRDUP(priv->vm_name, params[i].value.str) < 0) + return -1; + } else if (STREQ(params[i].key, "id")) { + priv->vm_id =3D params[i].value.ui; + } else if (STREQ(params[i].key, "pid")) { + priv->vm_pid =3D params[i].value.iv; + } else if (STREQ(params[i].key, "uri")) { + /* there would be a warning in some case according to the hist= ory patch, + * so ignored + */ + } else { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unexpected parameter %s for object"), + params[i].key); + } + } + + /* check the following to prevent some unexpexted state in some case */ + if (priv->vm_pid =3D=3D 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing PID parameter for domain object")); + return -1; + } + if (!priv->vm_name) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing name parameter for domain object")); + return -1; + } + + if (priv->vm_id =3D=3D 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing ID parameter for domain object")); + return -1; + } + if (!virUUIDIsValid(priv->vm_uuid)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing UUID parameter for domain object")); + return -1; + } + + lock->privateData =3D priv; + + return 0; +} + +static void virLockManagerDlmFree(virLockManagerPtr lock) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + size_t i; + + if (!priv) + return; + + for (i =3D 0; i < priv->nresources; i++) + VIR_FREE(priv->resources[i].name); + + VIR_FREE(priv->resources); + VIR_FREE(priv->vm_name); + VIR_FREE(priv); + lock->privateData =3D NULL; + + return; +} + +static int virLockManagerDlmAddResource(virLockManagerPtr lock, + unsigned int type, const char *nam= e, + size_t nparams, + virLockManagerParamPtr params, + unsigned int flags) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + char *newName =3D NULL; + + virCheckFlags(VIR_LOCK_MANAGER_RESOURCE_READONLY | + VIR_LOCK_MANAGER_RESOURCE_SHARED, -1); + + /* Treat read only resources as a no-op lock request */ + if (flags & VIR_LOCK_MANAGER_RESOURCE_READONLY) + return 0; + + switch (type) { + case VIR_LOCK_MANAGER_RESOURCE_TYPE_DISK: + if (params || nparams) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unexpected parameters for disk resource"= )); + return -1; + } + + if (!driver->autoDiskLease) { + if (!(flags & (VIR_LOCK_MANAGER_RESOURCE_SHARED | + VIR_LOCK_MANAGER_RESOURCE_READONLY))) { + priv->hasRWDisks =3D true; + /* ignore disk resource without error */ + return 0; + } + } + + if (virCryptoHashString(VIR_CRYPTO_HASH_SHA256, name, &newName= ) < 0) + goto cleanup; + + break; + + case VIR_LOCK_MANAGER_RESOURCE_TYPE_LEASE: + /* we need format the lock information, so the lock name must be t= he constant length */ + if (virCryptoHashString(VIR_CRYPTO_HASH_SHA256, name, &newName) < = 0) + goto cleanup; + + break; + + default: + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unknown lock manager object type %d"), + type); + return -1; + } + + if (VIR_EXPAND_N(priv->resources, priv->nresources, 1) < 0) + goto cleanup; + + priv->resources[priv->nresources-1].name =3D newName; + + if (!!(flags & VIR_LOCK_MANAGER_RESOURCE_SHARED)) + priv->resources[priv->nresources-1].mode =3D LKM_PRMODE; + else + priv->resources[priv->nresources-1].mode =3D LKM_EXMODE; + + return 0; + + cleanup: + VIR_FREE(newName); + + return -1; +} + +static int virLockManagerDlmAcquire(virLockManagerPtr lock, + const char *state ATTRIBUTE_UNUSED, + unsigned int flags, + virDomainLockFailureAction action ATTR= IBUTE_UNUSED, + int *fd) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + virLockInformationPtr theLock =3D NULL; + struct dlm_lksb lksb =3D {0}; + int rv =3D -1, theFd =3D -1; + size_t i; + + virCheckFlags(VIR_LOCK_MANAGER_ACQUIRE_REGISTER_ONLY | + VIR_LOCK_MANAGER_ACQUIRE_RESTRICT, -1); + + /* allowed to start a guest which has read/write disks, but without an= y leases */ + if (priv->nresources =3D=3D 0 && + priv->hasRWDisks && + driver->requireLeaseForDisks) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("read/write, exclusive access, disks were present= , but no leases specified")); + return -1; + } + + /* accorting to git patch history, add `fd` parameter in order to + * 'ensure sanlock socket is labelled with the VM process label', + * however, fixing sanlock socket security labelling remove related + * code. Now, `fd` parameter is useless. + */ + if (fd) + *fd =3D -1; + + if(!lockspace) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("lockspace is not opened")); + return -1; + } + + if (!(flags & VIR_LOCK_MANAGER_ACQUIRE_REGISTER_ONLY)) { + VIR_DEBUG("Acquiring object %zu", priv->nresources); + + theFd =3D open(driver->lockRecordFilePath, O_RDWR); + if (theFd < 0) { + virReportSystemError(errno, + _("unable to open '%s'"), driver->lockRec= ordFilePath); + return -1; + } + + for (i =3D 0; i < priv->nresources; i++) { + VIR_DEBUG("Acquiring object %zu", priv->nresources); + + memset(&lksb, 0, sizeof(lksb)); + rv =3D dlm_ls_lock_wait(lockspace, priv->resources[i].mode, + &lksb, LKF_NOQUEUE|LKF_PERSISTENT, + priv->resources[i].name, strlen(priv->re= sources[i].name), + 0, NULL, NULL, NULL); + /* both `rv` and `lksb.sb_status` equal 0 means lock sucessful= ly */ + if (rv || lksb.sb_status) { + if (lksb.sb_status =3D=3D EAGAIN) + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to acquire lock: the lock cou= ld not be granted")); + else { + virReportSystemError(errno, + _("failed to acquire lock: rv=3D%= d lockStatus=3D%d"), + rv, lksb.sb_status); + } + /* rv would be 0 although acquiring lock failed */ + rv =3D -1; + goto cleanup; + } + + theLock =3D virLockManagerDlmRecordLock(priv->resources[i].nam= e, + priv->resources[i].mode, + lksb.sb_lkid, + priv->vm_pid); + if (!theLock) { + virReportSystemError(errno, + _("unable to record lock information,= " + "lockName=3D%s lockMode=3D%s lockI= d=3D%d vm_pid=3D%jd"), + NULLSTR(priv->resources[i].name), + NULLSTR(virLockManagerDlmToModeText(p= riv->resources[i].mode)), + lksb.sb_lkid, (intmax_t)priv->vm_pid); + /* record lock failed, we can't save lock information in m= emory, so release it */ + rv =3D dlm_ls_unlock_wait(lockspace, lksb.sb_lkid, 0, &lks= b); + if (!rv) + virReportSystemError(errno, + _("failed to release lock: rv=3D%= d lockStatue=3D%d"), + rv, lksb.sb_status); + rv =3D -1; + goto cleanup; + } + + virMutexLock(&(lockListWait.fileMutex)); + virLockManagerDlmWriteLock(theLock, theFd, 1); + virMutexUnlock(&(lockListWait.fileMutex)); + } + + if(VIR_CLOSE(theFd) < 0) { + virReportSystemError(errno, + _("unable to save file '%s'"), + driver->lockRecordFilePath); + goto cleanup; + } + } + + if (flags & VIR_LOCK_MANAGER_ACQUIRE_RESTRICT) { + /* no daemon watches this fd, do nothing here, just close lockspac= e before `execv` + * `dlm_close_lockspace` always return 0, so ignore return value + */ + ignore_value(dlm_close_lockspace(lockspace)); + lockspace =3D NULL; + } + + rv =3D 0; + + cleanup: + if (rv) + VIR_FORCE_CLOSE(theFd); + return rv; +} + +static void virLockManagerDlmDeleteLock(const virLockInformationPtr lock, + const char *lockRecordFilePath) +{ + int fd =3D -1; + + if (!lock) + return; + + virMutexLock(&(lockListWait.listMutex)); + virListDelete(&(lock->entry)); + virMutexUnlock(&(lockListWait.listMutex)); + + fd =3D open(lockRecordFilePath, O_RDWR); + if (fd < 0) { + virReportSystemError(errno, + _("unable to open '%s'"), lockRecordFilePath); + goto cleanup; + } + + virMutexLock(&(lockListWait.fileMutex)); + virLockManagerDlmWriteLock(lock, fd, 0); + virMutexUnlock(&(lockListWait.fileMutex)); + + if (VIR_CLOSE(fd) < 0) { + virReportSystemError(errno, + _("unable to save file '%s'"), + lockRecordFilePath); + VIR_FORCE_CLOSE(fd); + } + + cleanup: + VIR_FREE(lock->name); + VIR_FREE(lock); +} + +static int virLockManagerDlmRelease(virLockManagerPtr lock, + char **state, + unsigned int flags) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + virLockManagerDlmResourcePtr resource =3D NULL; + virLockInformationPtr theLock =3D NULL; + struct dlm_lksb lksb =3D {0}; + int rv =3D -1; + size_t i; + + virCheckFlags(0, -1); + + if(state) + *state =3D NULL; + + if(!lockspace) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("lockspace is not opened")); + return -1; + } + + for (i =3D 0; i < priv->nresources; i++) { + resource =3D priv->resources + i; + + virListForEachEntry (theLock, &(lockListWait.list), entry) { + if((theLock->vm_pid =3D=3D priv->vm_pid) && + STREQ(theLock->name, resource->name) && + (theLock->mode =3D=3D resource->mode)) { + + /* + * there are some locks from adopting, the existence of `(= void *)1` + * when adopting makes 'terminated by signal SIGSEGV (Addr= ess + * boundary error)' error appear. + * + * The following code reference to lvm2 project's implemen= t. + */ + lksb.sb_lkid =3D theLock->lkid; + rv =3D dlm_ls_lock_wait(lockspace, LKM_NLMODE, + &lksb, LKF_CONVERT, + resource->name, + strlen(resource->name), + 0, NULL, NULL, NULL); + + if (rv < 0) { + virReportSystemError(errno, + _("failed to convert lock: rv=3D%= d lockStatus=3D%d"), + rv, lksb.sb_status); + goto cleanup; + } + + /* don't care whether the lock is released or not, + * it will be automatically released after the libvirtd de= ad + */ + virLockManagerDlmDeleteLock(theLock, driver->lockRecordFil= ePath); + + rv =3D dlm_ls_unlock_wait(lockspace, lksb.sb_lkid, 0, &lks= b); + if (rv < 0) { + virReportSystemError(errno, + _("failed to release lock: rv=3D%= d lockStatus=3D%d"), + rv, lksb.sb_status); + goto cleanup; + } + + break; + } + } + } + + rv =3D 0; + + cleanup: + return rv; +} + +static int virLockManagerDlmInquire(virLockManagerPtr lock ATTRIBUTE_UNUSE= D, + char **state, + unsigned int flags) +{ + /* not support mannual lock, so this function almost does nothing */ + virCheckFlags(0, -1); + + if (state) + *state =3D NULL; + + return 0; +} + +virLockDriver virLockDriverImpl =3D +{ + .version =3D VIR_LOCK_MANAGER_VERSION, + + .flags =3D VIR_LOCK_MANAGER_USES_STATE, // currently not used + + .drvInit =3D virLockManagerDlmInit, + .drvDeinit =3D virLockManagerDlmDeinit, + + .drvNew =3D virLockManagerDlmNew, + .drvFree =3D virLockManagerDlmFree, + + .drvAddResource =3D virLockManagerDlmAddResource, + + .drvAcquire =3D virLockManagerDlmAcquire, + .drvRelease =3D virLockManagerDlmRelease, + .drvInquire =3D virLockManagerDlmInquire, +}; diff --git a/src/util/virlist.h b/src/util/virlist.h new file mode 100644 index 000000000..4ac3626c7 --- /dev/null +++ b/src/util/virlist.h @@ -0,0 +1,110 @@ +/* + * virlist.h: methods for managing list, logic is copied from Linux Kernel + * + * Copyright (C) 2018 SUSE LINUX Products, Beijing, China. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + * + */ + +#ifndef __VIR_LIST_H +#define __VIR_LIST_H + +#include + +typedef struct _virListHead virListHead; +typedef virListHead *virListHeadPtr; + +struct _virListHead { + struct _virListHead *next, *prev; +}; + +static inline void +virListHeadInit(virListHeadPtr name) +{ + name->next =3D name; + name->prev =3D name; +} + +static inline void +__virListAdd(virListHeadPtr entry, + virListHeadPtr prev, virListHeadPtr next) +{ + next->prev =3D entry; + entry->next =3D next; + entry->prev =3D prev; + prev->next =3D entry; +} + +static inline void +virListAdd(virListHeadPtr entry, virListHeadPtr head) +{ + __virListAdd(entry, head, head->next); +} + +static inline void +virListAddTail(virListHeadPtr entry, virListHeadPtr head) +{ + __virListAdd(entry, head->prev, head); +} + +static inline void +__virListDelete(virListHeadPtr prev, virListHeadPtr next) +{ + next->prev =3D prev; + prev->next =3D next; +} + +static inline void +virListDelete(virListHeadPtr entry) +{ + __virListDelete(entry->prev, entry->next); +} + +static inline bool +virListEmpty(virListHeadPtr head) +{ + return head->next =3D=3D head; +} + +#ifndef virContainerOf +#define virContainerOf(ptr, type, member) \ + (type *)((char *)(ptr) - (char *) &((type *)0)->member) +#endif + +#define virListEntry(ptr, type, member) \ + virContainerOf(ptr, type, member) + +#define virListFirstEntry(ptr, type, member) \ + virListEntry((ptr)->next, type, member) + +#define virListLastEntry(ptr, type, member) \ + virListEntry((ptr)->prev, type, member) + +#define __virContainerOf(ptr, sample, member) \ + (void *)virContainerOf((ptr), typeof(*(sample)), member) + +#define virListForEachEntry(pos, head, member) \ + for (pos =3D __virContainerOf((head)->next, pos, member); \ + &pos->member !=3D (head); \ + pos =3D __virContainerOf(pos->member.next, pos, member)) + +#define virListForEachEntrySafe(pos, tmp, head, member) \ + for (pos =3D __virContainerOf((head)->next, pos, member), \ + tmp =3D __virContainerOf(pos->member.next, pos, member); \ + &pos->member !=3D (head); \ + pos =3D tmp, tmp =3D __virContainerOf(pos->member.next, tmp, member)) + +#endif --=20 2.15.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list