From nobody Fri Apr 26 21:04:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=virtuozzo.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1538986225877138.2478628608419; Mon, 8 Oct 2018 01:10:25 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DE2AB88312; Mon, 8 Oct 2018 08:10:23 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9F1EF308BDA1; Mon, 8 Oct 2018 08:10:23 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 10DF64BB79; Mon, 8 Oct 2018 08:10:23 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w988ALuP006521 for ; Mon, 8 Oct 2018 04:10:21 -0400 Received: by smtp.corp.redhat.com (Postfix) id 02B4A6AEA1; Mon, 8 Oct 2018 08:10:21 +0000 (UTC) Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 67DD96F4E4; Mon, 8 Oct 2018 08:10:16 +0000 (UTC) Received: from relay.sw.ru (relay.sw.ru [185.231.240.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 125253082A23; Mon, 8 Oct 2018 08:10:13 +0000 (UTC) Received: from [10.94.3.220] (helo=dim-vz7.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.90_1) (envelope-from ) id 1g9QcA-0003bt-MU; Mon, 08 Oct 2018 11:10:07 +0300 From: Nikolay Shirokovskiy To: libvir-list@redhat.com Date: Mon, 8 Oct 2018 11:10:05 +0300 Message-Id: <1538986205-419724-1-git-send-email-nshirokovskiy@virtuozzo.com> X-Greylist: Sender passed SPF test, ACL 236 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Mon, 08 Oct 2018 08:10:14 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Mon, 08 Oct 2018 08:10:14 +0000 (UTC) for IP:'185.231.240.75' DOMAIN:'relay.sw.ru' HELO:'relay.sw.ru' FROM:'nshirokovskiy@virtuozzo.com' RCPT:'' X-RedHat-Spam-Score: -0.001 (SPF_PASS) 185.231.240.75 relay.sw.ru 185.231.240.75 relay.sw.ru X-Scanned-By: MIMEDefang 2.84 on 10.5.110.45 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: libvir-list@redhat.com Cc: Jiri Denemark Subject: [libvirt] [PATCH RFC v2] qemu: fix deadlock when waiting in non async jobs X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 08 Oct 2018 08:10:24 +0000 (UTC) X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" Block job abort operation can not handle properly qemu crashes when waiting= for abort/pivot completion. Deadlock scenario is next: - qemuDomainBlockJobAbort waits for pivot/abort completion - qemu crashes, then qemuProcessBeginStopJob broadcasts for VM condition and then waits for job condition (taken by qemuDomainBlockJobAbort) - qemuDomainBlockJobAbort awakes but nothing really changed, VM is still active (vm->def->id !=3D -1) so thread starts waiting for completion agai= n. Now two threads are in deadlock. First let's remove broadcast in qemuProcessBeginStopJob. It is simply wrong because it is not set any condition before broadcast so that awaked threads= can not detect any changes. Crashing domain during async job will continue to be handled properly because destroy job can run concurrently with async job and destroy job calls qemuProcessStop which sets vm->def->id to -1 and broadcas= ts. Second let's introduce flag that EOF is received and broadcast after that. Now non async jobs can check this flag in wait loop. Signed-off-by: Nikolay Shirokovskiy --- Diff from v1: - patches 1 and 2 are already merged - don't bother with reporting monitor EOF reason to user as most of time it is simply "unexpected eof" (this implies dropping patch 3) - drop patch 5 as we now always report "domain is being stopped" in qemuDomainObjWait - don't signal on monitor error for simplicity (otherwise we need to report something more elaborate that "domain is being stopped" as we don't kill domain on monitor errors. On the other hand I guess monitor error is rare case to handle it right now) - keep virDomainObjWait for async jobs It's a bit uneven that for async jobs domain is destroyed concurrently and = for non async jobs it will be actually destroyed after job get completed. Also= if non async job needs issuing commands to qemu on cleanup then we will send t= hese commands in vain polluting logs etc because qemu process in not running at = this moment but typical check (virDomainObjIsActive) will think it is still runn= ing. Domain is destroyed (qemuProcessStop) in a job due to patches [1] and [2]. However AFAIU it is not neccessary. If qemuProcessStop does not drop VM lock then we don't need extra job to make qemuProcessStop and main job not interleave. And we can drop the lock now only in qemuDomainObjBeginNestedJo= b in qemuProcessStop which is introduced in [2]. AFAIU we can fix issues mention= ed in [2] the other way for example like it is done for qemu agent - we save agent monitor reference on stack for entering/exiting agent monitor. So I wonder can we instead of this fix remove job for qemuProcessStop and r= un destroying domain cuncurrently for non async jobs too. [1] commit 8c9ff9960b29d4703a99efdd1cadcf6f48799cc0 Author: Jiri Denemark Date: Thu Feb 11 15:32:48 2016 +0100 qemu: Process monitor EOF in a job [2] commit 81f50cb92d16643bcd749e3ab5b404b8b7cec643 Author: Jiri Denemark Date: Thu Feb 11 11:20:28 2016 +0100 qemu: Avoid calling qemuProcessStop without a job src/qemu/qemu_domain.c | 39 +++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_domain.h | 4 ++++ src/qemu/qemu_driver.c | 2 +- src/qemu/qemu_hotplug.c | 4 ++-- src/qemu/qemu_process.c | 9 +++++---- 5 files changed, 51 insertions(+), 7 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 939b2a3..aead72b 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -13534,3 +13534,42 @@ qemuDomainRunningReasonToResumeEvent(virDomainRunn= ingReason reason) =20 return VIR_DOMAIN_EVENT_RESUMED_UNPAUSED; } + + +/** + * Waits for domain condition to be triggered for a specific period of tim= e. + * if @until is 0 then waits indefinetely. + * + * Returns: + * -1 on error + * 0 on success + * 1 on timeout + */ +int +qemuDomainObjWait(virDomainObjPtr vm, unsigned long long until) +{ + qemuDomainObjPrivatePtr priv =3D vm->privateData; + int rc; + + if (until) + rc =3D virCondWaitUntil(&vm->cond, &vm->parent.lock, until); + else + rc =3D virCondWait(&vm->cond, &vm->parent.lock); + + if (rc < 0) { + if (until && errno =3D=3D ETIMEDOUT) + return 1; + + virReportSystemError(errno, "%s", + _("failed to wait for domain condition")); + return -1; + } + + if (priv->monEOF) { + virReportError(VIR_ERR_OPERATION_FAILED, "%s", + _("domain is being stopped")); + return -1; + } + + return 0; +} diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 2f8a1bf..36ab294 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -281,6 +281,7 @@ struct _qemuDomainObjPrivate { virDomainChrSourceDefPtr monConfig; bool monJSON; bool monError; + bool monEOF; unsigned long long monStart; =20 qemuAgentPtr agent; @@ -1085,4 +1086,7 @@ void qemuDomainStorageIdReset(qemuDomainObjPrivatePtr= priv); virDomainEventResumedDetailType qemuDomainRunningReasonToResumeEvent(virDomainRunningReason reason); =20 +int +qemuDomainObjWait(virDomainObjPtr vm, unsigned long long until); + #endif /* __QEMU_DOMAIN_H__ */ diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index b238309..f4250da 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -17142,7 +17142,7 @@ qemuDomainBlockJobAbort(virDomainPtr dom, qemuDomainDiskPrivatePtr diskPriv =3D QEMU_DOMAIN_DISK_PRIVATE(dis= k); qemuBlockJobUpdate(vm, QEMU_ASYNC_JOB_NONE, disk, NULL); while (diskPriv->blockjob) { - if (virDomainObjWait(vm) < 0) { + if (qemuDomainObjWait(vm, 0) < 0) { ret =3D -1; goto endjob; } diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 4558a3c..8189629 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -165,7 +165,7 @@ qemuHotplugWaitForTrayEject(virDomainObjPtr vm, return -1; =20 while (disk->tray_status !=3D VIR_DOMAIN_DISK_TRAY_OPEN) { - if ((rc =3D virDomainObjWaitUntil(vm, now + CHANGE_MEDIA_TIMEOUT))= < 0) + if ((rc =3D qemuDomainObjWait(vm, now + CHANGE_MEDIA_TIMEOUT)) < 0) return -1; =20 if (rc > 0) { @@ -5002,7 +5002,7 @@ qemuDomainWaitForDeviceRemoval(virDomainObjPtr vm) until +=3D qemuDomainRemoveDeviceWaitTime; =20 while (priv->unplug.alias) { - if ((rc =3D virDomainObjWaitUntil(vm, until)) =3D=3D 1) + if ((rc =3D qemuDomainObjWait(vm, until)) =3D=3D 1) return 0; =20 if (rc < 0) { diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 29b0ba1..dd03269 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -290,9 +290,12 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon, =20 virObjectLock(vm); =20 + priv =3D vm->privateData; + priv->monEOF =3D true; + virDomainObjBroadcast(vm); + VIR_DEBUG("Received EOF on %p '%s'", vm, vm->def->name); =20 - priv =3D vm->privateData; if (priv->beingDestroyed) { VIR_DEBUG("Domain is being destroyed, EOF is expected"); goto cleanup; @@ -5996,6 +5999,7 @@ qemuProcessPrepareDomain(virQEMUDriverPtr driver, =20 priv->monJSON =3D true; priv->monError =3D false; + priv->monEOF =3D false; priv->monStart =3D 0; priv->gotShutdown =3D false; priv->runningReason =3D VIR_DOMAIN_RUNNING_UNKNOWN; @@ -6965,9 +6969,6 @@ qemuProcessBeginStopJob(virQEMUDriverPtr driver, if (qemuProcessKill(vm, killFlags) < 0) goto cleanup; =20 - /* Wake up anything waiting on domain condition */ - virDomainObjBroadcast(vm); - if (qemuDomainObjBeginJob(driver, vm, job) < 0) goto cleanup; =20 --=20 1.8.3.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list