From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631199590624.9217803078778; Fri, 22 Apr 2022 05:39:59 -0700 (PDT) Received: from localhost ([::1]:60904 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhsZq-00050W-4H for importer2@patchew.org; Fri, 22 Apr 2022 08:39:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58862) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr2-0001fr-3x for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:40 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39354) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr0-0005Ex-Cp for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:39 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id E41F2407624D; Fri, 22 Apr 2022 11:53:14 +0000 (UTC) Subject: [PATCH 1/9] replay: fix event queue flush for qemu shutdown From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:14 +0300 Message-ID: <165062839471.526882.6289802323800289333.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631200290100001 This patch fixes event queue flush in the case of emulator shutdown. replay_finish_events should be called when replay_mode is not cleared. Signed-off-by: Pavel Dovgalyuk Reviewed-by: Richard Henderson --- replay/replay.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/replay/replay.c b/replay/replay.c index 6df2abc18c..2d3607998a 100644 --- a/replay/replay.c +++ b/replay/replay.c @@ -387,9 +387,8 @@ void replay_finish(void) g_free(replay_snapshot); replay_snapshot =3D NULL; =20 - replay_mode =3D REPLAY_MODE_NONE; - replay_finish_events(); + replay_mode =3D REPLAY_MODE_NONE; } =20 void replay_add_blocker(Error *reason) From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631468292739.4311835668357; Fri, 22 Apr 2022 05:44:28 -0700 (PDT) Received: from localhost ([::1]:39220 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhseB-0001fK-Ar for importer2@patchew.org; Fri, 22 Apr 2022 08:44:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58924) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr6-0001qU-Qe for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:47 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39378) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr5-0005F4-4Z for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:44 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id 69A1C4076261; Fri, 22 Apr 2022 11:53:20 +0000 (UTC) Subject: [PATCH 2/9] replay: notify vCPU when BH is scheduled From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:20 +0300 Message-ID: <165062840023.526882.4524922830180183891.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631469573100001 vCPU execution should be suspended when new BH is scheduled. This is needed to avoid guest timeouts caused by the long cycles of the execution. In replay mode execution may hang when vCPU sleeps and block event comes to the queue. This patch adds notification which wakes up vCPU or interrupts execution of guest code. Signed-off-by: Pavel Dovgalyuk --- include/sysemu/cpu-timers.h | 1 + softmmu/icount.c | 8 ++++++++ stubs/icount.c | 4 ++++ util/async.c | 8 ++++++++ 4 files changed, 21 insertions(+) diff --git a/include/sysemu/cpu-timers.h b/include/sysemu/cpu-timers.h index ed6ee5c46c..2e786fe7fb 100644 --- a/include/sysemu/cpu-timers.h +++ b/include/sysemu/cpu-timers.h @@ -59,6 +59,7 @@ int64_t icount_round(int64_t count); /* if the CPUs are idle, start accounting real time to virtual clock. */ void icount_start_warp_timer(void); void icount_account_warp_timer(void); +void icount_notify_exit(void); =20 /* * CPU Ticks and Clock diff --git a/softmmu/icount.c b/softmmu/icount.c index 5ca271620d..880ed04900 100644 --- a/softmmu/icount.c +++ b/softmmu/icount.c @@ -486,3 +486,11 @@ void icount_configure(QemuOpts *opts, Error **errp) qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + NANOSECONDS_PER_SECOND / 10); } + +void icount_notify_exit(void) +{ + if (icount_enabled() && first_cpu) { + cpu_exit(first_cpu); + qemu_clock_notify(QEMU_CLOCK_VIRTUAL); + } +} diff --git a/stubs/icount.c b/stubs/icount.c index f13c43568b..6df8c2bf7d 100644 --- a/stubs/icount.c +++ b/stubs/icount.c @@ -43,3 +43,7 @@ void icount_account_warp_timer(void) { abort(); } + +void icount_notify_exit(void) +{ +} diff --git a/util/async.c b/util/async.c index 2ea1172f3e..c376b6c2b4 100644 --- a/util/async.c +++ b/util/async.c @@ -33,6 +33,7 @@ #include "block/raw-aio.h" #include "qemu/coroutine_int.h" #include "qemu/coroutine-tls.h" +#include "sysemu/cpu-timers.h" #include "trace.h" =20 /***********************************************************/ @@ -185,6 +186,13 @@ void qemu_bh_schedule_idle(QEMUBH *bh) void qemu_bh_schedule(QEMUBH *bh) { aio_bh_enqueue(bh, BH_SCHEDULED); + /* + * Workaround for record/replay. + * vCPU execution should be suspended when new BH is set. + * This is needed to avoid guest timeouts caused + * by the long cycles of the execution. + */ + icount_notify_exit(); } =20 /* This func is async. From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631569116583.0696171075772; Fri, 22 Apr 2022 05:46:09 -0700 (PDT) Received: from localhost ([::1]:42342 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhsfn-000483-QN for importer2@patchew.org; Fri, 22 Apr 2022 08:46:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58976) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrE-0001th-6c for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:55 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39394) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrB-0005FE-Bw for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:51 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id CC1B94076267; Fri, 22 Apr 2022 11:53:25 +0000 (UTC) Subject: [PATCH 3/9] replay: rewrite async event handling From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:25 +0300 Message-ID: <165062840564.526882.15249141058569885051.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631570437100001 This patch decouples checkpoints and async events. It was a tricky part of replay implementation. Now it becomes much simpler and easier to maintain. Signed-off-by: Pavel Dovgalyuk Acked-by: Richard Henderson --- accel/tcg/tcg-accel-ops-icount.c | 5 +-- docs/replay.txt | 11 ++---- include/sysemu/replay.h | 9 ++++- replay/replay-events.c | 20 +++-------- replay/replay-internal.h | 6 +-- replay/replay-snapshot.c | 1 - replay/replay.c | 72 +++++++++++++++-------------------= ---- softmmu/icount.c | 4 ++ 8 files changed, 52 insertions(+), 76 deletions(-) diff --git a/accel/tcg/tcg-accel-ops-icount.c b/accel/tcg/tcg-accel-ops-ico= unt.c index 24520ea112..8f1dda4344 100644 --- a/accel/tcg/tcg-accel-ops-icount.c +++ b/accel/tcg/tcg-accel-ops-icount.c @@ -84,8 +84,7 @@ void icount_handle_deadline(void) * Don't interrupt cpu thread, when these events are waiting * (i.e., there is no checkpoint) */ - if (deadline =3D=3D 0 - && (replay_mode !=3D REPLAY_MODE_PLAY || replay_has_checkpoint()))= { + if (deadline =3D=3D 0) { icount_notify_aio_contexts(); } } @@ -109,7 +108,7 @@ void icount_prepare_for_run(CPUState *cpu) =20 replay_mutex_lock(); =20 - if (cpu->icount_budget =3D=3D 0 && replay_has_checkpoint()) { + if (cpu->icount_budget =3D=3D 0) { icount_notify_aio_contexts(); } } diff --git a/docs/replay.txt b/docs/replay.txt index 5b008ca491..6c9fdff09d 100644 --- a/docs/replay.txt +++ b/docs/replay.txt @@ -366,11 +366,9 @@ Here is the list of events that are written into the l= og: Argument: 4-byte number of executed instructions. - EVENT_INTERRUPT. Used to synchronize interrupt processing. - EVENT_EXCEPTION. Used to synchronize exception handling. - - EVENT_ASYNC. This is a group of events. They are always processed - together with checkpoints. When such an event is generated, it is - stored in the queue and processed only when checkpoint occurs. - Every such event is followed by 1-byte checkpoint id and 1-byte - async event id from the following list: + - EVENT_ASYNC. This is a group of events. When such an event is generated, + it is stored in the queue and processed in icount_account_warp_timer(). + Every such event has it's own id from the following list: - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes callbacks that affect virtual machine state, but normally called asynchronously. @@ -405,6 +403,5 @@ Here is the list of events that are written into the lo= g: - EVENT_CLOCK + clock_id. Group of events for host clock read operations. Argument: 8-byte clock value. - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of - CPU, internal threads, and asynchronous input events. May be followed - by one or more EVENT_ASYNC events. + CPU, internal threads, and asynchronous input events. - EVENT_END. Last event in the log. diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h index 032256533b..9af0ac32f0 100644 --- a/include/sysemu/replay.h +++ b/include/sysemu/replay.h @@ -161,9 +161,14 @@ void replay_shutdown_request(ShutdownCause cause); Returns 0 in PLAY mode if checkpoint was not found. Returns 1 in all other cases. */ bool replay_checkpoint(ReplayCheckpoint checkpoint); -/*! Used to determine that checkpoint is pending. +/*! Used to determine that checkpoint or async event is pending. Does not proceed to the next event in the log. */ -bool replay_has_checkpoint(void); +bool replay_has_event(void); +/* + * Processes the async events added to the queue (while recording) + * or reads the events from the file (while replaying). + */ +void replay_async_events(void); =20 /* Asynchronous events queue */ =20 diff --git a/replay/replay-events.c b/replay/replay-events.c index ac47c89834..db1decf9dd 100644 --- a/replay/replay-events.c +++ b/replay/replay-events.c @@ -170,12 +170,11 @@ void replay_block_event(QEMUBH *bh, uint64_t id) } } =20 -static void replay_save_event(Event *event, int checkpoint) +static void replay_save_event(Event *event) { if (replay_mode !=3D REPLAY_MODE_PLAY) { /* put the event into the file */ replay_put_event(EVENT_ASYNC); - replay_put_byte(checkpoint); replay_put_byte(event->event_kind); =20 /* save event-specific data */ @@ -206,34 +205,27 @@ static void replay_save_event(Event *event, int check= point) } =20 /* Called with replay mutex locked */ -void replay_save_events(int checkpoint) +void replay_save_events(void) { g_assert(replay_mutex_locked()); - g_assert(checkpoint !=3D CHECKPOINT_CLOCK_WARP_START); - g_assert(checkpoint !=3D CHECKPOINT_CLOCK_VIRTUAL); while (!QTAILQ_EMPTY(&events_list)) { Event *event =3D QTAILQ_FIRST(&events_list); - replay_save_event(event, checkpoint); + replay_save_event(event); replay_run_event(event); QTAILQ_REMOVE(&events_list, event, events); g_free(event); } } =20 -static Event *replay_read_event(int checkpoint) +static Event *replay_read_event(void) { Event *event; if (replay_state.read_event_kind =3D=3D -1) { - replay_state.read_event_checkpoint =3D replay_get_byte(); replay_state.read_event_kind =3D replay_get_byte(); replay_state.read_event_id =3D -1; replay_check_error(); } =20 - if (checkpoint !=3D replay_state.read_event_checkpoint) { - return NULL; - } - /* Events that has not to be in the queue */ switch (replay_state.read_event_kind) { case REPLAY_ASYNC_EVENT_BH: @@ -294,11 +286,11 @@ static Event *replay_read_event(int checkpoint) } =20 /* Called with replay mutex locked */ -void replay_read_events(int checkpoint) +void replay_read_events(void) { g_assert(replay_mutex_locked()); while (replay_state.data_kind =3D=3D EVENT_ASYNC) { - Event *event =3D replay_read_event(checkpoint); + Event *event =3D replay_read_event(); if (!event) { break; } diff --git a/replay/replay-internal.h b/replay/replay-internal.h index dada623527..59797c86cf 100644 --- a/replay/replay-internal.h +++ b/replay/replay-internal.h @@ -87,8 +87,6 @@ typedef struct ReplayState { int32_t read_event_kind; /*! Asynchronous event id read from the log */ uint64_t read_event_id; - /*! Asynchronous event checkpoint id read from the log */ - int32_t read_event_checkpoint; } ReplayState; extern ReplayState replay_state; =20 @@ -151,9 +149,9 @@ void replay_finish_events(void); /*! Returns true if there are any unsaved events in the queue */ bool replay_has_events(void); /*! Saves events from queue into the file */ -void replay_save_events(int checkpoint); +void replay_save_events(void); /*! Read events from the file into the input queue */ -void replay_read_events(int checkpoint); +void replay_read_events(void); /*! Adds specified async event to the queue */ void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque, void *opaque2, uint64_t id); diff --git a/replay/replay-snapshot.c b/replay/replay-snapshot.c index e8767a1937..7e935deb15 100644 --- a/replay/replay-snapshot.c +++ b/replay/replay-snapshot.c @@ -61,7 +61,6 @@ static const VMStateDescription vmstate_replay =3D { VMSTATE_UINT64(block_request_id, ReplayState), VMSTATE_INT32(read_event_kind, ReplayState), VMSTATE_UINT64(read_event_id, ReplayState), - VMSTATE_INT32(read_event_checkpoint, ReplayState), VMSTATE_END_OF_LIST() }, }; diff --git a/replay/replay.c b/replay/replay.c index 2d3607998a..cb1b68c46d 100644 --- a/replay/replay.c +++ b/replay/replay.c @@ -22,7 +22,7 @@ =20 /* Current version of the replay mechanism. Increase it when file format changes. */ -#define REPLAY_VERSION 0xe0200a +#define REPLAY_VERSION 0xe0200b /* Size of replay log header */ #define HEADER_SIZE (sizeof(uint32_t) + sizeof(uint64_t)) =20 @@ -171,64 +171,47 @@ void replay_shutdown_request(ShutdownCause cause) =20 bool replay_checkpoint(ReplayCheckpoint checkpoint) { - bool res =3D false; - static bool in_checkpoint; assert(EVENT_CHECKPOINT + checkpoint <=3D EVENT_CHECKPOINT_LAST); =20 - if (!replay_file) { - return true; - } - - if (in_checkpoint) { - /* - Recursion occurs when HW event modifies timers. - Prevent performing icount warp in this case and - wait for another invocation of the checkpoint. - */ - g_assert(replay_mode =3D=3D REPLAY_MODE_PLAY); - return false; - } - in_checkpoint =3D true; - replay_save_instructions(); =20 if (replay_mode =3D=3D REPLAY_MODE_PLAY) { g_assert(replay_mutex_locked()); if (replay_next_event_is(EVENT_CHECKPOINT + checkpoint)) { replay_finish_event(); - } else if (replay_state.data_kind !=3D EVENT_ASYNC) { - res =3D false; - goto out; + } else { + return false; } - replay_read_events(checkpoint); - /* replay_read_events may leave some unread events. - Return false if not all of the events associated with - checkpoint were processed */ - res =3D replay_state.data_kind !=3D EVENT_ASYNC; } else if (replay_mode =3D=3D REPLAY_MODE_RECORD) { g_assert(replay_mutex_locked()); replay_put_event(EVENT_CHECKPOINT + checkpoint); - /* This checkpoint belongs to several threads. - Processing events from different threads is - non-deterministic */ - if (checkpoint !=3D CHECKPOINT_CLOCK_WARP_START - /* FIXME: this is temporary fix, other checkpoints - may also be invoked from the different threads somed= ay. - Asynchronous event processing should be refactored - to create additional replay event kind which is - nailed to the one of the threads and which processes - the event queue. */ - && checkpoint !=3D CHECKPOINT_CLOCK_VIRTUAL) { - replay_save_events(checkpoint); - } - res =3D true; } -out: - in_checkpoint =3D false; - return res; + return true; +} + +void replay_async_events(void) +{ + static bool processing =3D false; + /* If we are already processing the events, recursion may occur + in case of incorrect implementation when HW event modifies timers. + Timer modification may invoke the icount warp, event processing, + and cause the recursion. */ + g_assert(!processing); + processing =3D true; + + replay_save_instructions(); + + if (replay_mode =3D=3D REPLAY_MODE_PLAY) { + g_assert(replay_mutex_locked()); + replay_read_events(); + } else if (replay_mode =3D=3D REPLAY_MODE_RECORD) { + g_assert(replay_mutex_locked()); + replay_save_events(); + } + processing =3D false; } =20 -bool replay_has_checkpoint(void) +bool replay_has_event(void) { bool res =3D false; if (replay_mode =3D=3D REPLAY_MODE_PLAY) { @@ -236,6 +219,7 @@ bool replay_has_checkpoint(void) replay_account_executed_instructions(); res =3D EVENT_CHECKPOINT <=3D replay_state.data_kind && replay_state.data_kind <=3D EVENT_CHECKPOINT_LAST; + res =3D res || replay_state.data_kind =3D=3D EVENT_ASYNC; } return res; } diff --git a/softmmu/icount.c b/softmmu/icount.c index 880ed04900..3c9ff8164b 100644 --- a/softmmu/icount.c +++ b/softmmu/icount.c @@ -322,7 +322,7 @@ void icount_start_warp_timer(void) * to vCPU was processed in advance and vCPU went to sleep. * Therefore we have to wake it up for doing someting. */ - if (replay_has_checkpoint()) { + if (replay_has_event()) { qemu_clock_notify(QEMU_CLOCK_VIRTUAL); } return; @@ -404,6 +404,8 @@ void icount_account_warp_timer(void) return; } =20 + replay_async_events(); + /* warp clock deterministically in record/replay mode */ if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_ACCOUNT)) { return; From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16506317726651005.2583086908346; Fri, 22 Apr 2022 05:49:32 -0700 (PDT) Received: from localhost ([::1]:48946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhsj5-0001ED-M2 for importer2@patchew.org; Fri, 22 Apr 2022 08:49:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59008) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrL-0001wJ-4U for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:59 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39414) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrH-0005FS-I2 for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:58 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id 351B5407626A; Fri, 22 Apr 2022 11:53:31 +0000 (UTC) Subject: [PATCH 4/9] replay: simplify async event processing From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:31 +0300 Message-ID: <165062841103.526882.9023955155829649867.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631773247100001 This patch joins replay event id and async event id into single byte in the= log. It makes processing a bit faster and log a bit smaller. Signed-off-by: Pavel Dovgalyuk --- replay/replay-events.c | 36 ++++++++++++++---------------------- replay/replay-internal.h | 29 ++++++++++++++--------------- replay/replay-snapshot.c | 1 - replay/replay.c | 5 +++-- 4 files changed, 31 insertions(+), 40 deletions(-) diff --git a/replay/replay-events.c b/replay/replay-events.c index db1decf9dd..7bc9d6c2ff 100644 --- a/replay/replay-events.c +++ b/replay/replay-events.c @@ -174,8 +174,8 @@ static void replay_save_event(Event *event) { if (replay_mode !=3D REPLAY_MODE_PLAY) { /* put the event into the file */ - replay_put_event(EVENT_ASYNC); - replay_put_byte(event->event_kind); + g_assert(event->event_kind < REPLAY_ASYNC_COUNT); + replay_put_event(EVENT_ASYNC + event->event_kind); =20 /* save event-specific data */ switch (event->event_kind) { @@ -220,14 +220,10 @@ void replay_save_events(void) static Event *replay_read_event(void) { Event *event; - if (replay_state.read_event_kind =3D=3D -1) { - replay_state.read_event_kind =3D replay_get_byte(); - replay_state.read_event_id =3D -1; - replay_check_error(); - } + int event_kind =3D replay_state.data_kind - EVENT_ASYNC; =20 /* Events that has not to be in the queue */ - switch (replay_state.read_event_kind) { + switch (event_kind) { case REPLAY_ASYNC_EVENT_BH: case REPLAY_ASYNC_EVENT_BH_ONESHOT: if (replay_state.read_event_id =3D=3D -1) { @@ -236,17 +232,17 @@ static Event *replay_read_event(void) break; case REPLAY_ASYNC_EVENT_INPUT: event =3D g_new0(Event, 1); - event->event_kind =3D replay_state.read_event_kind; + event->event_kind =3D event_kind; event->opaque =3D replay_read_input_event(); return event; case REPLAY_ASYNC_EVENT_INPUT_SYNC: event =3D g_new0(Event, 1); - event->event_kind =3D replay_state.read_event_kind; + event->event_kind =3D event_kind; event->opaque =3D 0; return event; case REPLAY_ASYNC_EVENT_CHAR_READ: event =3D g_new0(Event, 1); - event->event_kind =3D replay_state.read_event_kind; + event->event_kind =3D event_kind; event->opaque =3D replay_event_char_read_load(); return event; case REPLAY_ASYNC_EVENT_BLOCK: @@ -256,18 +252,17 @@ static Event *replay_read_event(void) break; case REPLAY_ASYNC_EVENT_NET: event =3D g_new0(Event, 1); - event->event_kind =3D replay_state.read_event_kind; + event->event_kind =3D event_kind; event->opaque =3D replay_event_net_load(); return event; default: - error_report("Unknown ID %d of replay event", - replay_state.read_event_kind); + error_report("Unknown ID %d of replay event", event_kind); exit(1); break; } =20 QTAILQ_FOREACH(event, &events_list, events) { - if (event->event_kind =3D=3D replay_state.read_event_kind + if (event->event_kind =3D=3D event_kind && (replay_state.read_event_id =3D=3D -1 || replay_state.read_event_id =3D=3D event->id)) { break; @@ -276,12 +271,8 @@ static Event *replay_read_event(void) =20 if (event) { QTAILQ_REMOVE(&events_list, event, events); - } else { - return NULL; } =20 - /* Read event-specific data */ - return event; } =20 @@ -289,13 +280,14 @@ static Event *replay_read_event(void) void replay_read_events(void) { g_assert(replay_mutex_locked()); - while (replay_state.data_kind =3D=3D EVENT_ASYNC) { + while (replay_state.data_kind >=3D EVENT_ASYNC + && replay_state.data_kind <=3D EVENT_ASYNC_LAST) { Event *event =3D replay_read_event(); if (!event) { break; } replay_finish_event(); - replay_state.read_event_kind =3D -1; + replay_state.read_event_id =3D -1; replay_run_event(event); =20 g_free(event); @@ -304,7 +296,7 @@ void replay_read_events(void) =20 void replay_init_events(void) { - replay_state.read_event_kind =3D -1; + replay_state.read_event_id =3D -1; } =20 void replay_finish_events(void) diff --git a/replay/replay-internal.h b/replay/replay-internal.h index 59797c86cf..3be2e077ad 100644 --- a/replay/replay-internal.h +++ b/replay/replay-internal.h @@ -12,6 +12,19 @@ * */ =20 +/* Asynchronous events IDs */ + +enum ReplayAsyncEventKind { + REPLAY_ASYNC_EVENT_BH, + REPLAY_ASYNC_EVENT_BH_ONESHOT, + REPLAY_ASYNC_EVENT_INPUT, + REPLAY_ASYNC_EVENT_INPUT_SYNC, + REPLAY_ASYNC_EVENT_CHAR_READ, + REPLAY_ASYNC_EVENT_BLOCK, + REPLAY_ASYNC_EVENT_NET, + REPLAY_ASYNC_COUNT +}; + /* Any changes to order/number of events will need to bump REPLAY_VERSION = */ enum ReplayEvents { /* for instruction event */ @@ -22,6 +35,7 @@ enum ReplayEvents { EVENT_EXCEPTION, /* for async events */ EVENT_ASYNC, + EVENT_ASYNC_LAST =3D EVENT_ASYNC + REPLAY_ASYNC_COUNT - 1, /* for shutdown requests, range allows recovery of ShutdownCause */ EVENT_SHUTDOWN, EVENT_SHUTDOWN_LAST =3D EVENT_SHUTDOWN + SHUTDOWN_CAUSE__MAX, @@ -49,19 +63,6 @@ enum ReplayEvents { EVENT_COUNT }; =20 -/* Asynchronous events IDs */ - -enum ReplayAsyncEventKind { - REPLAY_ASYNC_EVENT_BH, - REPLAY_ASYNC_EVENT_BH_ONESHOT, - REPLAY_ASYNC_EVENT_INPUT, - REPLAY_ASYNC_EVENT_INPUT_SYNC, - REPLAY_ASYNC_EVENT_CHAR_READ, - REPLAY_ASYNC_EVENT_BLOCK, - REPLAY_ASYNC_EVENT_NET, - REPLAY_ASYNC_COUNT -}; - typedef enum ReplayAsyncEventKind ReplayAsyncEventKind; =20 typedef struct ReplayState { @@ -83,8 +84,6 @@ typedef struct ReplayState { uint64_t block_request_id; /*! Prior value of the host clock */ uint64_t host_clock_last; - /*! Asynchronous event type read from the log */ - int32_t read_event_kind; /*! Asynchronous event id read from the log */ uint64_t read_event_id; } ReplayState; diff --git a/replay/replay-snapshot.c b/replay/replay-snapshot.c index 7e935deb15..10a7cf7992 100644 --- a/replay/replay-snapshot.c +++ b/replay/replay-snapshot.c @@ -59,7 +59,6 @@ static const VMStateDescription vmstate_replay =3D { VMSTATE_UINT32(has_unread_data, ReplayState), VMSTATE_UINT64(file_offset, ReplayState), VMSTATE_UINT64(block_request_id, ReplayState), - VMSTATE_INT32(read_event_kind, ReplayState), VMSTATE_UINT64(read_event_id, ReplayState), VMSTATE_END_OF_LIST() }, diff --git a/replay/replay.c b/replay/replay.c index cb1b68c46d..f01f79cd38 100644 --- a/replay/replay.c +++ b/replay/replay.c @@ -22,7 +22,7 @@ =20 /* Current version of the replay mechanism. Increase it when file format changes. */ -#define REPLAY_VERSION 0xe0200b +#define REPLAY_VERSION 0xe0200c /* Size of replay log header */ #define HEADER_SIZE (sizeof(uint32_t) + sizeof(uint64_t)) =20 @@ -219,7 +219,8 @@ bool replay_has_event(void) replay_account_executed_instructions(); res =3D EVENT_CHECKPOINT <=3D replay_state.data_kind && replay_state.data_kind <=3D EVENT_CHECKPOINT_LAST; - res =3D res || replay_state.data_kind =3D=3D EVENT_ASYNC; + res =3D res || (EVENT_ASYNC <=3D replay_state.data_kind + && replay_state.data_kind <=3D EVENT_ASYNC_LAST); } return res; } From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650629751152797.9004329328466; Fri, 22 Apr 2022 05:15:51 -0700 (PDT) Received: from localhost ([::1]:45234 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhsCT-0006AJ-Oi for importer2@patchew.org; Fri, 22 Apr 2022 08:15:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58906) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr5-0001o9-AH for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:43 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39466) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr2-0005Gw-V6 for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:42 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id 94C11407626E; Fri, 22 Apr 2022 11:53:36 +0000 (UTC) Subject: [PATCH 5/9] docs: convert docs/devel/replay page to rst From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:36 +0300 Message-ID: <165062841642.526882.11679119764732049695.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650629752232100001 This patch converts prior .txt replay devel documentation to .rst. Signed-off-by: Pavel Dovgalyuk Reviewed-by: Richard Henderson --- docs/devel/index-tcg.rst | 2 ++ docs/devel/replay.rst | 54 ++++++++++++++++++++++++++++++++++++++++++= ++++ docs/devel/replay.txt | 46 --------------------------------------- 3 files changed, 56 insertions(+), 46 deletions(-) create mode 100644 docs/devel/replay.rst delete mode 100644 docs/devel/replay.txt diff --git a/docs/devel/index-tcg.rst b/docs/devel/index-tcg.rst index 0b0ad12c22..52af5444d6 100644 --- a/docs/devel/index-tcg.rst +++ b/docs/devel/index-tcg.rst @@ -8,8 +8,10 @@ are only implementing things for HW accelerated hypervisor= s. .. toctree:: :maxdepth: 2 =20 + tcg decodetree multi-thread-tcg tcg-icount tcg-plugins + replay diff --git a/docs/devel/replay.rst b/docs/devel/replay.rst new file mode 100644 index 0000000000..dd8bf3b195 --- /dev/null +++ b/docs/devel/replay.rst @@ -0,0 +1,54 @@ +.. + Copyright (c) 2022, ISP RAS + Written by Pavel Dovgalyuk + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Execution Record/Replay +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Record/replay mechanism, that could be enabled through icount mode, expects +the virtual devices to satisfy the following requirements. + +The main idea behind this document is that everything that affects +the guest state during execution in icount mode should be deterministic. + +Timers +------ + +All virtual devices should use virtual clock for timers that change the gu= est +state. Virtual clock is deterministic, therefore such timers are determini= stic +too. + +Virtual devices can also use realtime clock for the events that do not cha= nge +the guest state directly. When the clock ticking should depend on VM execu= tion +speed, use virtual clock with EXTERNAL attribute. It is not deterministic, +but its speed depends on the guest execution. This clock is used by +the virtual devices (e.g., slirp routing device) that lie outside the +replayed guest. + +Bottom halves +------------- + +Bottom half callbacks, that affect the guest state, should be invoked thro= ugh +replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions. +Their invocations are saved in record mode and synchronized with the exist= ing +log in replay mode. + +Saving/restoring the VM state +----------------------------- + +All fields in the device state structure (including virtual timers) +should be restored by loadvm to the same values they had before savevm. + +Avoid accessing other devices' state, because the order of saving/restoring +is not defined. It means that you should not call functions like +'update_irq' in post_load callback. Save everything explicitly to avoid +the dependencies that may make restoring the VM state non-deterministic. + +Stopping the VM +--------------- + +Stopping the guest should not interfere with its state (with the exception +of the network connections, that could be broken by the remote timeouts). +VM can be stopped at any moment of replay by the user. Restarting the VM +after that stop should not break the replay by the unneeded guest state ch= ange. diff --git a/docs/devel/replay.txt b/docs/devel/replay.txt deleted file mode 100644 index e641c35add..0000000000 --- a/docs/devel/replay.txt +++ /dev/null @@ -1,46 +0,0 @@ -Record/replay mechanism, that could be enabled through icount mode, expects -the virtual devices to satisfy the following requirements. - -The main idea behind this document is that everything that affects -the guest state during execution in icount mode should be deterministic. - -Timers -=3D=3D=3D=3D=3D=3D - -All virtual devices should use virtual clock for timers that change the gu= est -state. Virtual clock is deterministic, therefore such timers are determini= stic -too. - -Virtual devices can also use realtime clock for the events that do not cha= nge -the guest state directly. When the clock ticking should depend on VM execu= tion -speed, use virtual clock with EXTERNAL attribute. It is not deterministic, -but its speed depends on the guest execution. This clock is used by -the virtual devices (e.g., slirp routing device) that lie outside the -replayed guest. - -Bottom halves -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Bottom half callbacks, that affect the guest state, should be invoked thro= ugh -replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions. -Their invocations are saved in record mode and synchronized with the exist= ing -log in replay mode. - -Saving/restoring the VM state -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D - -All fields in the device state structure (including virtual timers) -should be restored by loadvm to the same values they had before savevm. - -Avoid accessing other devices' state, because the order of saving/restoring -is not defined. It means that you should not call functions like -'update_irq' in post_load callback. Save everything explicitly to avoid -the dependencies that may make restoring the VM state non-deterministic. - -Stopping the VM -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Stopping the guest should not interfere with its state (with the exception -of the network connections, that could be broken by the remote timeouts). -VM can be stopped at any moment of replay by the user. Restarting the VM -after that stop should not break the replay by the unneeded guest state ch= ange. From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631383179237.03639392191587; Fri, 22 Apr 2022 05:43:03 -0700 (PDT) Received: from localhost ([::1]:35526 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhsco-0007AX-1B for importer2@patchew.org; Fri, 22 Apr 2022 08:43:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58934) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrA-0001rZ-C0 for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:49 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39474) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrr5-0005HB-IM for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:48 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id 00CF84076271; Fri, 22 Apr 2022 11:53:41 +0000 (UTC) Subject: [PATCH 6/9] docs: move replay docs to docs/system/replay.rst From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:41 +0300 Message-ID: <165062842182.526882.9554594086439389972.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631385365100001 This patch adds replay description page, converting prior text from docs/replay.txt. The text was also updated and some sections were moved to devel part of the docs. Signed-off-by: Pavel Dovgalyuk Acked-by: Richard Henderson --- docs/devel/replay.rst | 264 ++++++++++++++++++++++++++++++- docs/replay.txt | 407 --------------------------------------------= ---- docs/system/index.rst | 1=20 docs/system/replay.rst | 237 ++++++++++++++++++++++++++++ 4 files changed, 496 insertions(+), 413 deletions(-) delete mode 100644 docs/replay.txt create mode 100644 docs/system/replay.rst diff --git a/docs/devel/replay.rst b/docs/devel/replay.rst index dd8bf3b195..0244be8b9c 100644 --- a/docs/devel/replay.rst +++ b/docs/devel/replay.rst @@ -1,20 +1,149 @@ .. Copyright (c) 2022, ISP RAS - Written by Pavel Dovgalyuk + Written by Pavel Dovgalyuk and Alex Benn=C3=A9e =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Execution Record/Replay =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -Record/replay mechanism, that could be enabled through icount mode, expects -the virtual devices to satisfy the following requirements. +Core concepts +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Record/replay functions are used for the deterministic replay of qemu +execution. Execution recording writes a non-deterministic events log, which +can be later used for replaying the execution anywhere and for unlimited +number of times. Execution replaying reads the log and replays all +non-deterministic events including external input, hardware clocks, +and interrupts. + +Several parts of QEMU include function calls to make event log recording +and replaying. +Devices' models that have non-deterministic input from external devices we= re +changed to write every external event into the execution log immediately. +E.g. network packets are written into the log when they arrive into the vi= rtual +network adapter. + +All non-deterministic events are coming from these devices. But to +replay them we need to know at which moments they occur. We specify +these moments by counting the number of instructions executed between +every pair of consecutive events. + +Academic papers with description of deterministic replay implementation: + +* `Deterministic Replay of System's Execution with Multi-target QEMU Simul= ator for Dynamic Analysis and Reverse Debugging `_ +* `Don't panic: reverse debugging of kernel drivers `_ + +Modifications of qemu include: + + * wrappers for clock and time functions to save their return values in th= e log + * saving different asynchronous events (e.g. system shutdown) into the log + * synchronization of the bottom halves execution + * synchronization of the threads from thread pool + * recording/replaying user input (mouse, keyboard, and microphone) + * adding internal checkpoints for cpu and io synchronization + * network filter for recording and replaying the packets + * block driver for making block layer deterministic + * serial port input record and replay + * recording of random numbers obtained from the external sources + +Instruction counting +-------------------- + +QEMU should work in icount mode to use record/replay feature. icount was +designed to allow deterministic execution in absence of external inputs +of the virtual machine. We also use icount to control the occurrence of the +non-deterministic events. The number of instructions elapsed from the last= event +is written to the log while recording the execution. In replay mode we +can predict when to inject that event using the instruction counter. + +Locking and thread synchronisation +---------------------------------- + +Previously the synchronisation of the main thread and the vCPU thread +was ensured by the holding of the BQL. However the trend has been to +reduce the time the BQL was held across the system including under TCG +system emulation. As it is important that batches of events are kept +in sequence (e.g. expiring timers and checkpoints in the main thread +while instruction checkpoints are written by the vCPU thread) we need +another lock to keep things in lock-step. This role is now handled by +the replay_mutex_lock. It used to be held only for each event being +written but now it is held for a whole execution period. This results +in a deterministic ping-pong between the two main threads. + +As the BQL is now a finer grained lock than the replay_lock it is almost +certainly a bug, and a source of deadlocks, to take the +replay_mutex_lock while the BQL is held. This is enforced by an assert. +While the unlocks are usually in the reverse order, this is not +necessary; you can drop the replay_lock while holding the BQL, without +doing a more complicated unlock_iothread/replay_unlock/lock_iothread +sequence. + +Checkpoints +----------- + +Replaying the execution of virtual machine is bound by sources of +non-determinism. These are inputs from clock and peripheral devices, +and QEMU thread scheduling. Thread scheduling affect on processing events +from timers, asynchronous input-output, and bottom halves. =20 -The main idea behind this document is that everything that affects +Invocations of timers are coupled with clock reads and changing the state +of the virtual machine. Reads produce non-deterministic data taken from +host clock. And VM state changes should preserve their order. Their relati= ve +order in replay mode must replicate the order of callbacks in record mode. +To preserve this order we use checkpoints. When a specific clock is proces= sed +in record mode we save to the log special "checkpoint" event. +Checkpoints here do not refer to virtual machine snapshots. They are just +record/replay events used for synchronization. + +QEMU in replay mode will try to invoke timers processing in random moment +of time. That's why we do not process a group of timers until the checkpoi= nt +event will be read from the log. Such an event allows synchronizing CPU +execution and timer events. + +Two other checkpoints govern the "warping" of the virtual clock. +While the virtual machine is idle, the virtual clock increments at +1 ns per *real time* nanosecond. This is done by setting up a timer +(called the warp timer) on the virtual real time clock, so that the +timer fires at the next deadline of the virtual clock; the virtual clock +is then incremented (which is called "warping" the virtual clock) as +soon as the timer fires or the CPUs need to go out of the idle state. +Two functions are used for this purpose; because these actions change +virtual machine state and must be deterministic, each of them creates a +checkpoint. ``icount_start_warp_timer`` checks if the CPUs are idle and if= so +starts accounting real time to virtual clock. ``icount_account_warp_timer`` +is called when the CPUs get an interrupt or when the warp timer fires, +and it warps the virtual clock by the amount of real time that has passed +since ``icount_start_warp_timer``. + +Virtual devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Record/replay mechanism, that could be enabled through icount mode, expects +the virtual devices to satisfy the following requirement: +everything that affects the guest state during execution in icount mode should be deterministic. =20 Timers ------ =20 +Timers are used to execute callbacks from different subsystems of QEMU +at the specified moments of time. There are several kinds of timers: + + * Real time clock. Based on host time and used only for callbacks that + do not change the virtual machine state. For this reason real time + clock and timers does not affect deterministic replay at all. + * Virtual clock. These timers run only during the emulation. In icount + mode virtual clock value is calculated using executed instructions coun= ter. + That is why it is completely deterministic and does not have to be reco= rded. + * Host clock. This clock is used by device models that simulate real time + sources (e.g. real time clock chip). Host clock is the one of the sourc= es + of non-determinism. Host clock read operations should be logged to + make the execution deterministic. + * Virtual real time clock. This clock is similar to real time clock but + it is used only for increasing virtual clock while virtual machine is + sleeping. Due to its nature it is also non-deterministic as the host cl= ock + and has to be logged too. + All virtual devices should use virtual clock for timers that change the gu= est state. Virtual clock is deterministic, therefore such timers are determini= stic too. @@ -26,14 +155,50 @@ but its speed depends on the guest execution. This clo= ck is used by the virtual devices (e.g., slirp routing device) that lie outside the replayed guest. =20 +Block devices +------------- + +Block devices record/replay module (``blkreplay``) intercepts calls of +bdrv coroutine functions at the top of block drivers stack. + +All block completion operations are added to the queue in the coroutines. +When the queue is flushed the information about processed requests +is recorded to the log. In replay phase the queue is matched with +events read from the log. Therefore block devices requests are processed +deterministically. + Bottom halves ------------- =20 Bottom half callbacks, that affect the guest state, should be invoked thro= ugh -replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions. +``replay_bh_schedule_event`` or ``replay_bh_schedule_oneshot_event`` funct= ions. Their invocations are saved in record mode and synchronized with the exist= ing log in replay mode. =20 +Disk I/O events are completely deterministic in our model, because +in both record and replay modes we start virtual machine from the same +disk state. But callbacks that virtual disk controller uses for reading and +writing the disk may occur at different moments of time in record and repl= ay +modes. + +Reading and writing requests are created by CPU thread of QEMU. Later these +requests proceed to block layer which creates "bottom halves". Bottom +halves consist of callback and its parameters. They are processed when +main loop locks the global mutex. These locks are not synchronized with +replaying process because main loop also processes the events that do not +affect the virtual machine state (like user interaction with monitor). + +That is why we had to implement saving and replaying bottom halves callbac= ks +synchronously to the CPU execution. When the callback is about to execute +it is added to the queue in the replay module. This queue is written to the +log when its callbacks are executed. In replay mode callbacks are not proc= essed +until the corresponding event is read from the events log file. + +Sometimes the block layer uses asynchronous callbacks for its internal pur= poses +(like reading or writing VM snapshots or disk image cluster tables). In th= is +case bottom halves are not marked as "replayable" and do not saved +into the log. + Saving/restoring the VM state ----------------------------- =20 @@ -42,7 +207,7 @@ should be restored by loadvm to the same values they had= before savevm. =20 Avoid accessing other devices' state, because the order of saving/restoring is not defined. It means that you should not call functions like -'update_irq' in post_load callback. Save everything explicitly to avoid +``update_irq`` in ``post_load`` callback. Save everything explicitly to av= oid the dependencies that may make restoring the VM state non-deterministic. =20 Stopping the VM @@ -52,3 +217,90 @@ Stopping the guest should not interfere with its state = (with the exception of the network connections, that could be broken by the remote timeouts). VM can be stopped at any moment of replay by the user. Restarting the VM after that stop should not break the replay by the unneeded guest state ch= ange. + +Replay log format +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Record/replay log consists of the header and the sequence of execution +events. The header includes 4-byte replay version id and 8-byte reserved +field. Version is updated every time replay log format changes to prevent +using replay log created by another build of qemu. + +The sequence of the events describes virtual machine state changes. +It includes all non-deterministic inputs of VM, synchronization marks and +instruction counts used to correctly inject inputs at replay. + +Synchronization marks (checkpoints) are used for synchronizing qemu threads +that perform operations with virtual hardware. These operations may change +system's state (e.g., change some register or generate interrupt) and +therefore should execute synchronously with CPU thread. + +Every event in the log includes 1-byte event id and optional arguments. +When argument is an array, it is stored as 4-byte array length +and corresponding number of bytes with data. +Here is the list of events that are written into the log: + + - EVENT_INSTRUCTION. Instructions executed since last event. Followed by: + + - 4-byte number of executed instructions. + + - EVENT_INTERRUPT. Used to synchronize interrupt processing. + - EVENT_EXCEPTION. Used to synchronize exception handling. + - EVENT_ASYNC. This is a group of events. When such an event is generated, + it is stored in the queue and processed in icount_account_warp_timer(). + Every such event has it's own id from the following list: + + - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes + callbacks that affect virtual machine state, but normally called + asynchronously. Followed by: + + - 8-byte operation id. + + - REPLAY_ASYNC_EVENT_INPUT. Input device event. Contains + parameters of keyboard and mouse input operations + (key press/release, mouse pointer movement). Followed by: + + - 9-16 bytes depending of input event. + + - REPLAY_ASYNC_EVENT_INPUT_SYNC. Internal input synchronization event. + - REPLAY_ASYNC_EVENT_CHAR_READ. Character (e.g., serial port) device = input + initiated by the sender. Followed by: + + - 1-byte character device id. + - Array with bytes were read. + + - REPLAY_ASYNC_EVENT_BLOCK. Block device operation. Used to synchroni= ze + operations with disk and flash drives with CPU. Followed by: + + - 8-byte operation id. + + - REPLAY_ASYNC_EVENT_NET. Incoming network packet. Followed by: + + - 1-byte network adapter id. + - 4-byte packet flags. + - Array with packet bytes. + + - EVENT_SHUTDOWN. Occurs when user sends shutdown event to qemu, + e.g., by closing the window. + - EVENT_CHAR_WRITE. Used to synchronize character output operations. Foll= owed by: + + - 4-byte output function return value. + - 4-byte offset in the output array. + + - EVENT_CHAR_READ_ALL. Used to synchronize character input operations, + initiated by qemu. Followed by: + + - Array with bytes that were read. + + - EVENT_CHAR_READ_ALL_ERROR. Unsuccessful character input operation, + initiated by qemu. Followed by: + + - 4-byte error code. + + - EVENT_CLOCK + clock_id. Group of events for host clock read operations.= Followed by: + + - 8-byte clock value. + + - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of + CPU, internal threads, and asynchronous input events. + - EVENT_END. Last event in the log. diff --git a/docs/replay.txt b/docs/replay.txt deleted file mode 100644 index 6c9fdff09d..0000000000 --- a/docs/replay.txt +++ /dev/null @@ -1,407 +0,0 @@ -Copyright (c) 2010-2015 Institute for System Programming - of the Russian Academy of Sciences. - -This work is licensed under the terms of the GNU GPL, version 2 or later. -See the COPYING file in the top-level directory. - -Record/replay -------------- - -Record/replay functions are used for the deterministic replay of qemu exec= ution. -Execution recording writes a non-deterministic events log, which can be la= ter -used for replaying the execution anywhere and for unlimited number of time= s. -It also supports checkpointing for faster rewind to the specific replay mo= ment. -Execution replaying reads the log and replays all non-deterministic events -including external input, hardware clocks, and interrupts. - -Deterministic replay has the following features: - * Deterministically replays whole system execution and all contents of - the memory, state of the hardware devices, clocks, and screen of the VM. - * Writes execution log into the file for later replaying for multiple tim= es - on different machines. - * Supports i386, x86_64, and Arm hardware platforms. - * Performs deterministic replay of all operations with keyboard and mouse - input devices. - -Usage of the record/replay: - * First, record the execution with the following command line: - qemu-system-i386 \ - -icount shift=3D7,rr=3Drecord,rrfile=3Dreplay.bin \ - -drive file=3Ddisk.qcow2,if=3Dnone,snapshot,id=3Dimg-direct \ - -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-blkre= play \ - -device ide-hd,drive=3Dimg-blkreplay \ - -netdev user,id=3Dnet1 -device rtl8139,netdev=3Dnet1 \ - -object filter-replay,id=3Dreplay,netdev=3Dnet1 - * After recording, you can replay it by using another command line: - qemu-system-i386 \ - -icount shift=3D7,rr=3Dreplay,rrfile=3Dreplay.bin \ - -drive file=3Ddisk.qcow2,if=3Dnone,snapshot,id=3Dimg-direct \ - -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-blkre= play \ - -device ide-hd,drive=3Dimg-blkreplay \ - -netdev user,id=3Dnet1 -device rtl8139,netdev=3Dnet1 \ - -object filter-replay,id=3Dreplay,netdev=3Dnet1 - The only difference with recording is changing the rr option - from record to replay. - * Block device images are not actually changed in the recording mode, - because all of the changes are written to the temporary overlay file. - This behavior is enabled by using blkreplay driver. It should be used - for every enabled block device, as described in 'Block devices' section. - * '-net none' option should be specified when network is not used, - because QEMU adds network card by default. When network is needed, - it should be configured explicitly with replay filter, as described - in 'Network devices' section. - * Interaction with audio devices and serial ports are recorded and replay= ed - automatically when such devices are enabled. - -Academic papers with description of deterministic replay implementation: -http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.ht= ml -http://dl.acm.org/citation.cfm?id=3D2786805.2803179 - -Modifications of qemu include: - * wrappers for clock and time functions to save their return values in th= e log - * saving different asynchronous events (e.g. system shutdown) into the log - * synchronization of the bottom halves execution - * synchronization of the threads from thread pool - * recording/replaying user input (mouse, keyboard, and microphone) - * adding internal checkpoints for cpu and io synchronization - * network filter for recording and replaying the packets - * block driver for making block layer deterministic - * serial port input record and replay - * recording of random numbers obtained from the external sources - -Locking and thread synchronisation ----------------------------------- - -Previously the synchronisation of the main thread and the vCPU thread -was ensured by the holding of the BQL. However the trend has been to -reduce the time the BQL was held across the system including under TCG -system emulation. As it is important that batches of events are kept -in sequence (e.g. expiring timers and checkpoints in the main thread -while instruction checkpoints are written by the vCPU thread) we need -another lock to keep things in lock-step. This role is now handled by -the replay_mutex_lock. It used to be held only for each event being -written but now it is held for a whole execution period. This results -in a deterministic ping-pong between the two main threads. - -As the BQL is now a finer grained lock than the replay_lock it is almost -certainly a bug, and a source of deadlocks, to take the -replay_mutex_lock while the BQL is held. This is enforced by an assert. -While the unlocks are usually in the reverse order, this is not -necessary; you can drop the replay_lock while holding the BQL, without -doing a more complicated unlock_iothread/replay_unlock/lock_iothread -sequence. - -Non-deterministic events ------------------------- - -Our record/replay system is based on saving and replaying non-deterministic -events (e.g. keyboard input) and simulating deterministic ones (e.g. readi= ng -from HDD or memory of the VM). Saving only non-deterministic events makes -log file smaller and simulation faster. - -The following non-deterministic data from peripheral devices is saved into -the log: mouse and keyboard input, network packets, audio controller input, -serial port input, and hardware clocks (they are non-deterministic -too, because their values are taken from the host machine). Inputs from -simulated hardware, memory of VM, software interrupts, and execution of -instructions are not saved into the log, because they are deterministic and -can be replayed by simulating the behavior of virtual machine starting from -initial state. - -We had to solve three tasks to implement deterministic replay: recording -non-deterministic events, replaying non-deterministic events, and checking -that there is no divergence between record and replay modes. - -We changed several parts of QEMU to make event log recording and replaying. -Devices' models that have non-deterministic input from external devices we= re -changed to write every external event into the execution log immediately. -E.g. network packets are written into the log when they arrive into the vi= rtual -network adapter. - -All non-deterministic events are coming from these devices. But to -replay them we need to know at which moments they occur. We specify -these moments by counting the number of instructions executed between -every pair of consecutive events. - -Instruction counting --------------------- - -QEMU should work in icount mode to use record/replay feature. icount was -designed to allow deterministic execution in absence of external inputs -of the virtual machine. We also use icount to control the occurrence of the -non-deterministic events. The number of instructions elapsed from the last= event -is written to the log while recording the execution. In replay mode we -can predict when to inject that event using the instruction counter. - -Timers ------- - -Timers are used to execute callbacks from different subsystems of QEMU -at the specified moments of time. There are several kinds of timers: - * Real time clock. Based on host time and used only for callbacks that - do not change the virtual machine state. For this reason real time - clock and timers does not affect deterministic replay at all. - * Virtual clock. These timers run only during the emulation. In icount - mode virtual clock value is calculated using executed instructions coun= ter. - That is why it is completely deterministic and does not have to be reco= rded. - * Host clock. This clock is used by device models that simulate real time - sources (e.g. real time clock chip). Host clock is the one of the sourc= es - of non-determinism. Host clock read operations should be logged to - make the execution deterministic. - * Virtual real time clock. This clock is similar to real time clock but - it is used only for increasing virtual clock while virtual machine is - sleeping. Due to its nature it is also non-deterministic as the host cl= ock - and has to be logged too. - -Checkpoints ------------ - -Replaying of the execution of virtual machine is bound by sources of -non-determinism. These are inputs from clock and peripheral devices, -and QEMU thread scheduling. Thread scheduling affect on processing events -from timers, asynchronous input-output, and bottom halves. - -Invocations of timers are coupled with clock reads and changing the state -of the virtual machine. Reads produce non-deterministic data taken from -host clock. And VM state changes should preserve their order. Their relati= ve -order in replay mode must replicate the order of callbacks in record mode. -To preserve this order we use checkpoints. When a specific clock is proces= sed -in record mode we save to the log special "checkpoint" event. -Checkpoints here do not refer to virtual machine snapshots. They are just -record/replay events used for synchronization. - -QEMU in replay mode will try to invoke timers processing in random moment -of time. That's why we do not process a group of timers until the checkpoi= nt -event will be read from the log. Such an event allows synchronizing CPU -execution and timer events. - -Two other checkpoints govern the "warping" of the virtual clock. -While the virtual machine is idle, the virtual clock increments at -1 ns per *real time* nanosecond. This is done by setting up a timer -(called the warp timer) on the virtual real time clock, so that the -timer fires at the next deadline of the virtual clock; the virtual clock -is then incremented (which is called "warping" the virtual clock) as -soon as the timer fires or the CPUs need to go out of the idle state. -Two functions are used for this purpose; because these actions change -virtual machine state and must be deterministic, each of them creates a -checkpoint. icount_start_warp_timer checks if the CPUs are idle and if so -starts accounting real time to virtual clock. icount_account_warp_timer -is called when the CPUs get an interrupt or when the warp timer fires, -and it warps the virtual clock by the amount of real time that has passed -since icount_start_warp_timer. - -Bottom halves -------------- - -Disk I/O events are completely deterministic in our model, because -in both record and replay modes we start virtual machine from the same -disk state. But callbacks that virtual disk controller uses for reading and -writing the disk may occur at different moments of time in record and repl= ay -modes. - -Reading and writing requests are created by CPU thread of QEMU. Later these -requests proceed to block layer which creates "bottom halves". Bottom -halves consist of callback and its parameters. They are processed when -main loop locks the global mutex. These locks are not synchronized with -replaying process because main loop also processes the events that do not -affect the virtual machine state (like user interaction with monitor). - -That is why we had to implement saving and replaying bottom halves callbac= ks -synchronously to the CPU execution. When the callback is about to execute -it is added to the queue in the replay module. This queue is written to the -log when its callbacks are executed. In replay mode callbacks are not proc= essed -until the corresponding event is read from the events log file. - -Sometimes the block layer uses asynchronous callbacks for its internal pur= poses -(like reading or writing VM snapshots or disk image cluster tables). In th= is -case bottom halves are not marked as "replayable" and do not saved -into the log. - -Block devices -------------- - -Block devices record/replay module intercepts calls of -bdrv coroutine functions at the top of block drivers stack. -To record and replay block operations the drive must be configured -as following: - -drive file=3Ddisk.qcow2,if=3Dnone,snapshot,id=3Dimg-direct - -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-blkreplay - -device ide-hd,drive=3Dimg-blkreplay - -blkreplay driver should be inserted between disk image and virtual driver -controller. Therefore all disk requests may be recorded and replayed. - -All block completion operations are added to the queue in the coroutines. -Queue is flushed at checkpoints and information about processed requests -is recorded to the log. In replay phase the queue is matched with -events read from the log. Therefore block devices requests are processed -deterministically. - -Snapshotting ------------- - -New VM snapshots may be created in replay mode. They can be used later -to recover the desired VM state. All VM states created in replay mode -are associated with the moment of time in the replay scenario. -After recovering the VM state replay will start from that position. - -Default starting snapshot name may be specified with icount field -rrsnapshot as follows: - -icount shift=3D7,rr=3Drecord,rrfile=3Dreplay.bin,rrsnapshot=3Dsnapshot_n= ame - -This snapshot is created at start of recording and restored at start -of replaying. It also can be loaded while replaying to roll back -the execution. - -'snapshot' flag of the disk image must be removed to save the snapshots -in the overlay (or original image) instead of using the temporary overlay. - -drive file=3Ddisk.ovl,if=3Dnone,id=3Dimg-direct - -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-blkreplay - -device ide-hd,drive=3Dimg-blkreplay - -Use QEMU monitor to create additional snapshots. 'savevm ' command -created the snapshot and 'loadvm ' restores it. To prevent corruption -of the original disk image, use overlay files linked to the original image= s. -Therefore all new snapshots (including the starting one) will be saved in -overlays and the original image remains unchanged. - -When you need to use snapshots with diskless virtual machine, -it must be started with 'orphan' qcow2 image. This image will be used -for storing VM snapshots. Here is the example of the command line for this: - - qemu-system-i386 -icount shift=3D3,rr=3Dreplay,rrfile=3Drecord.bin,rrsna= pshot=3Dinit \ - -net none -drive file=3Dempty.qcow2,if=3Dnone,id=3Drr - -empty.qcow2 drive does not connected to any virtual block device and used -for VM snapshots only. - -Network devices ---------------- - -Record and replay for network interactions is performed with the network f= ilter. -Each backend must have its own instance of the replay filter as follows: - -netdev user,id=3Dnet1 -device rtl8139,netdev=3Dnet1 - -object filter-replay,id=3Dreplay,netdev=3Dnet1 - -Replay network filter is used to record and replay network packets. While -recording the virtual machine this filter puts all packets coming from -the outer world into the log. In replay mode packets from the log are -injected into the network device. All interactions with network backend -in replay mode are disabled. - -Audio devices -------------- - -Audio data is recorded and replay automatically. The command line for reco= rding -and replaying must contain identical specifications of audio hardware, e.g= .: - -soundhw ac97 - -Serial ports ------------- - -Serial ports input is recorded and replay automatically. The command lines -for recording and replaying must contain identical number of ports in reco= rd -and replay modes, but their backends may differ. -E.g., '-serial stdio' in record mode, and '-serial null' in replay mode. - -Reverse debugging ------------------ - -Reverse debugging allows "executing" the program in reverse direction. -GDB remote protocol supports "reverse step" and "reverse continue" -commands. The first one steps single instruction backwards in time, -and the second one finds the last breakpoint in the past. - -Recorded executions may be used to enable reverse debugging. QEMU can't -execute the code in backwards direction, but can load a snapshot and -replay forward to find the desired position or breakpoint. - -The following GDB commands are supported: - - reverse-stepi (or rsi) - step one instruction backwards - - reverse-continue (or rc) - find last breakpoint in the past - -Reverse step loads the nearest snapshot and replays the execution until -the required instruction is met. - -Reverse continue may include several passes of examining the execution -between the snapshots. Each of the passes include the following steps: - 1. loading the snapshot - 2. replaying to examine the breakpoints - 3. if breakpoint or watchpoint was met - - loading the snapshot again - - replaying to the required breakpoint - 4. else - - proceeding to the p.1 with the earlier snapshot - -Therefore usage of the reverse debugging requires at least one snapshot -created in advance. This can be done by omitting 'snapshot' option -for the block drives and adding 'rrsnapshot' for both record and replay -command lines. -See the "Snapshotting" section to learn more about running record/replay -and creating the snapshot in these modes. - -Replay log format ------------------ - -Record/replay log consists of the header and the sequence of execution -events. The header includes 4-byte replay version id and 8-byte reserved -field. Version is updated every time replay log format changes to prevent -using replay log created by another build of qemu. - -The sequence of the events describes virtual machine state changes. -It includes all non-deterministic inputs of VM, synchronization marks and -instruction counts used to correctly inject inputs at replay. - -Synchronization marks (checkpoints) are used for synchronizing qemu threads -that perform operations with virtual hardware. These operations may change -system's state (e.g., change some register or generate interrupt) and -therefore should execute synchronously with CPU thread. - -Every event in the log includes 1-byte event id and optional arguments. -When argument is an array, it is stored as 4-byte array length -and corresponding number of bytes with data. -Here is the list of events that are written into the log: - - - EVENT_INSTRUCTION. Instructions executed since last event. - Argument: 4-byte number of executed instructions. - - EVENT_INTERRUPT. Used to synchronize interrupt processing. - - EVENT_EXCEPTION. Used to synchronize exception handling. - - EVENT_ASYNC. This is a group of events. When such an event is generated, - it is stored in the queue and processed in icount_account_warp_timer(). - Every such event has it's own id from the following list: - - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes - callbacks that affect virtual machine state, but normally called - asynchronously. - Argument: 8-byte operation id. - - REPLAY_ASYNC_EVENT_INPUT. Input device event. Contains - parameters of keyboard and mouse input operations - (key press/release, mouse pointer movement). - Arguments: 9-16 bytes depending of input event. - - REPLAY_ASYNC_EVENT_INPUT_SYNC. Internal input synchronization event. - - REPLAY_ASYNC_EVENT_CHAR_READ. Character (e.g., serial port) device = input - initiated by the sender. - Arguments: 1-byte character device id. - Array with bytes were read. - - REPLAY_ASYNC_EVENT_BLOCK. Block device operation. Used to synchroni= ze - operations with disk and flash drives with CPU. - Argument: 8-byte operation id. - - REPLAY_ASYNC_EVENT_NET. Incoming network packet. - Arguments: 1-byte network adapter id. - 4-byte packet flags. - Array with packet bytes. - - EVENT_SHUTDOWN. Occurs when user sends shutdown event to qemu, - e.g., by closing the window. - - EVENT_CHAR_WRITE. Used to synchronize character output operations. - Arguments: 4-byte output function return value. - 4-byte offset in the output array. - - EVENT_CHAR_READ_ALL. Used to synchronize character input operations, - initiated by qemu. - Argument: Array with bytes that were read. - - EVENT_CHAR_READ_ALL_ERROR. Unsuccessful character input operation, - initiated by qemu. - Argument: 4-byte error code. - - EVENT_CLOCK + clock_id. Group of events for host clock read operations. - Argument: 8-byte clock value. - - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of - CPU, internal threads, and asynchronous input events. - - EVENT_END. Last event in the log. diff --git a/docs/system/index.rst b/docs/system/index.rst index 23e30e26e5..e3695649c5 100644 --- a/docs/system/index.rst +++ b/docs/system/index.rst @@ -27,6 +27,7 @@ or Hypervisor.Framework. secrets authz gdb + replay managed-startup bootindex cpu-hotplug diff --git a/docs/system/replay.rst b/docs/system/replay.rst new file mode 100644 index 0000000000..3105327423 --- /dev/null +++ b/docs/system/replay.rst @@ -0,0 +1,237 @@ +.. _replay: + +.. + Copyright (c) 2010-2022 Institute for System Programming + of the Russian Academy of Sciences. + + This work is licensed under the terms of the GNU GPL, version 2 or lat= er. + See the COPYING file in the top-level directory. + +Record/replay +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Record/replay functions are used for the deterministic replay of qemu exec= ution. +Execution recording writes a non-deterministic events log, which can be la= ter +used for replaying the execution anywhere and for unlimited number of time= s. +It also supports checkpointing for faster rewind to the specific replay mo= ment. +Execution replaying reads the log and replays all non-deterministic events +including external input, hardware clocks, and interrupts. + +Deterministic replay has the following features: + + * Deterministically replays whole system execution and all contents of + the memory, state of the hardware devices, clocks, and screen of the VM. + * Writes execution log into the file for later replaying for multiple tim= es + on different machines. + * Supports i386, x86_64, ARM, AArch64, Risc-V, MIPS, MIPS64, S390X, Alpha, + PowerPC, PowerPC64, M68000, Microblaze, OpenRISC, Nios II, SPARC, + and Xtensa hardware platforms. + * Performs deterministic replay of all operations with keyboard and mouse + input devices, serial ports, and network. + +Usage of the record/replay: + + * First, record the execution with the following command line: + + .. parsed-literal:: + |qemu_system| \\ + -icount shift=3Dauto,rr=3Drecord,rrfile=3Dreplay.bin \\ + -drive file=3Ddisk.qcow2,if=3Dnone,snapshot,id=3Dimg-direct \\ + -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-bl= kreplay \\ + -device ide-hd,drive=3Dimg-blkreplay \\ + -netdev user,id=3Dnet1 -device rtl8139,netdev=3Dnet1 \\ + -object filter-replay,id=3Dreplay,netdev=3Dnet1 + + * After recording, you can replay it by using another command line: + + .. parsed-literal:: + |qemu_system| \\ + -icount shift=3Dauto,rr=3Dreplay,rrfile=3Dreplay.bin \\ + -drive file=3Ddisk.qcow2,if=3Dnone,snapshot,id=3Dimg-direct \\ + -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-bl= kreplay \\ + -device ide-hd,drive=3Dimg-blkreplay \\ + -netdev user,id=3Dnet1 -device rtl8139,netdev=3Dnet1 \\ + -object filter-replay,id=3Dreplay,netdev=3Dnet1 + + The only difference with recording is changing the rr option + from record to replay. + * Block device images are not actually changed in the recording mode, + because all of the changes are written to the temporary overlay file. + This behavior is enabled by using blkreplay driver. It should be used + for every enabled block device, as described in :ref:`block-label` sect= ion. + * ``-net none`` option should be specified when network is not used, + because QEMU adds network card by default. When network is needed, + it should be configured explicitly with replay filter, as described + in :ref:`network-label` section. + * Interaction with audio devices and serial ports are recorded and replay= ed + automatically when such devices are enabled. + +Core idea +--------- + +Record/replay system is based on saving and replaying non-deterministic +events (e.g. keyboard input) and simulating deterministic ones (e.g. readi= ng +from HDD or memory of the VM). Saving only non-deterministic events makes +log file smaller and simulation faster. + +The following non-deterministic data from peripheral devices is saved into +the log: mouse and keyboard input, network packets, audio controller input, +serial port input, and hardware clocks (they are non-deterministic +too, because their values are taken from the host machine). Inputs from +simulated hardware, memory of VM, software interrupts, and execution of +instructions are not saved into the log, because they are deterministic and +can be replayed by simulating the behavior of virtual machine starting from +initial state. + +Instruction counting +-------------------- + +QEMU should work in icount mode to use record/replay feature. icount was +designed to allow deterministic execution in absence of external inputs +of the virtual machine. Record/replay feature is enabled through ``-icount= `` +command-line option, making possible deterministic execution of the machin= e, +interacting with user or network. + +.. _block-label: + +Block devices +------------- + +Block devices record/replay module intercepts calls of +bdrv coroutine functions at the top of block drivers stack. +To record and replay block operations the drive must be configured +as following: + +.. parsed-literal:: + -drive file=3Ddisk.qcow2,if=3Dnone,snapshot,id=3Dimg-direct + -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-blkrep= lay + -device ide-hd,drive=3Dimg-blkreplay + +blkreplay driver should be inserted between disk image and virtual driver +controller. Therefore all disk requests may be recorded and replayed. + +.. _snapshotting-label: + +Snapshotting +------------ + +New VM snapshots may be created in replay mode. They can be used later +to recover the desired VM state. All VM states created in replay mode +are associated with the moment of time in the replay scenario. +After recovering the VM state replay will start from that position. + +Default starting snapshot name may be specified with icount field +rrsnapshot as follows: + +.. parsed-literal:: + -icount shift=3Dauto,rr=3Drecord,rrfile=3Dreplay.bin,rrsnapshot=3Dsnap= shot_name + +This snapshot is created at start of recording and restored at start +of replaying. It also can be loaded while replaying to roll back +the execution. + +``snapshot`` flag of the disk image must be removed to save the snapshots +in the overlay (or original image) instead of using the temporary overlay. + +.. parsed-literal:: + -drive file=3Ddisk.ovl,if=3Dnone,id=3Dimg-direct + -drive driver=3Dblkreplay,if=3Dnone,image=3Dimg-direct,id=3Dimg-blkrep= lay + -device ide-hd,drive=3Dimg-blkreplay + +Use QEMU monitor to create additional snapshots. ``savevm `` command +created the snapshot and ``loadvm `` restores it. To prevent corrupt= ion +of the original disk image, use overlay files linked to the original image= s. +Therefore all new snapshots (including the starting one) will be saved in +overlays and the original image remains unchanged. + +When you need to use snapshots with diskless virtual machine, +it must be started with "orphan" qcow2 image. This image will be used +for storing VM snapshots. Here is the example of the command line for this: + +.. parsed-literal:: + |qemu_system| \\ + -icount shift=3Dauto,rr=3Dreplay,rrfile=3Drecord.bin,rrsnapshot=3Din= it \\ + -net none -drive file=3Dempty.qcow2,if=3Dnone,id=3Drr + +``empty.qcow2`` drive does not connected to any virtual block device and u= sed +for VM snapshots only. + +.. _network-label: + +Network devices +--------------- + +Record and replay for network interactions is performed with the network f= ilter. +Each backend must have its own instance of the replay filter as follows: + +.. parsed-literal:: + -netdev user,id=3Dnet1 -device rtl8139,netdev=3Dnet1 + -object filter-replay,id=3Dreplay,netdev=3Dnet1 + +Replay network filter is used to record and replay network packets. While +recording the virtual machine this filter puts all packets coming from +the outer world into the log. In replay mode packets from the log are +injected into the network device. All interactions with network backend +in replay mode are disabled. + +Audio devices +------------- + +Audio data is recorded and replay automatically. The command line for reco= rding +and replaying must contain identical specifications of audio hardware, e.g= .: + +.. parsed-literal:: + -soundhw ac97 + +Serial ports +------------ + +Serial ports input is recorded and replay automatically. The command lines +for recording and replaying must contain identical number of ports in reco= rd +and replay modes, but their backends may differ. +E.g., ``-serial stdio`` in record mode, and ``-serial null`` in replay mod= e. + +Reverse debugging +----------------- + +Reverse debugging allows "executing" the program in reverse direction. +GDB remote protocol supports "reverse step" and "reverse continue" +commands. The first one steps single instruction backwards in time, +and the second one finds the last breakpoint in the past. + +Recorded executions may be used to enable reverse debugging. QEMU can't +execute the code in backwards direction, but can load a snapshot and +replay forward to find the desired position or breakpoint. + +The following GDB commands are supported: + + - ``reverse-stepi`` (or ``rsi``) - step one instruction backwards + - ``reverse-continue`` (or ``rc``) - find last breakpoint in the past + +Reverse step loads the nearest snapshot and replays the execution until +the required instruction is met. + +Reverse continue may include several passes of examining the execution +between the snapshots. Each of the passes include the following steps: + + #. loading the snapshot + #. replaying to examine the breakpoints + #. if breakpoint or watchpoint was met + + * loading the snapshot again + * replaying to the required breakpoint + + #. else + + * proceeding to the p.1 with the earlier snapshot + +Therefore usage of the reverse debugging requires at least one snapshot +created. This can be done by omitting ``snapshot`` option +for the block drives and adding ``rrsnapshot`` for both record and replay +command lines. +See the :ref:`snapshotting-label` section to learn more about running reco= rd/replay +and creating the snapshot in these modes. + +When ``rrsnapshot`` is not used, then snapshot named ``start_debugging`` +created in temporary overlay. This allows using reverse debugging, but with +temporary snapshots (existing within the session). From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631464139885.4952212655672; Fri, 22 Apr 2022 05:44:24 -0700 (PDT) Received: from localhost ([::1]:38930 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhse7-0001Tc-2x for importer2@patchew.org; Fri, 22 Apr 2022 08:44:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58974) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrD-0001te-Ox for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:55 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39498) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrB-0005He-CA for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:50 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id 6DA9C4076274; Fri, 22 Apr 2022 11:53:47 +0000 (UTC) Subject: [PATCH 7/9] tests/avocado: update replay_linux test From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:47 +0300 Message-ID: <165062842725.526882.9389797416793788047.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631465552100001 This patch updates replay_linux test to make it compatible with new LinuxTest class. Signed-off-by: Pavel Dovgalyuk --- tests/avocado/replay_linux.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py index 15953f9e49..1099b5647f 100644 --- a/tests/avocado/replay_linux.py +++ b/tests/avocado/replay_linux.py @@ -32,9 +32,16 @@ class ReplayLinux(LinuxTest): bus =3D 'ide' =20 def setUp(self): - super(ReplayLinux, self).setUp() + # LinuxTest does many replay-incompatible things, but includes + # useful methods. Do not setup LinuxTest here and just + # call some functions. + super(LinuxTest, self).setUp() + self._set_distro() self.boot_path =3D self.download_boot() - self.cloudinit_path =3D self.prepare_cloudinit() + self.phone_server =3D cloudinit.PhoneHomeServer(('0.0.0.0', 0), + self.name) + ssh_pubkey, self.ssh_key =3D self.set_up_existing_ssh_keys() + self.cloudinit_path =3D self.prepare_cloudinit(ssh_pubkey) =20 def vm_add_disk(self, vm, path, id, device): bus_string =3D '' @@ -50,7 +57,9 @@ def launch_and_wait(self, record, args, shift): vm =3D self.get_vm() vm.add_args('-smp', '1') vm.add_args('-m', '1024') - vm.add_args('-object', 'filter-replay,id=3Dreplay,netdev=3Dhub0por= t0') + vm.add_args('-netdev', 'user,id=3Dvnet,hostfwd=3D:127.0.0.1:0-:22', + '-device', 'virtio-net,netdev=3Dvnet') + vm.add_args('-object', 'filter-replay,id=3Dreplay,netdev=3Dvnet') if args: vm.add_args(*args) self.vm_add_disk(vm, self.boot_path, 0, self.hdd) @@ -75,8 +84,8 @@ def launch_and_wait(self, record, args, shift): stop_check=3D(lambda : not vm.is_runni= ng())) console_drainer.start() if record: - cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port= ), - self.name) + while not self.phone_server.instance_phoned_back: + self.phone_server.handle_request() vm.shutdown() logger.info('finished the recording with log size %s bytes' % os.path.getsize(replay_path)) From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631691905518.6606025711394; Fri, 22 Apr 2022 05:48:11 -0700 (PDT) Received: from localhost ([::1]:45476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhshm-0006my-QG for importer2@patchew.org; Fri, 22 Apr 2022 08:48:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59006) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrL-0001wI-1Z for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:59 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39534) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrH-0005I2-IC for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:53:58 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id DD4434076B22; Fri, 22 Apr 2022 11:53:52 +0000 (UTC) Subject: [PATCH 8/9] tests/avocado: add replay Linux tests for virtio machine From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:52 +0300 Message-ID: <165062843267.526882.9799586788310156794.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631692969100001 This patch adds two tests for replaying Linux boot process on x86_64 virtio platform. Signed-off-by: Pavel Dovgalyuk --- tests/avocado/replay_linux.py | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py index 1099b5647f..3bb1bc8816 100644 --- a/tests/avocado/replay_linux.py +++ b/tests/avocado/replay_linux.py @@ -123,3 +123,29 @@ def test_pc_q35(self): :avocado: tags=3Dmachine:q35 """ self.run_rr(shift=3D3) + +@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout') +class ReplayLinuxX8664Virtio(ReplayLinux): + """ + :avocado: tags=3Darch:x86_64 + :avocado: tags=3Dvirtio + :avocado: tags=3Daccel:tcg + """ + + hdd =3D 'virtio-blk-pci' + cd =3D 'virtio-blk-pci' + bus =3D None + + chksum =3D 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c= 5c27a0' + + def test_pc_i440fx(self): + """ + :avocado: tags=3Dmachine:pc + """ + self.run_rr(shift=3D1) + + def test_pc_q35(self): + """ + :avocado: tags=3Dmachine:q35 + """ + self.run_rr(shift=3D3) From nobody Thu May 2 22:13:05 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ispras.ru Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1650631908065655.2322069628148; Fri, 22 Apr 2022 05:51:48 -0700 (PDT) Received: from localhost ([::1]:51888 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nhslH-0003xx-1N for importer2@patchew.org; Fri, 22 Apr 2022 08:51:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59034) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrO-00021F-80 for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:54:02 -0400 Received: from mail.ispras.ru ([83.149.199.84]:39554) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nhrrM-0005IR-HT for qemu-devel@nongnu.org; Fri, 22 Apr 2022 07:54:01 -0400 Received: from [127.0.1.1] (unknown [85.142.117.226]) by mail.ispras.ru (Postfix) with ESMTPSA id 4B4194076B21; Fri, 22 Apr 2022 11:53:58 +0000 (UTC) Subject: [PATCH 9/9] tests/avocado: add replay Linux test for Aarch64 machines From: Pavel Dovgalyuk To: qemu-devel@nongnu.org Date: Fri, 22 Apr 2022 14:53:58 +0300 Message-ID: <165062843812.526882.14512197869729908793.stgit@pasha-ThinkPad-X280> In-Reply-To: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> References: <165062838915.526882.13230207960407998257.stgit@pasha-ThinkPad-X280> User-Agent: StGit/0.23 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=83.149.199.84; envelope-from=pavel.dovgalyuk@ispras.ru; helo=mail.ispras.ru X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pavel.dovgalyuk@ispras.ru, philmd@redhat.com, wrampazz@redhat.com, crosa@redhat.com, pbonzini@redhat.com, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1650631909219100001 This patch adds two tests for replaying Linux boot process on Aarch64 platform. Signed-off-by: Pavel Dovgalyuk --- tests/avocado/replay_linux.py | 41 +++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 41 insertions(+) diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py index 3bb1bc8816..e1f9981a34 100644 --- a/tests/avocado/replay_linux.py +++ b/tests/avocado/replay_linux.py @@ -13,6 +13,7 @@ import time =20 from avocado import skipUnless +from avocado_qemu import BUILD_DIR from avocado.utils import cloudinit from avocado.utils import network from avocado.utils import vmimage @@ -149,3 +150,43 @@ def test_pc_q35(self): :avocado: tags=3Dmachine:q35 """ self.run_rr(shift=3D3) + +@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout') +class ReplayLinuxAarch64(ReplayLinux): + """ + :avocado: tags=3Daccel:tcg + :avocado: tags=3Darch:aarch64 + :avocado: tags=3Dmachine:virt + :avocado: tags=3Dcpu:max + """ + + chksum =3D '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf= 16fe49' + + hdd =3D 'virtio-blk-device' + cd =3D 'virtio-blk-device' + bus =3D None + + def get_common_args(self): + return ('-bios', + os.path.join(BUILD_DIR, 'pc-bios', 'edk2-aarch64-code.fd'), + "-cpu", "max,lpa2=3Doff", + '-device', 'virtio-rng-pci,rng=3Drng0', + '-object', 'rng-builtin,id=3Drng0') + + def test_virt_gicv2(self): + """ + :avocado: tags=3Dmachine:gic-version=3D2 + """ + + self.run_rr(shift=3D3, + args=3D(*self.get_common_args(), + "-machine", "virt,gic-version=3D2")) + + def test_virt_gicv3(self): + """ + :avocado: tags=3Dmachine:gic-version=3D3 + """ + + self.run_rr(shift=3D3, + args=3D(*self.get_common_args(), + "-machine", "virt,gic-version=3D3"))