From nobody Tue May 7 21:57:21 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623805881078174.83348216956665; Tue, 15 Jun 2021 18:11:21 -0700 (PDT) Received: from localhost ([::1]:55272 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK5Q-0000zU-2B for importer2@patchew.org; Tue, 15 Jun 2021 21:11:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51344) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltK2q-0006z9-1M for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:40 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.223]:51218 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK2i-0005zt-4u for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:39 -0400 Received: from clientip-202.80.192.38?logid-6d78197d793c4667afc4678d3a3390f7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 312942800C9; Wed, 16 Jun 2021 09:08:26 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id 87312b0dfe6e40d597b3135324201afc for qemu-devel@nongnu.org; Wed Jun 16 09:08:26 2021 HMM_SOURCE_IP: 172.18.0.218:55640.1447589156 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 87312b0dfe6e40d597b3135324201afc X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v4 1/6] KVM: introduce dirty_pages and kvm_dirty_ring_enabled Date: Wed, 16 Jun 2021 09:12:27 +0800 Message-Id: <32c3266ebfd75aaf3f2caaf6b5fad0fb6dd8a96a.1623804189.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) dirty_pages is used to calculate dirtyrate via dirty ring, when enabled, kvm-reaper will increase the dirty pages after gfns being dirtied. kvm_dirty_ring_enabled shows if kvm-reaper is working. dirtyrate thread could use it to check if measurement can base on dirty ring feature. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- accel/kvm/kvm-all.c | 7 +++++++ include/hw/core/cpu.h | 1 + include/sysemu/kvm.h | 1 + 3 files changed, 9 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index e5b10dd..e0e88a2 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -469,6 +469,7 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp) cpu->kvm_fd =3D ret; cpu->kvm_state =3D s; cpu->vcpu_dirty =3D true; + cpu->dirty_pages =3D 0; =20 mmap_size =3D kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0); if (mmap_size < 0) { @@ -743,6 +744,7 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CP= UState *cpu) count++; } cpu->kvm_fetch_index =3D fetch; + cpu->dirty_pages +=3D count; =20 return count; } @@ -2293,6 +2295,11 @@ bool kvm_vcpu_id_is_valid(int vcpu_id) return vcpu_id >=3D 0 && vcpu_id < kvm_max_vcpu_id(s); } =20 +bool kvm_dirty_ring_enabled(void) +{ + return kvm_state->kvm_dirty_ring_size ? true : false; +} + static int kvm_init(MachineState *ms) { MachineClass *mc =3D MACHINE_GET_CLASS(ms); diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 4e0ea68..80fcb1d 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -374,6 +374,7 @@ struct CPUState { struct kvm_run *kvm_run; struct kvm_dirty_gfn *kvm_dirty_gfns; uint32_t kvm_fetch_index; + uint64_t dirty_pages; =20 /* Used for events with 'vcpu' and *without* the 'disabled' properties= */ DECLARE_BITMAP(trace_dstate_delayed, CPU_TRACE_DSTATE_MAX_EVENTS); diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index a1ab1ee..7b22aeb 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -547,4 +547,5 @@ bool kvm_cpu_check_are_resettable(void); =20 bool kvm_arch_cpu_check_are_resettable(void); =20 +bool kvm_dirty_ring_enabled(void); #endif --=20 1.8.3.1 From nobody Tue May 7 21:57:21 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623806221673591.8351903225574; Tue, 15 Jun 2021 18:17:01 -0700 (PDT) Received: from localhost ([::1]:42746 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKAu-0003Hx-Lo for importer2@patchew.org; Tue, 15 Jun 2021 21:17:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51374) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltK30-000726-8l for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:50 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.223]:51325 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK2m-00064j-HV for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:49 -0400 Received: from clientip-202.80.192.38?logid-6d78197d793c4667afc4678d3a3390f7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 3C1302800AC; Wed, 16 Jun 2021 09:08:31 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id 33d7b114a3c745d59bf35548c6ec1791 for qemu-devel@nongnu.org; Wed Jun 16 09:08:33 2021 HMM_SOURCE_IP: 172.18.0.218:55640.1447589156 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 33d7b114a3c745d59bf35548c6ec1791 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v4 2/6] memory: make global_dirty_log a bitmask Date: Wed, 16 Jun 2021 09:12:28 +0800 Message-Id: <3848028c3029e2e870286c84c2b3e9a93967567e.1623804189.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) dirty rate measurement may start or stop dirty logging during calculation. this conflict with migration because stop dirty log make migration leave dirty pages out then that'll be a problem. make global_dirty_log a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty log aims for, migration or dirty rate. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- include/exec/memory.h | 14 +++++++++++--- migration/ram.c | 8 ++++---- softmmu/memory.c | 31 ++++++++++++++++++++++--------- softmmu/trace-events | 1 + 4 files changed, 38 insertions(+), 16 deletions(-) diff --git a/include/exec/memory.h b/include/exec/memory.h index b114f54..e31eef2 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -55,7 +55,11 @@ static inline void fuzz_dma_read_cb(size_t addr, } #endif =20 -extern bool global_dirty_log; +/* what is the purpose of current dirty log, migration or dirty rate ? */ +#define GLOBAL_DIRTY_MIGRATION (1U << 0) +#define GLOBAL_DIRTY_DIRTY_RATE (1U << 1) + +extern unsigned int global_dirty_log; =20 typedef struct MemoryRegionOps MemoryRegionOps; =20 @@ -2099,13 +2103,17 @@ void memory_listener_unregister(MemoryListener *lis= tener); =20 /** * memory_global_dirty_log_start: begin dirty logging for all regions + * + * @flags: purpose of starting dirty log, migration or dirty rate */ -void memory_global_dirty_log_start(void); +void memory_global_dirty_log_start(unsigned int flags); =20 /** * memory_global_dirty_log_stop: end dirty logging for all regions + * + * @flags: purpose of stopping dirty log, migration or dirty rate */ -void memory_global_dirty_log_stop(void); +void memory_global_dirty_log_stop(unsigned int flags); =20 void mtree_info(bool flatview, bool dispatch_tree, bool owner, bool disabl= ed); =20 diff --git a/migration/ram.c b/migration/ram.c index 60ea913..9ce31af 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2190,7 +2190,7 @@ static void ram_save_cleanup(void *opaque) /* caller have hold iothread lock or is in a bh, so there is * no writing race against the migration bitmap */ - memory_global_dirty_log_stop(); + memory_global_dirty_log_stop(GLOBAL_DIRTY_MIGRATION); } =20 RAMBLOCK_FOREACH_NOT_IGNORED(block) { @@ -2652,7 +2652,7 @@ static void ram_init_bitmaps(RAMState *rs) ram_list_init_bitmaps(); /* We don't use dirty log with background snapshots */ if (!migrate_background_snapshot()) { - memory_global_dirty_log_start(); + memory_global_dirty_log_start(GLOBAL_DIRTY_MIGRATION); migration_bitmap_sync_precopy(rs); } } @@ -3393,7 +3393,7 @@ void colo_incoming_start_dirty_log(void) /* Discard this dirty bitmap record */ bitmap_zero(block->bmap, block->max_length >> TARGET_PAGE_BITS= ); } - memory_global_dirty_log_start(); + memory_global_dirty_log_start(GLOBAL_DIRTY_MIGRATION); } ram_state->migration_dirty_pages =3D 0; qemu_mutex_unlock_ramlist(); @@ -3405,7 +3405,7 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 - memory_global_dirty_log_stop(); + memory_global_dirty_log_stop(GLOBAL_DIRTY_MIGRATION); RAMBLOCK_FOREACH_NOT_IGNORED(block) { g_free(block->bmap); block->bmap =3D NULL; diff --git a/softmmu/memory.c b/softmmu/memory.c index c19b0be..d74172f 100644 --- a/softmmu/memory.c +++ b/softmmu/memory.c @@ -39,7 +39,7 @@ static unsigned memory_region_transaction_depth; static bool memory_region_update_pending; static bool ioeventfd_update_pending; -bool global_dirty_log; +unsigned int global_dirty_log; =20 static QTAILQ_HEAD(, MemoryListener) memory_listeners =3D QTAILQ_HEAD_INITIALIZER(memory_listeners); @@ -2659,14 +2659,19 @@ void memory_global_after_dirty_log_sync(void) =20 static VMChangeStateEntry *vmstate_change; =20 -void memory_global_dirty_log_start(void) +void memory_global_dirty_log_start(unsigned int flags) { if (vmstate_change) { qemu_del_vm_change_state_handler(vmstate_change); vmstate_change =3D NULL; } =20 - global_dirty_log =3D true; +#define GLOBAL_DIRTY_MASK (0x3) + assert(flags && !(flags & (~GLOBAL_DIRTY_MASK))); + assert(global_dirty_log ^ flags); + global_dirty_log |=3D flags; + + trace_global_dirty_changed(global_dirty_log); =20 MEMORY_LISTENER_CALL_GLOBAL(log_global_start, Forward); =20 @@ -2676,9 +2681,12 @@ void memory_global_dirty_log_start(void) memory_region_transaction_commit(); } =20 -static void memory_global_dirty_log_do_stop(void) +static void memory_global_dirty_log_do_stop(unsigned int flags) { - global_dirty_log =3D false; + assert(flags && !(flags & (~GLOBAL_DIRTY_MASK))); + global_dirty_log &=3D ~flags; + + trace_global_dirty_changed(global_dirty_log); =20 /* Refresh DIRTY_MEMORY_MIGRATION bit. */ memory_region_transaction_begin(); @@ -2691,8 +2699,10 @@ static void memory_global_dirty_log_do_stop(void) static void memory_vm_change_state_handler(void *opaque, bool running, RunState state) { + int *flags =3D opaque; if (running) { - memory_global_dirty_log_do_stop(); + memory_global_dirty_log_do_stop(*flags); + g_free(opaque); =20 if (vmstate_change) { qemu_del_vm_change_state_handler(vmstate_change); @@ -2701,18 +2711,21 @@ static void memory_vm_change_state_handler(void *op= aque, bool running, } } =20 -void memory_global_dirty_log_stop(void) +void memory_global_dirty_log_stop(unsigned int flags) { + int *opaque =3D NULL; if (!runstate_is_running()) { if (vmstate_change) { return; } + opaque =3D g_malloc0(sizeof(opaque)); + *opaque =3D flags; vmstate_change =3D qemu_add_vm_change_state_handler( - memory_vm_change_state_handler, NULL); + memory_vm_change_state_handler, opaque); return; } =20 - memory_global_dirty_log_do_stop(); + memory_global_dirty_log_do_stop(flags); } =20 static void listener_add_address_space(MemoryListener *listener, diff --git a/softmmu/trace-events b/softmmu/trace-events index 5262828..4431f7f 100644 --- a/softmmu/trace-events +++ b/softmmu/trace-events @@ -18,6 +18,7 @@ memory_region_ram_device_write(int cpu_index, void *mr, u= int64_t addr, uint64_t flatview_new(void *view, void *root) "%p (root %p)" flatview_destroy(void *view, void *root) "%p (root %p)" flatview_destroy_rcu(void *view, void *root) "%p (root %p)" +global_dirty_changed(unsigned int bitmask) "bitmask 0x%"PRIx32 =20 # vl.c vm_state_notify(int running, int reason, const char *reason_str) "running = %d reason %d (%s)" --=20 1.8.3.1 From nobody Tue May 7 21:57:21 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623805888789838.9358530156399; Tue, 15 Jun 2021 18:11:28 -0700 (PDT) Received: from localhost ([::1]:55908 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK5X-0001Or-Jt for importer2@patchew.org; Tue, 15 Jun 2021 21:11:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51416) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltK35-00075y-QO for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:56 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.223]:51425 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK32-00066q-2c for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:55 -0400 Received: from clientip-202.80.192.38?logid-6d78197d793c4667afc4678d3a3390f7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 367282800BC; Wed, 16 Jun 2021 09:08:38 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id 02bb3b5364f14f7590a6f31c058cab8a for qemu-devel@nongnu.org; Wed Jun 16 09:08:47 2021 HMM_SOURCE_IP: 172.18.0.218:55640.1447589156 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 02bb3b5364f14f7590a6f31c058cab8a X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v4 3/6] migration/dirtyrate: introduce struct and adjust DirtyRateStat Date: Wed, 16 Jun 2021 09:12:29 +0800 Message-Id: <5b9aeb59a3bda83a374608ba44f92e5d242ee6da.1623804189.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) introduce "DirtyRateMeasureMode" to specify what method should be used to calculate dirty rate, introduce "DirtyRateVcpu" to store dirty rate fore each vcpu. use union to store stat data of specific mode Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- migration/dirtyrate.c | 47 +++++++++++++++++++++++++++-------------------- migration/dirtyrate.h | 19 ++++++++++++++++--- qapi/migration.json | 30 ++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 23 deletions(-) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index 320c56b..14ffac9 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -88,33 +88,43 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) return info; } =20 -static void init_dirtyrate_stat(int64_t start_time, int64_t calc_time, - uint64_t sample_pages) +static void init_dirtyrate_stat(int64_t start_time, + struct DirtyRateConfig config) { - DirtyStat.total_dirty_samples =3D 0; - DirtyStat.total_sample_count =3D 0; - DirtyStat.total_block_mem_MB =3D 0; DirtyStat.dirty_rate =3D -1; DirtyStat.start_time =3D start_time; - DirtyStat.calc_time =3D calc_time; - DirtyStat.sample_pages =3D sample_pages; + DirtyStat.calc_time =3D config.sample_period_seconds; + DirtyStat.sample_pages =3D config.sample_pages_per_gigabytes; + + switch (config.mode) { + case DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING: + DirtyStat.page_sampling.total_dirty_samples =3D 0; + DirtyStat.page_sampling.total_sample_count =3D 0; + DirtyStat.page_sampling.total_block_mem_MB =3D 0; + break; + case DIRTY_RATE_MEASURE_MODE_DIRTY_RING: + DirtyStat.dirty_ring.nvcpu =3D -1; + DirtyStat.dirty_ring.rates =3D NULL; + default: + break; + } } =20 static void update_dirtyrate_stat(struct RamblockDirtyInfo *info) { - DirtyStat.total_dirty_samples +=3D info->sample_dirty_count; - DirtyStat.total_sample_count +=3D info->sample_pages_count; + DirtyStat.page_sampling.total_dirty_samples +=3D info->sample_dirty_co= unt; + DirtyStat.page_sampling.total_sample_count +=3D info->sample_pages_cou= nt; /* size of total pages in MB */ - DirtyStat.total_block_mem_MB +=3D (info->ramblock_pages * - TARGET_PAGE_SIZE) >> 20; + DirtyStat.page_sampling.total_block_mem_MB +=3D (info->ramblock_pages * + TARGET_PAGE_SIZE) >> 20; } =20 static void update_dirtyrate(uint64_t msec) { uint64_t dirtyrate; - uint64_t total_dirty_samples =3D DirtyStat.total_dirty_samples; - uint64_t total_sample_count =3D DirtyStat.total_sample_count; - uint64_t total_block_mem_MB =3D DirtyStat.total_block_mem_MB; + uint64_t total_dirty_samples =3D DirtyStat.page_sampling.total_dirty_s= amples; + uint64_t total_sample_count =3D DirtyStat.page_sampling.total_sample_c= ount; + uint64_t total_block_mem_MB =3D DirtyStat.page_sampling.total_block_me= m_MB; =20 dirtyrate =3D total_dirty_samples * total_block_mem_MB * 1000 / (total_sample_count * msec); @@ -327,7 +337,7 @@ static bool compare_page_hash_info(struct RamblockDirty= Info *info, update_dirtyrate_stat(block_dinfo); } =20 - if (DirtyStat.total_sample_count =3D=3D 0) { + if (DirtyStat.page_sampling.total_sample_count =3D=3D 0) { return false; } =20 @@ -372,8 +382,6 @@ void *get_dirtyrate_thread(void *arg) struct DirtyRateConfig config =3D *(struct DirtyRateConfig *)arg; int ret; int64_t start_time; - int64_t calc_time; - uint64_t sample_pages; =20 ret =3D dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTA= RTED, DIRTY_RATE_STATUS_MEASURING); @@ -383,9 +391,7 @@ void *get_dirtyrate_thread(void *arg) } =20 start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000; - calc_time =3D config.sample_period_seconds; - sample_pages =3D config.sample_pages_per_gigabytes; - init_dirtyrate_stat(start_time, calc_time, sample_pages); + init_dirtyrate_stat(start_time, config); =20 calculate_dirtyrate(config); =20 @@ -442,6 +448,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sa= mple_pages, =20 config.sample_period_seconds =3D calc_time; config.sample_pages_per_gigabytes =3D sample_pages; + config.mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; qemu_thread_create(&thread, "get_dirtyrate", get_dirtyrate_thread, (void *)&config, QEMU_THREAD_DETACHED); } diff --git a/migration/dirtyrate.h b/migration/dirtyrate.h index e1fd290..69d4c5b 100644 --- a/migration/dirtyrate.h +++ b/migration/dirtyrate.h @@ -43,6 +43,7 @@ struct DirtyRateConfig { uint64_t sample_pages_per_gigabytes; /* sample pages per GB */ int64_t sample_period_seconds; /* time duration between two sampling */ + DirtyRateMeasureMode mode; /* mode of dirtyrate measurement */ }; =20 /* @@ -58,17 +59,29 @@ struct RamblockDirtyInfo { uint32_t *hash_result; /* array of hash result for sampled pages */ }; =20 +typedef struct SampleVMStat { + uint64_t total_dirty_samples; /* total dirty sampled page */ + uint64_t total_sample_count; /* total sampled pages */ + uint64_t total_block_mem_MB; /* size of total sampled pages in MB */ +} SampleVMStat; + +typedef struct VcpuStat { + int nvcpu; /* number of vcpu */ + DirtyRateVcpu *rates; /* array of dirty rate for each vcpu */ +} VcpuStat; + /* * Store calculation statistics for each measure. */ struct DirtyRateStat { - uint64_t total_dirty_samples; /* total dirty sampled page */ - uint64_t total_sample_count; /* total sampled pages */ - uint64_t total_block_mem_MB; /* size of total sampled pages in MB */ int64_t dirty_rate; /* dirty rate in MB/s */ int64_t start_time; /* calculation start time in units of second */ int64_t calc_time; /* time duration of two sampling in units of second= */ uint64_t sample_pages; /* sample pages per GB */ + union { + SampleVMStat page_sampling; + VcpuStat dirty_ring; + }; }; =20 void *get_dirtyrate_thread(void *arg); diff --git a/qapi/migration.json b/qapi/migration.json index 1124a2d..7395305 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1709,6 +1709,21 @@ 'data': { 'device-id': 'str' } } =20 ## +# @DirtyRateVcpu: +# +# Dirty rate of vcpu. +# +# @id: vcpu index. +# +# @dirty-rate: dirty rate. +# +# Since: 6.1 +# +## +{ 'struct': 'DirtyRateVcpu', + 'data': { 'id': 'int', 'dirty-rate': 'int64' } } + +## # @DirtyRateStatus: # # An enumeration of dirtyrate status. @@ -1726,6 +1741,21 @@ 'data': [ 'unstarted', 'measuring', 'measured'] } =20 ## +# @DirtyRateMeasureMode: +# +# An enumeration of mode of measuring dirtyrate. +# +# @page-sampling: calculate dirtyrate by sampling pages. +# +# @dirty-ring: calculate dirtyrate by via dirty ring. +# +# Since: 6.1 +# +## +{ 'enum': 'DirtyRateMeasureMode', + 'data': [ 'none', 'page-sampling', 'dirty-ring'] } + +## # @DirtyRateInfo: # # Information about current dirty page rate of vm. --=20 1.8.3.1 From nobody Tue May 7 21:57:21 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623805888636844.259321572365; Tue, 15 Jun 2021 18:11:28 -0700 (PDT) Received: from localhost ([::1]:55956 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK5X-0001Qz-Gd for importer2@patchew.org; Tue, 15 Jun 2021 21:11:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51414) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltK34-00075v-Pb for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:56 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.223]:51548 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK33-0006BV-2K for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:08:54 -0400 Received: from clientip-202.80.192.38?logid-6d78197d793c4667afc4678d3a3390f7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id D4B112800C5; Wed, 16 Jun 2021 09:08:52 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id 45491c6819e14c1bb0e74c234f8fd3d0 for qemu-devel@nongnu.org; Wed Jun 16 09:08:51 2021 HMM_SOURCE_IP: 172.18.0.218:55640.1447589156 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 45491c6819e14c1bb0e74c234f8fd3d0 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v4 4/6] migration/dirtyrate: adjust order of registering thread Date: Wed, 16 Jun 2021 09:12:30 +0800 Message-Id: <5df3137cdd9661186e1df0ca0e2bb049e3c30308.1623804189.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) registering get_dirtyrate thread in advance so that both page-sampling and dirty-ring mode can be covered. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Reviewed-by: Peter Xu --- migration/dirtyrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index 14ffac9..b97f6a5 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -351,7 +351,6 @@ static void calculate_dirtyrate(struct DirtyRateConfig = config) int64_t msec =3D 0; int64_t initial_time; =20 - rcu_register_thread(); rcu_read_lock(); initial_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); if (!record_ramblock_hash_info(&block_dinfo, config, &block_count)) { @@ -374,7 +373,6 @@ static void calculate_dirtyrate(struct DirtyRateConfig = config) out: rcu_read_unlock(); free_ramblock_dirty_info(block_dinfo, block_count); - rcu_unregister_thread(); } =20 void *get_dirtyrate_thread(void *arg) @@ -382,6 +380,7 @@ void *get_dirtyrate_thread(void *arg) struct DirtyRateConfig config =3D *(struct DirtyRateConfig *)arg; int ret; int64_t start_time; + rcu_register_thread(); =20 ret =3D dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTA= RTED, DIRTY_RATE_STATUS_MEASURING); @@ -400,6 +399,8 @@ void *get_dirtyrate_thread(void *arg) if (ret =3D=3D -1) { error_report("change dirtyrate state failed."); } + + rcu_unregister_thread(); return NULL; } =20 --=20 1.8.3.1 From nobody Tue May 7 21:57:21 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623806374682896.0102889585576; Tue, 15 Jun 2021 18:19:34 -0700 (PDT) Received: from localhost ([::1]:51082 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKDN-0000Vr-Jr for importer2@patchew.org; Tue, 15 Jun 2021 21:19:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51446) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltK3C-00077G-DT for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:09:03 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.223]:51594 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK3A-0006F0-6K for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:09:02 -0400 Received: from clientip-202.80.192.38?logid-6d78197d793c4667afc4678d3a3390f7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 863452800BC; Wed, 16 Jun 2021 09:08:57 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id f124fa0c9e48417eb7959d86d170a6ec for qemu-devel@nongnu.org; Wed Jun 16 09:08:56 2021 HMM_SOURCE_IP: 172.18.0.218:55640.1447589156 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: f124fa0c9e48417eb7959d86d170a6ec X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v4 5/6] migration/dirtyrate: move init step of calculation to main thread Date: Wed, 16 Jun 2021 09:12:31 +0800 Message-Id: <0e5ece3a7a2c235611e398086334a908bc63c4de.1623804189.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) since main thread could "query dirty rate" at any time, then it's better to move init step into main thead so that synchronization overhead of dirty stat can be reduced. since not sure whether "only one QMP iothread" will still keep true forever, always introduce a mutex and protect dirty stat. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- migration/dirtyrate.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index b97f6a5..d7b41bd 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -26,6 +26,8 @@ =20 static int CalculatingState =3D DIRTY_RATE_STATUS_UNSTARTED; static struct DirtyRateStat DirtyStat; +static QemuMutex dirtyrate_lock; +static DirtyRateMeasureMode dirtyrate_mode =3D DIRTY_RATE_MEASURE_MODE_NON= E; =20 static int64_t set_sample_page_period(int64_t msec, int64_t initial_time) { @@ -70,6 +72,7 @@ static int dirtyrate_set_state(int *state, int old_state,= int new_state) =20 static struct DirtyRateInfo *query_dirty_rate_info(void) { + qemu_mutex_lock(&dirtyrate_lock); int64_t dirty_rate =3D DirtyStat.dirty_rate; struct DirtyRateInfo *info =3D g_malloc0(sizeof(DirtyRateInfo)); =20 @@ -83,6 +86,8 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) info->calc_time =3D DirtyStat.calc_time; info->sample_pages =3D DirtyStat.sample_pages; =20 + qemu_mutex_unlock(&dirtyrate_lock); + trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState)); =20 return info; @@ -91,6 +96,7 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) static void init_dirtyrate_stat(int64_t start_time, struct DirtyRateConfig config) { + qemu_mutex_lock(&dirtyrate_lock); DirtyStat.dirty_rate =3D -1; DirtyStat.start_time =3D start_time; DirtyStat.calc_time =3D config.sample_period_seconds; @@ -108,6 +114,12 @@ static void init_dirtyrate_stat(int64_t start_time, default: break; } + qemu_mutex_unlock(&dirtyrate_lock); +} + +static void cleanup_dirtyrate_stat(struct DirtyRateConfig config) +{ + /* TODO */ } =20 static void update_dirtyrate_stat(struct RamblockDirtyInfo *info) @@ -379,7 +391,6 @@ void *get_dirtyrate_thread(void *arg) { struct DirtyRateConfig config =3D *(struct DirtyRateConfig *)arg; int ret; - int64_t start_time; rcu_register_thread(); =20 ret =3D dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTA= RTED, @@ -389,9 +400,6 @@ void *get_dirtyrate_thread(void *arg) return NULL; } =20 - start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000; - init_dirtyrate_stat(start_time, config); - calculate_dirtyrate(config); =20 ret =3D dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_MEASU= RING, @@ -410,6 +418,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sa= mple_pages, static struct DirtyRateConfig config; QemuThread thread; int ret; + int64_t start_time; =20 /* * If the dirty rate is already being measured, don't attempt to start. @@ -450,6 +459,23 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_s= ample_pages, config.sample_period_seconds =3D calc_time; config.sample_pages_per_gigabytes =3D sample_pages; config.mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + + if (unlikely(dirtyrate_mode =3D=3D DIRTY_RATE_MEASURE_MODE_NONE)) { + /* first time to calculate dirty rate */ + qemu_mutex_init(&dirtyrate_lock); + } + + cleanup_dirtyrate_stat(config); + + /* + * update dirty rate mode so that we can figure out what mode has + * been used in last calculation + **/ + dirtyrate_mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + + start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000; + init_dirtyrate_stat(start_time, config); + qemu_thread_create(&thread, "get_dirtyrate", get_dirtyrate_thread, (void *)&config, QEMU_THREAD_DETACHED); } --=20 1.8.3.1 From nobody Tue May 7 21:57:21 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623805892029287.24624637988995; Tue, 15 Jun 2021 18:11:32 -0700 (PDT) Received: from localhost ([::1]:56230 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK5a-0001bE-Ve for importer2@patchew.org; Tue, 15 Jun 2021 21:11:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51480) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltK3G-00079H-B9 for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:09:06 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.223]:51643 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltK3D-0006Gz-6l for qemu-devel@nongnu.org; Tue, 15 Jun 2021 21:09:06 -0400 Received: from clientip-202.80.192.38?logid-6d78197d793c4667afc4678d3a3390f7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 398432800AC; Wed, 16 Jun 2021 09:09:01 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id 9d03674cd2aa4fc187c364b5d45d9e1e for qemu-devel@nongnu.org; Wed Jun 16 09:09:01 2021 HMM_SOURCE_IP: 172.18.0.218:55640.1447589156 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 9d03674cd2aa4fc187c364b5d45d9e1e X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v4 6/6] migration/dirtyrate: implement dirty-ring dirtyrate calculation Date: Wed, 16 Jun 2021 09:12:32 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) use dirty ring feature to implement dirtyrate calculation. introduce mode option in qmp calc_dirty_rate to specify what method should be used when calculating dirtyrate, either page-sampling or dirty-ring should be passed. introduce "dirty_ring:-r" option in hmp calc_dirty_rate to indicate dirty ring method should be used for calculation. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- hmp-commands.hx | 7 +- migration/dirtyrate.c | 183 +++++++++++++++++++++++++++++++++++++++++++++= +--- migration/trace-events | 2 + qapi/migration.json | 16 ++++- 4 files changed, 195 insertions(+), 13 deletions(-) diff --git a/hmp-commands.hx b/hmp-commands.hx index 8e45bce..f7fc9d7 100644 --- a/hmp-commands.hx +++ b/hmp-commands.hx @@ -1738,8 +1738,9 @@ ERST =20 { .name =3D "calc_dirty_rate", - .args_type =3D "second:l,sample_pages_per_GB:l?", - .params =3D "second [sample_pages_per_GB]", - .help =3D "start a round of guest dirty rate measurement", + .args_type =3D "dirty_ring:-r,second:l,sample_pages_per_GB:l?", + .params =3D "[-r] second [sample_pages_per_GB]", + .help =3D "start a round of guest dirty rate measurement (us= ing -d to" + "\n\t\t\t specify dirty ring as the method of calcul= ation)", .cmd =3D hmp_calc_dirty_rate, }, diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index d7b41bd..7c9515b 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -16,6 +16,7 @@ #include "cpu.h" #include "exec/ramblock.h" #include "qemu/rcu_queue.h" +#include "qemu/main-loop.h" #include "qapi/qapi-commands-migration.h" #include "ram.h" #include "trace.h" @@ -23,11 +24,20 @@ #include "monitor/hmp.h" #include "monitor/monitor.h" #include "qapi/qmp/qdict.h" +#include "sysemu/kvm.h" +#include "sysemu/runstate.h" +#include "exec/memory.h" + +typedef struct DirtyPageRecord { + uint64_t start_pages; + uint64_t end_pages; +} DirtyPageRecord; =20 static int CalculatingState =3D DIRTY_RATE_STATUS_UNSTARTED; static struct DirtyRateStat DirtyStat; static QemuMutex dirtyrate_lock; static DirtyRateMeasureMode dirtyrate_mode =3D DIRTY_RATE_MEASURE_MODE_NON= E; +static DirtyPageRecord *dirty_pages; =20 static int64_t set_sample_page_period(int64_t msec, int64_t initial_time) { @@ -72,9 +82,11 @@ static int dirtyrate_set_state(int *state, int old_state= , int new_state) =20 static struct DirtyRateInfo *query_dirty_rate_info(void) { + int i; qemu_mutex_lock(&dirtyrate_lock); int64_t dirty_rate =3D DirtyStat.dirty_rate; struct DirtyRateInfo *info =3D g_malloc0(sizeof(DirtyRateInfo)); + DirtyRateVcpuList *head =3D NULL, **tail =3D &head; =20 if (qatomic_read(&CalculatingState) =3D=3D DIRTY_RATE_STATUS_MEASURED)= { info->has_dirty_rate =3D true; @@ -85,9 +97,22 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) info->start_time =3D DirtyStat.start_time; info->calc_time =3D DirtyStat.calc_time; info->sample_pages =3D DirtyStat.sample_pages; + info->mode =3D dirtyrate_mode; + + if (dirtyrate_mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) { + /* set sample_pages with 0 to indicate page sampling isn't enabled= */ + info->sample_pages =3D 0; + info->has_vcpu_dirty_rate =3D true; + for (i =3D 0; i < DirtyStat.dirty_ring.nvcpu; i++) { + DirtyRateVcpu *rate =3D g_malloc0(sizeof(DirtyRateVcpu)); + rate->id =3D DirtyStat.dirty_ring.rates[i].id; + rate->dirty_rate =3D DirtyStat.dirty_ring.rates[i].dirty_rate; + QAPI_LIST_APPEND(tail, rate); + } + info->vcpu_dirty_rate =3D head; + } =20 qemu_mutex_unlock(&dirtyrate_lock); - trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState)); =20 return info; @@ -119,7 +144,11 @@ static void init_dirtyrate_stat(int64_t start_time, =20 static void cleanup_dirtyrate_stat(struct DirtyRateConfig config) { - /* TODO */ + /* last calc-dirty-rate qmp use dirty ring mode */ + if (dirtyrate_mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) { + free(DirtyStat.dirty_ring.rates); + DirtyStat.dirty_ring.rates =3D NULL; + } } =20 static void update_dirtyrate_stat(struct RamblockDirtyInfo *info) @@ -356,7 +385,97 @@ static bool compare_page_hash_info(struct RamblockDirt= yInfo *info, return true; } =20 -static void calculate_dirtyrate(struct DirtyRateConfig config) +static void record_dirtypages(CPUState *cpu, bool start) +{ + if (start) { + dirty_pages[cpu->cpu_index].start_pages =3D cpu->dirty_pages; + } else { + dirty_pages[cpu->cpu_index].end_pages =3D cpu->dirty_pages; + } +} + +static void dirtyrate_global_dirty_log_start(void) +{ + qemu_mutex_lock_iothread(); + memory_global_dirty_log_start(GLOBAL_DIRTY_DIRTY_RATE); + qemu_mutex_unlock_iothread(); +} + +static void dirtyrate_global_dirty_log_stop(void) +{ + qemu_mutex_lock_iothread(); + memory_global_dirty_log_stop(GLOBAL_DIRTY_DIRTY_RATE); + qemu_mutex_unlock_iothread(); +} + +static int64_t do_calculate_dirtyrate_vcpu(int idx) +{ + uint64_t memory_size_MB; + int64_t time_s; + uint64_t start_pages =3D dirty_pages[idx].start_pages; + uint64_t end_pages =3D dirty_pages[idx].end_pages; + uint64_t dirty_pages =3D 0; + + dirty_pages =3D end_pages - start_pages; + + memory_size_MB =3D (dirty_pages * TARGET_PAGE_SIZE) >> 20; + time_s =3D DirtyStat.calc_time; + + trace_dirtyrate_do_calculate_vcpu(idx, dirty_pages, time_s); + + return memory_size_MB / time_s; +} + +static void calculate_dirtyrate_dirty_ring(struct DirtyRateConfig config) +{ + CPUState *cpu; + int64_t msec =3D 0; + int64_t start_time; + uint64_t dirtyrate =3D 0; + uint64_t dirtyrate_sum =3D 0; + int nvcpu =3D 0; + int i =3D 0; + + CPU_FOREACH(cpu) { + nvcpu++; + } + + dirty_pages =3D malloc(sizeof(*dirty_pages) * nvcpu); + + DirtyStat.dirty_ring.nvcpu =3D nvcpu; + DirtyStat.dirty_ring.rates =3D malloc(sizeof(DirtyRateVcpu) * nvcpu); + + dirtyrate_global_dirty_log_start(); + + CPU_FOREACH(cpu) { + record_dirtypages(cpu, true); + } + + start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + DirtyStat.start_time =3D start_time / 1000; + + msec =3D config.sample_period_seconds * 1000; + msec =3D set_sample_page_period(msec, start_time); + DirtyStat.calc_time =3D msec / 1000; + + CPU_FOREACH(cpu) { + record_dirtypages(cpu, false); + } + + dirtyrate_global_dirty_log_stop(); + + for (i =3D 0; i < DirtyStat.dirty_ring.nvcpu; i++) { + dirtyrate =3D do_calculate_dirtyrate_vcpu(i); + DirtyStat.dirty_ring.rates[i].id =3D i; + DirtyStat.dirty_ring.rates[i].dirty_rate =3D dirtyrate; + dirtyrate_sum +=3D dirtyrate; + } + + DirtyStat.dirty_rate =3D dirtyrate_sum; + free(dirty_pages); +} + +static void calculate_dirtyrate_sample_vm(struct DirtyRateConfig config) { struct RamblockDirtyInfo *block_dinfo =3D NULL; int block_count =3D 0; @@ -387,6 +506,17 @@ out: free_ramblock_dirty_info(block_dinfo, block_count); } =20 +static void calculate_dirtyrate(struct DirtyRateConfig config) +{ + if (config.mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) { + calculate_dirtyrate_dirty_ring(config); + } else { + calculate_dirtyrate_sample_vm(config); + } + + trace_dirtyrate_calculate(DirtyStat.dirty_rate); +} + void *get_dirtyrate_thread(void *arg) { struct DirtyRateConfig config =3D *(struct DirtyRateConfig *)arg; @@ -412,8 +542,12 @@ void *get_dirtyrate_thread(void *arg) return NULL; } =20 -void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, - int64_t sample_pages, Error **errp) +void qmp_calc_dirty_rate(int64_t calc_time, + bool has_sample_pages, + int64_t sample_pages, + bool has_mode, + DirtyRateMeasureMode mode, + Error **errp) { static struct DirtyRateConfig config; QemuThread thread; @@ -435,6 +569,15 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_s= ample_pages, return; } =20 + if (!has_mode) { + mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + } + + if (has_sample_pages && mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING= ) { + error_setg(errp, "either sample-pages or dirty-ring can be specifi= ed."); + return; + } + if (has_sample_pages) { if (!is_sample_pages_valid(sample_pages)) { error_setg(errp, "sample-pages is out of range[%d, %d].", @@ -447,6 +590,16 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_s= ample_pages, } =20 /* + * dirty ring mode only works when kvm dirty ring is enabled. + */ + if ((mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) && + !kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty ring is disabled, use sample-pages method " + "or remeasure later."); + return; + } + + /* * Init calculation state as unstarted. */ ret =3D dirtyrate_set_state(&CalculatingState, CalculatingState, @@ -458,7 +611,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sa= mple_pages, =20 config.sample_period_seconds =3D calc_time; config.sample_pages_per_gigabytes =3D sample_pages; - config.mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + config.mode =3D mode; =20 if (unlikely(dirtyrate_mode =3D=3D DIRTY_RATE_MEASURE_MODE_NONE)) { /* first time to calculate dirty rate */ @@ -471,7 +624,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sa= mple_pages, * update dirty rate mode so that we can figure out what mode has * been used in last calculation **/ - dirtyrate_mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + dirtyrate_mode =3D mode; =20 start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000; init_dirtyrate_stat(start_time, config); @@ -497,9 +650,18 @@ void hmp_info_dirty_rate(Monitor *mon, const QDict *qd= ict) info->sample_pages); monitor_printf(mon, "Period: %"PRIi64" (sec)\n", info->calc_time); + monitor_printf(mon, "Mode: %s\n", + DirtyRateMeasureMode_str(info->mode)); monitor_printf(mon, "Dirty rate: "); if (info->has_dirty_rate) { monitor_printf(mon, "%"PRIi64" (MB/s)\n", info->dirty_rate); + if (info->has_vcpu_dirty_rate) { + DirtyRateVcpuList *rate, *head =3D info->vcpu_dirty_rate; + for (rate =3D head; rate !=3D NULL; rate =3D rate->next) { + monitor_printf(mon, "vcpu[%"PRIi64"], Dirty rate: %"PRIi64= "\n", + rate->value->id, rate->value->dirty_rate); + } + } } else { monitor_printf(mon, "(not ready)\n"); } @@ -511,6 +673,10 @@ void hmp_calc_dirty_rate(Monitor *mon, const QDict *qd= ict) int64_t sec =3D qdict_get_try_int(qdict, "second", 0); int64_t sample_pages =3D qdict_get_try_int(qdict, "sample_pages_per_GB= ", -1); bool has_sample_pages =3D (sample_pages !=3D -1); + bool dirty_ring =3D qdict_get_try_bool(qdict, "dirty_ring", false); + DirtyRateMeasureMode mode =3D + (dirty_ring ? DIRTY_RATE_MEASURE_MODE_DIRTY_RING : + DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING); Error *err =3D NULL; =20 if (!sec) { @@ -518,7 +684,8 @@ void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdi= ct) return; } =20 - qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, &err); + qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, true, + mode, &err); if (err) { hmp_handle_error(mon, err); return; diff --git a/migration/trace-events b/migration/trace-events index 860c4f4..e51ebe1 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -330,6 +330,8 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, = uint32_t crc) "ramblock n calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc= ) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32 skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock = name: %s, ramblock size: %" PRIu64 find_page_matched(const char *idstr) "ramblock %s addr or size changed" +dirtyrate_calculate(int64_t dirtyrate) "dirty rate: %" PRIi64 +dirtyrate_do_calculate_vcpu(int idx, uint64_t pages, int64_t seconds) "vcp= u[%d]: dirty %"PRIu64 " pages in %"PRIi64 " seconds" =20 # block.c migration_block_init_shared(const char *blk_device_name) "Start migration = for %s with shared base image" diff --git a/qapi/migration.json b/qapi/migration.json index 7395305..e3d21a8 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1773,6 +1773,12 @@ # @sample-pages: page count per GB for sample dirty pages # the default value is 512 (since 6.1) # +# @mode: mode containing method of calculate dirtyrate includes +# 'page-sampling' and 'dirty-ring' (Since 6.1) +# +# @vcpu-dirty-rate: dirtyrate for each vcpu if dirty-ring +# mode specified (Since 6.1) +# # Since: 5.2 # ## @@ -1781,7 +1787,9 @@ 'status': 'DirtyRateStatus', 'start-time': 'int64', 'calc-time': 'int64', - 'sample-pages': 'uint64'} } + 'sample-pages': 'uint64', + 'mode': 'DirtyRateMeasureMode', + '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } } =20 ## # @calc-dirty-rate: @@ -1793,6 +1801,9 @@ # @sample-pages: page count per GB for sample dirty pages # the default value is 512 (since 6.1) # +# @mode: mechanism of calculating dirtyrate includes +# 'page-sampling' and 'dirty-ring' (Since 6.1) +# # Since: 5.2 # # Example: @@ -1801,7 +1812,8 @@ # ## { 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', - '*sample-pages': 'int'} } + '*sample-pages': 'int', + '*mode': 'DirtyRateMeasureMode'} } =20 ## # @query-dirty-rate: --=20 1.8.3.1