From nobody Wed May 8 01:26:32 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1673452557; cv=none; d=zohomail.com; s=zohoarc; b=XiD8lGtTXUDDBmmlKhnvA+SWvu6D875Am1bYHG9H36xJbRRYhe8pRaSpUkllXXvYSUa7Mxg5KToEM3FSqKV0NlDRaYRDT04yy9vwWSlOFHu6rixVlI8BCuFy3g6IgjINnxIlMLDzrlfoHxXT7Jwb9oinBG3Pp8Ejyi3WVvXKfYI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673452557; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=QwzyW7oZ6debHOnGp+1MstVKgcAlJPD9/UuYM93GsSg=; b=GuvoFihp0TaFRbjJvrsA/i8jv7Tavn7R7m6BLKQk5cHKQ/KXu7NjB0RQZoHaUoHjSSqh9I6zcBHoHAm3WI6hcsnFf7u1MKlbPC8Ub/qoUd/0G+prDDqW1b7Z5Ehf6fPx8biM23+vZxoU6O2/xtupaKClqajzTZS8kh5Bnlgrk7I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673452557671146.81900183433686; Wed, 11 Jan 2023 07:55:57 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pFcqd-0004w2-Ho; Wed, 11 Jan 2023 10:17:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pFcqb-0004ty-39 for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:01 -0500 Received: from mail-io1-xd34.google.com ([2607:f8b0:4864:20::d34]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pFcqX-0001OF-AW for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:00 -0500 Received: by mail-io1-xd34.google.com with SMTP id p9so7702225iod.13 for ; Wed, 11 Jan 2023 07:16:56 -0800 (PST) Received: from localhost ([37.19.210.30]) by smtp.gmail.com with ESMTPSA id p15-20020a0566022b0f00b006e00556dc9esm5146477iov.16.2023.01.11.07.16.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 07:16:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QwzyW7oZ6debHOnGp+1MstVKgcAlJPD9/UuYM93GsSg=; b=gRx049Ojzl6oRk4IJwclZMQyYXdiaBXqWsN1Or8ZLw7Kzh/eLgcI7+2mv3CZDLvLtn gyki6pQMSNSkGQP65BMBa2k/5W/mGDsGa/NPvlEuDEm0WZITII8EVwjFV/xrDCGaG7RK t1k8V2VRcAtctF/Yexpu2M+ic61tYrQ/QEaRBMbvuJf2A3xPQahGNC3Mb+tvfZrKOqm/ +1aepaoX+ip2+/8nQti1EyFcTXy4KLEY3uZAYVnSFlZbFx6ijyS4q6EXSufD+StfD4xX 2fG4RfkelANTRURls1WnnliW6P0h9En7s3PWNtcE4GtpCaOYJF6EOAzJR7v/nrwAPJn6 SnDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QwzyW7oZ6debHOnGp+1MstVKgcAlJPD9/UuYM93GsSg=; b=wWwI/wybbzsn0zspGwguKbl84oCC4QVnJArQOiSAUcs0gqbuBj4OFF/xa9D5Wpw5yt ECsPWy/PzZv3Nds75k5zKDGxRMv0nUPNmxZEL5coo/2tMaEyaYv13wXMXbe1cNHA7V3v +VYbux9MIRrw1lNg5zjv1yJJVMFlW1ky5NklsIhushP8ibeEZucTI3flpnbY+kRo3cHW bCkmDZtXNASLjI74LpEna/p45Dj9jVTZOex2/wxX7PkLhadkgCuShsJTkrXFva3dY21w etpmubHFYBRh9dpojPft8ije0lmYc8g7MKiauDHINsCrH1GIYbu414O+c8dU13WVfLL3 cuOQ== X-Gm-Message-State: AFqh2ko9xfPlmmILLYW8p4FvVCLo6m30BbxKyYe+vRyq0r6yul8w8nHh /UaBVeos3MkmPFt+E+hP7PNZ+rCCmcPaqXbww9k= X-Google-Smtp-Source: AMrXdXtyfeDVojRHViQnqpQjNC79wVsQka9dN6ikI0tvwvMNUz7i9vi3TjfAS64kXagyaBPlpue9RA== X-Received: by 2002:a05:6602:87:b0:704:57c7:e440 with SMTP id h7-20020a056602008700b0070457c7e440mr1775647iob.5.1673450215855; Wed, 11 Jan 2023 07:16:55 -0800 (PST) From: Emilio Cota To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Paolo Bonzini , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Richard Henderson , Emilio Cota Subject: [PATCH v3 1/5] cpu: free cpu->tb_jmp_cache with RCU Date: Wed, 11 Jan 2023 10:16:24 -0500 Message-Id: <20230111151628.320011-2-cota@braap.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230111151628.320011-1-cota@braap.org> References: <20230111151628.320011-1-cota@braap.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: softfail client-ip=2607:f8b0:4864:20::d34; envelope-from=cota@braap.org; helo=mail-io1-xd34.google.com X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @braap-org.20210112.gappssmtp.com) X-ZM-MESSAGEID: 1673452558719100003 Content-Type: text/plain; charset="utf-8" Fixes the appended use-after-free. The root cause is that during tb invalidation we use CPU_FOREACH, and therefore to safely free a vCPU we must wait for an RCU grace period to elapse. $ x86_64-linux-user/qemu-x86_64 tests/tcg/x86_64-linux-user/munmap-pthread =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D1800604=3D=3DERROR: AddressSanitizer: heap-use-after-free on address = 0x62d0005f7418 at pc 0x5593da6704eb bp 0x7f4961a7ac70 sp 0x7f4961a7ac60 READ of size 8 at 0x62d0005f7418 thread T2 #0 0x5593da6704ea in tb_jmp_cache_inval_tb ../accel/tcg/tb-maint.c:244 #1 0x5593da6704ea in do_tb_phys_invalidate ../accel/tcg/tb-maint.c:290 #2 0x5593da670631 in tb_phys_invalidate__locked ../accel/tcg/tb-maint.c= :306 #3 0x5593da670631 in tb_invalidate_phys_page_range__locked ../accel/tcg= /tb-maint.c:542 #4 0x5593da67106d in tb_invalidate_phys_range ../accel/tcg/tb-maint.c:6= 14 #5 0x5593da6a64d4 in target_munmap ../linux-user/mmap.c:766 #6 0x5593da6dba05 in do_syscall1 ../linux-user/syscall.c:10105 #7 0x5593da6f564c in do_syscall ../linux-user/syscall.c:13329 #8 0x5593da49e80c in cpu_loop ../linux-user/x86_64/../i386/cpu_loop.c:2= 33 #9 0x5593da6be28c in clone_func ../linux-user/syscall.c:6633 #10 0x7f496231cb42 in start_thread nptl/pthread_create.c:442 #11 0x7f49623ae9ff (/lib/x86_64-linux-gnu/libc.so.6+0x1269ff) 0x62d0005f7418 is located 28696 bytes inside of 32768-byte region [0x62d000= 5f0400,0x62d0005f8400) freed by thread T148 here: #0 0x7f49627b6460 in __interceptor_free ../../../../src/libsanitizer/as= an/asan_malloc_linux.cpp:52 #1 0x5593da5ac057 in cpu_exec_unrealizefn ../cpu.c:180 #2 0x5593da81f851 (/home/cota/src/qemu/build/qemu-x86_64+0x484851) Signed-off-by: Emilio Cota --- accel/tcg/cpu-exec.c | 3 +-- accel/tcg/tb-jmp-cache.h | 1 + cpu.c | 11 ++++++++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index 356fe348de..6bd29227f3 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -1064,13 +1064,12 @@ void tcg_exec_realizefn(CPUState *cpu, Error **errp) /* undo the initializations in reverse order */ void tcg_exec_unrealizefn(CPUState *cpu) { - qemu_plugin_vcpu_exit_hook(cpu); #ifndef CONFIG_USER_ONLY tcg_iommu_free_notifier_list(cpu); #endif /* !CONFIG_USER_ONLY */ =20 tlb_destroy(cpu); - g_free(cpu->tb_jmp_cache); + g_free_rcu(cpu->tb_jmp_cache, rcu); } =20 #ifndef CONFIG_USER_ONLY diff --git a/accel/tcg/tb-jmp-cache.h b/accel/tcg/tb-jmp-cache.h index ff5ffc8fc2..b3f6e78835 100644 --- a/accel/tcg/tb-jmp-cache.h +++ b/accel/tcg/tb-jmp-cache.h @@ -18,6 +18,7 @@ * a load_acquire/store_release to 'tb'. */ struct CPUJumpCache { + struct rcu_head rcu; struct { TranslationBlock *tb; #if TARGET_TB_PCREL diff --git a/cpu.c b/cpu.c index 4a7d865427..21cf809614 100644 --- a/cpu.c +++ b/cpu.c @@ -176,11 +176,20 @@ void cpu_exec_unrealizefn(CPUState *cpu) vmstate_unregister(NULL, &vmstate_cpu_common, cpu); } #endif + + /* Call the plugin hook before clearing cpu->cpu_index in cpu_list_rem= ove */ if (tcg_enabled()) { - tcg_exec_unrealizefn(cpu); + qemu_plugin_vcpu_exit_hook(cpu); } =20 cpu_list_remove(cpu); + /* + * Now that the vCPU has been removed from the RCU list, we can call + * tcg_exec_unrealizefn, which may free fields using call_rcu. + */ + if (tcg_enabled()) { + tcg_exec_unrealizefn(cpu); + } } =20 /* --=20 2.34.1 From nobody Wed May 8 01:26:32 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1673452425; cv=none; d=zohomail.com; s=zohoarc; b=O1gJ3NXMbJYsvKT8Ic/w07lg9FwvpVIYvBFTXV+UScZt267kbuylIWtiiy11rEww5GWqq9xaPPRRr77L2ZSkLMtE1F3pQaQcKZfq2iIaBgcjd9NZMI5IiMTY1vCWWM2vS1fFyJaDqFl0Nn6FWxWzofftWF/oMmEqV0v25WYFqDU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673452425; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=A2T9PH4pJZ9eRkBl3md21WXq4XJ379Vi9ay+aelpYvo=; b=IkxGFKARdsFQ/YVv/RMY4P6yvQ9Xm7ovrLXm6aqGsJcPbKbBYnXZ1B7Gv8H2l6QykKlr+5EQlu43qfWlxVvP5CEPsvPypLMNg/DiqimVXAk6koPlOdrwS5ItZnsrfUu7/lr7zKFfzStz37oXMJnkEWMrZG5kFKpRGShZVUwFdvI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673452425927141.49376197588026; Wed, 11 Jan 2023 07:53:45 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pFcqe-0004yl-T9; Wed, 11 Jan 2023 10:17:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pFcqd-0004w1-Db for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:03 -0500 Received: from mail-il1-x132.google.com ([2607:f8b0:4864:20::132]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pFcqa-0001ON-EZ for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:03 -0500 Received: by mail-il1-x132.google.com with SMTP id x6so3415604ill.10 for ; Wed, 11 Jan 2023 07:16:59 -0800 (PST) Received: from localhost ([37.19.210.30]) by smtp.gmail.com with ESMTPSA id j21-20020a023215000000b003748d3552e1sm4545772jaa.154.2023.01.11.07.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 07:16:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A2T9PH4pJZ9eRkBl3md21WXq4XJ379Vi9ay+aelpYvo=; b=V9VbEq1o0Nif0/J3SG+QsOqnpHFRKKAxKMhTtjMjRZciYez5TwRZRMO0HCwvmxyzWg eg2Alem8MpAKeVmZQN8wiKhI4TKRgmhaJTkjWxjCUvvSUvn5eVhHWByoAHK7Wtvvp7xW gwXIH57uUQpjER86u+tbNR99AhvBzR8dj3pTtz+8tzqZaxwSlRglac9m3xSzUjHSzgw1 xjKh6GbY0n2M3t3IwykFHz6FQ1Ck6lQWvAoLXpHV3nMnN+3ksZV9MAISbc7v7t8Hr/yU cwCJjyfUApokgSjoJl/8viSFnAq+F+WSjUqz10wNMj3nSyINgLSL2bT9n0fjpNzrAl2M i8yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A2T9PH4pJZ9eRkBl3md21WXq4XJ379Vi9ay+aelpYvo=; b=fEUs51dLxSEbJnfluYo5awoZLNLK+tkwmvSGdhMQvky2c4ZhpdfV5ak0kKI4qQhS0c gUBA5C0yfP/UADec2/fQGZeybwwM/iup9/G+l/wOnlX4ZQQyuHdBbs+ZAR0Hx/yZpdKK mNcQ/LHdVcdYVVhSJUpDgQs+oQkI7WtNYkzzDGtXNTfWmjlueWoxVeVu+XP5GArJBbIh Jvh6rQuIJTpRpOjieeGTlRAsvIPi3cdI+6XtuyyAr5WHjbsiHht78eraOfz+onqTFMb0 p13MaOlluBAc/IB3v9YFugQqx9tp7A5qBUoVeMWJDRJOkmdVKYB34Pxez68LhPajWRb7 JdzA== X-Gm-Message-State: AFqh2kq31wWfaOKobH990ggZCQPM9fFa3HWVW15uOVmwRO0GKbGsl4bm ZDYxL7udaJ+mo9UIH9HVgU7lADeYxoKfyhTPzLA= X-Google-Smtp-Source: AMrXdXvIYL9qrcWCtoR0sOt4nj6/JUZg8PBZYJn8x8LOdtv5C17PKTrlWOKsBnFqQhiR0GFmmbvAnw== X-Received: by 2002:a92:ab05:0:b0:30b:c4f0:f7f2 with SMTP id v5-20020a92ab05000000b0030bc4f0f7f2mr44070925ilh.11.1673450218839; Wed, 11 Jan 2023 07:16:58 -0800 (PST) From: Emilio Cota To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Paolo Bonzini , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Richard Henderson , Emilio Cota Subject: [PATCH v3 2/5] util/qht: add missing atomic_set(hashes[i]) Date: Wed, 11 Jan 2023 10:16:25 -0500 Message-Id: <20230111151628.320011-3-cota@braap.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230111151628.320011-1-cota@braap.org> References: <20230111151628.320011-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: softfail client-ip=2607:f8b0:4864:20::132; envelope-from=cota@braap.org; helo=mail-il1-x132.google.com X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @braap-org.20210112.gappssmtp.com) X-ZM-MESSAGEID: 1673452427253100001 We forgot to add this one in "a890643958 util/qht: atomically set b->hashes= ". Detected with tsan. Reviewed-by: Richard Henderson Reviewed-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Emilio Cota --- util/qht.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/util/qht.c b/util/qht.c index 065fc501f4..15866299e6 100644 --- a/util/qht.c +++ b/util/qht.c @@ -688,7 +688,7 @@ static inline void qht_bucket_remove_entry(struct qht_b= ucket *orig, int pos) int i; =20 if (qht_entry_is_last(orig, pos)) { - orig->hashes[pos] =3D 0; + qatomic_set(&orig->hashes[pos], 0); qatomic_set(&orig->pointers[pos], NULL); return; } --=20 2.34.1 From nobody Wed May 8 01:26:32 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1673452570; cv=none; d=zohomail.com; s=zohoarc; b=RZi3WcJlVNNWFZweiQVSC1TyZjm52+hzsSKoVclt7ldwJB/OdSrAwM+/HonZKWnWlLHPIz1EP7XY6lgXlz1OvnVXahEv8ynM1upPCv/WvtpIeDUAtk1g0UGoMZCsHhTl3fpmFqb94DdX/3KGUg34Gigy5NUW88zvt07AawugHAc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673452570; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=DNzttjb32EOmWWwU7OuFf4Ug13G2VErbHqLK1dunUrI=; b=f8c3VEkqZUrLrmhuVmqDsQPUd7QrLALGayNMJ+S214/1cgq43jWIAK2/gAmt0k3CfXbhL+OxiEClnceXO5v4W+gK6tfHMUtq9PWmghe7XzqUED9pzdvsrBO5EjzwQR8BldhveC2/SAVK3oe8C2mBwF1NO/5zmWkehStt63UI4hU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673452570034773.4993714932909; Wed, 11 Jan 2023 07:56:10 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pFcqf-00052V-Qr; Wed, 11 Jan 2023 10:17:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pFcqe-0004yN-K7 for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:04 -0500 Received: from mail-il1-x136.google.com ([2607:f8b0:4864:20::136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pFcqd-0001Oc-7D for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:04 -0500 Received: by mail-il1-x136.google.com with SMTP id u8so8011665ilq.13 for ; Wed, 11 Jan 2023 07:17:02 -0800 (PST) Received: from localhost ([37.19.210.30]) by smtp.gmail.com with ESMTPSA id a17-20020a056e020e1100b0030c186ea94fsm4504531ilk.55.2023.01.11.07.17.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 07:17:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DNzttjb32EOmWWwU7OuFf4Ug13G2VErbHqLK1dunUrI=; b=5OnCJx2DQ2YPZddkymZOYHAixFsEZCcKJypXzfB6F2jyUJTeej9sqlPJiZsJTRGJ8E 1SZQ8cbE9xSt9/KfxM7tWL5vXab8bJFROF5XGRhq7Ef7D4nksRVh96iMyV7TRZo7+jgD +ZN4SghyaxKoHQdhr+sBzFG1Bg0OHGI8DGjUHJzHrCgzaCZDpkPVxgNQbMd650cAd4D2 IqLwErKu8f2UbYtSB3tB95HbdFsnkUUOrKJw6X6nDZyH7mVKZel16dyqg6Y+IHErFTvi cIhi+n2oqrWqBOvYCiLStE1KrMjCN3Hw/FC52GIioSz9rlJ5M4WJdTSY+1tATwjHfQgi wzKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DNzttjb32EOmWWwU7OuFf4Ug13G2VErbHqLK1dunUrI=; b=7gV0QvzBEzlSdTraO/cUIO+KOCIKsoxTg1iSBvHjqIcxgvkXuoFXmY5Med4Kz7zrOz uWA/tKIUDUiVHQ9wPUxJB0EgseD3kr+ZNNwCwi0FOCYOt1Y5gvxmMVlAImiMWAFeDOMv n7l/YrMuYAIMpEsS/bsMxuUZVzz1rJ2P6vv0s8RsXt70Sj2JcIkKID9hmEwF8mZUFG0D +23bYVQNrkRDwQ+qx/TK7Pv40rD0JGa6MtgjHgdquCeYwq41sBS94idZ3Ju2nBhF6u/R NhIUlWaFTfVuFudzmfZhVCf9RDL09d+ppBBKcpQ1BOXxxKKUX+FTrrowh5bk3abpErNu U1mA== X-Gm-Message-State: AFqh2krg7Sm76xu/bGYTdh4fhg8j9j87+sl+j5q1oFcSYtpn+TxRGAAe cnZgOULg9Tf5hl3+ti1jl5oScOfkSgHuxO2Ie3U= X-Google-Smtp-Source: AMrXdXtLfAbnDW23WVy0MZnKRnV3//x08cYhLCRzmXDd/4h1t6Svo9v17dU+GTjBSM/EiLl66IOa2g== X-Received: by 2002:a05:6e02:18cd:b0:30b:fb78:6f0 with SMTP id s13-20020a056e0218cd00b0030bfb7806f0mr57105945ilu.5.1673450221798; Wed, 11 Jan 2023 07:17:01 -0800 (PST) From: Emilio Cota To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Paolo Bonzini , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Richard Henderson , Emilio Cota Subject: [PATCH v3 3/5] thread: de-const qemu_spin_destroy Date: Wed, 11 Jan 2023 10:16:26 -0500 Message-Id: <20230111151628.320011-4-cota@braap.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230111151628.320011-1-cota@braap.org> References: <20230111151628.320011-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: softfail client-ip=2607:f8b0:4864:20::136; envelope-from=cota@braap.org; helo=mail-il1-x136.google.com X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @braap-org.20210112.gappssmtp.com) X-ZM-MESSAGEID: 1673452570787100003 Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Emilio Cota --- include/qemu/thread.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/include/qemu/thread.h b/include/qemu/thread.h index 7c6703bce3..7841084199 100644 --- a/include/qemu/thread.h +++ b/include/qemu/thread.h @@ -237,11 +237,10 @@ static inline void qemu_spin_init(QemuSpin *spin) #endif } =20 -/* const parameter because the only purpose here is the TSAN annotation */ -static inline void qemu_spin_destroy(const QemuSpin *spin) +static inline void qemu_spin_destroy(QemuSpin *spin) { #ifdef CONFIG_TSAN - __tsan_mutex_destroy((void *)spin, __tsan_mutex_not_static); + __tsan_mutex_destroy(spin, __tsan_mutex_not_static); #endif } =20 --=20 2.34.1 From nobody Wed May 8 01:26:32 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1673452560; cv=none; d=zohomail.com; s=zohoarc; b=j7wC4jtWWzTI/ZYWHXbnNfeXaM0MCIz8sbvz3VPk+rY+1YEse6RLCsXAiiK0pTNVcfJaqcniyVlF630aH72JsQze6muIiA6VNNuJVrVL2WQfETDA+xdRqcCIF1sOhnvSweLzll22/eyXo09hdelUTOZJXOsR7feiONLrd0yLOZ0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673452560; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=LtPWCngx1skeijoNISC9sQENv6JcG3kFfpbvAsry54A=; b=YZJBRjsCWLOjj+5iya9EhCyBTdimqaXG1JxQdvKRSEaZjB2U8LihDBHDI0u+GOhrCY7DpgZy4Lr5MqmsoUkNzjDYokzDBNdpIhdRs21DJAkJgswTGz41NRzwLpSagTY/VULIs7KDNLQT3wSbc26gLLb2QJhrE2VGDNMhYkynkdw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673452560139391.94408559099634; Wed, 11 Jan 2023 07:56:00 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pFcqk-000547-Av; Wed, 11 Jan 2023 10:17:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pFcqi-00053b-ID for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:08 -0500 Received: from mail-io1-xd2c.google.com ([2607:f8b0:4864:20::d2c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pFcqg-0001Ox-JA for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:08 -0500 Received: by mail-io1-xd2c.google.com with SMTP id r72so7759122iod.5 for ; Wed, 11 Jan 2023 07:17:06 -0800 (PST) Received: from localhost ([37.19.210.30]) by smtp.gmail.com with ESMTPSA id x20-20020a0566022c5400b006bc039e3224sm5066986iov.17.2023.01.11.07.17.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 07:17:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LtPWCngx1skeijoNISC9sQENv6JcG3kFfpbvAsry54A=; b=SS9nBIqKEds2KOpXRl6wswALjYLP6dOd3aIEkgilN3wr6K9bkQ8CHUeUlW7J5r/0BJ o0pGHSxuyJOI6UealtEWklInAF7aTGM2rk/TX7L8uhgq+8zqHk/EADK0w1UGbBLC3ipO VfVLr7Rmr5ykj1tgF1RC9jatJGjxFz8voJE+MK3uaMwOrg3RjI5BglTCWgvzQKldKkWs 5I/WKOajJNVjyViKiQz6v5kWS0cmCbE0BekIGsSKWsxpDZGiavx0OeD3WVn3VkxyWtag uqiH/4vQxGNgpiOCKEXv4+YwX/NF9izwD4EwWi8yQwfAZwXHO2VcpzrKYyrmaUiYhNuX IuCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LtPWCngx1skeijoNISC9sQENv6JcG3kFfpbvAsry54A=; b=ocKnfQMwH7Lu97/S+sbFesAbvd+cG/Or5bObF79LdMS05NuDoavFpAANKQlietGExp PFVp90PLGxj5ERpmgkMW9uFGsi/GnD4spfUI3iENMSToECgJp8hWvEF9QKHh3YR2XeGA et959HZckGRdx1uw+oA183Fl7fw1exoYC3pSCwB6YUKHB6O1vn5zNKtwkl1W+OZs7Op+ 1oyJsoHETSSEFyhxhoiciC2KtP3XJVCxMe8HA5d6YD+9qIP564gOt2gu5o39HQX0j1nT s4Y/3H2c26/pfDW8gco0sKp79124GbjjFOaH1xEjKASLqpABaj6CO5eQoDWj1zGw8KvO Vplg== X-Gm-Message-State: AFqh2krLZ0G2PXX15o4sVIM6QucxvfRIN95ys/4sX/zAUdof1edeNsXT Qv70cUaQsDe2AAymr5sIoIot/d36zRkYchfR/L4= X-Google-Smtp-Source: AMrXdXuou6VM2NnypYbSKUo04Wh4kLSZG8KWDbf3p/kfjvb6mFPqalqB7BSg09MZXYd7Rf/cbz5KJg== X-Received: by 2002:a05:6602:398e:b0:6ee:f9e1:641e with SMTP id bw14-20020a056602398e00b006eef9e1641emr2475398iob.15.1673450224864; Wed, 11 Jan 2023 07:17:04 -0800 (PST) From: Emilio Cota To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Paolo Bonzini , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Richard Henderson , Emilio Cota Subject: [PATCH v3 4/5] util/qht: use striped locks under TSAN Date: Wed, 11 Jan 2023 10:16:27 -0500 Message-Id: <20230111151628.320011-5-cota@braap.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230111151628.320011-1-cota@braap.org> References: <20230111151628.320011-1-cota@braap.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: softfail client-ip=2607:f8b0:4864:20::d2c; envelope-from=cota@braap.org; helo=mail-io1-xd2c.google.com X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @braap-org.20210112.gappssmtp.com) X-ZM-MESSAGEID: 1673452560815100007 Content-Type: text/plain; charset="utf-8" Fixes this tsan crash, easy to reproduce with any large enough program: $ tests/unit/test-qht 1..2 ThreadSanitizer: CHECK failed: sanitizer_deadlock_detector.h:67 "((n_all_lo= cks_)) < (((sizeof(all_locks_with_contexts_)/sizeof((all_locks_with_context= s_)[0]))))" (0x40, 0x40) (tid=3D1821568) #0 __tsan::CheckUnwind() ../../../../src/libsanitizer/tsan/tsan_rtl.cpp= :353 (libtsan.so.2+0x90034) #1 __sanitizer::CheckFailed(char const*, int, char const*, unsigned lon= g long, unsigned long long) ../../../../src/libsanitizer/sanitizer_common/s= anitizer_termination.cpp:86 (libtsan.so.2+0xca555) #2 __sanitizer::DeadlockDetectorTLS<__sanitizer::TwoLevelBitVector<1ul,= __sanitizer::BasicBitVector > >::addLock(unsigned long, uns= igned long, unsigned int) ../../../../src/libsanitizer/sanitizer_common/san= itizer_deadlock_detector.h:67 (libtsan.so.2+0xb3616) #3 __sanitizer::DeadlockDetectorTLS<__sanitizer::TwoLevelBitVector<1ul,= __sanitizer::BasicBitVector > >::addLock(unsigned long, uns= igned long, unsigned int) ../../../../src/libsanitizer/sanitizer_common/san= itizer_deadlock_detector.h:59 (libtsan.so.2+0xb3616) #4 __sanitizer::DeadlockDetector<__sanitizer::TwoLevelBitVector<1ul, __= sanitizer::BasicBitVector > >::onLockAfter(__sanitizer::Dead= lockDetectorTLS<__sanitizer::TwoLevelBitVector<1ul, __sanitizer::BasicBitVe= ctor > >*, unsigned long, unsigned int) ../../../../src/libs= anitizer/sanitizer_common/sanitizer_deadlock_detector.h:216 (libtsan.so.2+0= xb3616) #5 __sanitizer::DD::MutexAfterLock(__sanitizer::DDCallback*, __sanitize= r::DDMutex*, bool, bool) ../../../../src/libsanitizer/sanitizer_common/sani= tizer_deadlock_detector1.cpp:169 (libtsan.so.2+0xb3616) #6 __tsan::MutexPostLock(__tsan::ThreadState*, unsigned long, unsigned = long, unsigned int, int) ../../../../src/libsanitizer/tsan/tsan_rtl_mutex.c= pp:200 (libtsan.so.2+0xa3382) #7 __tsan_mutex_post_lock ../../../../src/libsanitizer/tsan/tsan_interf= ace_ann.cpp:384 (libtsan.so.2+0x76bc3) #8 qemu_spin_lock /home/cota/src/qemu/include/qemu/thread.h:259 (test-q= ht+0x44a97) #9 qht_map_lock_buckets ../util/qht.c:253 (test-qht+0x44a97) #10 do_qht_iter ../util/qht.c:809 (test-qht+0x45f33) #11 qht_iter ../util/qht.c:821 (test-qht+0x45f33) #12 iter_check ../tests/unit/test-qht.c:121 (test-qht+0xe473) #13 qht_do_test ../tests/unit/test-qht.c:202 (test-qht+0xe473) #14 qht_test ../tests/unit/test-qht.c:240 (test-qht+0xe7c1) #15 test_default ../tests/unit/test-qht.c:246 (test-qht+0xe828) #16 (libglib-2.0.so.0+0x7daed) #17 (libglib-2.0.so.0+0x7d80a) #18 (libglib-2.0.so.0+0x7d80a) #19 g_test_run_suite (libglib-2.0.so.0+0x7dfe9) #20 g_test_run (libglib-2.0.so.0+0x7e055) #21 main ../tests/unit/test-qht.c:259 (test-qht+0xd2c6) #22 __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 (l= ibc.so.6+0x29d8f) #23 __libc_start_main_impl ../csu/libc-start.c:392 (libc.so.6+0x29e3f) #24 _start (test-qht+0xdb44) Signed-off-by: Emilio Cota --- util/qht.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 81 insertions(+), 14 deletions(-) diff --git a/util/qht.c b/util/qht.c index 15866299e6..92c6b78759 100644 --- a/util/qht.c +++ b/util/qht.c @@ -151,6 +151,22 @@ struct qht_bucket { =20 QEMU_BUILD_BUG_ON(sizeof(struct qht_bucket) > QHT_BUCKET_ALIGN); =20 +/* + * Under TSAN, we use striped locks instead of one lock per bucket chain. + * This avoids crashing under TSAN, since TSAN aborts the program if more = than + * 64 locks are held (this is a hardcoded limit in TSAN). + * When resizing a QHT we grab all the buckets' locks, which can easily + * go over TSAN's limit. By using striped locks, we avoid this problem. + * + * Note: this number must be a power of two for easy index computation. + */ +#define QHT_TSAN_BUCKET_LOCKS_BITS 4 +#define QHT_TSAN_BUCKET_LOCKS (1 << QHT_TSAN_BUCKET_LOCKS_BITS) + +struct qht_tsan_lock { + QemuSpin lock; +} QEMU_ALIGNED(QHT_BUCKET_ALIGN); + /** * struct qht_map - structure to track an array of buckets * @rcu: used by RCU. Keep it as the top field in the struct to help valgr= ind @@ -160,6 +176,7 @@ QEMU_BUILD_BUG_ON(sizeof(struct qht_bucket) > QHT_BUCKE= T_ALIGN); * @n_added_buckets: number of added (i.e. "non-head") buckets * @n_added_buckets_threshold: threshold to trigger an upward resize once = the * number of added buckets surpasses it. + * @tsan_bucket_locks: Array of striped locks to be used only under TSAN. * * Buckets are tracked in what we call a "map", i.e. this structure. */ @@ -169,6 +186,9 @@ struct qht_map { size_t n_buckets; size_t n_added_buckets; size_t n_added_buckets_threshold; +#ifdef CONFIG_TSAN + struct qht_tsan_lock tsan_bucket_locks[QHT_TSAN_BUCKET_LOCKS]; +#endif }; =20 /* trigger a resize when n_added_buckets > n_buckets / div */ @@ -229,10 +249,56 @@ static inline size_t qht_elems_to_buckets(size_t n_el= ems) return pow2ceil(n_elems / QHT_BUCKET_ENTRIES); } =20 -static inline void qht_head_init(struct qht_bucket *b) +/* + * When using striped locks (i.e. under TSAN), we have to be careful not + * to operate on the same lock twice (e.g. when iterating through all buck= ets). + * We achieve this by operating only on each stripe's first matching lock. + */ +static inline void qht_do_if_first_in_stripe(struct qht_map *map, + struct qht_bucket *b, + void (*func)(QemuSpin *spin)) +{ +#ifdef CONFIG_TSAN + unsigned long bucket_idx =3D b - map->buckets; + bool is_first_in_stripe =3D (bucket_idx >> QHT_TSAN_BUCKET_LOCKS_BITS)= =3D=3D 0; + if (is_first_in_stripe) { + unsigned long lock_idx =3D bucket_idx & (QHT_TSAN_BUCKET_LOCKS - 1= ); + func(&map->tsan_bucket_locks[lock_idx].lock); + } +#else + func(&b->lock); +#endif +} + +static inline void qht_bucket_lock_do(struct qht_map *map, + struct qht_bucket *b, + void (*func)(QemuSpin *lock)) +{ +#ifdef CONFIG_TSAN + unsigned long bucket_idx =3D b - map->buckets; + unsigned long lock_idx =3D bucket_idx & (QHT_TSAN_BUCKET_LOCKS - 1); + func(&map->tsan_bucket_locks[lock_idx].lock); +#else + func(&b->lock); +#endif +} + +static inline void qht_bucket_lock(struct qht_map *map, + struct qht_bucket *b) +{ + qht_bucket_lock_do(map, b, qemu_spin_lock); +} + +static inline void qht_bucket_unlock(struct qht_map *map, + struct qht_bucket *b) +{ + qht_bucket_lock_do(map, b, qemu_spin_unlock); +} + +static inline void qht_head_init(struct qht_map *map, struct qht_bucket *b) { memset(b, 0, sizeof(*b)); - qemu_spin_init(&b->lock); + qht_do_if_first_in_stripe(map, b, qemu_spin_init); seqlock_init(&b->sequence); } =20 @@ -250,7 +316,7 @@ static void qht_map_lock_buckets(struct qht_map *map) for (i =3D 0; i < map->n_buckets; i++) { struct qht_bucket *b =3D &map->buckets[i]; =20 - qemu_spin_lock(&b->lock); + qht_do_if_first_in_stripe(map, b, qemu_spin_lock); } } =20 @@ -261,7 +327,7 @@ static void qht_map_unlock_buckets(struct qht_map *map) for (i =3D 0; i < map->n_buckets; i++) { struct qht_bucket *b =3D &map->buckets[i]; =20 - qemu_spin_unlock(&b->lock); + qht_do_if_first_in_stripe(map, b, qemu_spin_unlock); } } =20 @@ -308,7 +374,7 @@ void qht_map_lock_buckets__no_stale(struct qht *ht, str= uct qht_map **pmap) * Get a head bucket and lock it, making sure its parent map is not stale. * @pmap is filled with a pointer to the bucket's parent map. * - * Unlock with qemu_spin_unlock(&b->lock). + * Unlock with qht_bucket_unlock. * * Note: callers cannot have ht->lock held. */ @@ -322,18 +388,18 @@ struct qht_bucket *qht_bucket_lock__no_stale(struct q= ht *ht, uint32_t hash, map =3D qatomic_rcu_read(&ht->map); b =3D qht_map_to_bucket(map, hash); =20 - qemu_spin_lock(&b->lock); + qht_bucket_lock(map, b); if (likely(!qht_map_is_stale__locked(ht, map))) { *pmap =3D map; return b; } - qemu_spin_unlock(&b->lock); + qht_bucket_unlock(map, b); =20 /* we raced with a resize; acquire ht->lock to see the updated ht->map= */ qht_lock(ht); map =3D ht->map; b =3D qht_map_to_bucket(map, hash); - qemu_spin_lock(&b->lock); + qht_bucket_lock(map, b); qht_unlock(ht); *pmap =3D map; return b; @@ -345,12 +411,13 @@ static inline bool qht_map_needs_resize(const struct = qht_map *map) map->n_added_buckets_threshold; } =20 -static inline void qht_chain_destroy(const struct qht_bucket *head) +static inline void qht_chain_destroy(struct qht_map *map, + struct qht_bucket *head) { struct qht_bucket *curr =3D head->next; struct qht_bucket *prev; =20 - qemu_spin_destroy(&head->lock); + qht_do_if_first_in_stripe(map, head, qemu_spin_destroy); while (curr) { prev =3D curr; curr =3D curr->next; @@ -364,7 +431,7 @@ static void qht_map_destroy(struct qht_map *map) size_t i; =20 for (i =3D 0; i < map->n_buckets; i++) { - qht_chain_destroy(&map->buckets[i]); + qht_chain_destroy(map, &map->buckets[i]); } qemu_vfree(map->buckets); g_free(map); @@ -390,7 +457,7 @@ static struct qht_map *qht_map_create(size_t n_buckets) map->buckets =3D qemu_memalign(QHT_BUCKET_ALIGN, sizeof(*map->buckets) * n_buckets); for (i =3D 0; i < n_buckets; i++) { - qht_head_init(&map->buckets[i]); + qht_head_init(map, &map->buckets[i]); } return map; } @@ -638,7 +705,7 @@ bool qht_insert(struct qht *ht, void *p, uint32_t hash,= void **existing) b =3D qht_bucket_lock__no_stale(ht, hash, &map); prev =3D qht_insert__locked(ht, map, b, p, hash, &needs_resize); qht_bucket_debug__locked(b); - qemu_spin_unlock(&b->lock); + qht_bucket_unlock(map, b); =20 if (unlikely(needs_resize) && ht->mode & QHT_MODE_AUTO_RESIZE) { qht_grow_maybe(ht); @@ -749,7 +816,7 @@ bool qht_remove(struct qht *ht, const void *p, uint32_t= hash) b =3D qht_bucket_lock__no_stale(ht, hash, &map); ret =3D qht_remove__locked(b, p, hash); qht_bucket_debug__locked(b); - qemu_spin_unlock(&b->lock); + qht_bucket_unlock(map, b); return ret; } =20 --=20 2.34.1 From nobody Wed May 8 01:26:32 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1673452551; cv=none; d=zohomail.com; s=zohoarc; b=ZcLmJ4L45g+dABaueAFkse9YMa2vCthnQAFzIlGXg+Vnt25drcQNP5LpIvHmGZNEpF2eEtNJa8fDFoZbXLlT3Q67vxyYFtRPRyvajdDMlDFnNn++TbyNGTGFkTuKsFjnv95VETYjhfIJBu0S1tzWS9gAedhYjgcERANSDB+I/zg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673452551; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=gTtAtxGY4Bs65WVYgMhynTWfuOk2bggcnt6aho5Sk7c=; b=SmPucmcU20/e9y+wl4M6Zq58Yi+4P1xEd2+wHVfZpbOsUyW5nJZBTr7sRyzV1kxeRnuZfusWv/Xl/gQh5tzx62vE8tr4guI1yYQcrnZOBoFNKS7ZH5ORHKnQ7T8oBBh/lDPZSYB0q5Rs1JuwrOPR3f309mKh37zZv30wn9hbiBw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673452551810313.73742687574656; Wed, 11 Jan 2023 07:55:51 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pFcr2-00058W-UF; Wed, 11 Jan 2023 10:17:28 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pFcql-00054G-9P for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:13 -0500 Received: from mail-il1-x134.google.com ([2607:f8b0:4864:20::134]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pFcqj-0001PF-Hy for qemu-devel@nongnu.org; Wed, 11 Jan 2023 10:17:10 -0500 Received: by mail-il1-x134.google.com with SMTP id i1so4552244ilu.8 for ; Wed, 11 Jan 2023 07:17:09 -0800 (PST) Received: from localhost ([37.19.210.30]) by smtp.gmail.com with ESMTPSA id q20-20020a056e02097400b0030bf7ae141esm4495355ilt.2.2023.01.11.07.17.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 07:17:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gTtAtxGY4Bs65WVYgMhynTWfuOk2bggcnt6aho5Sk7c=; b=AohEby0FGsmGJBvdrsXaMZUADjXpw0O6HAYYvg1lEf8/s3Pj+JdABRK+NCw3mjXscy dzuJt9vLCsg3nHLTtrXKGP7SM+SiFdhAHP4+0iUTP4o2FzU7BluiUR7i1/X1+naVNxNF lhuOFxHXIMnPsgEByFzWD4Kf6pmB9qP84eaIEjSdB+CcLTMYFGY5moZzTfIPA2IDH7y+ 9wB4tC77MVCOvc6H+KyrBh+St8LdqKvkDgPGJCC0Ek7AExU1LRBXyog8IcdWjAT9979N /fKGrFfKKkITF8STPYNIucLWWIC5F3+gILQG2GIQVEvklz7h6oO8vhiLB+UQN80uiDb7 f/wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gTtAtxGY4Bs65WVYgMhynTWfuOk2bggcnt6aho5Sk7c=; b=ceR3ol5ZWYy3XuAy0bf2lMuioYGesP+EZ+YIgo9ma63S5FSTlCtufN1jRupBFwxi4M qH8hdGVnkI1VybbHiJMqEh2fikEsythMOaK52hO/UdaPoozd3OrI6lkQuCEl+fSNjOC1 +Fy4n3kEhSPv4pKlZ8IM5RW6mQYadU/2Lxd98ryESf4jrfqKcZR78Ipmq1TxJ6LKhGx+ ciaWrY6CreCHDD8StcHvYhYiM+rTTGqucybpuUpve4oKyRUnMzt2C3r6uCdD5qmawk7q N7tq80zvAqyLB9VWlWBBORAM9ge9DhI7CPO7W2w0vdkr5SP8bZPtubxONTFImFSsubLx APPA== X-Gm-Message-State: AFqh2kqsm3A39O1qCcxYjHvLbRjqRz8UNGPrE46SQkdvA1XQXtBzfpEL xUhaRfmiJprm0VQAMd76fxvioTGdwtWoeZEL4Z0= X-Google-Smtp-Source: AMrXdXs0iwc6v6l2gY0O+gh8smnBrHi0TPiNegAvirnhju7dfPEGxvZCJRYoA5+F7LeudPOCiaX5jw== X-Received: by 2002:a92:d111:0:b0:30b:b664:d0e9 with SMTP id a17-20020a92d111000000b0030bb664d0e9mr46884880ilb.24.1673450228263; Wed, 11 Jan 2023 07:17:08 -0800 (PST) From: Emilio Cota To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Paolo Bonzini , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Richard Henderson , Emilio Cota Subject: [PATCH v3 5/5] plugins: make qemu_plugin_user_exit's locking order consistent with fork_start's Date: Wed, 11 Jan 2023 10:16:28 -0500 Message-Id: <20230111151628.320011-6-cota@braap.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230111151628.320011-1-cota@braap.org> References: <20230111151628.320011-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: softfail client-ip=2607:f8b0:4864:20::134; envelope-from=cota@braap.org; helo=mail-il1-x134.google.com X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @braap-org.20210112.gappssmtp.com) X-ZM-MESSAGEID: 1673452552722100001 To fix potential deadlocks as reported by tsan. Reviewed-by: Richard Henderson Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Emilio Cota --- plugins/core.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/plugins/core.c b/plugins/core.c index ccb770a485..728bacef95 100644 --- a/plugins/core.c +++ b/plugins/core.c @@ -500,10 +500,17 @@ void qemu_plugin_user_exit(void) enum qemu_plugin_event ev; CPUState *cpu; =20 - QEMU_LOCK_GUARD(&plugin.lock); - + /* + * Locking order: we must acquire locks in an order that is consistent + * with the one in fork_start(). That is: + * - start_exclusive(), which acquires qemu_cpu_list_lock, + * must be called before acquiring plugin.lock. + * - tb_flush(), which acquires mmap_lock(), must be called + * while plugin.lock is not held. + */ start_exclusive(); =20 + qemu_rec_mutex_lock(&plugin.lock); /* un-register all callbacks except the final AT_EXIT one */ for (ev =3D 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { if (ev !=3D QEMU_PLUGIN_EV_ATEXIT) { @@ -513,13 +520,12 @@ void qemu_plugin_user_exit(void) } } } - - tb_flush(current_cpu); - CPU_FOREACH(cpu) { qemu_plugin_disable_mem_helpers(cpu); } + qemu_rec_mutex_unlock(&plugin.lock); =20 + tb_flush(current_cpu); end_exclusive(); =20 /* now it's safe to handle the exit case */ --=20 2.34.1