From nobody Mon Sep 16 19:13:27 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1673878518; cv=pass; d=zohomail.com; s=zohoarc; b=nQopl6k7ABO8CWSCBDvsSDG8TZFtg6vksLUJ+n76L352XzstP8diRQhqvz1LZ+cpBA0ajNXtPC00IZfTyICnDdJ2xDI0tZ4+n/ZVeqITmXmzaUGdC8jW4qbGw8NcFo/9Qf1XbTPa/eqax/iCnmfFyqa6FCLf7swrsBDjsYtErZ4= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673878518; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=BaJ38liVsUzOA3vHwYlHsSh8cwLPTulKqyW2kq98Nxw=; b=FcUdzaRCaFlrJJB5pPA0TeI8VYf1dZMHsCDmN6WRbFwSiS4iJczllOa7HtRct+VTGNGaEGayloxB6oYQkZ7N1qNH5hixGxphVUpMmXEmK7GPjrotxE0fXyXc84tuzDrt3Xjo+KQJcwUMzepYfI8Gv6CAzez+5/ZjIfqnSWL2IRQ= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673878518324863.5233708493903; Mon, 16 Jan 2023 06:15:18 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pHQE1-0002Zj-Fj; Mon, 16 Jan 2023 09:12:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pHQDv-0002XW-PZ; Mon, 16 Jan 2023 09:12:33 -0500 Received: from mail-co1nam11on2088.outbound.protection.outlook.com ([40.107.220.88] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pHQDt-0002O4-GU; Mon, 16 Jan 2023 09:12:31 -0500 Received: from DS7PR05CA0044.namprd05.prod.outlook.com (2603:10b6:8:2f::10) by IA0PR12MB7700.namprd12.prod.outlook.com (2603:10b6:208:430::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.22; Mon, 16 Jan 2023 14:12:24 +0000 Received: from DM6NAM11FT036.eop-nam11.prod.protection.outlook.com (2603:10b6:8:2f:cafe::3b) by DS7PR05CA0044.outlook.office365.com (2603:10b6:8:2f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6023.12 via Frontend Transport; Mon, 16 Jan 2023 14:12:23 +0000 Received: from mail.nvidia.com (216.228.117.160) by DM6NAM11FT036.mail.protection.outlook.com (10.13.172.64) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13 via Frontend Transport; Mon, 16 Jan 2023 14:12:23 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Mon, 16 Jan 2023 06:12:13 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Mon, 16 Jan 2023 06:12:12 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Mon, 16 Jan 2023 06:12:05 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EB9ogsmL8pxz6YYXdnOpLpscnRd19NA+Q+cfwdMEGhtDgd73yUkp7/olBfTdP0KbIPDYjTGY9CSiTI9L07adrE82yXlYkv/wovIoaL3FZwyU+J5xr+TVfPl/DIJUY+dRBrh38RxMqElkxohTarVMXGYeKh68Ey5uho4ly9mwuLBenwTSl4BwbJ47qMwekHTVumFze6khWRGSRkhH3az68Tny7ZSzKEPdw93i3h5PoddYEflLjEMKKaW1lULZ5uLN1ePgt3s8l3+8QWkyl/72uLWiJCbdKdGb9qggJTVsgHBOZSYhptgFDhXJDTAr6R+QnGGws4cjj+1WMKSliWTqIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BaJ38liVsUzOA3vHwYlHsSh8cwLPTulKqyW2kq98Nxw=; b=FqZcXATnDLfnwa7bkpgeENTmPRjYCwiEXZ2x7iCXYaqIplyg6avORNJR5wkMFVKyEbTSha2yiMPFSZUtg9tsVfYDJ4oGmEr+jNSslopLtU/V29MzXfYtSIADSgUsSEgsMFF1CDvvZ4nKAtDAo7d0avIjqe286WCF72+isAGxOwLGDSs9xmx11K5Av6acIZ81b51mTO2p73ZqHEggDAh4gYbsDQvwwVHPHnBpXdqkQjTN4z2P/QL2NojpQS/Xla/hbGxvYDwWMuVQ7wBOpI3oIShfEBeIBv2uPMseFFAM8/7H7sZjFWVqhHyU+kjPIIZN7QrDyaXKhO77weEzb3z/Bg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BaJ38liVsUzOA3vHwYlHsSh8cwLPTulKqyW2kq98Nxw=; b=pQ6xsk18rwboUW1PpLKg6UphSlZY86R4rX0xt+UCh8GQ2+DGWLSZYqP3fS+O5vDeMIQVZZ3jNZUtxmMYKXi/wYwGSQps6FnldGbiKZg9s8xVaH8GGQGaiSxy1ZZegH8jrrPDNGMIHDpnoBM2/mKvAz1MUaQEah7yU2aEN9vnklylamSD+7V7WgGrtlH7ojSTAEh5ZFnOxeXxFTWoB9sEcw9bJ3z0UgvLGNfr3Xxdyp5wjfcGQ0xIKBxLvH1T5bNeXwXaVMI5dWxfOr6BC2/S3YlslG3Vknj4u6wejkPRQC0iSLej8Mbh9dQlYIB7N8JAIaDHB9lc5PvqvvQbKJNdYA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , Halil Pasic , Christian Borntraeger , Eric Farman , Richard Henderson , David Hildenbrand , "Ilya Leoshkevich" , Thomas Huth , "Juan Quintela" , "Dr. David Alan Gilbert" , "Michael S. Tsirkin" , Cornelia Huck , Paolo Bonzini , Stefan Hajnoczi , Fam Zheng , Eric Blake , Vladimir Sementsov-Ogievskiy , John Snow , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , , , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH v8 04/13] vfio/migration: Allow migration without VFIO IOMMU dirty tracking support Date: Mon, 16 Jan 2023 16:11:26 +0200 Message-ID: <20230116141135.12021-5-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230116141135.12021-1-avihaih@nvidia.com> References: <20230116141135.12021-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT036:EE_|IA0PR12MB7700:EE_ X-MS-Office365-Filtering-Correlation-Id: 3320d1fc-53d0-4a18-03d4-08daf7cbb081 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qA05bPXu/s1FoZXGE6XTX/KE49f/lgt7chsQD/aEExX6dBKWLLFcIM+19FaO1/cPjiix+sM0Q48Jn+HxcrUfNA48n2j9ozlutTVOtNL6ODyNrpG4Va4OCxqu7VwzTt9pLGA390PIMzNDds7TYrB81ChGKxRsq+t+wYv+CjzBjqpXEhoXcAm7jxt22ztrtLpumwn4syp278bnRVPnU8RNSXA6yDxcasVZ+YWUF/IJ98/6trD/WeCjDcRGeaIVJdt8GVboIkmor/2pmc54K5iib4V8cAIz6QG3Fs/iWKcKQiiHIVEBohDA4me0P2val07Jq3uWZw+yk9l9Nkvm2pe9uXLTImu6Dxvgk3PUriVlgr40Rlq0sUmD0CFV2bc6xJTaKSTPo8KGiwhS3FhuxZZ9IUwwrfRlxoZfnmiESAXP/dLsQ89cDP7pzCOM0P0TsdpYpUOtko9r6hSbo3Ij3/cciLQnHcJ0UtXzznq74CaJGH06/Ptbe1WXwlCY9O0BlJbEonVAv0l5n6JJ4D4Vc3MBZZ1EC3TKxntLIl5D6QunnMLO0KF4+QO3gQygGWRfm4C4Hu/mRaZzhAYYW/FL46aNnZvpmJFo4qM4MM5m8tVHZWWd6R8H1pcgj40ZbxtMB8vwm/nhLMcbAlzXEbbh4WLi4gG9HLRYmtea4TsaHz514EXMljvj5Te7NDDhTDGg2lPqmoRX9ihAdRGay5Z0oDuhz4WgTxLtYojdix1/euMHpeiM9pCPaECDEQupAsKCTiwefQTgYIZB2CwOSZvJzfbjgNV7mnYpfnrsSKLX4EsAaQYHjS79iR9qQ/9tmJae1NQY X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(346002)(396003)(376002)(39860400002)(136003)(451199015)(40470700004)(46966006)(36840700001)(7636003)(82740400003)(40460700003)(36756003)(40480700001)(82310400005)(356005)(86362001)(966005)(70206006)(70586007)(7696005)(6916009)(4326008)(26005)(8676002)(336012)(186003)(478600001)(1076003)(54906003)(2616005)(36860700001)(316002)(41300700001)(8936002)(426003)(47076005)(2906002)(83380400001)(6666004)(7416002)(5660300002)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2023 14:12:23.6607 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3320d1fc-53d0-4a18-03d4-08daf7cbb081 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT036.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7700 Received-SPF: softfail client-ip=40.107.220.88; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1673878520253100004 Content-Type: text/plain; charset="utf-8" Currently, if IOMMU of a VFIO container doesn't support dirty page tracking, migration is blocked. This is because a DMA-able VFIO device can dirty RAM pages without updating QEMU about it, thus breaking the migration. However, this doesn't mean that migration can't be done at all. In such case, allow migration and let QEMU VFIO code mark all pages dirty. This guarantees that all pages that might have gotten dirty are reported back, and thus guarantees a valid migration even without VFIO IOMMU dirty tracking support. The motivation for this patch is the introduction of iommufd [1]. iommufd can directly implement the /dev/vfio/vfio container IOCTLs by mapping them into its internal ops, allowing the usage of these IOCTLs over iommufd. However, VFIO IOMMU dirty tracking is not supported by this VFIO compatibility API. This patch will allow migration by hosts that use the VFIO compatibility API and prevent migration regressions caused by the lack of VFIO IOMMU dirty tracking support. [1] https://lore.kernel.org/kvm/0-v6-a196d26f289e+11787-iommufd_jgg@nvidia.com/ Signed-off-by: Avihai Horon Reviewed-by: C=C3=A9dric Le Goater --- hw/vfio/common.c | 20 ++++++++++++++++++-- hw/vfio/migration.c | 3 +-- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 130e5d1dc7..f6dd571549 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -488,6 +488,12 @@ static int vfio_dma_unmap(VFIOContainer *container, return -errno; } =20 + if (iotlb && vfio_devices_all_running_and_saving(container)) { + cpu_physical_memory_set_dirty_range(iotlb->translated_addr, size, + tcg_enabled() ? DIRTY_CLIENTS_= ALL : + DIRTY_CLIENTS_NOCODE); + } + return 0; } =20 @@ -1201,6 +1207,10 @@ static void vfio_set_dirty_page_tracking(VFIOContain= er *container, bool start) .argsz =3D sizeof(dirty), }; =20 + if (!container->dirty_pages_supported) { + return; + } + if (start) { dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; } else { @@ -1236,6 +1246,13 @@ static int vfio_get_dirty_bitmap(VFIOContainer *cont= ainer, uint64_t iova, uint64_t pages; int ret; =20 + if (!container->dirty_pages_supported) { + cpu_physical_memory_set_dirty_range(ram_addr, size, + tcg_enabled() ? DIRTY_CLIENTS_= ALL : + DIRTY_CLIENTS_NOCODE); + return 0; + } + dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); =20 dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); @@ -1409,8 +1426,7 @@ static void vfio_listener_log_sync(MemoryListener *li= stener, { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); =20 - if (vfio_listener_skipped_section(section) || - !container->dirty_pages_supported) { + if (vfio_listener_skipped_section(section)) { return; } =20 diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 09fe7c1de2..552c2313b2 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -860,11 +860,10 @@ int64_t vfio_mig_bytes_transferred(void) =20 int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) { - VFIOContainer *container =3D vbasedev->group->container; struct vfio_region_info *info =3D NULL; int ret =3D -ENOTSUP; =20 - if (!vbasedev->enable_migration || !container->dirty_pages_supported) { + if (!vbasedev->enable_migration) { goto add_blocker; } =20 --=20 2.26.3