From nobody Mon Sep 16 18:56:24 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1673878475; cv=pass; d=zohomail.com; s=zohoarc; b=oAqlghaG2rg3Qbn0X0OzRzO/utgkHRhwil2mGUn3kxDVwxBSxHTwiQszlYwM2bujdGkd6p01UbYd1Dlz/mNRjUrPfn7Nlru7oZvNOdnDtgNktwHT0WTq9O+/N8WddUmaJciiRnX2xerJfFoJDkgO7qt/kt0PCWzCWzTAAt1tYns= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1673878475; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=W54cKSR8JEenlKyzzT121+sHOWUN4QvnWRZT3QRZi84=; b=QGpQCR3hi8wLmth9ocxfrfme6C9AJ4bFJwRmuR37NbdQ+EUU/YuKrkCn8xhz4kIs7xCBy4B1kz7+3xvdTNm1NGCS9cpngpuCBWmDnhRJhhslx8jdagNHGJEOB3GDYvGDgQ/poeqXBpHpx9SPFQUcyfsF2Hak/kGRF1g3J7bwDEY= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1673878475070952.0701824893488; Mon, 16 Jan 2023 06:14:35 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pHQEm-0004He-S4; Mon, 16 Jan 2023 09:13:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pHQEk-00044H-II; Mon, 16 Jan 2023 09:13:22 -0500 Received: from mail-bn8nam12on2084.outbound.protection.outlook.com ([40.107.237.84] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pHQEh-0002UB-7b; Mon, 16 Jan 2023 09:13:22 -0500 Received: from DM6PR11CA0040.namprd11.prod.outlook.com (2603:10b6:5:14c::17) by PH7PR12MB6788.namprd12.prod.outlook.com (2603:10b6:510:1ae::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18; Mon, 16 Jan 2023 14:13:15 +0000 Received: from DM6NAM11FT084.eop-nam11.prod.protection.outlook.com (2603:10b6:5:14c:cafe::98) by DM6PR11CA0040.outlook.office365.com (2603:10b6:5:14c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.19 via Frontend Transport; Mon, 16 Jan 2023 14:13:15 +0000 Received: from mail.nvidia.com (216.228.117.160) by DM6NAM11FT084.mail.protection.outlook.com (10.13.172.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13 via Frontend Transport; Mon, 16 Jan 2023 14:13:14 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Mon, 16 Jan 2023 06:13:04 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Mon, 16 Jan 2023 06:13:04 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Mon, 16 Jan 2023 06:12:57 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=haepxXYgpTHk2o3uXMJ4gh6iykj4I6Co2Q9i9GSsIav3LQV4FZFiiLkxTDq4IjpScoKOQJ4f1PaImf79COmpvymzE0voiiLarbz35s378ictmBZQsQqOhn9mtqo/D+8TxIHAOY5htafZdsbDtWKoZna71WxXGH9PqEQ7p7otr7/Dy/QDJD/K6fVXbW9JpwAb6whiivkHOfFQoViRTa79k3K+vG/+WmMm6Rn0G9Y/w3B0w2JQe0fJ4hNxzxx9QFm9lraFEKCksf2n02v3/fTyr0BiJ+qLdMshr/OAi7Nxz10F9ney5wUKhizbMfI1159vm5XgVIhn9rsUe7McfQtEgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=W54cKSR8JEenlKyzzT121+sHOWUN4QvnWRZT3QRZi84=; b=hnfxuGxRDMZWT/aTM9c9w4QqQhJxnhtCSzRocHtM5G7yrFjf0do6jnHDf5KMhTQzOjHKJZK3icBPgMfkKaI1iNURKyTprpfVKVTWdrMSoXJoNneRUoMMKnAtwPZwBGLvAP0jDt7UnGvHhGGo/1MgJDvriw4WE5nyo3yqHoJnKO3Oca2gxqhjFBxdtWLvNso19mxuRdTTBX5zkX+3A7ohpaek+3/EKBaHFBR+PtJCgZLMH/mLRiuXGPq9E+bM/kOwJY+oR+xf0d+VgJyo05g9GSkh6NpmX41cMhUU5+TuC7jrp+dlECEDiPf/htBXtom5G5y/TFdZE2zzoxyWl/2GWw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W54cKSR8JEenlKyzzT121+sHOWUN4QvnWRZT3QRZi84=; b=pZ9Bg9GkFdURlCq6QudiwM1IKdYSqAidMtMDHIWfnCnivcPL3j96oMP+TiOcXL/3CFUW0RXuDTDoAtmNWYh5cOMkfGuKkUETpIwpJyQ6M3G7/yjZVzEaNSjMafu4EwfOSH7qFc0+Mod87eWb9FQjBQ/uanI9EW1kyycSWpjWkUPWBp2FfprogfbgI6+tjLfOqe1yKIllkS3SkSj8QhIeDZ6O0UVcPFu7TSq09Ssh63TTGB0+WMLaV9gsjl8FU3ngWvpv+LfBkPSUjsJOp9FKBa/rjfC2sbIrYirZJ1zJPx7c6QOyUgHSHD6zzAnigkB3B8v1DUQSK8M1Jw//z4407Q== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , Halil Pasic , Christian Borntraeger , Eric Farman , Richard Henderson , David Hildenbrand , "Ilya Leoshkevich" , Thomas Huth , "Juan Quintela" , "Dr. David Alan Gilbert" , "Michael S. Tsirkin" , Cornelia Huck , Paolo Bonzini , Stefan Hajnoczi , Fam Zheng , Eric Blake , Vladimir Sementsov-Ogievskiy , John Snow , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , , , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH v8 11/13] vfio/migration: Remove VFIO migration protocol v1 Date: Mon, 16 Jan 2023 16:11:33 +0200 Message-ID: <20230116141135.12021-12-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230116141135.12021-1-avihaih@nvidia.com> References: <20230116141135.12021-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT084:EE_|PH7PR12MB6788:EE_ X-MS-Office365-Filtering-Correlation-Id: 56780f95-f589-4421-2bdc-08daf7cbcf00 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zt8FMu9xKr4UlxkwWALKPc2n6mkEwlExsFaNqIAwbFaAGsY7B+WVZ1yTVnwqmZRewyR6mgBL1Gvi5Xt1mrSH69MnTXHfJN1hYU7DkIafuuGFUklxOC2Y/IYBlKKKVRbZEPpP/Sw5Il8a+W0k3YbTCQFLacKviFfDlu0n9c76zFEU2D60l+eKw44gGTJzg53ws2jfyZ/gU3WGGBnSYqMC2waG/pXV1qG9qbdGPwKKIw+GwuVFK5NPJ/iimC8Tdw+QPOfUZa8vEvonurBnoy70LPmn4oAlRYBuWjx0I4DR5RuojdE75swvPOmS0D4X+bUCrtB5dS53jKXLx1Y/C6nkzwELqOOsxldm1Wf7hmknVecx7lWiUMwugL/VjJe0NxxXtQ0XQ0C/GZHQBM4bAu+3QoKp0ZQUT38uYjoPPam+gLv0V5yeSQuPWVWymkvKKXCLn6WmoJmfc0vU9F/a+MgKTyoGKZahwDz87KiVeJbySHjy/gcpSRwUWi0mr+4s8qvuicvoD0hyTznRhA1HR+VqhW6PgetE8n4EMGxVUZ1yT2RKVXpxt410F7j5vpB9vTZQsXJt52BnVhw5smKXYLWV1+1ZKwmLljQV7po9fEkjqQ4/ArCfA2+Kp3x5oKYU5u6dXHTH95eRHkKQ0J+ZFWycPoxc+SGrT8x9dugi7boYMIOetpfPTOYVHt0KIbad/KIjh9wdQFGMRBiLpeGR+CfQFQ== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(136003)(346002)(39860400002)(376002)(396003)(451199015)(36840700001)(40470700004)(46966006)(70206006)(82310400005)(2906002)(7416002)(5660300002)(36756003)(2616005)(70586007)(8676002)(1076003)(4326008)(54906003)(316002)(41300700001)(30864003)(40480700001)(478600001)(66574015)(26005)(8936002)(336012)(186003)(6916009)(6666004)(426003)(7696005)(86362001)(83380400001)(36860700001)(47076005)(356005)(7636003)(40460700003)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2023 14:13:14.8100 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 56780f95-f589-4421-2bdc-08daf7cbcf00 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT084.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6788 Received-SPF: softfail client-ip=40.107.237.84; envelope-from=avihaih@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1673878476124100011 Now that v2 protocol implementation has been added, remove the deprecated v1 implementation. Signed-off-by: Avihai Horon Reviewed-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-common.h | 5 - hw/vfio/common.c | 19 +- hw/vfio/migration.c | 703 +--------------------------------- hw/vfio/trace-events | 9 - 4 files changed, 24 insertions(+), 712 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1f4c41bf9a..5f8e7a02fe 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -61,18 +61,13 @@ typedef struct VFIORegion { typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; - VFIORegion region; - uint32_t device_state_v1; - int vm_running; Notifier migration_state; NotifierWithReturn migration_data; - uint64_t pending_bytes; uint32_t device_state; int data_fd; void *data_buffer; size_t data_buffer_size; uint64_t stop_copy_size; - bool v2; } VFIOMigration; =20 typedef struct VFIOAddressSpace { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index dcaa77d2a8..9a0dbee6b4 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -355,14 +355,7 @@ static bool vfio_devices_all_dirty_tracking(VFIOContai= ner *container) return false; } =20 - if (!migration->v2 && - (vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) && - (migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING= )) { - return false; - } - - if (migration->v2 && - (vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) && + if ((vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) && (migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING = || migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_= P2P)) { return false; @@ -393,14 +386,8 @@ static bool vfio_devices_all_running_and_mig_active(VF= IOContainer *container) return false; } =20 - if (!migration->v2 && - migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING)= { - continue; - } - - if (migration->v2 && - (migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING = || - migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_= P2P)) { + if (migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING || + migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_P= 2P) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 87ef2b44ef..7489fcb03a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -142,220 +142,6 @@ static int vfio_migration_set_state(VFIODevice *vbase= dev, return 0; } =20 -static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int cou= nt, - off_t off, bool iswrite) -{ - int ret; - - ret =3D iswrite ? pwrite(vbasedev->fd, val, count, off) : - pread(vbasedev->fd, val, count, off); - if (ret < count) { - error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" - HWADDR_PRIx", err: %s", iswrite ? "write" : "read", c= ount, - vbasedev->name, off, strerror(errno)); - return (ret < 0) ? ret : -EINVAL; - } - return 0; -} - -static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, - off_t off, bool iswrite) -{ - int ret, done =3D 0; - __u8 *tbuf =3D buf; - - while (count) { - int bytes =3D 0; - - if (count >=3D 8 && !(off % 8)) { - bytes =3D 8; - } else if (count >=3D 4 && !(off % 4)) { - bytes =3D 4; - } else if (count >=3D 2 && !(off % 2)) { - bytes =3D 2; - } else { - bytes =3D 1; - } - - ret =3D vfio_mig_access(vbasedev, tbuf, bytes, off, iswrite); - if (ret) { - return ret; - } - - count -=3D bytes; - done +=3D bytes; - off +=3D bytes; - tbuf +=3D bytes; - } - return done; -} - -#define vfio_mig_read(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, fa= lse) -#define vfio_mig_write(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, tr= ue) - -#define VFIO_MIG_STRUCT_OFFSET(f) \ - offsetof(struct vfio_device_migration_inf= o, f) -/* - * Change the device_state register for device @vbasedev. Bits set in @mask - * are preserved, bits set in @value are set, and bits not set in either @= mask - * or @value are cleared in device_state. If the register cannot be access= ed, - * the resulting state would be invalid, or the device enters an error sta= te, - * an error is returned. - */ - -static int vfio_migration_v1_set_state(VFIODevice *vbasedev, uint32_t mask, - uint32_t value) -{ - VFIOMigration *migration =3D vbasedev->migration; - VFIORegion *region =3D &migration->region; - off_t dev_state_off =3D region->fd_offset + - VFIO_MIG_STRUCT_OFFSET(device_state); - uint32_t device_state; - int ret; - - ret =3D vfio_mig_read(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - if (ret < 0) { - return ret; - } - - device_state =3D (device_state & mask) | value; - - if (!VFIO_DEVICE_STATE_VALID(device_state)) { - return -EINVAL; - } - - ret =3D vfio_mig_write(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - if (ret < 0) { - int rret; - - rret =3D vfio_mig_read(vbasedev, &device_state, sizeof(device_stat= e), - dev_state_off); - - if ((rret < 0) || (VFIO_DEVICE_STATE_IS_ERROR(device_state))) { - hw_error("%s: Device in error state 0x%x", vbasedev->name, - device_state); - return rret ? rret : -EIO; - } - return ret; - } - - migration->device_state_v1 =3D device_state; - trace_vfio_migration_v1_set_state(vbasedev->name, device_state); - return 0; -} - -static void *get_data_section_size(VFIORegion *region, uint64_t data_offse= t, - uint64_t data_size, uint64_t *size) -{ - void *ptr =3D NULL; - uint64_t limit =3D 0; - int i; - - if (!region->mmaps) { - if (size) { - *size =3D MIN(data_size, region->size - data_offset); - } - return ptr; - } - - for (i =3D 0; i < region->nr_mmaps; i++) { - VFIOMmap *map =3D region->mmaps + i; - - if ((data_offset >=3D map->offset) && - (data_offset < map->offset + map->size)) { - - /* check if data_offset is within sparse mmap areas */ - ptr =3D map->mmap + data_offset - map->offset; - if (size) { - *size =3D MIN(data_size, map->offset + map->size - data_of= fset); - } - break; - } else if ((data_offset < map->offset) && - (!limit || limit > map->offset)) { - /* - * data_offset is not within sparse mmap areas, find size of - * non-mapped area. Check through all list since region->mmaps= list - * is not sorted. - */ - limit =3D map->offset; - } - } - - if (!ptr && size) { - *size =3D limit ? MIN(data_size, limit - data_offset) : data_size; - } - return ptr; -} - -static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *s= ize) -{ - VFIOMigration *migration =3D vbasedev->migration; - VFIORegion *region =3D &migration->region; - uint64_t data_offset =3D 0, data_size =3D 0, sz; - int ret; - - ret =3D vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offs= et)); - if (ret < 0) { - return ret; - } - - ret =3D vfio_mig_read(vbasedev, &data_size, sizeof(data_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_si= ze)); - if (ret < 0) { - return ret; - } - - trace_vfio_save_buffer(vbasedev->name, data_offset, data_size, - migration->pending_bytes); - - qemu_put_be64(f, data_size); - sz =3D data_size; - - while (sz) { - void *buf; - uint64_t sec_size; - bool buf_allocated =3D false; - - buf =3D get_data_section_size(region, data_offset, sz, &sec_size); - - if (!buf) { - buf =3D g_try_malloc(sec_size); - if (!buf) { - error_report("%s: Error allocating buffer ", __func__); - return -ENOMEM; - } - buf_allocated =3D true; - - ret =3D vfio_mig_read(vbasedev, buf, sec_size, - region->fd_offset + data_offset); - if (ret < 0) { - g_free(buf); - return ret; - } - } - - qemu_put_buffer(f, buf, sec_size); - - if (buf_allocated) { - g_free(buf); - } - sz -=3D sec_size; - data_offset +=3D sec_size; - } - - ret =3D qemu_file_get_error(f); - - if (!ret && size) { - *size =3D data_size; - } - - bytes_transferred +=3D data_size; - return ret; -} - static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t data_size) { @@ -368,96 +154,6 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *v= basedev, return ret; } =20 -static int vfio_v1_load_buffer(QEMUFile *f, VFIODevice *vbasedev, - uint64_t data_size) -{ - VFIORegion *region =3D &vbasedev->migration->region; - uint64_t data_offset =3D 0, size, report_size; - int ret; - - do { - ret =3D vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offs= et)); - if (ret < 0) { - return ret; - } - - if (data_offset + data_size > region->size) { - /* - * If data_size is greater than the data section of migration = region - * then iterate the write buffer operation. This case can occu= r if - * size of migration region at destination is smaller than siz= e of - * migration region at source. - */ - report_size =3D size =3D region->size - data_offset; - data_size -=3D size; - } else { - report_size =3D size =3D data_size; - data_size =3D 0; - } - - trace_vfio_v1_load_state_device_data(vbasedev->name, data_offset, = size); - - while (size) { - void *buf; - uint64_t sec_size; - bool buf_alloc =3D false; - - buf =3D get_data_section_size(region, data_offset, size, &sec_= size); - - if (!buf) { - buf =3D g_try_malloc(sec_size); - if (!buf) { - error_report("%s: Error allocating buffer ", __func__); - return -ENOMEM; - } - buf_alloc =3D true; - } - - qemu_get_buffer(f, buf, sec_size); - - if (buf_alloc) { - ret =3D vfio_mig_write(vbasedev, buf, sec_size, - region->fd_offset + data_offset); - g_free(buf); - - if (ret < 0) { - return ret; - } - } - size -=3D sec_size; - data_offset +=3D sec_size; - } - - ret =3D vfio_mig_write(vbasedev, &report_size, sizeof(report_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_si= ze)); - if (ret < 0) { - return ret; - } - } while (data_size); - - return 0; -} - -static int vfio_update_pending(VFIODevice *vbasedev) -{ - VFIOMigration *migration =3D vbasedev->migration; - VFIORegion *region =3D &migration->region; - uint64_t pending_bytes =3D 0; - int ret; - - ret =3D vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_byt= es)); - if (ret < 0) { - migration->pending_bytes =3D 0; - return ret; - } - - migration->pending_bytes =3D pending_bytes; - trace_vfio_update_pending(vbasedev->name, pending_bytes); - return 0; -} - static int vfio_save_device_config_state(QEMUFile *f, void *opaque) { VFIODevice *vbasedev =3D opaque; @@ -510,15 +206,6 @@ static void vfio_migration_cleanup(VFIODevice *vbasede= v) migration->data_fd =3D -1; } =20 -static void vfio_migration_v1_cleanup(VFIODevice *vbasedev) -{ - VFIOMigration *migration =3D vbasedev->migration; - - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); - } -} - static int vfio_query_stop_copy_size(VFIODevice *vbasedev, uint64_t *stop_copy_size) { @@ -593,49 +280,6 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } =20 -static int vfio_v1_save_setup(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev =3D opaque; - VFIOMigration *migration =3D vbasedev->migration; - int ret; - - trace_vfio_v1_save_setup(vbasedev->name); - - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); - - if (migration->region.mmaps) { - /* - * Calling vfio_region_mmap() from migration thread. Memory API ca= lled - * from this function require locking the iothread when called from - * outside the main loop thread. - */ - qemu_mutex_lock_iothread(); - ret =3D vfio_region_mmap(&migration->region); - qemu_mutex_unlock_iothread(); - if (ret) { - error_report("%s: Failed to mmap VFIO migration region: %s", - vbasedev->name, strerror(-ret)); - error_report("%s: Falling back to slow path", vbasedev->name); - } - } - - ret =3D vfio_migration_v1_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_SAVING); - if (ret) { - error_report("%s: Failed to set state SAVING", vbasedev->name); - return ret; - } - - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - - ret =3D qemu_file_get_error(f); - if (ret) { - return ret; - } - - return 0; -} - static void vfio_save_cleanup(void *opaque) { VFIODevice *vbasedev =3D opaque; @@ -647,14 +291,6 @@ static void vfio_save_cleanup(void *opaque) trace_vfio_save_cleanup(vbasedev->name); } =20 -static void vfio_v1_save_cleanup(void *opaque) -{ - VFIODevice *vbasedev =3D opaque; - - vfio_migration_v1_cleanup(vbasedev); - trace_vfio_save_cleanup(vbasedev->name); -} - static void vfio_save_pending(void *opaque, uint64_t threshold_size, uint64_t *res_precopy_only, uint64_t *res_compatible, @@ -670,73 +306,6 @@ static void vfio_save_pending(void *opaque, uint64_t t= hreshold_size, migration->stop_copy_size); } =20 -static void vfio_v1_save_pending(void *opaque, uint64_t threshold_size, - uint64_t *res_precopy_only, - uint64_t *res_compatible, - uint64_t *res_postcopy_only) -{ - VFIODevice *vbasedev =3D opaque; - VFIOMigration *migration =3D vbasedev->migration; - int ret; - - ret =3D vfio_update_pending(vbasedev); - if (ret) { - return; - } - - *res_precopy_only +=3D migration->pending_bytes; - - trace_vfio_v1_save_pending(vbasedev->name, *res_precopy_only, - *res_postcopy_only, *res_compatible); -} - -static int vfio_save_iterate(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev =3D opaque; - VFIOMigration *migration =3D vbasedev->migration; - uint64_t data_size; - int ret; - - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); - - if (migration->pending_bytes =3D=3D 0) { - ret =3D vfio_update_pending(vbasedev); - if (ret) { - return ret; - } - - if (migration->pending_bytes =3D=3D 0) { - qemu_put_be64(f, 0); - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - /* indicates data finished, goto complete phase */ - return 1; - } - } - - ret =3D vfio_save_buffer(f, vbasedev, &data_size); - if (ret) { - error_report("%s: vfio_save_buffer failed %s", vbasedev->name, - strerror(errno)); - return ret; - } - - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - - ret =3D qemu_file_get_error(f); - if (ret) { - return ret; - } - - /* - * Reset pending_bytes as .save_live_pending is not called during save= vm or - * snapshot case, in such case vfio_update_pending() at the start of t= his - * function updates pending_bytes. - */ - migration->pending_bytes =3D 0; - trace_vfio_save_iterate(vbasedev->name, data_size); - return 0; -} - static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) { VFIODevice *vbasedev =3D opaque; @@ -772,62 +341,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, voi= d *opaque) return ret; } =20 -static int vfio_v1_save_complete_precopy(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev =3D opaque; - VFIOMigration *migration =3D vbasedev->migration; - uint64_t data_size; - int ret; - - ret =3D vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_RU= NNING, - VFIO_DEVICE_STATE_V1_SAVING); - if (ret) { - error_report("%s: Failed to set state STOP and SAVING", - vbasedev->name); - return ret; - } - - ret =3D vfio_update_pending(vbasedev); - if (ret) { - return ret; - } - - while (migration->pending_bytes > 0) { - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); - ret =3D vfio_save_buffer(f, vbasedev, &data_size); - if (ret < 0) { - error_report("%s: Failed to save buffer", vbasedev->name); - return ret; - } - - if (data_size =3D=3D 0) { - break; - } - - ret =3D vfio_update_pending(vbasedev); - if (ret) { - return ret; - } - } - - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - - ret =3D qemu_file_get_error(f); - if (ret) { - return ret; - } - - ret =3D vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SA= VING, - 0); - if (ret) { - error_report("%s: Failed to set state STOPPED", vbasedev->name); - return ret; - } - - trace_vfio_v1_save_complete_precopy(vbasedev->name); - return ret; -} - static void vfio_save_state(QEMUFile *f, void *opaque) { VFIODevice *vbasedev =3D opaque; @@ -849,33 +362,6 @@ static int vfio_load_setup(QEMUFile *f, void *opaque) vbasedev->migration->device_state); } =20 -static int vfio_v1_load_setup(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev =3D opaque; - VFIOMigration *migration =3D vbasedev->migration; - int ret =3D 0; - - if (migration->region.mmaps) { - ret =3D vfio_region_mmap(&migration->region); - if (ret) { - error_report("%s: Failed to mmap VFIO migration region %d: %s", - vbasedev->name, migration->region.nr, - strerror(-ret)); - error_report("%s: Falling back to slow path", vbasedev->name); - } - } - - ret =3D vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_RESUMING); - if (ret) { - error_report("%s: Failed to set state RESUMING", vbasedev->name); - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); - } - } - return ret; -} - static int vfio_load_cleanup(void *opaque) { VFIODevice *vbasedev =3D opaque; @@ -886,15 +372,6 @@ static int vfio_load_cleanup(void *opaque) return 0; } =20 -static int vfio_v1_load_cleanup(void *opaque) -{ - VFIODevice *vbasedev =3D opaque; - - vfio_migration_v1_cleanup(vbasedev); - trace_vfio_load_cleanup(vbasedev->name); - return 0; -} - static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) { VFIODevice *vbasedev =3D opaque; @@ -928,11 +405,7 @@ static int vfio_load_state(QEMUFile *f, void *opaque, = int version_id) uint64_t data_size =3D qemu_get_be64(f); =20 if (data_size) { - if (vbasedev->migration->v2) { - ret =3D vfio_load_buffer(f, vbasedev, data_size); - } else { - ret =3D vfio_v1_load_buffer(f, vbasedev, data_size); - } + ret =3D vfio_load_buffer(f, vbasedev, data_size); if (ret < 0) { return ret; } @@ -964,18 +437,6 @@ static const SaveVMHandlers savevm_vfio_handlers =3D { .load_state =3D vfio_load_state, }; =20 -static SaveVMHandlers savevm_vfio_v1_handlers =3D { - .save_setup =3D vfio_v1_save_setup, - .save_cleanup =3D vfio_v1_save_cleanup, - .save_live_pending =3D vfio_v1_save_pending, - .save_live_iterate =3D vfio_save_iterate, - .save_live_complete_precopy =3D vfio_v1_save_complete_precopy, - .save_state =3D vfio_save_state, - .load_setup =3D vfio_v1_load_setup, - .load_cleanup =3D vfio_v1_load_cleanup, - .load_state =3D vfio_load_state, -}; - /* ---------------------------------------------------------------------- = */ =20 static void vfio_vmstate_change(void *opaque, bool running, RunState state) @@ -1006,70 +467,12 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) mig_state_to_str(new_state)); } =20 -static void vfio_v1_vmstate_change(void *opaque, bool running, RunState st= ate) -{ - VFIODevice *vbasedev =3D opaque; - VFIOMigration *migration =3D vbasedev->migration; - uint32_t value, mask; - int ret; - - if (vbasedev->migration->vm_running =3D=3D running) { - return; - } - - if (running) { - /* - * Here device state can have one of _SAVING, _RESUMING or _STOP b= it. - * Transition from _SAVING to _RUNNING can happen if there is migr= ation - * failure, in that case clear _SAVING bit. - * Transition from _RESUMING to _RUNNING occurs during resuming - * phase, in that case clear _RESUMING bit. - * In both the above cases, set _RUNNING bit. - */ - mask =3D ~VFIO_DEVICE_STATE_MASK; - value =3D VFIO_DEVICE_STATE_V1_RUNNING; - } else { - /* - * Here device state could be either _RUNNING or _SAVING|_RUNNING.= Reset - * _RUNNING bit - */ - mask =3D ~VFIO_DEVICE_STATE_V1_RUNNING; - - /* - * When VM state transition to stop for savevm command, device sho= uld - * start saving data. - */ - if (state =3D=3D RUN_STATE_SAVE_VM) { - value =3D VFIO_DEVICE_STATE_V1_SAVING; - } else { - value =3D 0; - } - } - - ret =3D vfio_migration_v1_set_state(vbasedev, mask, value); - if (ret) { - /* - * Migration should be aborted in this case, but vm_state_notify() - * currently does not support reporting failures. - */ - error_report("%s: Failed to set device state 0x%x", vbasedev->name, - (migration->device_state_v1 & mask) | value); - if (migrate_get_current()->to_dst_file) { - qemu_file_set_error(migrate_get_current()->to_dst_file, ret); - } - } - vbasedev->migration->vm_running =3D running; - trace_vfio_v1_vmstate_change(vbasedev->name, running, RunState_str(sta= te), - (migration->device_state_v1 & mask) | value); -} - static void vfio_migration_state_notifier(Notifier *notifier, void *data) { MigrationState *s =3D data; VFIOMigration *migration =3D container_of(notifier, VFIOMigration, migration_state); VFIODevice *vbasedev =3D migration->vbasedev; - int ret; =20 trace_vfio_migration_state_notifier(vbasedev->name, MigrationStatus_str(s->state)); @@ -1079,18 +482,8 @@ static void vfio_migration_state_notifier(Notifier *n= otifier, void *data) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: bytes_transferred =3D 0; - if (migration->v2) { - vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING, - VFIO_DEVICE_STATE_ERROR); - } else { - ret =3D vfio_migration_v1_set_state(vbasedev, - ~(VFIO_DEVICE_STATE_V1_SAVIN= G | - VFIO_DEVICE_STATE_V1_RESUM= ING), - VFIO_DEVICE_STATE_V1_RUNNING= ); - if (ret) { - error_report("%s: Failed to set state RUNNING", vbasedev->= name); - } - } + vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING, + VFIO_DEVICE_STATE_ERROR); } } =20 @@ -1130,12 +523,6 @@ static int vfio_migration_data_notifier(NotifierWithR= eturn *n, void *data) =20 static void vfio_migration_exit(VFIODevice *vbasedev) { - VFIOMigration *migration =3D vbasedev->migration; - - if (!migration->v2) { - vfio_region_exit(&migration->region); - vfio_region_finalize(&migration->region); - } g_free(vbasedev->migration); vbasedev->migration =3D NULL; } @@ -1175,7 +562,6 @@ static int vfio_migration_init(VFIODevice *vbasedev) VFIOMigration *migration; char id[256] =3D ""; g_autofree char *path =3D NULL, *oid =3D NULL; - struct vfio_region_info *info; uint64_t mig_flags =3D 0; =20 if (!vbasedev->ops->vfio_get_object) { @@ -1188,52 +574,20 @@ static int vfio_migration_init(VFIODevice *vbasedev) } =20 ret =3D vfio_migration_query_flags(vbasedev, &mig_flags); - if (!ret) { - /* Migration v2 */ - /* Basic migration functionality must be supported */ - if (!(mig_flags & VFIO_MIGRATION_STOP_COPY)) { - return -EOPNOTSUPP; - } - vbasedev->migration =3D g_new0(VFIOMigration, 1); - vbasedev->migration->device_state =3D VFIO_DEVICE_STATE_RUNNING; - vbasedev->migration->data_fd =3D -1; - vbasedev->migration->v2 =3D true; - } else if (ret =3D=3D -ENOTTY) { - /* Migration v1 */ - ret =3D vfio_get_dev_region_info(vbasedev, - VFIO_REGION_TYPE_MIGRATION_DEPRECAT= ED, - VFIO_REGION_SUBTYPE_MIGRATION_DEPRE= CATED, - &info); - if (ret) { - return ret; - } - - vbasedev->migration =3D g_new0(VFIOMigration, 1); - vbasedev->migration->device_state_v1 =3D VFIO_DEVICE_STATE_V1_RUNN= ING; - vbasedev->migration->vm_running =3D runstate_is_running(); - - ret =3D vfio_region_setup(obj, vbasedev, &vbasedev->migration->reg= ion, - info->index, "migration"); - if (ret) { - error_report("%s: Failed to setup VFIO migration region %d: %s= ", - vbasedev->name, info->index, strerror(-ret)); - goto err; - } - - if (!vbasedev->migration->region.size) { - error_report("%s: Invalid zero-sized VFIO migration region %d", - vbasedev->name, info->index); - ret =3D -EINVAL; - goto err; - } - - g_free(info); - } else { + if (ret) { return ret; } =20 + /* Basic migration functionality must be supported */ + if (!(mig_flags & VFIO_MIGRATION_STOP_COPY)) { + return -EOPNOTSUPP; + } + + vbasedev->migration =3D g_new0(VFIOMigration, 1); migration =3D vbasedev->migration; migration->vbasedev =3D vbasedev; + migration->device_state =3D VFIO_DEVICE_STATE_RUNNING; + migration->data_fd =3D -1; =20 oid =3D vmstate_if_get_id(VMSTATE_IF(DEVICE(obj))); if (oid) { @@ -1243,31 +597,18 @@ static int vfio_migration_init(VFIODevice *vbasedev) } strpadcpy(id, sizeof(id), path, '\0'); =20 - if (migration->v2) { - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, - &savevm_vfio_handlers, vbasedev); - - migration->vm_state =3D qdev_add_vm_change_state_handler( - vbasedev->dev, vfio_vmstate_change, vbasedev); - - migration->migration_data.notify =3D vfio_migration_data_notifier; - precopy_add_notifier(&migration->migration_data); - } else { - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, - &savevm_vfio_v1_handlers, vbasedev); - - migration->vm_state =3D qdev_add_vm_change_state_handler( - vbasedev->dev, vfio_v1_vmstate_change, vbasedev); - } + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_hand= lers, + vbasedev); =20 + migration->vm_state =3D qdev_add_vm_change_state_handler(vbasedev->dev, + vfio_vmstate_ch= ange, + vbasedev); migration->migration_state.notify =3D vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); - return 0; + migration->migration_data.notify =3D vfio_migration_data_notifier; + precopy_add_notifier(&migration->migration_data); =20 -err: - g_free(info); - vfio_migration_exit(vbasedev); - return ret; + return 0; } =20 /* ---------------------------------------------------------------------- = */ @@ -1310,9 +651,7 @@ void vfio_migration_finalize(VFIODevice *vbasedev) if (vbasedev->migration) { VFIOMigration *migration =3D vbasedev->migration; =20 - if (migration->v2) { - precopy_remove_notifier(&migration->migration_data); - } + precopy_remove_notifier(&migration->migration_data); remove_migration_state_change_notifier(&migration->migration_state= ); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index a8a64f0627..60c49b2ecf 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -150,24 +150,15 @@ vfio_display_edid_write_error(void) "" # migration.c vfio_migration_probe(const char *name) " (%s)" vfio_migration_set_state(const char *name, const char *state) " (%s) state= %s" -vfio_migration_v1_set_state(const char *name, uint32_t state) " (%s) state= %d" vfio_vmstate_change(const char *name, int running, const char *reason, con= st char *dev_state) " (%s) running %d reason %s device state %s" -vfio_v1_vmstate_change(const char *name, int running, const char *reason, = uint32_t dev_state) " (%s) running %d reason %s device state %d" vfio_migration_state_notifier(const char *name, const char *state) " (%s) = state %s" vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data b= uffer size 0x%"PRIx64 -vfio_v1_save_setup(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)" -vfio_save_buffer(const char *name, uint64_t data_offset, uint64_t data_siz= e, uint64_t pending) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%= "PRIx64 -vfio_update_pending(const char *name, uint64_t pending) " (%s) pending 0x%= "PRIx64 vfio_save_device_config_state(const char *name) " (%s)" vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, u= int64_t compatible, uint64_t stopcopy_size) " (%s) precopy 0x%"PRIx64" post= copy 0x%"PRIx64" compatible 0x%"PRIx64" stopcopy size 0x%"PRIx64 -vfio_v1_save_pending(const char *name, uint64_t precopy, uint64_t postcopy= , uint64_t compatible) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" comp= atible 0x%"PRIx64 -vfio_save_iterate(const char *name, int data_size) " (%s) data_size %d" vfio_save_complete_precopy(const char *name, int ret) " (%s) ret %d" -vfio_v1_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 -vfio_v1_load_state_device_data(const char *name, uint64_t data_offset, uin= t64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_size, int ret)= " (%s) size 0x%"PRIx64" ret %d" vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitma= p_size, uint64_t start) "container fd=3D%d, iova=3D0x%"PRIx64" size=3D 0x%"= PRIx64" bitmap_size=3D0x%"PRIx64" start=3D0x%"PRIx64 --=20 2.26.3