From nobody Fri Apr 19 23:30:42 2024 Delivered-To: importer2@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675198486; cv=none; d=zohomail.com; s=zohoarc; b=lpssgf2s7gt+V4DDaNLnwHHU0d0QQQCG8uHxtjjUIy/NBeD+HVQdEKRA3rTv13yYKir1r6YapBik8t0wId+ac3YY7tt/YWCV2WZV6gz5M0wrnLSPJZy1uWUAIjEhO0j1779ui1rzv+uVfWm99iGGxhlKs+eOKuj31oARVGTvokM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675198486; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=x6JUhFYLn9+eyNOwIm3a5Pl14NwSTGas+ArIgpJYKyw=; b=C0WOqZ0GOFTEy23nnbiAVC7qOO6prBKKhwVrhwoKar8QCtXGgOgSp5s/EWN2KwMXMiw1d9PixC8BkyE8svlC4n2LqvQDAMhY1KXpABLrbZNZDR1MKr34+Cryj+0FVv2a/M4ONMwC0yxzZugo1Ohrw3CMYQ80MsBvRxLOGcPSS6s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer2=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1675198486610206.21024799042584; Tue, 31 Jan 2023 12:54:46 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pMxdi-0002pR-KH; Tue, 31 Jan 2023 15:54:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pMxdg-0002pJ-Sa for qemu-devel@nongnu.org; Tue, 31 Jan 2023 15:54:00 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pMxdc-0004VT-L1 for qemu-devel@nongnu.org; Tue, 31 Jan 2023 15:54:00 -0500 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-74-OHFnxmw7OvCFySOhaoe1Kw-1; Tue, 31 Jan 2023 15:53:50 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 829882A5955A; Tue, 31 Jan 2023 20:53:49 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.39.193.239]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4CD2540C2064; Tue, 31 Jan 2023 20:53:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675198436; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x6JUhFYLn9+eyNOwIm3a5Pl14NwSTGas+ArIgpJYKyw=; b=XtdTkIxapF3ezpr6VBf3UJE6MF/9sMAr0KfBj1rwz+hhT94+RZ+C8u7hizEnvDqaxr8w1J YggoZF9jLc/wv6gIJjR6abb7CwvqH7bbVqIHDrPOLZkNln0LGyrZRTKdouBXuIa3FlKtUT /5fErldDR3D6l1v/3MXSEV7zf0bTJZc= X-MC-Unique: OHFnxmw7OvCFySOhaoe1Kw-1 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, yi.l.liu@intel.com, yi.y.sun@intel.com, alex.williamson@redhat.com, clg@redhat.com, qemu-devel@nongnu.org Cc: david@gibson.dropbear.id.au, thuth@redhat.com, farman@linux.ibm.com, mjrosato@linux.ibm.com, akrowiak@linux.ibm.com, pasic@linux.ibm.com, jjherne@linux.ibm.com, jasowang@redhat.com, kvm@vger.kernel.org, jgg@nvidia.com, nicolinc@nvidia.com, kevin.tian@intel.com, chao.p.peng@intel.com, peterx@redhat.com, shameerali.kolothum.thodi@huawei.com, zhangfei.gao@linaro.org, berrange@redhat.com, apopple@nvidia.com, suravee.suthikulpanit@amd.com Subject: [RFC v3 05/18] vfio/common: Move legacy VFIO backend code into separate container.c Date: Tue, 31 Jan 2023 21:52:52 +0100 Message-Id: <20230131205305.2726330-6-eric.auger@redhat.com> In-Reply-To: <20230131205305.2726330-1-eric.auger@redhat.com> References: <20230131205305.2726330-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer2=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer2=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer2=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675198488405100007 Content-Type: text/plain; charset="utf-8" From: Yi Liu Move all the code really dependent on the legacy VFIO container/group into a separate file: container.c. What does remain in common.c is the code related to VFIOAddressSpace and MemoryListeners. Signed-off-by: Eric Auger Signed-off-by: Yi Liu --- - vfio_group_list init before int vfio_kvm_device_fd =3D -1; - vfio_kvm_device_fd is not static since used in common.c/vfio_listener_reg= ion_add - vfio_ram_block_discard_disable comes fisrt before vfio_mig_active following upstream order - vfio_get_iommu_type1_info_cap and vfio_get_info_dma_avail now in container.c because they use vfio iommu type1 structs - vfio_container_add_section_window and vfio_container_del_section_window does not exist anymore --- include/hw/vfio/vfio-common.h | 28 + hw/vfio/common.c | 1197 +------------------------------- hw/vfio/container.c | 1225 +++++++++++++++++++++++++++++++++ hw/vfio/meson.build | 1 + 4 files changed, 1261 insertions(+), 1190 deletions(-) create mode 100644 hw/vfio/container.c diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 2cf6a9e928..bc70e8748a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -33,6 +33,8 @@ =20 #define VFIO_MSG_PREFIX "vfio %s: " =20 +extern const MemoryListener vfio_memory_listener; + enum { VFIO_DEVICE_TYPE_PCI =3D 0, VFIO_DEVICE_TYPE_PLATFORM =3D 1, @@ -190,6 +192,32 @@ typedef struct VFIODisplay { } dmabuf; } VFIODisplay; =20 +void vfio_host_win_add(VFIOContainer *container, + hwaddr min_iova, hwaddr max_iova, + uint64_t iova_pgsizes); +int vfio_host_win_del(VFIOContainer *container, hwaddr min_iova, + hwaddr max_iova); +VFIOAddressSpace *vfio_get_address_space(AddressSpace *as); +void vfio_put_address_space(VFIOAddressSpace *space); +bool vfio_devices_all_running_and_saving(VFIOContainer *container); +bool vfio_devices_all_dirty_tracking(VFIOContainer *container); + +/* container->fd */ +int vfio_dma_unmap(VFIOContainer *container, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb); +int vfio_dma_map(VFIOContainer *container, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly); +void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start); +int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr); + +int vfio_container_add_section_window(VFIOContainer *container, + MemoryRegionSection *section, + Error **errp); +void vfio_container_del_section_window(VFIOContainer *container, + MemoryRegionSection *section); + void vfio_put_base_device(VFIODevice *vbasedev); void vfio_disable_irqindex(VFIODevice *vbasedev, int index); void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index); diff --git a/hw/vfio/common.c b/hw/vfio/common.c index f976a1b662..d2899151db 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -42,256 +42,11 @@ #include "migration/migration.h" #include "sysemu/tpm.h" =20 -VFIOGroupList vfio_group_list =3D - QLIST_HEAD_INITIALIZER(vfio_group_list); static QLIST_HEAD(, VFIOAddressSpace) vfio_address_spaces =3D QLIST_HEAD_INITIALIZER(vfio_address_spaces); =20 -#ifdef CONFIG_KVM -/* - * We have a single VFIO pseudo device per KVM VM. Once created it lives - * for the life of the VM. Closing the file descriptor only drops our - * reference to it and the device's reference to kvm. Therefore once - * initialized, this file descriptor is only released on QEMU exit and - * we'll re-use it should another vfio device be attached before then. - */ -static int vfio_kvm_device_fd =3D -1; -#endif - -static int vfio_ram_block_discard_disable(VFIOContainer *container, bool s= tate) -{ - switch (container->iommu_type) { - case VFIO_TYPE1v2_IOMMU: - case VFIO_TYPE1_IOMMU: - /* - * We support coordinated discarding of RAM via the RamDiscardMana= ger. - */ - return ram_block_uncoordinated_discard_disable(state); - default: - /* - * VFIO_SPAPR_TCE_IOMMU most probably works just fine with - * RamDiscardManager, however, it is completely untested. - * - * VFIO_SPAPR_TCE_v2_IOMMU with "DMA memory preregistering" does - * completely the opposite of managing mapping/pinning dynamically= as - * required by RamDiscardManager. We would have to special-case se= ctions - * with a RamDiscardManager. - */ - return ram_block_discard_disable(state); - } -} - -/* - * Device state interfaces - */ - -bool vfio_mig_active(void) -{ - VFIOGroup *group; - VFIODevice *vbasedev; - - if (QLIST_EMPTY(&vfio_group_list)) { - return false; - } - - QLIST_FOREACH(group, &vfio_group_list, next) { - QLIST_FOREACH(vbasedev, &group->device_list, next) { - if (vbasedev->migration_blocker) { - return false; - } - } - } - return true; -} - -static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) -{ - VFIOGroup *group; - VFIODevice *vbasedev; - MigrationState *ms =3D migrate_get_current(); - - if (!migration_is_setup_or_active(ms->state)) { - return false; - } - - QLIST_FOREACH(group, &container->group_list, container_next) { - QLIST_FOREACH(vbasedev, &group->device_list, next) { - VFIOMigration *migration =3D vbasedev->migration; - - if (!migration) { - return false; - } - - if ((vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) - && (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING= )) { - return false; - } - } - } - return true; -} - -static bool vfio_devices_all_running_and_saving(VFIOContainer *container) -{ - VFIOGroup *group; - VFIODevice *vbasedev; - MigrationState *ms =3D migrate_get_current(); - - if (!migration_is_setup_or_active(ms->state)) { - return false; - } - - QLIST_FOREACH(group, &container->group_list, container_next) { - QLIST_FOREACH(vbasedev, &group->device_list, next) { - VFIOMigration *migration =3D vbasedev->migration; - - if (!migration) { - return false; - } - - if ((migration->device_state & VFIO_DEVICE_STATE_V1_SAVING) && - (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING)) { - continue; - } else { - return false; - } - } - } - return true; -} - -static int vfio_dma_unmap_bitmap(VFIOContainer *container, - hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) -{ - struct vfio_iommu_type1_dma_unmap *unmap; - struct vfio_bitmap *bitmap; - uint64_t pages =3D REAL_HOST_PAGE_ALIGN(size) / qemu_real_host_page_si= ze(); - int ret; - - unmap =3D g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); - - unmap->argsz =3D sizeof(*unmap) + sizeof(*bitmap); - unmap->iova =3D iova; - unmap->size =3D size; - unmap->flags |=3D VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP; - bitmap =3D (struct vfio_bitmap *)&unmap->data; - - /* - * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of - * qemu_real_host_page_size to mark those dirty. Hence set bitmap_pgsi= ze - * to qemu_real_host_page_size. - */ - - bitmap->pgsize =3D qemu_real_host_page_size(); - bitmap->size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / - BITS_PER_BYTE; - - if (bitmap->size > container->max_dirty_bitmap_size) { - error_report("UNMAP: Size of bitmap too big 0x%"PRIx64, - (uint64_t)bitmap->size); - ret =3D -E2BIG; - goto unmap_exit; - } - - bitmap->data =3D g_try_malloc0(bitmap->size); - if (!bitmap->data) { - ret =3D -ENOMEM; - goto unmap_exit; - } - - ret =3D ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); - if (!ret) { - cpu_physical_memory_set_dirty_lebitmap((unsigned long *)bitmap->da= ta, - iotlb->translated_addr, pages); - } else { - error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); - } - - g_free(bitmap->data); -unmap_exit: - g_free(unmap); - return ret; -} - -/* - * DMA - Mapping and unmapping for the "type1" IOMMU interface used on x86 - */ -static int vfio_dma_unmap(VFIOContainer *container, - hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) -{ - struct vfio_iommu_type1_dma_unmap unmap =3D { - .argsz =3D sizeof(unmap), - .flags =3D 0, - .iova =3D iova, - .size =3D size, - }; - - if (iotlb && container->dirty_pages_supported && - vfio_devices_all_running_and_saving(container)) { - return vfio_dma_unmap_bitmap(container, iova, size, iotlb); - } - - while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { - /* - * The type1 backend has an off-by-one bug in the kernel (71a7d3d7= 8e3c - * v4.15) where an overflow in its wrap-around check prevents us f= rom - * unmapping the last page of the address space. Test for the err= or - * condition and re-try the unmap excluding the last page. The - * expectation is that we've never mapped the last page anyway and= this - * unmap request comes via vIOMMU support which also makes it unli= kely - * that this page is used. This bug was introduced well after typ= e1 v2 - * support was introduced, so we shouldn't need to test for v1. A= fix - * is queued for kernel v5.0 so this workaround can be removed once - * affected kernels are sufficiently deprecated. - */ - if (errno =3D=3D EINVAL && unmap.size && !(unmap.iova + unmap.size= ) && - container->iommu_type =3D=3D VFIO_TYPE1v2_IOMMU) { - trace_vfio_dma_unmap_overflow_workaround(); - unmap.size -=3D 1ULL << ctz64(container->pgsizes); - continue; - } - error_report("VFIO_UNMAP_DMA failed: %s", strerror(errno)); - return -errno; - } - - return 0; -} - -static int vfio_dma_map(VFIOContainer *container, hwaddr iova, - ram_addr_t size, void *vaddr, bool readonly) -{ - struct vfio_iommu_type1_dma_map map =3D { - .argsz =3D sizeof(map), - .flags =3D VFIO_DMA_MAP_FLAG_READ, - .vaddr =3D (__u64)(uintptr_t)vaddr, - .iova =3D iova, - .size =3D size, - }; - - if (!readonly) { - map.flags |=3D VFIO_DMA_MAP_FLAG_WRITE; - } - - /* - * Try the mapping, if it fails with EBUSY, unmap the region and try - * again. This shouldn't be necessary, but we sometimes see it in - * the VGA ROM space. - */ - if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0 || - (errno =3D=3D EBUSY && vfio_dma_unmap(container, iova, size, NULL)= =3D=3D 0 && - ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0)) { - return 0; - } - - error_report("VFIO_MAP_DMA failed: %s", strerror(errno)); - return -errno; -} - -static void vfio_host_win_add(VFIOContainer *container, - hwaddr min_iova, hwaddr max_iova, - uint64_t iova_pgsizes) +void vfio_host_win_add(VFIOContainer *container, hwaddr min_iova, + hwaddr max_iova, uint64_t iova_pgsizes) { VFIOHostDMAWindow *hostwin; =20 @@ -312,8 +67,8 @@ static void vfio_host_win_add(VFIOContainer *container, QLIST_INSERT_HEAD(&container->hostwin_list, hostwin, hostwin_next); } =20 -static int vfio_host_win_del(VFIOContainer *container, hwaddr min_iova, - hwaddr max_iova) +int vfio_host_win_del(VFIOContainer *container, + hwaddr min_iova, hwaddr max_iova) { VFIOHostDMAWindow *hostwin; =20 @@ -590,92 +345,6 @@ static bool vfio_known_safe_misalignment(MemoryRegionS= ection *section) return true; } =20 -static int vfio_container_add_section_window(VFIOContainer *container, - MemoryRegionSection *section, - Error **errp) -{ - VFIOHostDMAWindow *hostwin; - hwaddr pgsize =3D 0; - int ret; - - if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { - return 0; - } - - /* For now intersections are not allowed, we may relax this later */ - QLIST_FOREACH(hostwin, &container->hostwin_list, hostwin_next) { - if (ranges_overlap(hostwin->min_iova, - hostwin->max_iova - hostwin->min_iova + 1, - section->offset_within_address_space, - int128_get64(section->size))) { - error_setg(errp, - "region [0x%"PRIx64",0x%"PRIx64"] overlaps with existing" - "host DMA window [0x%"PRIx64",0x%"PRIx64"]", - section->offset_within_address_space, - section->offset_within_address_space + - int128_get64(section->size) - 1, - hostwin->min_iova, hostwin->max_iova); - return -EINVAL; - } - } - - ret =3D vfio_spapr_create_window(container, section, &pgsize); - if (ret) { - error_setg_errno(errp, -ret, "Failed to create SPAPR window"); - return ret; - } - - vfio_host_win_add(container, section->offset_within_address_space, - section->offset_within_address_space + - int128_get64(section->size) - 1, pgsize); -#ifdef CONFIG_KVM - if (kvm_enabled()) { - VFIOGroup *group; - IOMMUMemoryRegion *iommu_mr =3D IOMMU_MEMORY_REGION(section->mr); - struct kvm_vfio_spapr_tce param; - struct kvm_device_attr attr =3D { - .group =3D KVM_DEV_VFIO_GROUP, - .attr =3D KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE, - .addr =3D (uint64_t)(unsigned long)¶m, - }; - - if (!memory_region_iommu_get_attr(iommu_mr, IOMMU_ATTR_SPAPR_TCE_F= D, - ¶m.tablefd)) { - QLIST_FOREACH(group, &container->group_list, container_next) { - param.groupfd =3D group->fd; - if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr))= { - error_report("vfio: failed to setup fd %d " - "for a group with fd %d: %s", - param.tablefd, param.groupfd, - strerror(errno)); - return 0; - } - trace_vfio_spapr_group_attach(param.groupfd, param.tablefd= ); - } - } - } -#endif - return 0; -} - -static void vfio_container_del_section_window(VFIOContainer *container, - MemoryRegionSection *section) -{ - if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { - return; - } - - vfio_spapr_remove_window(container, - section->offset_within_address_space); - if (vfio_host_win_del(container, - section->offset_within_address_space, - section->offset_within_address_space + - int128_get64(section->size) - 1) < 0) { - hw_error("%s: Cannot delete missing window at %"HWADDR_PRIx, - __func__, section->offset_within_address_space); - } -} - static void vfio_listener_region_add(MemoryListener *listener, MemoryRegionSection *section) { @@ -984,26 +653,6 @@ static void vfio_listener_region_del(MemoryListener *l= istener, vfio_container_del_section_window(container, section); } =20 -static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool st= art) -{ - int ret; - struct vfio_iommu_type1_dirty_bitmap dirty =3D { - .argsz =3D sizeof(dirty), - }; - - if (start) { - dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; - } else { - dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; - } - - ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); - if (ret) { - error_report("Failed to set dirty tracking flag 0x%x errno: %d", - dirty.flags, errno); - } -} - static void vfio_listener_log_global_start(MemoryListener *listener) { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); @@ -1018,58 +667,6 @@ static void vfio_listener_log_global_stop(MemoryListe= ner *listener) vfio_set_dirty_page_tracking(container, false); } =20 -static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, - uint64_t size, ram_addr_t ram_addr) -{ - struct vfio_iommu_type1_dirty_bitmap *dbitmap; - struct vfio_iommu_type1_dirty_bitmap_get *range; - uint64_t pages; - int ret; - - dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); - - dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); - dbitmap->flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; - range =3D (struct vfio_iommu_type1_dirty_bitmap_get *)&dbitmap->data; - range->iova =3D iova; - range->size =3D size; - - /* - * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of - * qemu_real_host_page_size to mark those dirty. Hence set bitmap's pg= size - * to qemu_real_host_page_size. - */ - range->bitmap.pgsize =3D qemu_real_host_page_size(); - - pages =3D REAL_HOST_PAGE_ALIGN(range->size) / qemu_real_host_page_size= (); - range->bitmap.size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / - BITS_PER_BYTE; - range->bitmap.data =3D g_try_malloc0(range->bitmap.size); - if (!range->bitmap.data) { - ret =3D -ENOMEM; - goto err_out; - } - - ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); - if (ret) { - error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 - " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, - (uint64_t)range->size, errno); - goto err_out; - } - - cpu_physical_memory_set_dirty_lebitmap((unsigned long *)range->bitmap.= data, - ram_addr, pages); - - trace_vfio_get_dirty_bitmap(container->fd, range->iova, range->size, - range->bitmap.size, ram_addr); -err_out: - g_free(range->bitmap.data); - g_free(dbitmap); - - return ret; -} - typedef struct { IOMMUNotifier n; VFIOGuestIOMMU *giommu; @@ -1209,7 +806,7 @@ static void vfio_listener_log_sync(MemoryListener *lis= tener, } } =20 -static const MemoryListener vfio_memory_listener =3D { +const MemoryListener vfio_memory_listener =3D { .name =3D "vfio", .region_add =3D vfio_listener_region_add, .region_del =3D vfio_listener_region_del, @@ -1218,121 +815,7 @@ static const MemoryListener vfio_memory_listener =3D= { .log_sync =3D vfio_listener_log_sync, }; =20 -static void vfio_listener_release(VFIOContainer *container) -{ - memory_listener_unregister(&container->listener); - if (container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { - memory_listener_unregister(&container->prereg_listener); - } -} - -static struct vfio_info_cap_header * -vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t= id) -{ - if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { - return NULL; - } - - return vfio_get_cap((void *)info, info->cap_offset, id); -} - -bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, - unsigned int *avail) -{ - struct vfio_info_cap_header *hdr; - struct vfio_iommu_type1_info_dma_avail *cap; - - /* If the capability cannot be found, assume no DMA limiting */ - hdr =3D vfio_get_iommu_type1_info_cap(info, - VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL); - if (hdr =3D=3D NULL) { - return false; - } - - if (avail !=3D NULL) { - cap =3D (void *) hdr; - *avail =3D cap->avail; - } - - return true; -} - -void vfio_reset_handler(void *opaque) -{ - VFIOGroup *group; - VFIODevice *vbasedev; - - QLIST_FOREACH(group, &vfio_group_list, next) { - QLIST_FOREACH(vbasedev, &group->device_list, next) { - if (vbasedev->dev->realized) { - vbasedev->ops->vfio_compute_needs_reset(vbasedev); - } - } - } - - QLIST_FOREACH(group, &vfio_group_list, next) { - QLIST_FOREACH(vbasedev, &group->device_list, next) { - if (vbasedev->dev->realized && vbasedev->needs_reset) { - vbasedev->ops->vfio_hot_reset_multi(vbasedev); - } - } - } -} - -static void vfio_kvm_device_add_group(VFIOGroup *group) -{ -#ifdef CONFIG_KVM - struct kvm_device_attr attr =3D { - .group =3D KVM_DEV_VFIO_GROUP, - .attr =3D KVM_DEV_VFIO_GROUP_ADD, - .addr =3D (uint64_t)(unsigned long)&group->fd, - }; - - if (!kvm_enabled()) { - return; - } - - if (vfio_kvm_device_fd < 0) { - struct kvm_create_device cd =3D { - .type =3D KVM_DEV_TYPE_VFIO, - }; - - if (kvm_vm_ioctl(kvm_state, KVM_CREATE_DEVICE, &cd)) { - error_report("Failed to create KVM VFIO device: %m"); - return; - } - - vfio_kvm_device_fd =3D cd.fd; - } - - if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr)) { - error_report("Failed to add group %d to KVM VFIO device: %m", - group->groupid); - } -#endif -} - -static void vfio_kvm_device_del_group(VFIOGroup *group) -{ -#ifdef CONFIG_KVM - struct kvm_device_attr attr =3D { - .group =3D KVM_DEV_VFIO_GROUP, - .attr =3D KVM_DEV_VFIO_GROUP_DEL, - .addr =3D (uint64_t)(unsigned long)&group->fd, - }; - - if (vfio_kvm_device_fd < 0) { - return; - } - - if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr)) { - error_report("Failed to remove group %d from KVM VFIO device: %m", - group->groupid); - } -#endif -} - -static VFIOAddressSpace *vfio_get_address_space(AddressSpace *as) +VFIOAddressSpace *vfio_get_address_space(AddressSpace *as) { VFIOAddressSpace *space; =20 @@ -1352,676 +835,10 @@ static VFIOAddressSpace *vfio_get_address_space(Add= ressSpace *as) return space; } =20 -static void vfio_put_address_space(VFIOAddressSpace *space) +void vfio_put_address_space(VFIOAddressSpace *space) { if (QLIST_EMPTY(&space->containers)) { QLIST_REMOVE(space, list); g_free(space); } } - -/* - * vfio_get_iommu_type - selects the richest iommu_type (v2 first) - */ -static int vfio_get_iommu_type(VFIOContainer *container, - Error **errp) -{ - int iommu_types[] =3D { VFIO_TYPE1v2_IOMMU, VFIO_TYPE1_IOMMU, - VFIO_SPAPR_TCE_v2_IOMMU, VFIO_SPAPR_TCE_IOMMU }; - int i; - - for (i =3D 0; i < ARRAY_SIZE(iommu_types); i++) { - if (ioctl(container->fd, VFIO_CHECK_EXTENSION, iommu_types[i])) { - return iommu_types[i]; - } - } - error_setg(errp, "No available IOMMU models"); - return -EINVAL; -} - -static int vfio_init_container(VFIOContainer *container, int group_fd, - Error **errp) -{ - int iommu_type, ret; - - iommu_type =3D vfio_get_iommu_type(container, errp); - if (iommu_type < 0) { - return iommu_type; - } - - ret =3D ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); - if (ret) { - error_setg_errno(errp, errno, "Failed to set group container"); - return -errno; - } - - while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { - if (iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { - /* - * On sPAPR, despite the IOMMU subdriver always advertises v1 = and - * v2, the running platform may not support v2 and there is no - * way to guess it until an IOMMU group gets added to the cont= ainer. - * So in case it fails with v2, try v1 as a fallback. - */ - iommu_type =3D VFIO_SPAPR_TCE_IOMMU; - continue; - } - error_setg_errno(errp, errno, "Failed to set iommu for container"); - return -errno; - } - - container->iommu_type =3D iommu_type; - return 0; -} - -static int vfio_get_iommu_info(VFIOContainer *container, - struct vfio_iommu_type1_info **info) -{ - - size_t argsz =3D sizeof(struct vfio_iommu_type1_info); - - *info =3D g_new0(struct vfio_iommu_type1_info, 1); -again: - (*info)->argsz =3D argsz; - - if (ioctl(container->fd, VFIO_IOMMU_GET_INFO, *info)) { - g_free(*info); - *info =3D NULL; - return -errno; - } - - if (((*info)->argsz > argsz)) { - argsz =3D (*info)->argsz; - *info =3D g_realloc(*info, argsz); - goto again; - } - - return 0; -} - -static struct vfio_info_cap_header * -vfio_get_iommu_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) -{ - struct vfio_info_cap_header *hdr; - void *ptr =3D info; - - if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { - return NULL; - } - - for (hdr =3D ptr + info->cap_offset; hdr !=3D ptr; hdr =3D ptr + hdr->= next) { - if (hdr->id =3D=3D id) { - return hdr; - } - } - - return NULL; -} - -static void vfio_get_iommu_info_migration(VFIOContainer *container, - struct vfio_iommu_type1_info *inf= o) -{ - struct vfio_info_cap_header *hdr; - struct vfio_iommu_type1_info_cap_migration *cap_mig; - - hdr =3D vfio_get_iommu_info_cap(info, VFIO_IOMMU_TYPE1_INFO_CAP_MIGRAT= ION); - if (!hdr) { - return; - } - - cap_mig =3D container_of(hdr, struct vfio_iommu_type1_info_cap_migrati= on, - header); - - /* - * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of - * qemu_real_host_page_size to mark those dirty. - */ - if (cap_mig->pgsize_bitmap & qemu_real_host_page_size()) { - container->dirty_pages_supported =3D true; - container->max_dirty_bitmap_size =3D cap_mig->max_dirty_bitmap_siz= e; - container->dirty_pgsizes =3D cap_mig->pgsize_bitmap; - } -} - -static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, - Error **errp) -{ - VFIOContainer *container; - int ret, fd; - VFIOAddressSpace *space; - - space =3D vfio_get_address_space(as); - - /* - * VFIO is currently incompatible with discarding of RAM insofar as the - * madvise to purge (zap) the page from QEMU's address space does not - * interact with the memory API and therefore leaves stale virtual to - * physical mappings in the IOMMU if the page was previously pinned. = We - * therefore set discarding broken for each group added to a container, - * whether the container is used individually or shared. This provides - * us with options to allow devices within a group to opt-in and allow - * discarding, so long as it is done consistently for a group (for ins= tance - * if the device is an mdev device where it is known that the host ven= dor - * driver will never pin pages outside of the working set of the guest - * driver, which would thus not be discarding candidates). - * - * The first opportunity to induce pinning occurs here where we attemp= t to - * attach the group to existing containers within the AddressSpace. I= f any - * pages are already zapped from the virtual address space, such as fr= om - * previous discards, new pinning will cause valid mappings to be - * re-established. Likewise, when the overall MemoryListener for a new - * container is registered, a replay of mappings within the AddressSpa= ce - * will occur, re-establishing any previously zapped pages as well. - * - * Especially virtio-balloon is currently only prevented from discardi= ng - * new memory, it will not yet set ram_block_discard_set_required() and - * therefore, neither stops us here or deals with the sudden memory - * consumption of inflated memory. - * - * We do support discarding of memory coordinated via the RamDiscardMa= nager - * with some IOMMU types. vfio_ram_block_discard_disable() handles the - * details once we know which type of IOMMU we are using. - */ - - QLIST_FOREACH(container, &space->containers, next) { - if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { - ret =3D vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, - "Cannot set discarding of RAM broken"); - if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, - &container->fd)) { - error_report("vfio: error disconnecting group %d from" - " container", group->groupid); - } - return ret; - } - group->container =3D container; - QLIST_INSERT_HEAD(&container->group_list, group, container_nex= t); - vfio_kvm_device_add_group(group); - return 0; - } - } - - fd =3D qemu_open_old("/dev/vfio/vfio", O_RDWR); - if (fd < 0) { - error_setg_errno(errp, errno, "failed to open /dev/vfio/vfio"); - ret =3D -errno; - goto put_space_exit; - } - - ret =3D ioctl(fd, VFIO_GET_API_VERSION); - if (ret !=3D VFIO_API_VERSION) { - error_setg(errp, "supported vfio version: %d, " - "reported version: %d", VFIO_API_VERSION, ret); - ret =3D -EINVAL; - goto close_fd_exit; - } - - container =3D g_malloc0(sizeof(*container)); - container->space =3D space; - container->fd =3D fd; - container->error =3D NULL; - container->dirty_pages_supported =3D false; - container->dma_max_mappings =3D 0; - QLIST_INIT(&container->giommu_list); - QLIST_INIT(&container->hostwin_list); - QLIST_INIT(&container->vrdl_list); - - ret =3D vfio_init_container(container, group->fd, errp); - if (ret) { - goto free_container_exit; - } - - ret =3D vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"= ); - goto free_container_exit; - } - - switch (container->iommu_type) { - case VFIO_TYPE1v2_IOMMU: - case VFIO_TYPE1_IOMMU: - { - struct vfio_iommu_type1_info *info; - - ret =3D vfio_get_iommu_info(container, &info); - if (ret) { - error_setg_errno(errp, -ret, "Failed to get VFIO IOMMU info"); - goto enable_discards_exit; - } - - if (info->flags & VFIO_IOMMU_INFO_PGSIZES) { - container->pgsizes =3D info->iova_pgsizes; - } else { - container->pgsizes =3D qemu_real_host_page_size(); - } - - if (!vfio_get_info_dma_avail(info, &container->dma_max_mappings)) { - container->dma_max_mappings =3D 65535; - } - vfio_get_iommu_info_migration(container, info); - g_free(info); - - /* - * FIXME: We should parse VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE - * information to get the actual window extent rather than assume - * a 64-bit IOVA address space. - */ - vfio_host_win_add(container, 0, (hwaddr)-1, container->pgsizes); - - break; - } - case VFIO_SPAPR_TCE_v2_IOMMU: - case VFIO_SPAPR_TCE_IOMMU: - { - struct vfio_iommu_spapr_tce_info info; - bool v2 =3D container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU; - - /* - * The host kernel code implementing VFIO_IOMMU_DISABLE is called - * when container fd is closed so we do not call it explicitly - * in this file. - */ - if (!v2) { - ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); - if (ret) { - error_setg_errno(errp, errno, "failed to enable container"= ); - ret =3D -errno; - goto enable_discards_exit; - } - } else { - container->prereg_listener =3D vfio_prereg_listener; - - memory_listener_register(&container->prereg_listener, - &address_space_memory); - if (container->error) { - memory_listener_unregister(&container->prereg_listener); - ret =3D -1; - error_propagate_prepend(errp, container->error, - "RAM memory listener initialization failed: "); - goto enable_discards_exit; - } - } - - info.argsz =3D sizeof(info); - ret =3D ioctl(fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info); - if (ret) { - error_setg_errno(errp, errno, - "VFIO_IOMMU_SPAPR_TCE_GET_INFO failed"); - ret =3D -errno; - if (v2) { - memory_listener_unregister(&container->prereg_listener); - } - goto enable_discards_exit; - } - - if (v2) { - container->pgsizes =3D info.ddw.pgsizes; - /* - * There is a default window in just created container. - * To make region_add/del simpler, we better remove this - * window now and let those iommu_listener callbacks - * create/remove them when needed. - */ - ret =3D vfio_spapr_remove_window(container, info.dma32_window_= start); - if (ret) { - error_setg_errno(errp, -ret, - "failed to remove existing window"); - goto enable_discards_exit; - } - } else { - /* The default table uses 4K pages */ - container->pgsizes =3D 0x1000; - vfio_host_win_add(container, info.dma32_window_start, - info.dma32_window_start + - info.dma32_window_size - 1, - 0x1000); - } - } - } - - vfio_kvm_device_add_group(group); - - QLIST_INIT(&container->group_list); - QLIST_INSERT_HEAD(&space->containers, container, next); - - group->container =3D container; - QLIST_INSERT_HEAD(&container->group_list, group, container_next); - - container->listener =3D vfio_memory_listener; - - memory_listener_register(&container->listener, container->space->as); - - if (container->error) { - ret =3D -1; - error_propagate_prepend(errp, container->error, - "memory listener initialization failed: "); - goto listener_release_exit; - } - - container->initialized =3D true; - - return 0; -listener_release_exit: - QLIST_REMOVE(group, container_next); - QLIST_REMOVE(container, next); - vfio_kvm_device_del_group(group); - vfio_listener_release(container); - -enable_discards_exit: - vfio_ram_block_discard_disable(container, false); - -free_container_exit: - g_free(container); - -close_fd_exit: - close(fd); - -put_space_exit: - vfio_put_address_space(space); - - return ret; -} - -static void vfio_disconnect_container(VFIOGroup *group) -{ - VFIOContainer *container =3D group->container; - - QLIST_REMOVE(group, container_next); - group->container =3D NULL; - - /* - * Explicitly release the listener first before unset container, - * since unset may destroy the backend container if it's the last - * group. - */ - if (QLIST_EMPTY(&container->group_list)) { - vfio_listener_release(container); - } - - if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, &container->fd)) { - error_report("vfio: error disconnecting group %d from container", - group->groupid); - } - - if (QLIST_EMPTY(&container->group_list)) { - VFIOAddressSpace *space =3D container->space; - VFIOGuestIOMMU *giommu, *tmp; - VFIOHostDMAWindow *hostwin, *next; - - QLIST_REMOVE(container, next); - - QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, t= mp) { - memory_region_unregister_iommu_notifier( - MEMORY_REGION(giommu->iommu_mr), &giommu->n); - QLIST_REMOVE(giommu, giommu_next); - g_free(giommu); - } - - QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, - next) { - QLIST_REMOVE(hostwin, hostwin_next); - g_free(hostwin); - } - - trace_vfio_disconnect_container(container->fd); - close(container->fd); - g_free(container); - - vfio_put_address_space(space); - } -} - -VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) -{ - VFIOGroup *group; - char path[32]; - struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; - - QLIST_FOREACH(group, &vfio_group_list, next) { - if (group->groupid =3D=3D groupid) { - /* Found it. Now is it already in the right context? */ - if (group->container->space->as =3D=3D as) { - return group; - } else { - error_setg(errp, "group %d used in multiple address spaces= ", - group->groupid); - return NULL; - } - } - } - - group =3D g_malloc0(sizeof(*group)); - - snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); - group->fd =3D qemu_open_old(path, O_RDWR); - if (group->fd < 0) { - error_setg_errno(errp, errno, "failed to open %s", path); - goto free_group_exit; - } - - if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { - error_setg_errno(errp, errno, "failed to get group %d status", gro= upid); - goto close_fd_exit; - } - - if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { - error_setg(errp, "group %d is not viable", groupid); - error_append_hint(errp, - "Please ensure all devices within the iommu_grou= p " - "are bound to their vfio bus driver.\n"); - goto close_fd_exit; - } - - group->groupid =3D groupid; - QLIST_INIT(&group->device_list); - - if (vfio_connect_container(group, as, errp)) { - error_prepend(errp, "failed to setup container for group %d: ", - groupid); - goto close_fd_exit; - } - - if (QLIST_EMPTY(&vfio_group_list)) { - qemu_register_reset(vfio_reset_handler, NULL); - } - - QLIST_INSERT_HEAD(&vfio_group_list, group, next); - - return group; - -close_fd_exit: - close(group->fd); - -free_group_exit: - g_free(group); - - return NULL; -} - -void vfio_put_group(VFIOGroup *group) -{ - if (!group || !QLIST_EMPTY(&group->device_list)) { - return; - } - - if (!group->ram_block_discard_allowed) { - vfio_ram_block_discard_disable(group->container, false); - } - vfio_kvm_device_del_group(group); - vfio_disconnect_container(group); - QLIST_REMOVE(group, next); - trace_vfio_put_group(group->fd); - close(group->fd); - g_free(group); - - if (QLIST_EMPTY(&vfio_group_list)) { - qemu_unregister_reset(vfio_reset_handler, NULL); - } -} - -int vfio_get_device(VFIOGroup *group, const char *name, - VFIODevice *vbasedev, Error **errp) -{ - struct vfio_device_info dev_info =3D { .argsz =3D sizeof(dev_info) }; - int ret, fd; - - fd =3D ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); - if (fd < 0) { - error_setg_errno(errp, errno, "error getting device from group %d", - group->groupid); - error_append_hint(errp, - "Verify all devices in group %d are bound to vfio- " - "or pci-stub and not already in use\n", group->group= id); - return fd; - } - - ret =3D ioctl(fd, VFIO_DEVICE_GET_INFO, &dev_info); - if (ret) { - error_setg_errno(errp, errno, "error getting device info"); - close(fd); - return ret; - } - - /* - * Set discarding of RAM as not broken for this group if the driver kn= ows - * the device operates compatibly with discarding. Setting must be - * consistent per group, but since compatibility is really only possib= le - * with mdev currently, we expect singleton groups. - */ - if (vbasedev->ram_block_discard_allowed !=3D - group->ram_block_discard_allowed) { - if (!QLIST_EMPTY(&group->device_list)) { - error_setg(errp, "Inconsistent setting of support for discardi= ng " - "RAM (e.g., balloon) within group"); - close(fd); - return -1; - } - - if (!group->ram_block_discard_allowed) { - group->ram_block_discard_allowed =3D true; - vfio_ram_block_discard_disable(group->container, false); - } - } - - vbasedev->fd =3D fd; - vbasedev->group =3D group; - QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); - - vbasedev->num_irqs =3D dev_info.num_irqs; - vbasedev->num_regions =3D dev_info.num_regions; - vbasedev->flags =3D dev_info.flags; - - trace_vfio_get_device(name, dev_info.flags, dev_info.num_regions, - dev_info.num_irqs); - - vbasedev->reset_works =3D !!(dev_info.flags & VFIO_DEVICE_FLAGS_RESET); - return 0; -} - -void vfio_put_base_device(VFIODevice *vbasedev) -{ - if (!vbasedev->group) { - return; - } - QLIST_REMOVE(vbasedev, next); - vbasedev->group =3D NULL; - trace_vfio_put_base_device(vbasedev->fd); - close(vbasedev->fd); -} - -/* - * Interfaces for IBM EEH (Enhanced Error Handling) - */ -static bool vfio_eeh_container_ok(VFIOContainer *container) -{ - /* - * As of 2016-03-04 (linux-4.5) the host kernel EEH/VFIO - * implementation is broken if there are multiple groups in a - * container. The hardware works in units of Partitionable - * Endpoints (=3D=3D IOMMU groups) and the EEH operations naively - * iterate across all groups in the container, without any logic - * to make sure the groups have their state synchronized. For - * certain operations (ENABLE) that might be ok, until an error - * occurs, but for others (GET_STATE) it's clearly broken. - */ - - /* - * XXX Once fixed kernels exist, test for them here - */ - - if (QLIST_EMPTY(&container->group_list)) { - return false; - } - - if (QLIST_NEXT(QLIST_FIRST(&container->group_list), container_next)) { - return false; - } - - return true; -} - -static int vfio_eeh_container_op(VFIOContainer *container, uint32_t op) -{ - struct vfio_eeh_pe_op pe_op =3D { - .argsz =3D sizeof(pe_op), - .op =3D op, - }; - int ret; - - if (!vfio_eeh_container_ok(container)) { - error_report("vfio/eeh: EEH_PE_OP 0x%x: " - "kernel requires a container with exactly one group",= op); - return -EPERM; - } - - ret =3D ioctl(container->fd, VFIO_EEH_PE_OP, &pe_op); - if (ret < 0) { - error_report("vfio/eeh: EEH_PE_OP 0x%x failed: %m", op); - return -errno; - } - - return ret; -} - -static VFIOContainer *vfio_eeh_as_container(AddressSpace *as) -{ - VFIOAddressSpace *space =3D vfio_get_address_space(as); - VFIOContainer *container =3D NULL; - - if (QLIST_EMPTY(&space->containers)) { - /* No containers to act on */ - goto out; - } - - container =3D QLIST_FIRST(&space->containers); - - if (QLIST_NEXT(container, next)) { - /* We don't yet have logic to synchronize EEH state across - * multiple containers */ - container =3D NULL; - goto out; - } - -out: - vfio_put_address_space(space); - return container; -} - -bool vfio_eeh_as_ok(AddressSpace *as) -{ - VFIOContainer *container =3D vfio_eeh_as_container(as); - - return (container !=3D NULL) && vfio_eeh_container_ok(container); -} - -int vfio_eeh_as_op(AddressSpace *as, uint32_t op) -{ - VFIOContainer *container =3D vfio_eeh_as_container(as); - - if (!container) { - return -ENODEV; - } - return vfio_eeh_container_op(container, op); -} diff --git a/hw/vfio/container.c b/hw/vfio/container.c new file mode 100644 index 0000000000..68d4e360ed --- /dev/null +++ b/hw/vfio/container.c @@ -0,0 +1,1225 @@ +/* + * generic functions used by VFIO devices + * + * Copyright Red Hat, Inc. 2012 + * + * Authors: + * Alex Williamson + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + * Based on qemu-kvm device-assignment: + * Adapted for KVM by Qumranet. + * Copyright (c) 2007, Neocleus, Alex Novik (alex@neocleus.com) + * Copyright (c) 2007, Neocleus, Guy Zana (guy@neocleus.com) + * Copyright (C) 2008, Qumranet, Amit Shah (amit.shah@qumranet.com) + * Copyright (C) 2008, Red Hat, Amit Shah (amit.shah@redhat.com) + * Copyright (C) 2008, IBM, Muli Ben-Yehuda (muli@il.ibm.com) + */ + +#include "qemu/osdep.h" +#include +#ifdef CONFIG_KVM +#include +#endif +#include + +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio.h" +#include "exec/address-spaces.h" +#include "exec/memory.h" +#include "exec/ram_addr.h" +#include "hw/hw.h" +#include "qemu/error-report.h" +#include "qemu/range.h" +#include "sysemu/kvm.h" +#include "sysemu/reset.h" +#include "trace.h" +#include "qapi/error.h" +#include "migration/migration.h" + +VFIOGroupList vfio_group_list =3D + QLIST_HEAD_INITIALIZER(vfio_group_list); + +#ifdef CONFIG_KVM +/* + * We have a single VFIO pseudo device per KVM VM. Once created it lives + * for the life of the VM. Closing the file descriptor only drops our + * reference to it and the device's reference to kvm. Therefore once + * initialized, this file descriptor is only released on QEMU exit and + * we'll re-use it should another vfio device be attached before then. + */ +static int vfio_kvm_device_fd =3D -1; +#endif + +static int vfio_ram_block_discard_disable(VFIOContainer *container, bool s= tate) +{ + switch (container->iommu_type) { + case VFIO_TYPE1v2_IOMMU: + case VFIO_TYPE1_IOMMU: + /* + * We support coordinated discarding of RAM via the RamDiscardMana= ger. + */ + return ram_block_uncoordinated_discard_disable(state); + default: + /* + * VFIO_SPAPR_TCE_IOMMU most probably works just fine with + * RamDiscardManager, however, it is completely untested. + * + * VFIO_SPAPR_TCE_v2_IOMMU with "DMA memory preregistering" does + * completely the opposite of managing mapping/pinning dynamically= as + * required by RamDiscardManager. We would have to special-case se= ctions + * with a RamDiscardManager. + */ + return ram_block_discard_disable(state); + } +} + +/* + * Device state interfaces + */ + +bool vfio_mig_active(void) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + + if (QLIST_EMPTY(&vfio_group_list)) { + return false; + } + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->migration_blocker) { + return false; + } + } + } + return true; +} + +bool vfio_devices_all_dirty_tracking(VFIOContainer *container) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + MigrationState *ms =3D migrate_get_current(); + + if (!migration_is_setup_or_active(ms->state)) { + return false; + } + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + VFIOMigration *migration =3D vbasedev->migration; + + if (!migration) { + return false; + } + + if ((vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) + && (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING= )) { + return false; + } + } + } + return true; +} + +bool vfio_devices_all_running_and_saving(VFIOContainer *container) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + MigrationState *ms =3D migrate_get_current(); + + if (!migration_is_setup_or_active(ms->state)) { + return false; + } + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + VFIOMigration *migration =3D vbasedev->migration; + + if (!migration) { + return false; + } + + if ((migration->device_state & VFIO_DEVICE_STATE_V1_SAVING) && + (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING)) { + continue; + } else { + return false; + } + } + } + return true; +} + +static int vfio_dma_unmap_bitmap(VFIOContainer *container, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb) +{ + struct vfio_iommu_type1_dma_unmap *unmap; + struct vfio_bitmap *bitmap; + uint64_t pages =3D REAL_HOST_PAGE_ALIGN(size) / qemu_real_host_page_si= ze(); + int ret; + + unmap =3D g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); + + unmap->argsz =3D sizeof(*unmap) + sizeof(*bitmap); + unmap->iova =3D iova; + unmap->size =3D size; + unmap->flags |=3D VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP; + bitmap =3D (struct vfio_bitmap *)&unmap->data; + + /* + * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of + * qemu_real_host_page_size to mark those dirty. Hence set bitmap_pgsi= ze + * to qemu_real_host_page_size. + */ + + bitmap->pgsize =3D qemu_real_host_page_size(); + bitmap->size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / + BITS_PER_BYTE; + + if (bitmap->size > container->max_dirty_bitmap_size) { + error_report("UNMAP: Size of bitmap too big 0x%"PRIx64, + (uint64_t)bitmap->size); + ret =3D -E2BIG; + goto unmap_exit; + } + + bitmap->data =3D g_try_malloc0(bitmap->size); + if (!bitmap->data) { + ret =3D -ENOMEM; + goto unmap_exit; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + if (!ret) { + cpu_physical_memory_set_dirty_lebitmap((unsigned long *)bitmap->da= ta, + iotlb->translated_addr, pages); + } else { + error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); + } + + g_free(bitmap->data); +unmap_exit: + g_free(unmap); + return ret; +} + +/* + * DMA - Mapping and unmapping for the "type1" IOMMU interface used on x86 + */ +int vfio_dma_unmap(VFIOContainer *container, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb) +{ + struct vfio_iommu_type1_dma_unmap unmap =3D { + .argsz =3D sizeof(unmap), + .flags =3D 0, + .iova =3D iova, + .size =3D size, + }; + + if (iotlb && container->dirty_pages_supported && + vfio_devices_all_running_and_saving(container)) { + return vfio_dma_unmap_bitmap(container, iova, size, iotlb); + } + + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { + /* + * The type1 backend has an off-by-one bug in the kernel (71a7d3d7= 8e3c + * v4.15) where an overflow in its wrap-around check prevents us f= rom + * unmapping the last page of the address space. Test for the err= or + * condition and re-try the unmap excluding the last page. The + * expectation is that we've never mapped the last page anyway and= this + * unmap request comes via vIOMMU support which also makes it unli= kely + * that this page is used. This bug was introduced well after typ= e1 v2 + * support was introduced, so we shouldn't need to test for v1. A= fix + * is queued for kernel v5.0 so this workaround can be removed once + * affected kernels are sufficiently deprecated. + */ + if (errno =3D=3D EINVAL && unmap.size && !(unmap.iova + unmap.size= ) && + container->iommu_type =3D=3D VFIO_TYPE1v2_IOMMU) { + trace_vfio_dma_unmap_overflow_workaround(); + unmap.size -=3D 1ULL << ctz64(container->pgsizes); + continue; + } + error_report("VFIO_UNMAP_DMA failed: %s", strerror(errno)); + return -errno; + } + + return 0; +} + +int vfio_dma_map(VFIOContainer *container, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly) +{ + struct vfio_iommu_type1_dma_map map =3D { + .argsz =3D sizeof(map), + .flags =3D VFIO_DMA_MAP_FLAG_READ, + .vaddr =3D (__u64)(uintptr_t)vaddr, + .iova =3D iova, + .size =3D size, + }; + + if (!readonly) { + map.flags |=3D VFIO_DMA_MAP_FLAG_WRITE; + } + + /* + * Try the mapping, if it fails with EBUSY, unmap the region and try + * again. This shouldn't be necessary, but we sometimes see it in + * the VGA ROM space. + */ + if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0 || + (errno =3D=3D EBUSY && vfio_dma_unmap(container, iova, size, NULL)= =3D=3D 0 && + ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0)) { + return 0; + } + + error_report("VFIO_MAP_DMA failed: %s", strerror(errno)); + return -errno; +} + +void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start) +{ + int ret; + struct vfio_iommu_type1_dirty_bitmap dirty =3D { + .argsz =3D sizeof(dirty), + }; + + if (start) { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; + } else { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, errno); + } +} + +int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr) +{ + struct vfio_iommu_type1_dirty_bitmap *dbitmap; + struct vfio_iommu_type1_dirty_bitmap_get *range; + uint64_t pages; + int ret; + + dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); + + dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); + dbitmap->flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; + range =3D (struct vfio_iommu_type1_dirty_bitmap_get *)&dbitmap->data; + range->iova =3D iova; + range->size =3D size; + + /* + * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of + * qemu_real_host_page_size to mark those dirty. Hence set bitmap's pg= size + * to qemu_real_host_page_size. + */ + range->bitmap.pgsize =3D qemu_real_host_page_size(); + + pages =3D REAL_HOST_PAGE_ALIGN(range->size) / qemu_real_host_page_size= (); + range->bitmap.size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / + BITS_PER_BYTE; + range->bitmap.data =3D g_try_malloc0(range->bitmap.size); + if (!range->bitmap.data) { + ret =3D -ENOMEM; + goto err_out; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + if (ret) { + error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 + " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, + (uint64_t)range->size, errno); + goto err_out; + } + + cpu_physical_memory_set_dirty_lebitmap((unsigned long *)range->bitmap.= data, + ram_addr, pages); + + trace_vfio_get_dirty_bitmap(container->fd, range->iova, range->size, + range->bitmap.size, ram_addr); +err_out: + g_free(range->bitmap.data); + g_free(dbitmap); + + return ret; +} + +static void vfio_listener_release(VFIOContainer *container) +{ + memory_listener_unregister(&container->listener); + if (container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { + memory_listener_unregister(&container->prereg_listener); + } +} + +int vfio_container_add_section_window(VFIOContainer *container, + MemoryRegionSection *section, + Error **errp) +{ + VFIOHostDMAWindow *hostwin; + hwaddr pgsize =3D 0; + int ret; + + if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { + return 0; + } + + /* For now intersections are not allowed, we may relax this later */ + QLIST_FOREACH(hostwin, &container->hostwin_list, hostwin_next) { + if (ranges_overlap(hostwin->min_iova, + hostwin->max_iova - hostwin->min_iova + 1, + section->offset_within_address_space, + int128_get64(section->size))) { + error_setg(errp, + "region [0x%"PRIx64",0x%"PRIx64"] overlaps with existing" + "host DMA window [0x%"PRIx64",0x%"PRIx64"]", + section->offset_within_address_space, + section->offset_within_address_space + + int128_get64(section->size) - 1, + hostwin->min_iova, hostwin->max_iova); + return -EINVAL; + } + } + + ret =3D vfio_spapr_create_window(container, section, &pgsize); + if (ret) { + error_setg_errno(errp, -ret, "Failed to create SPAPR window"); + return ret; + } + + vfio_host_win_add(container, section->offset_within_address_space, + section->offset_within_address_space + + int128_get64(section->size) - 1, pgsize); +#ifdef CONFIG_KVM + if (kvm_enabled()) { + VFIOGroup *group; + IOMMUMemoryRegion *iommu_mr =3D IOMMU_MEMORY_REGION(section->mr); + struct kvm_vfio_spapr_tce param; + struct kvm_device_attr attr =3D { + .group =3D KVM_DEV_VFIO_GROUP, + .attr =3D KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE, + .addr =3D (uint64_t)(unsigned long)¶m, + }; + + if (!memory_region_iommu_get_attr(iommu_mr, IOMMU_ATTR_SPAPR_TCE_F= D, + ¶m.tablefd)) { + QLIST_FOREACH(group, &container->group_list, container_next) { + param.groupfd =3D group->fd; + if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr))= { + error_report("vfio: failed to setup fd %d " + "for a group with fd %d: %s", + param.tablefd, param.groupfd, + strerror(errno)); + return 0; + } + trace_vfio_spapr_group_attach(param.groupfd, param.tablefd= ); + } + } + } +#endif + return 0; +} + +void vfio_container_del_section_window(VFIOContainer *container, + MemoryRegionSection *section) +{ + if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { + return; + } + + vfio_spapr_remove_window(container, + section->offset_within_address_space); + if (vfio_host_win_del(container, + section->offset_within_address_space, + section->offset_within_address_space + + int128_get64(section->size) - 1) < 0) { + hw_error("%s: Cannot delete missing window at %"HWADDR_PRIx, + __func__, section->offset_within_address_space); + } +} + +static struct vfio_info_cap_header * +vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t= id) +{ + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { + return NULL; + } + + return vfio_get_cap((void *)info, info->cap_offset, id); +} + +bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, + unsigned int *avail) +{ + struct vfio_info_cap_header *hdr; + struct vfio_iommu_type1_info_dma_avail *cap; + + /* If the capability cannot be found, assume no DMA limiting */ + hdr =3D vfio_get_iommu_type1_info_cap(info, + VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL); + if (hdr =3D=3D NULL) { + return false; + } + + if (avail !=3D NULL) { + cap =3D (void *) hdr; + *avail =3D cap->avail; + } + + return true; +} + +void vfio_reset_handler(void *opaque) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->dev->realized) { + vbasedev->ops->vfio_compute_needs_reset(vbasedev); + } + } + } + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->dev->realized && vbasedev->needs_reset) { + vbasedev->ops->vfio_hot_reset_multi(vbasedev); + } + } + } +} + +static void vfio_kvm_device_add_group(VFIOGroup *group) +{ +#ifdef CONFIG_KVM + struct kvm_device_attr attr =3D { + .group =3D KVM_DEV_VFIO_GROUP, + .attr =3D KVM_DEV_VFIO_GROUP_ADD, + .addr =3D (uint64_t)(unsigned long)&group->fd, + }; + + if (!kvm_enabled()) { + return; + } + + if (vfio_kvm_device_fd < 0) { + struct kvm_create_device cd =3D { + .type =3D KVM_DEV_TYPE_VFIO, + }; + + if (kvm_vm_ioctl(kvm_state, KVM_CREATE_DEVICE, &cd)) { + error_report("Failed to create KVM VFIO device: %m"); + return; + } + + vfio_kvm_device_fd =3D cd.fd; + } + + if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr)) { + error_report("Failed to add group %d to KVM VFIO device: %m", + group->groupid); + } +#endif +} + +static void vfio_kvm_device_del_group(VFIOGroup *group) +{ +#ifdef CONFIG_KVM + struct kvm_device_attr attr =3D { + .group =3D KVM_DEV_VFIO_GROUP, + .attr =3D KVM_DEV_VFIO_GROUP_DEL, + .addr =3D (uint64_t)(unsigned long)&group->fd, + }; + + if (vfio_kvm_device_fd < 0) { + return; + } + + if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr)) { + error_report("Failed to remove group %d from KVM VFIO device: %m", + group->groupid); + } +#endif +} + +/* + * vfio_get_iommu_type - selects the richest iommu_type (v2 first) + */ +static int vfio_get_iommu_type(VFIOContainer *container, + Error **errp) +{ + int iommu_types[] =3D { VFIO_TYPE1v2_IOMMU, VFIO_TYPE1_IOMMU, + VFIO_SPAPR_TCE_v2_IOMMU, VFIO_SPAPR_TCE_IOMMU }; + int i; + + for (i =3D 0; i < ARRAY_SIZE(iommu_types); i++) { + if (ioctl(container->fd, VFIO_CHECK_EXTENSION, iommu_types[i])) { + return iommu_types[i]; + } + } + error_setg(errp, "No available IOMMU models"); + return -EINVAL; +} + +static int vfio_init_container(VFIOContainer *container, int group_fd, + Error **errp) +{ + int iommu_type, ret; + + iommu_type =3D vfio_get_iommu_type(container, errp); + if (iommu_type < 0) { + return iommu_type; + } + + ret =3D ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); + if (ret) { + error_setg_errno(errp, errno, "Failed to set group container"); + return -errno; + } + + while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { + if (iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { + /* + * On sPAPR, despite the IOMMU subdriver always advertises v1 = and + * v2, the running platform may not support v2 and there is no + * way to guess it until an IOMMU group gets added to the cont= ainer. + * So in case it fails with v2, try v1 as a fallback. + */ + iommu_type =3D VFIO_SPAPR_TCE_IOMMU; + continue; + } + error_setg_errno(errp, errno, "Failed to set iommu for container"); + return -errno; + } + + container->iommu_type =3D iommu_type; + return 0; +} + +static int vfio_get_iommu_info(VFIOContainer *container, + struct vfio_iommu_type1_info **info) +{ + + size_t argsz =3D sizeof(struct vfio_iommu_type1_info); + + *info =3D g_new0(struct vfio_iommu_type1_info, 1); +again: + (*info)->argsz =3D argsz; + + if (ioctl(container->fd, VFIO_IOMMU_GET_INFO, *info)) { + g_free(*info); + *info =3D NULL; + return -errno; + } + + if (((*info)->argsz > argsz)) { + argsz =3D (*info)->argsz; + *info =3D g_realloc(*info, argsz); + goto again; + } + + return 0; +} + +static struct vfio_info_cap_header * +vfio_get_iommu_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) +{ + struct vfio_info_cap_header *hdr; + void *ptr =3D info; + + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { + return NULL; + } + + for (hdr =3D ptr + info->cap_offset; hdr !=3D ptr; hdr =3D ptr + hdr->= next) { + if (hdr->id =3D=3D id) { + return hdr; + } + } + + return NULL; +} + +static void vfio_get_iommu_info_migration(VFIOContainer *container, + struct vfio_iommu_type1_info *inf= o) +{ + struct vfio_info_cap_header *hdr; + struct vfio_iommu_type1_info_cap_migration *cap_mig; + + hdr =3D vfio_get_iommu_info_cap(info, VFIO_IOMMU_TYPE1_INFO_CAP_MIGRAT= ION); + if (!hdr) { + return; + } + + cap_mig =3D container_of(hdr, struct vfio_iommu_type1_info_cap_migrati= on, + header); + + /* + * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of + * qemu_real_host_page_size to mark those dirty. + */ + if (cap_mig->pgsize_bitmap & qemu_real_host_page_size()) { + container->dirty_pages_supported =3D true; + container->max_dirty_bitmap_size =3D cap_mig->max_dirty_bitmap_siz= e; + container->dirty_pgsizes =3D cap_mig->pgsize_bitmap; + } +} + +static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, + Error **errp) +{ + VFIOContainer *container; + int ret, fd; + VFIOAddressSpace *space; + + space =3D vfio_get_address_space(as); + + /* + * VFIO is currently incompatible with discarding of RAM insofar as the + * madvise to purge (zap) the page from QEMU's address space does not + * interact with the memory API and therefore leaves stale virtual to + * physical mappings in the IOMMU if the page was previously pinned. = We + * therefore set discarding broken for each group added to a container, + * whether the container is used individually or shared. This provides + * us with options to allow devices within a group to opt-in and allow + * discarding, so long as it is done consistently for a group (for ins= tance + * if the device is an mdev device where it is known that the host ven= dor + * driver will never pin pages outside of the working set of the guest + * driver, which would thus not be discarding candidates). + * + * The first opportunity to induce pinning occurs here where we attemp= t to + * attach the group to existing containers within the AddressSpace. I= f any + * pages are already zapped from the virtual address space, such as fr= om + * previous discards, new pinning will cause valid mappings to be + * re-established. Likewise, when the overall MemoryListener for a new + * container is registered, a replay of mappings within the AddressSpa= ce + * will occur, re-establishing any previously zapped pages as well. + * + * Especially virtio-balloon is currently only prevented from discardi= ng + * new memory, it will not yet set ram_block_discard_set_required() and + * therefore, neither stops us here or deals with the sudden memory + * consumption of inflated memory. + * + * We do support discarding of memory coordinated via the RamDiscardMa= nager + * with some IOMMU types. vfio_ram_block_discard_disable() handles the + * details once we know which type of IOMMU we are using. + */ + + QLIST_FOREACH(container, &space->containers, next) { + if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { + ret =3D vfio_ram_block_discard_disable(container, true); + if (ret) { + error_setg_errno(errp, -ret, + "Cannot set discarding of RAM broken"); + if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, + &container->fd)) { + error_report("vfio: error disconnecting group %d from" + " container", group->groupid); + } + return ret; + } + group->container =3D container; + QLIST_INSERT_HEAD(&container->group_list, group, container_nex= t); + vfio_kvm_device_add_group(group); + return 0; + } + } + + fd =3D qemu_open_old("/dev/vfio/vfio", O_RDWR); + if (fd < 0) { + error_setg_errno(errp, errno, "failed to open /dev/vfio/vfio"); + ret =3D -errno; + goto put_space_exit; + } + + ret =3D ioctl(fd, VFIO_GET_API_VERSION); + if (ret !=3D VFIO_API_VERSION) { + error_setg(errp, "supported vfio version: %d, " + "reported version: %d", VFIO_API_VERSION, ret); + ret =3D -EINVAL; + goto close_fd_exit; + } + + container =3D g_malloc0(sizeof(*container)); + container->space =3D space; + container->fd =3D fd; + container->error =3D NULL; + container->dirty_pages_supported =3D false; + container->dma_max_mappings =3D 0; + QLIST_INIT(&container->giommu_list); + QLIST_INIT(&container->hostwin_list); + QLIST_INIT(&container->vrdl_list); + + ret =3D vfio_init_container(container, group->fd, errp); + if (ret) { + goto free_container_exit; + } + + ret =3D vfio_ram_block_discard_disable(container, true); + if (ret) { + error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"= ); + goto free_container_exit; + } + + switch (container->iommu_type) { + case VFIO_TYPE1v2_IOMMU: + case VFIO_TYPE1_IOMMU: + { + struct vfio_iommu_type1_info *info; + + ret =3D vfio_get_iommu_info(container, &info); + if (ret) { + error_setg_errno(errp, -ret, "Failed to get VFIO IOMMU info"); + goto enable_discards_exit; + } + + if (info->flags & VFIO_IOMMU_INFO_PGSIZES) { + container->pgsizes =3D info->iova_pgsizes; + } else { + container->pgsizes =3D qemu_real_host_page_size(); + } + + if (!vfio_get_info_dma_avail(info, &container->dma_max_mappings)) { + container->dma_max_mappings =3D 65535; + } + vfio_get_iommu_info_migration(container, info); + g_free(info); + + /* + * FIXME: We should parse VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE + * information to get the actual window extent rather than assume + * a 64-bit IOVA address space. + */ + vfio_host_win_add(container, 0, (hwaddr)-1, container->pgsizes); + + break; + } + case VFIO_SPAPR_TCE_v2_IOMMU: + case VFIO_SPAPR_TCE_IOMMU: + { + struct vfio_iommu_spapr_tce_info info; + bool v2 =3D container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU; + + /* + * The host kernel code implementing VFIO_IOMMU_DISABLE is called + * when container fd is closed so we do not call it explicitly + * in this file. + */ + if (!v2) { + ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); + if (ret) { + error_setg_errno(errp, errno, "failed to enable container"= ); + ret =3D -errno; + goto enable_discards_exit; + } + } else { + container->prereg_listener =3D vfio_prereg_listener; + + memory_listener_register(&container->prereg_listener, + &address_space_memory); + if (container->error) { + memory_listener_unregister(&container->prereg_listener); + ret =3D -1; + error_propagate_prepend(errp, container->error, + "RAM memory listener initialization failed: "); + goto enable_discards_exit; + } + } + + info.argsz =3D sizeof(info); + ret =3D ioctl(fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info); + if (ret) { + error_setg_errno(errp, errno, + "VFIO_IOMMU_SPAPR_TCE_GET_INFO failed"); + ret =3D -errno; + if (v2) { + memory_listener_unregister(&container->prereg_listener); + } + goto enable_discards_exit; + } + + if (v2) { + container->pgsizes =3D info.ddw.pgsizes; + /* + * There is a default window in just created container. + * To make region_add/del simpler, we better remove this + * window now and let those iommu_listener callbacks + * create/remove them when needed. + */ + ret =3D vfio_spapr_remove_window(container, info.dma32_window_= start); + if (ret) { + error_setg_errno(errp, -ret, + "failed to remove existing window"); + goto enable_discards_exit; + } + } else { + /* The default table uses 4K pages */ + container->pgsizes =3D 0x1000; + vfio_host_win_add(container, info.dma32_window_start, + info.dma32_window_start + + info.dma32_window_size - 1, + 0x1000); + } + } + } + + vfio_kvm_device_add_group(group); + + QLIST_INIT(&container->group_list); + QLIST_INSERT_HEAD(&space->containers, container, next); + + group->container =3D container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + + container->listener =3D vfio_memory_listener; + + memory_listener_register(&container->listener, container->space->as); + + if (container->error) { + ret =3D -1; + error_propagate_prepend(errp, container->error, + "memory listener initialization failed: "); + goto listener_release_exit; + } + + container->initialized =3D true; + + return 0; +listener_release_exit: + QLIST_REMOVE(group, container_next); + QLIST_REMOVE(container, next); + vfio_kvm_device_del_group(group); + vfio_listener_release(container); + +enable_discards_exit: + vfio_ram_block_discard_disable(container, false); + +free_container_exit: + g_free(container); + +close_fd_exit: + close(fd); + +put_space_exit: + vfio_put_address_space(space); + + return ret; +} + +static void vfio_disconnect_container(VFIOGroup *group) +{ + VFIOContainer *container =3D group->container; + + QLIST_REMOVE(group, container_next); + group->container =3D NULL; + + /* + * Explicitly release the listener first before unset container, + * since unset may destroy the backend container if it's the last + * group. + */ + if (QLIST_EMPTY(&container->group_list)) { + vfio_listener_release(container); + } + + if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, &container->fd)) { + error_report("vfio: error disconnecting group %d from container", + group->groupid); + } + + if (QLIST_EMPTY(&container->group_list)) { + VFIOAddressSpace *space =3D container->space; + VFIOGuestIOMMU *giommu, *tmp; + VFIOHostDMAWindow *hostwin, *next; + + QLIST_REMOVE(container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, t= mp) { + memory_region_unregister_iommu_notifier( + MEMORY_REGION(giommu->iommu_mr), &giommu->n); + QLIST_REMOVE(giommu, giommu_next); + g_free(giommu); + } + + QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, + next) { + QLIST_REMOVE(hostwin, hostwin_next); + g_free(hostwin); + } + + trace_vfio_disconnect_container(container->fd); + close(container->fd); + g_free(container); + + vfio_put_address_space(space); + } +} + +VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) +{ + VFIOGroup *group; + char path[32]; + struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; + + QLIST_FOREACH(group, &vfio_group_list, next) { + if (group->groupid =3D=3D groupid) { + /* Found it. Now is it already in the right context? */ + if (group->container->space->as =3D=3D as) { + return group; + } else { + error_setg(errp, "group %d used in multiple address spaces= ", + group->groupid); + return NULL; + } + } + } + + group =3D g_malloc0(sizeof(*group)); + + snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); + group->fd =3D qemu_open_old(path, O_RDWR); + if (group->fd < 0) { + error_setg_errno(errp, errno, "failed to open %s", path); + goto free_group_exit; + } + + if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { + error_setg_errno(errp, errno, "failed to get group %d status", gro= upid); + goto close_fd_exit; + } + + if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + error_setg(errp, "group %d is not viable", groupid); + error_append_hint(errp, + "Please ensure all devices within the iommu_grou= p " + "are bound to their vfio bus driver.\n"); + goto close_fd_exit; + } + + group->groupid =3D groupid; + QLIST_INIT(&group->device_list); + + if (vfio_connect_container(group, as, errp)) { + error_prepend(errp, "failed to setup container for group %d: ", + groupid); + goto close_fd_exit; + } + + if (QLIST_EMPTY(&vfio_group_list)) { + qemu_register_reset(vfio_reset_handler, NULL); + } + + QLIST_INSERT_HEAD(&vfio_group_list, group, next); + + return group; + +close_fd_exit: + close(group->fd); + +free_group_exit: + g_free(group); + + return NULL; +} + +void vfio_put_group(VFIOGroup *group) +{ + if (!group || !QLIST_EMPTY(&group->device_list)) { + return; + } + + if (!group->ram_block_discard_allowed) { + vfio_ram_block_discard_disable(group->container, false); + } + vfio_kvm_device_del_group(group); + vfio_disconnect_container(group); + QLIST_REMOVE(group, next); + trace_vfio_put_group(group->fd); + close(group->fd); + g_free(group); + + if (QLIST_EMPTY(&vfio_group_list)) { + qemu_unregister_reset(vfio_reset_handler, NULL); + } +} + +int vfio_get_device(VFIOGroup *group, const char *name, + VFIODevice *vbasedev, Error **errp) +{ + struct vfio_device_info dev_info =3D { .argsz =3D sizeof(dev_info) }; + int ret, fd; + + fd =3D ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); + if (fd < 0) { + error_setg_errno(errp, errno, "error getting device from group %d", + group->groupid); + error_append_hint(errp, + "Verify all devices in group %d are bound to vfio- " + "or pci-stub and not already in use\n", group->group= id); + return fd; + } + + ret =3D ioctl(fd, VFIO_DEVICE_GET_INFO, &dev_info); + if (ret) { + error_setg_errno(errp, errno, "error getting device info"); + close(fd); + return ret; + } + + /* + * Set discarding of RAM as not broken for this group if the driver kn= ows + * the device operates compatibly with discarding. Setting must be + * consistent per group, but since compatibility is really only possib= le + * with mdev currently, we expect singleton groups. + */ + if (vbasedev->ram_block_discard_allowed !=3D + group->ram_block_discard_allowed) { + if (!QLIST_EMPTY(&group->device_list)) { + error_setg(errp, "Inconsistent setting of support for discardi= ng " + "RAM (e.g., balloon) within group"); + close(fd); + return -1; + } + + if (!group->ram_block_discard_allowed) { + group->ram_block_discard_allowed =3D true; + vfio_ram_block_discard_disable(group->container, false); + } + } + + vbasedev->fd =3D fd; + vbasedev->group =3D group; + QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); + + vbasedev->num_irqs =3D dev_info.num_irqs; + vbasedev->num_regions =3D dev_info.num_regions; + vbasedev->flags =3D dev_info.flags; + + trace_vfio_get_device(name, dev_info.flags, dev_info.num_regions, + dev_info.num_irqs); + + vbasedev->reset_works =3D !!(dev_info.flags & VFIO_DEVICE_FLAGS_RESET); + return 0; +} + +void vfio_put_base_device(VFIODevice *vbasedev) +{ + if (!vbasedev->group) { + return; + } + QLIST_REMOVE(vbasedev, next); + vbasedev->group =3D NULL; + trace_vfio_put_base_device(vbasedev->fd); + close(vbasedev->fd); +} + +/* + * Interfaces for IBM EEH (Enhanced Error Handling) + */ +static bool vfio_eeh_container_ok(VFIOContainer *container) +{ + /* + * As of 2016-03-04 (linux-4.5) the host kernel EEH/VFIO + * implementation is broken if there are multiple groups in a + * container. The hardware works in units of Partitionable + * Endpoints (=3D=3D IOMMU groups) and the EEH operations naively + * iterate across all groups in the container, without any logic + * to make sure the groups have their state synchronized. For + * certain operations (ENABLE) that might be ok, until an error + * occurs, but for others (GET_STATE) it's clearly broken. + */ + + /* + * XXX Once fixed kernels exist, test for them here + */ + + if (QLIST_EMPTY(&container->group_list)) { + return false; + } + + if (QLIST_NEXT(QLIST_FIRST(&container->group_list), container_next)) { + return false; + } + + return true; +} + +static int vfio_eeh_container_op(VFIOContainer *container, uint32_t op) +{ + struct vfio_eeh_pe_op pe_op =3D { + .argsz =3D sizeof(pe_op), + .op =3D op, + }; + int ret; + + if (!vfio_eeh_container_ok(container)) { + error_report("vfio/eeh: EEH_PE_OP 0x%x: " + "kernel requires a container with exactly one group",= op); + return -EPERM; + } + + ret =3D ioctl(container->fd, VFIO_EEH_PE_OP, &pe_op); + if (ret < 0) { + error_report("vfio/eeh: EEH_PE_OP 0x%x failed: %m", op); + return -errno; + } + + return ret; +} + +static VFIOContainer *vfio_eeh_as_container(AddressSpace *as) +{ + VFIOAddressSpace *space =3D vfio_get_address_space(as); + VFIOContainer *container =3D NULL; + + if (QLIST_EMPTY(&space->containers)) { + /* No containers to act on */ + goto out; + } + + container =3D QLIST_FIRST(&space->containers); + + if (QLIST_NEXT(container, next)) { + /* + * We don't yet have logic to synchronize EEH state across + * multiple containers + */ + container =3D NULL; + goto out; + } + +out: + vfio_put_address_space(space); + return container; +} + +bool vfio_eeh_as_ok(AddressSpace *as) +{ + VFIOContainer *container =3D vfio_eeh_as_container(as); + + return (container !=3D NULL) && vfio_eeh_container_ok(container); +} + +int vfio_eeh_as_op(AddressSpace *as, uint32_t op) +{ + VFIOContainer *container =3D vfio_eeh_as_container(as); + + if (!container) { + return -ENODEV; + } + return vfio_eeh_container_op(container, op); +} diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index 3746c9f984..2a6912c940 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -2,6 +2,7 @@ vfio_ss =3D ss.source_set() vfio_ss.add(files( 'helpers.c', 'common.c', + 'container.c', 'spapr.c', 'migration.c', )) --=20 2.37.3