From nobody Thu Jul 3 23:58:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 152890818451569.93090058204064; Wed, 13 Jun 2018 09:43:04 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5B22030820DB; Wed, 13 Jun 2018 16:43:03 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1E8D0761E6; Wed, 13 Jun 2018 16:43:03 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id BAE0F4CA81; Wed, 13 Jun 2018 16:43:02 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w5DGgYD7029600 for ; Wed, 13 Jun 2018 12:42:34 -0400 Received: by smtp.corp.redhat.com (Postfix) id 983582024CA2; Wed, 13 Jun 2018 16:42:34 +0000 (UTC) Received: from red.redhat.com (ovpn-120-103.rdu2.redhat.com [10.10.120.103]) by smtp.corp.redhat.com (Postfix) with ESMTP id AE5972024CA1; Wed, 13 Jun 2018 16:42:33 +0000 (UTC) From: Eric Blake To: libvir-list@redhat.com Date: Wed, 13 Jun 2018 11:42:25 -0500 Message-Id: <20180613164229.1379979-5-eblake@redhat.com> In-Reply-To: <20180613164229.1379979-1-eblake@redhat.com> References: <20180613164229.1379979-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-loop: libvir-list@redhat.com Cc: vsementsov@virtuozzo.com, mnestratov@virtuozzo.com, nsoffer@redhat.com, pkrempa@redhat.com, nshirokovskiy@virtuozzo.com, den@openvz.org, jsnow@redhat.com Subject: [libvirt] [PATCH 4/8] backup: Document new XML for backups X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Wed, 13 Jun 2018 16:43:03 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" Prepare for new checkpoint and backup APIs by describing the XML that will represent a checkpoint. This is modeled heavily after the XML for virDomainSnapshotPtr, since both represent a point in time of the guest. But while a snapshot exists with the intent of rolling back to that state, a checkpoint instead makes it possible to create an incremental backup at a later time. Add testsuite coverage of a minimal use of the XML. Signed-off-by: Eric Blake --- docs/docs.html.in | 3 +- docs/domainstatecapture.html.in | 4 +- docs/formatcheckpoint.html.in | 273 +++++++++++++++++++++++++= ++++ docs/schemas/domaincheckpoint.rng | 89 ++++++++++ libvirt.spec.in | 1 + mingw-libvirt.spec.in | 2 + tests/domaincheckpointxml2xmlin/empty.xml | 1 + tests/domaincheckpointxml2xmlout/empty.xml | 10 ++ tests/virschematest.c | 2 + 9 files changed, 382 insertions(+), 3 deletions(-) create mode 100644 docs/formatcheckpoint.html.in create mode 100644 docs/schemas/domaincheckpoint.rng create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml diff --git a/docs/docs.html.in b/docs/docs.html.in index 4c46b74980..11dfd27ba6 100644 --- a/docs/docs.html.in +++ b/docs/docs.html.in @@ -79,7 +79,8 @@ domain capabilities, node devices, secrets, - snapshots + snapshots, + checkpoints
URI format
The URI formats used for connecting to libvirt
diff --git a/docs/domainstatecapture.html.in b/docs/domainstatecapture.html= .in index 00ab7e8ee1..4de93c87c8 100644 --- a/docs/domainstatecapture.html.in +++ b/docs/domainstatecapture.html.in @@ -154,9 +154,9 @@ time as a new backup, so that the next incremental backup can refer to the incremental state since the checkpoint created during the current backup. Guest state is then actually - captured using virDomainBackupBegin(). + this command.

Examples

diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in new file mode 100644 index 0000000000..34507a9f68 --- /dev/null +++ b/docs/formatcheckpoint.html.in @@ -0,0 +1,273 @@ + + + + +

Checkpoint and Backup XML format

+ +
    + +

    Checkpoint XML

    + +

    + Domain disk backups, including incremental backups, are one form + of domain state capture. +

    +

    + Libvirt is able to facilitate incremental backups by tracking + disk checkpoints, or points in time against which it is easy to + compute which portion of the disk has changed. Given a full + backup (a backup created from the creation of the disk to a + given point in time, coupled with the creation of a disk + checkpoint at that time), and an incremental backup (a backup + created from just the dirty portion of the disk between the + first checkpoint and the second backup operation), it is + possible to do an offline reconstruction of the state of the + disk at the time of the second backup, without having to copy as + much data as a second full backup would require. Most disk + checkpoints are created in concert with a backup, + via virDomainBackupBegin(); however, libvirt also + exposes enough support to create disk checkpoints independently + from a backup operation, + via virDomainCheckpointCreateXML(). +

    +

    + Attributes of libvirt checkpoints are stored as child elements of + the domaincheckpoint element. At checkpoint creation + time, normally only the name, description, + and disks elements are settable; the rest of the + fields are ignored on creation, and will be filled in by + libvirt in for informational purposes + by virDomainCheckpointGetXMLDesc(). However, when + redefining a checkpoint, + with the VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE flag + of virDomainCheckpointCreateXML(), all of the XML + described here is relevant. +

    +

    + Checkpoints are maintained in a hierarchy. A domain can have a + current checkpoint, which is the most recent checkpoint compared to + the current state of the domain (although a domain might have + checkpoints without a current checkpoint, if checkpoints have been + deleted in the meantime). Creating or reverting to a checkpoint + sets that checkpoint as current, and the prior current checkpoint is + the parent of the new checkpoint. Branches in the hierarchy can + be formed by reverting to a checkpoint with a child, then creating + another checkpoint. +

    +

    + The top-level domaincheckpoint element may contain + the following elements: +

    +
    +
    name
    +
    The name for this checkpoint. If the name is specified when + initially creating the checkpoint, then the checkpoint will have + that particular name. If the name is omitted when initially + creating the checkpoint, then libvirt will make up a name for + the checkpoint, based on the time when it was created. +
    +
    description
    +
    A human-readable description of the checkpoint. If the + description is omitted when initially creating the checkpoint, + then this field will be empty. +
    +
    disks
    +
    On input, this is an optional listing of specific + instructions for disk checkpoints; it is needed when making a + checkpoint on only a subset of the disks associated with a + domain (in particular, since qemu checkpoints require qcow2 + disks, this element may be needed on input for excluding guest + disks that are not in qcow2 format); if omitted on input, then + all disks participate in the checkpoint. On output, this is + fully populated to show the state of each disk in the + checkpoint. This element has a list of disk + sub-elements, describing anywhere from one to all of the disks + associated with the domain. +
    +
    disk
    +
    This sub-element describes the checkpoint properties of + a specific disk. The attribute name is + mandatory, and must match either the <target + dev=3D'name'/> or an unambiguous <source + file=3D'name'/> of one of + the disk + devices specified for the domain at the time of the + checkpoint. The attribute checkpoint is + optional on input; possible values are no + when the disk does not participate in this checkpoint; + or bitmap if the disk will track all changes + since the creation of this checkpoint via a bitmap, in + which case another attribute bitmap will be + the name of the tracking bitmap (defaulting to the + checkpoint name). +
    +
    +
    +
    creationTime
    +
    The time this checkpoint was created. The time is specified + in seconds since the Epoch, UTC (i.e. Unix time). Readonly. +
    +
    parent
    +
    The parent of this checkpoint. If present, this element + contains exactly one child element, name. This specifies the + name of the parent checkpoint of this one, and is used to + represent trees of checkpoints. Readonly. +
    +
    domain
    +
    The inactive domain + configuration at the time the checkpoint was created. + Readonly. +
    +
    + +

    Backup XML

    + +

    + Creating a backup, whether full or incremental, is done + via virDomainBackupBegin(), which takes an XML + description of the actions to perform. There are two general + modes for backups: a push mode (where the hypervisor writes out + the data to the destination file, which may be local or remote), + and a pull mode (where the hypervisor creates an NBD server that + a third-party client can then read as needed, and which requires + the use of temporary storage, typically local, until the backup + is complete). +

    +

    + The instructions for beginning a backup job are provided as + attributes and elements of the + top-level domainbackup element. This element + includes an optional attribute mode which can be + either "push" or "pull" (default push). Where elements are + optional on creation, virDomainBackupGetXMLDesc() + can be used to see the actual values selected (for example, + learning which port the NBD server is using in the pull model, + or what file names libvirt generated when none were supplied). + The following child elements are supported: +

    +
    +
    incremental
    +
    Optional. If this element is present, it must name an + existing checkpoint of the domain, which will be used to make + this backup an incremental one (in the push model, only + changes since the checkpoint are written to the destination; + in the pull model, the NBD server uses the + NBD_OPT_SET_META_CONTEXT extension to advertise to the client + which portions of the export contain changes since the + checkpoint). If omitted, a full backup is performed. +
    +
    server
    +
    Present only for a pull mode backup. Contains the same + attributes as the protocol element of a disk + attached via NBD in the domain (such as transport, socket, + name, port, or tls), necessary to set up an NBD server that + exposes the content of each disk at the time the backup + started. +
    +
    disks
    +
    This is an optional listing of instructions for disks + participating in the backup (if omitted, all disks + participate, and libvirt attempts to generate filenames by + appending the current timestamp as a suffix). When provided on + input, disks omitted from the list do not participate in the + backup. On output, the list is present but contains only the + disks participating in the backup job. This element has a + list of disk sub-elements, describing anywhere + from one to all of the disks associated with the domain. +
    +
    disk
    +
    This sub-element describes the checkpoint properties of + a specific disk. The attribute name is + mandatory, and must match either the <target + dev=3D'name'/> or an unambiguous <source + file=3D'name'/> of one of + the disk + devices specified for the domain at the time of the + checkpoint. The optional attribute type can + be file, block, + or networks, similar to a disk declaration + for a domain, controls what additional sub-elements are + needed to describe the destination (such + as protocol for a network destination). In + push mode backups, the primary subelement + is target; in pull mode, the primary sublement + is scratch; but either way, + the primary sub-element describes the file name to be used + during the backup operation, similar to + the source sub-element of a domain disk. An + optional sublement driver can also be used to + specify a destination format different from qcow2. +
    +
    +
    +
    + +

    Examples

    + +

    Using this XML to create a checkpoint of just vda on a qemu + domain with two disks and a prior checkpoint:

    + 
    +<domaincheckpoint>
+  <description>Completion of updates after OS install</descriptio=
n>
+  <disks>
+    <disk name=3D'vda' checkpoint=3D'bitmap'/>
+    <disk name=3D'vdb' checkpoint=3D'no'/>
+  </disks>
+</domaincheckpoint>
    + +

    will result in XML similar to this from + virDomainCheckpointGetXMLDesc():

    + 
    +<domaincheckpoint>
+  <name>1525889631</name>
+  <description>Completion of updates after OS install</descriptio=
n>
+  <creationTime>1525889631</creationTime>
+  <parent>
+    <name>1525111885</name>
+  </parent>
+  <disks>
+    <disk name=3D'vda' checkpoint=3D'bitmap' bitmap=3D'1525889631'/>
+    <disk name=3D'vdb' checkpoint=3D'no'/>
+  </disks>
+  <domain>
+    <name>fedora</name>
+    <uuid>93a5c045-6457-2c09-e56c-927cdf34e178</uuid>
+    <memory>1048576</memory>
+    ...
+    <devices>
+      <disk type=3D'file' device=3D'disk'>
+        <driver name=3D'qemu' type=3D'qcow2'/>
+        <source file=3D'/path/to/file1'/>
+        <target dev=3D'vda' bus=3D'virtio'/>
+      </disk>
+      <disk type=3D'file' device=3D'disk' snapshot=3D'external'>
+        <driver name=3D'qemu' type=3D'raw'/>
+        <source file=3D'/path/to/file2'/>
+        <target dev=3D'vdb' bus=3D'virtio'/>
+      </disk>
+      ...
+    </devices>
+  </domain>
+</domaincheckpoint>
    + +

    With that checkpoint created, the qcow2 image is now tracking + all changes that occur in the image since the checkpoint via + the persistent bitmap named 1525889631. Now, we + can make a subsequent call + to virDomainBackupBegin() to perform an incremental + backup of just this data, using the following XML to start a + pull model NBD export of the vda disk: +

    + 
    +<domainbackup mode=3D"pull">
+  <incremental>1525889631</incremental>
+  <server transport=3D"unix" socket=3D"/path/to/server"/>
+  <disks/>
+    <disk name=3D'vda' type=3D'file'/>
+      <scratch file=3D/path/to/file1.scratch'/>
+    </disk>
+  </disks/>
+</domainbackup>
+
    + + diff --git a/docs/schemas/domaincheckpoint.rng b/docs/schemas/domaincheckpo= int.rng new file mode 100644 index 0000000000..1e2c16e035 --- /dev/null +++ b/docs/schemas/domaincheckpoint.rng @@ -0,0 +1,89 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + no + + + + + bitmap + + + + + + + + + + + + + diff --git a/libvirt.spec.in b/libvirt.spec.in index ace05820aa..50bd79a7d7 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -2044,6 +2044,7 @@ exit 0 %{_datadir}/libvirt/schemas/cputypes.rng %{_datadir}/libvirt/schemas/domain.rng %{_datadir}/libvirt/schemas/domaincaps.rng +%{_datadir}/libvirt/schemas/domaincheckpoint.rng %{_datadir}/libvirt/schemas/domaincommon.rng %{_datadir}/libvirt/schemas/domainsnapshot.rng %{_datadir}/libvirt/schemas/interface.rng diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in index 917d2143d8..6912527cf7 100644 --- a/mingw-libvirt.spec.in +++ b/mingw-libvirt.spec.in @@ -241,6 +241,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-gue= sts.sh %{mingw32_datadir}/libvirt/schemas/cputypes.rng %{mingw32_datadir}/libvirt/schemas/domain.rng %{mingw32_datadir}/libvirt/schemas/domaincaps.rng +%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng %{mingw32_datadir}/libvirt/schemas/domaincommon.rng %{mingw32_datadir}/libvirt/schemas/domainsnapshot.rng %{mingw32_datadir}/libvirt/schemas/interface.rng @@ -326,6 +327,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-gue= sts.sh %{mingw64_datadir}/libvirt/schemas/cputypes.rng %{mingw64_datadir}/libvirt/schemas/domain.rng %{mingw64_datadir}/libvirt/schemas/domaincaps.rng +%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng %{mingw64_datadir}/libvirt/schemas/domaincommon.rng %{mingw64_datadir}/libvirt/schemas/domainsnapshot.rng %{mingw64_datadir}/libvirt/schemas/interface.rng diff --git a/tests/domaincheckpointxml2xmlin/empty.xml b/tests/domaincheckp= ointxml2xmlin/empty.xml new file mode 100644 index 0000000000..dc36449142 --- /dev/null +++ b/tests/domaincheckpointxml2xmlin/empty.xml @@ -0,0 +1 @@ + diff --git a/tests/domaincheckpointxml2xmlout/empty.xml b/tests/domaincheck= pointxml2xmlout/empty.xml new file mode 100644 index 0000000000..a26c7caab0 --- /dev/null +++ b/tests/domaincheckpointxml2xmlout/empty.xml @@ -0,0 +1,10 @@ + + 1525889631 + 1525889631 + + + + + 9d37b878-a7cc-9f9a-b78f-49b3abad25a8 + + diff --git a/tests/virschematest.c b/tests/virschematest.c index 2d35833919..b866db4326 100644 --- a/tests/virschematest.c +++ b/tests/virschematest.c @@ -223,6 +223,8 @@ mymain(void) "genericxml2xmloutdata", "xlconfigdata", "libxlxml2domconf= igdata", "qemuhotplugtestdomains"); DO_TEST_DIR("domaincaps.rng", "domaincapsschemadata"); + DO_TEST_DIR("domaincheckpoint.rng", "domaincheckpointxml2xmlin", + "domaincheckpointxml2xmlout"); DO_TEST_DIR("domainsnapshot.rng", "domainsnapshotxml2xmlin", "domainsnapshotxml2xmlout"); DO_TEST_DIR("interface.rng", "interfaceschemadata"); --=20 2.14.4 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list