Prepare for new checkpoint and backup APIs by describing the XML
that will represent a checkpoint. This is modeled heavily after
the XML for virDomainSnapshotPtr, since both represent a point in
time of the guest. But while a snapshot exists with the intent
of rolling back to that state, a checkpoint instead makes it
possible to create an incremental backup at a later time.
Add testsuite coverage of a minimal use of the XML.
Signed-off-by: Eric Blake <eblake@redhat.com>
---
docs/docs.html.in | 3 +-
docs/domainstatecapture.html.in | 4 +-
docs/formatcheckpoint.html.in | 273 +++++++++++++++++++++++++++++
docs/schemas/domaincheckpoint.rng | 89 ++++++++++
libvirt.spec.in | 1 +
mingw-libvirt.spec.in | 2 +
tests/domaincheckpointxml2xmlin/empty.xml | 1 +
tests/domaincheckpointxml2xmlout/empty.xml | 10 ++
tests/virschematest.c | 2 +
9 files changed, 382 insertions(+), 3 deletions(-)
create mode 100644 docs/formatcheckpoint.html.in
create mode 100644 docs/schemas/domaincheckpoint.rng
create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
diff --git a/docs/docs.html.in b/docs/docs.html.in
index 4c46b74980..11dfd27ba6 100644
--- a/docs/docs.html.in
+++ b/docs/docs.html.in
@@ -79,7 +79,8 @@
<a href="formatdomaincaps.html">domain capabilities</a>,
<a href="formatnode.html">node devices</a>,
<a href="formatsecret.html">secrets</a>,
- <a href="formatsnapshot.html">snapshots</a></dd>
+ <a href="formatsnapshot.html">snapshots</a>,
+ <a href="formatcheckpoint.html">checkpoints</a></dd>
<dt><a href="uri.html">URI format</a></dt>
<dd>The URI formats used for connecting to libvirt</dd>
diff --git a/docs/domainstatecapture.html.in b/docs/domainstatecapture.html.in
index 00ab7e8ee1..4de93c87c8 100644
--- a/docs/domainstatecapture.html.in
+++ b/docs/domainstatecapture.html.in
@@ -154,9 +154,9 @@
time as a new backup, so that the next incremental backup can
refer to the incremental state since the checkpoint created
during the current backup. Guest state is then actually
- captured using <code>virDomainBackupBegin()</code>. <!--See also
+ captured using <code>virDomainBackupBegin()</code>. See also
the <a href="formatcheckpoint.html">XML details</a> used with
- this command.--></dd>
+ this command.</dd>
</dl>
<h2><a id="examples">Examples</a></h2>
diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in
new file mode 100644
index 0000000000..34507a9f68
--- /dev/null
+++ b/docs/formatcheckpoint.html.in
@@ -0,0 +1,273 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <body>
+ <h1>Checkpoint and Backup XML format</h1>
+
+ <ul id="toc"></ul>
+
+ <h2><a id="CheckpointAttributes">Checkpoint XML</a></h2>
+
+ <p>
+ Domain disk backups, including incremental backups, are one form
+ of <a href="domainstatecapture.html">domain state capture</a>.
+ </p>
+ <p>
+ Libvirt is able to facilitate incremental backups by tracking
+ disk checkpoints, or points in time against which it is easy to
+ compute which portion of the disk has changed. Given a full
+ backup (a backup created from the creation of the disk to a
+ given point in time, coupled with the creation of a disk
+ checkpoint at that time), and an incremental backup (a backup
+ created from just the dirty portion of the disk between the
+ first checkpoint and the second backup operation), it is
+ possible to do an offline reconstruction of the state of the
+ disk at the time of the second backup, without having to copy as
+ much data as a second full backup would require. Most disk
+ checkpoints are created in concert with a backup,
+ via <code>virDomainBackupBegin()</code>; however, libvirt also
+ exposes enough support to create disk checkpoints independently
+ from a backup operation,
+ via <code>virDomainCheckpointCreateXML()</code>.
+ </p>
+ <p>
+ Attributes of libvirt checkpoints are stored as child elements of
+ the <code>domaincheckpoint</code> element. At checkpoint creation
+ time, normally only the <code>name</code>, <code>description</code>,
+ and <code>disks</code> elements are settable; the rest of the
+ fields are ignored on creation, and will be filled in by
+ libvirt in for informational purposes
+ by <code>virDomainCheckpointGetXMLDesc()</code>. However, when
+ redefining a checkpoint,
+ with the <code>VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE</code> flag
+ of <code>virDomainCheckpointCreateXML()</code>, all of the XML
+ described here is relevant.
+ </p>
+ <p>
+ Checkpoints are maintained in a hierarchy. A domain can have a
+ current checkpoint, which is the most recent checkpoint compared to
+ the current state of the domain (although a domain might have
+ checkpoints without a current checkpoint, if checkpoints have been
+ deleted in the meantime). Creating or reverting to a checkpoint
+ sets that checkpoint as current, and the prior current checkpoint is
+ the parent of the new checkpoint. Branches in the hierarchy can
+ be formed by reverting to a checkpoint with a child, then creating
+ another checkpoint.
+ </p>
+ <p>
+ The top-level <code>domaincheckpoint</code> element may contain
+ the following elements:
+ </p>
+ <dl>
+ <dt><code>name</code></dt>
+ <dd>The name for this checkpoint. If the name is specified when
+ initially creating the checkpoint, then the checkpoint will have
+ that particular name. If the name is omitted when initially
+ creating the checkpoint, then libvirt will make up a name for
+ the checkpoint, based on the time when it was created.
+ </dd>
+ <dt><code>description</code></dt>
+ <dd>A human-readable description of the checkpoint. If the
+ description is omitted when initially creating the checkpoint,
+ then this field will be empty.
+ </dd>
+ <dt><code>disks</code></dt>
+ <dd>On input, this is an optional listing of specific
+ instructions for disk checkpoints; it is needed when making a
+ checkpoint on only a subset of the disks associated with a
+ domain (in particular, since qemu checkpoints require qcow2
+ disks, this element may be needed on input for excluding guest
+ disks that are not in qcow2 format); if omitted on input, then
+ all disks participate in the checkpoint. On output, this is
+ fully populated to show the state of each disk in the
+ checkpoint. This element has a list of <code>disk</code>
+ sub-elements, describing anywhere from one to all of the disks
+ associated with the domain.
+ <dl>
+ <dt><code>disk</code></dt>
+ <dd>This sub-element describes the checkpoint properties of
+ a specific disk. The attribute <code>name</code> is
+ mandatory, and must match either the <code><target
+ dev='name'/></code> or an unambiguous <code><source
+ file='name'/></code> of one of
+ the <a href="formatdomain.html#elementsDisks">disk
+ devices</a> specified for the domain at the time of the
+ checkpoint. The attribute <code>checkpoint</code> is
+ optional on input; possible values are <code>no</code>
+ when the disk does not participate in this checkpoint;
+ or <code>bitmap</code> if the disk will track all changes
+ since the creation of this checkpoint via a bitmap, in
+ which case another attribute <code>bitmap</code> will be
+ the name of the tracking bitmap (defaulting to the
+ checkpoint name).
+ </dd>
+ </dl>
+ </dd>
+ <dt><code>creationTime</code></dt>
+ <dd>The time this checkpoint was created. The time is specified
+ in seconds since the Epoch, UTC (i.e. Unix time). Readonly.
+ </dd>
+ <dt><code>parent</code></dt>
+ <dd>The parent of this checkpoint. If present, this element
+ contains exactly one child element, name. This specifies the
+ name of the parent checkpoint of this one, and is used to
+ represent trees of checkpoints. Readonly.
+ </dd>
+ <dt><code>domain</code></dt>
+ <dd>The inactive <a href="formatdomain.html">domain
+ configuration</a> at the time the checkpoint was created.
+ Readonly.
+ </dd>
+ </dl>
+
+ <h2><a id="BackupAttributes">Backup XML</a></h2>
+
+ <p>
+ Creating a backup, whether full or incremental, is done
+ via <code>virDomainBackupBegin()</code>, which takes an XML
+ description of the actions to perform. There are two general
+ modes for backups: a push mode (where the hypervisor writes out
+ the data to the destination file, which may be local or remote),
+ and a pull mode (where the hypervisor creates an NBD server that
+ a third-party client can then read as needed, and which requires
+ the use of temporary storage, typically local, until the backup
+ is complete).
+ </p>
+ <p>
+ The instructions for beginning a backup job are provided as
+ attributes and elements of the
+ top-level <code>domainbackup</code> element. This element
+ includes an optional attribute <code>mode</code> which can be
+ either "push" or "pull" (default push). Where elements are
+ optional on creation, <code>virDomainBackupGetXMLDesc()</code>
+ can be used to see the actual values selected (for example,
+ learning which port the NBD server is using in the pull model,
+ or what file names libvirt generated when none were supplied).
+ The following child elements are supported:
+ </p>
+ <dl>
+ <dt><code>incremental</code></dt>
+ <dd>Optional. If this element is present, it must name an
+ existing checkpoint of the domain, which will be used to make
+ this backup an incremental one (in the push model, only
+ changes since the checkpoint are written to the destination;
+ in the pull model, the NBD server uses the
+ NBD_OPT_SET_META_CONTEXT extension to advertise to the client
+ which portions of the export contain changes since the
+ checkpoint). If omitted, a full backup is performed.
+ </dd>
+ <dt><code>server</code></dt>
+ <dd>Present only for a pull mode backup. Contains the same
+ attributes as the <code>protocol</code> element of a disk
+ attached via NBD in the domain (such as transport, socket,
+ name, port, or tls), necessary to set up an NBD server that
+ exposes the content of each disk at the time the backup
+ started.
+ </dd>
+ <dt><code>disks</code></dt>
+ <dd>This is an optional listing of instructions for disks
+ participating in the backup (if omitted, all disks
+ participate, and libvirt attempts to generate filenames by
+ appending the current timestamp as a suffix). When provided on
+ input, disks omitted from the list do not participate in the
+ backup. On output, the list is present but contains only the
+ disks participating in the backup job. This element has a
+ list of <code>disk</code> sub-elements, describing anywhere
+ from one to all of the disks associated with the domain.
+ <dl>
+ <dt><code>disk</code></dt>
+ <dd>This sub-element describes the checkpoint properties of
+ a specific disk. The attribute <code>name</code> is
+ mandatory, and must match either the <code><target
+ dev='name'/></code> or an unambiguous <code><source
+ file='name'/></code> of one of
+ the <a href="formatdomain.html#elementsDisks">disk
+ devices</a> specified for the domain at the time of the
+ checkpoint. The optional attribute <code>type</code> can
+ be <code>file</code>, <code>block</code>,
+ or <code>networks</code>, similar to a disk declaration
+ for a domain, controls what additional sub-elements are
+ needed to describe the destination (such
+ as <code>protocol</code> for a network destination). In
+ push mode backups, the primary subelement
+ is <code>target</code>; in pull mode, the primary sublement
+ is <code>scratch</code>; but either way,
+ the primary sub-element describes the file name to be used
+ during the backup operation, similar to
+ the <code>source</code> sub-element of a domain disk. An
+ optional sublement <code>driver</code> can also be used to
+ specify a destination format different from qcow2.
+ </dd>
+ </dl>
+ </dd>
+ </dl>
+
+ <h2><a id="example">Examples</a></h2>
+
+ <p>Using this XML to create a checkpoint of just vda on a qemu
+ domain with two disks and a prior checkpoint:</p>
+ <pre>
+<domaincheckpoint>
+ <description>Completion of updates after OS install</description>
+ <disks>
+ <disk name='vda' checkpoint='bitmap'/>
+ <disk name='vdb' checkpoint='no'/>
+ </disks>
+</domaincheckpoint></pre>
+
+ <p>will result in XML similar to this from
+ <code>virDomainCheckpointGetXMLDesc()</code>:</p>
+ <pre>
+<domaincheckpoint>
+ <name>1525889631</name>
+ <description>Completion of updates after OS install</description>
+ <creationTime>1525889631</creationTime>
+ <parent>
+ <name>1525111885</name>
+ </parent>
+ <disks>
+ <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
+ <disk name='vdb' checkpoint='no'/>
+ </disks>
+ <domain>
+ <name>fedora</name>
+ <uuid>93a5c045-6457-2c09-e56c-927cdf34e178</uuid>
+ <memory>1048576</memory>
+ ...
+ <devices>
+ <disk type='file' device='disk'>
+ <driver name='qemu' type='qcow2'/>
+ <source file='/path/to/file1'/>
+ <target dev='vda' bus='virtio'/>
+ </disk>
+ <disk type='file' device='disk' snapshot='external'>
+ <driver name='qemu' type='raw'/>
+ <source file='/path/to/file2'/>
+ <target dev='vdb' bus='virtio'/>
+ </disk>
+ ...
+ </devices>
+ </domain>
+</domaincheckpoint></pre>
+
+ <p>With that checkpoint created, the qcow2 image is now tracking
+ all changes that occur in the image since the checkpoint via
+ the persistent bitmap named <code>1525889631</code>. Now, we
+ can make a subsequent call
+ to <code>virDomainBackupBegin()</code> to perform an incremental
+ backup of just this data, using the following XML to start a
+ pull model NBD export of the vda disk:
+ </p>
+ <pre>
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport="unix" socket="/path/to/server"/>
+ <disks/>
+ <disk name='vda' type='file'/>
+ <scratch file=/path/to/file1.scratch'/>
+ </disk>
+ </disks/>
+</domainbackup>
+ </pre>
+ </body>
+</html>
diff --git a/docs/schemas/domaincheckpoint.rng b/docs/schemas/domaincheckpoint.rng
new file mode 100644
index 0000000000..1e2c16e035
--- /dev/null
+++ b/docs/schemas/domaincheckpoint.rng
@@ -0,0 +1,89 @@
+<?xml version="1.0"?>
+<!-- A Relax NG schema for the libvirt domain checkpoint properties XML format -->
+<grammar xmlns="http://relaxng.org/ns/structure/1.0">
+ <start>
+ <ref name='domaincheckpoint'/>
+ </start>
+
+ <include href='domaincommon.rng'/>
+
+ <define name='domaincheckpoint'>
+ <element name='domaincheckpoint'>
+ <interleave>
+ <optional>
+ <element name='name'>
+ <text/>
+ </element>
+ </optional>
+ <optional>
+ <element name='description'>
+ <text/>
+ </element>
+ </optional>
+ <optional>
+ <element name='creationTime'>
+ <text/>
+ </element>
+ </optional>
+ <optional>
+ <element name='disks'>
+ <zeroOrMore>
+ <ref name='diskcheckpoint'/>
+ </zeroOrMore>
+ </element>
+ </optional>
+ <optional>
+ <choice>
+ <element name='domain'>
+ <element name='uuid'>
+ <ref name="UUID"/>
+ </element>
+ </element>
+ <!-- Nested grammar ensures that any of our overrides of
+ storagecommon/domaincommon defines do not conflict
+ with any domain.rng overrides. -->
+ <grammar>
+ <include href='domain.rng'/>
+ </grammar>
+ </choice>
+ </optional>
+ <optional>
+ <element name='parent'>
+ <element name='name'>
+ <text/>
+ </element>
+ </element>
+ </optional>
+ </interleave>
+ </element>
+ </define>
+
+ <define name='diskcheckpoint'>
+ <element name='disk'>
+ <attribute name='name'>
+ <choice>
+ <ref name='diskTarget'/>
+ <ref name='absFilePath'/>
+ </choice>
+ </attribute>
+ <choice>
+ <attribute name='checkpoint'>
+ <value>no</value>
+ </attribute>
+ <group>
+ <optional>
+ <attribute name='checkpoint'>
+ <value>bitmap</value>
+ </attribute>
+ </optional>
+ <optional>
+ <attribute name='bitmap'>
+ <text/>
+ </attribute>
+ </optional>
+ </group>
+ </choice>
+ </element>
+ </define>
+
+</grammar>
diff --git a/libvirt.spec.in b/libvirt.spec.in
index ace05820aa..50bd79a7d7 100644
--- a/libvirt.spec.in
+++ b/libvirt.spec.in
@@ -2044,6 +2044,7 @@ exit 0
%{_datadir}/libvirt/schemas/cputypes.rng
%{_datadir}/libvirt/schemas/domain.rng
%{_datadir}/libvirt/schemas/domaincaps.rng
+%{_datadir}/libvirt/schemas/domaincheckpoint.rng
%{_datadir}/libvirt/schemas/domaincommon.rng
%{_datadir}/libvirt/schemas/domainsnapshot.rng
%{_datadir}/libvirt/schemas/interface.rng
diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
index 917d2143d8..6912527cf7 100644
--- a/mingw-libvirt.spec.in
+++ b/mingw-libvirt.spec.in
@@ -241,6 +241,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
%{mingw32_datadir}/libvirt/schemas/cputypes.rng
%{mingw32_datadir}/libvirt/schemas/domain.rng
%{mingw32_datadir}/libvirt/schemas/domaincaps.rng
+%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
%{mingw32_datadir}/libvirt/schemas/domaincommon.rng
%{mingw32_datadir}/libvirt/schemas/domainsnapshot.rng
%{mingw32_datadir}/libvirt/schemas/interface.rng
@@ -326,6 +327,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
%{mingw64_datadir}/libvirt/schemas/cputypes.rng
%{mingw64_datadir}/libvirt/schemas/domain.rng
%{mingw64_datadir}/libvirt/schemas/domaincaps.rng
+%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
%{mingw64_datadir}/libvirt/schemas/domaincommon.rng
%{mingw64_datadir}/libvirt/schemas/domainsnapshot.rng
%{mingw64_datadir}/libvirt/schemas/interface.rng
diff --git a/tests/domaincheckpointxml2xmlin/empty.xml b/tests/domaincheckpointxml2xmlin/empty.xml
new file mode 100644
index 0000000000..dc36449142
--- /dev/null
+++ b/tests/domaincheckpointxml2xmlin/empty.xml
@@ -0,0 +1 @@
+<domaincheckpoint/>
diff --git a/tests/domaincheckpointxml2xmlout/empty.xml b/tests/domaincheckpointxml2xmlout/empty.xml
new file mode 100644
index 0000000000..a26c7caab0
--- /dev/null
+++ b/tests/domaincheckpointxml2xmlout/empty.xml
@@ -0,0 +1,10 @@
+<domaincheckpoint>
+ <name>1525889631</name>
+ <creationTime>1525889631</creationTime>
+ <disks>
+ <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
+ </disks>
+ <domain>
+ <uuid>9d37b878-a7cc-9f9a-b78f-49b3abad25a8</uuid>
+ </domain>
+</domaincheckpoint>
diff --git a/tests/virschematest.c b/tests/virschematest.c
index 2d35833919..b866db4326 100644
--- a/tests/virschematest.c
+++ b/tests/virschematest.c
@@ -223,6 +223,8 @@ mymain(void)
"genericxml2xmloutdata", "xlconfigdata", "libxlxml2domconfigdata",
"qemuhotplugtestdomains");
DO_TEST_DIR("domaincaps.rng", "domaincapsschemadata");
+ DO_TEST_DIR("domaincheckpoint.rng", "domaincheckpointxml2xmlin",
+ "domaincheckpointxml2xmlout");
DO_TEST_DIR("domainsnapshot.rng", "domainsnapshotxml2xmlin",
"domainsnapshotxml2xmlout");
DO_TEST_DIR("interface.rng", "interfaceschemadata");
--
2.14.4
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
On Wed, Jun 13, 2018 at 7:42 PM Eric Blake <eblake@redhat.com> wrote:
> Prepare for new checkpoint and backup APIs by describing the XML
> that will represent a checkpoint. This is modeled heavily after
> the XML for virDomainSnapshotPtr, since both represent a point in
> time of the guest. But while a snapshot exists with the intent
> of rolling back to that state, a checkpoint instead makes it
> possible to create an incremental backup at a later time.
>
> Add testsuite coverage of a minimal use of the XML.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
> docs/docs.html.in | 3 +-
> docs/domainstatecapture.html.in | 4 +-
> docs/formatcheckpoint.html.in | 273
> +++++++++++++++++++++++++++++
> docs/schemas/domaincheckpoint.rng | 89 ++++++++++
> libvirt.spec.in | 1 +
> mingw-libvirt.spec.in | 2 +
> tests/domaincheckpointxml2xmlin/empty.xml | 1 +
> tests/domaincheckpointxml2xmlout/empty.xml | 10 ++
> tests/virschematest.c | 2 +
> 9 files changed, 382 insertions(+), 3 deletions(-)
> create mode 100644 docs/formatcheckpoint.html.in
> create mode 100644 docs/schemas/domaincheckpoint.rng
> create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
> create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
>
> diff --git a/docs/docs.html.in b/docs/docs.html.in
> index 4c46b74980..11dfd27ba6 100644
> --- a/docs/docs.html.in
> +++ b/docs/docs.html.in
> @@ -79,7 +79,8 @@
> <a href="formatdomaincaps.html">domain capabilities</a>,
> <a href="formatnode.html">node devices</a>,
> <a href="formatsecret.html">secrets</a>,
> - <a href="formatsnapshot.html">snapshots</a></dd>
> + <a href="formatsnapshot.html">snapshots</a>,
> + <a href="formatcheckpoint.html">checkpoints</a></dd>
>
> <dt><a href="uri.html">URI format</a></dt>
> <dd>The URI formats used for connecting to libvirt</dd>
> diff --git a/docs/domainstatecapture.html.in b/docs/
> domainstatecapture.html.in
> index 00ab7e8ee1..4de93c87c8 100644
> --- a/docs/domainstatecapture.html.in
> +++ b/docs/domainstatecapture.html.in
> @@ -154,9 +154,9 @@
> time as a new backup, so that the next incremental backup can
> refer to the incremental state since the checkpoint created
> during the current backup. Guest state is then actually
> - captured using <code>virDomainBackupBegin()</code>. <!--See also
> + captured using <code>virDomainBackupBegin()</code>. See also
> the <a href="formatcheckpoint.html">XML details</a> used with
> - this command.--></dd>
> + this command.</dd>
> </dl>
>
> <h2><a id="examples">Examples</a></h2>
> diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in
> new file mode 100644
> index 0000000000..34507a9f68
> --- /dev/null
> +++ b/docs/formatcheckpoint.html.in
> @@ -0,0 +1,273 @@
> +<?xml version="1.0" encoding="UTF-8"?>
> +<!DOCTYPE html>
> +<html xmlns="http://www.w3.org/1999/xhtml">
> + <body>
> + <h1>Checkpoint and Backup XML format</h1>
> +
> + <ul id="toc"></ul>
> +
> + <h2><a id="CheckpointAttributes">Checkpoint XML</a></h2>
>
id=CheckpointXML?
> +
> + <p>
> + Domain disk backups, including incremental backups, are one form
> + of <a href="domainstatecapture.html">domain state capture</a>.
> + </p>
> + <p>
> + Libvirt is able to facilitate incremental backups by tracking
> + disk checkpoints, or points in time against which it is easy to
> + compute which portion of the disk has changed. Given a full
> + backup (a backup created from the creation of the disk to a
> + given point in time, coupled with the creation of a disk
> + checkpoint at that time),
Not clear.
> and an incremental backup (a backup
> + created from just the dirty portion of the disk between the
> + first checkpoint and the second backup operation),
Also not clear.
> it is
> + possible to do an offline reconstruction of the state of the
> + disk at the time of the second backup, without having to copy as
> + much data as a second full backup would require. Most disk
> + checkpoints are created in concert with a backup,
> + via <code>virDomainBackupBegin()</code>; however, libvirt also
> + exposes enough support to create disk checkpoints independently
> + from a backup operation,
> + via <code>virDomainCheckpointCreateXML()</code>.
>
Thanks for the extra context.
> + </p>
> + <p>
> + Attributes of libvirt checkpoints are stored as child elements of
> + the <code>domaincheckpoint</code> element. At checkpoint creation
> + time, normally only the <code>name</code>, <code>description</code>,
> + and <code>disks</code> elements are settable; the rest of the
> + fields are ignored on creation, and will be filled in by
> + libvirt in for informational purposes
>
So the user is responsible for creating checkpoints names? Do we use these
the same name in qcow2?
> + by <code>virDomainCheckpointGetXMLDesc()</code>. However, when
> + redefining a checkpoint,
> + with the <code>VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE</code> flag
> + of <code>virDomainCheckpointCreateXML()</code>, all of the XML
> + described here is relevant.
> + </p>
> + <p>
> + Checkpoints are maintained in a hierarchy. A domain can have a
> + current checkpoint, which is the most recent checkpoint compared to
> + the current state of the domain (although a domain might have
> + checkpoints without a current checkpoint, if checkpoints have been
> + deleted in the meantime). Creating or reverting to a checkpoint
> + sets that checkpoint as current, and the prior current checkpoint is
> + the parent of the new checkpoint. Branches in the hierarchy can
> + be formed by reverting to a checkpoint with a child, then creating
> + another checkpoint.
>
This seems too complex. Why do we need arbitrary trees of checkpoints?
What is the meaning of reverting a checkpoint?
> + </p>
> + <p>
> + The top-level <code>domaincheckpoint</code> element may contain
> + the following elements:
> + </p>
> + <dl>
> + <dt><code>name</code></dt>
> + <dd>The name for this checkpoint. If the name is specified when
> + initially creating the checkpoint, then the checkpoint will have
> + that particular name. If the name is omitted when initially
> + creating the checkpoint, then libvirt will make up a name for
> + the checkpoint, based on the time when it was created.
> + </dd>
>
Why not simplify and require the use to provide a name?
> + <dt><code>description</code></dt>
> + <dd>A human-readable description of the checkpoint. If the
> + description is omitted when initially creating the checkpoint,
> + then this field will be empty.
> + </dd>
> + <dt><code>disks</code></dt>
> + <dd>On input, this is an optional listing of specific
> + instructions for disk checkpoints; it is needed when making a
> + checkpoint on only a subset of the disks associated with a
> + domain (in particular, since qemu checkpoints require qcow2
> + disks, this element may be needed on input for excluding guest
> + disks that are not in qcow2 format); if omitted on input, then
> + all disks participate in the checkpoint. On output, this is
> + fully populated to show the state of each disk in the
> + checkpoint. This element has a list of <code>disk</code>
> + sub-elements, describing anywhere from one to all of the disks
> + associated with the domain.
>
Why not always specify the disks?
> + <dl>
> + <dt><code>disk</code></dt>
> + <dd>This sub-element describes the checkpoint properties of
> + a specific disk. The attribute <code>name</code> is
> + mandatory, and must match either the <code><target
> + dev='name'/></code> or an unambiguous <code><source
> + file='name'/></code> of one of
> + the <a href="formatdomain.html#elementsDisks">disk
> + devices</a> specified for the domain at the time of the
> + checkpoint. The attribute <code>checkpoint</code> is
> + optional on input; possible values are <code>no</code>
> + when the disk does not participate in this checkpoint;
> + or <code>bitmap</code> if the disk will track all changes
> + since the creation of this checkpoint via a bitmap, in
> + which case another attribute <code>bitmap</code> will be
> + the name of the tracking bitmap (defaulting to the
> + checkpoint name).
>
Seems too complicated. Why do we need to support a checkpoint
referencing a bitmap with a different name?
Instead we can have a list of disk that will participate in the checkpoint.
Anything not specified will not participate in the snapshot. The name of
the checkpoint is always the name of the bitmap.
> + </dd>
> + </dl>
> + </dd>
> + <dt><code>creationTime</code></dt>
> + <dd>The time this checkpoint was created. The time is specified
> + in seconds since the Epoch, UTC (i.e. Unix time). Readonly.
> + </dd>
> + <dt><code>parent</code></dt>
> + <dd>The parent of this checkpoint. If present, this element
> + contains exactly one child element, name. This specifies the
> + name of the parent checkpoint of this one, and is used to
> + represent trees of checkpoints. Readonly.
> + </dd>
>
I think we are missing here the size of the underlying data in every
disk. This probably means how many dirty bits we have in the bitmaps
referenced by the checkpoint for every disk.
> + <dt><code>domain</code></dt>
> + <dd>The inactive <a href="formatdomain.html">domain
> + configuration</a> at the time the checkpoint was created.
> + Readonly.
>
What do you mean by "inactive domain configuration"?
> + </dd>
> + </dl>
> +
> + <h2><a id="BackupAttributes">Backup XML</a></h2>
> +
> + <p>
> + Creating a backup, whether full or incremental, is done
> + via <code>virDomainBackupBegin()</code>, which takes an XML
> + description of the actions to perform. There are two general
> + modes for backups: a push mode (where the hypervisor writes out
> + the data to the destination file, which may be local or remote),
> + and a pull mode (where the hypervisor creates an NBD server that
> + a third-party client can then read as needed, and which requires
> + the use of temporary storage, typically local, until the backup
> + is complete).
+ </p>
> + <p>
> + The instructions for beginning a backup job are provided as
> + attributes and elements of the
> + top-level <code>domainbackup</code> element. This element
> + includes an optional attribute <code>mode</code> which can be
> + either "push" or "pull" (default push). Where elements are
> + optional on creation, <code>virDomainBackupGetXMLDesc()</code>
> + can be used to see the actual values selected (for example,
> + learning which port the NBD server is using in the pull model,
> + or what file names libvirt generated when none were supplied).
> + The following child elements are supported:
> + </p>
> + <dl>
> + <dt><code>incremental</code></dt>
> + <dd>Optional. If this element is present, it must name an
> + existing checkpoint of the domain, which will be used to make
> + this backup an incremental one (in the push model, only
> + changes since the checkpoint are written to the destination;
> + in the pull model, the NBD server uses the
> + NBD_OPT_SET_META_CONTEXT extension to advertise to the client
> + which portions of the export contain changes since the
> + checkpoint). If omitted, a full backup is performed.
>
Just to make it clear:
For example we start with:
c1 c2 [c3]
c3 is the active checkpoint.
We create a new checkpoint:
c1 c2 c3 [c4]
So
- using incremental=c2, we will get data referenced by c2?
- using incremental=c1, we will get data reference by both c1 and c2?
What if we want to backup only data from c1 to c2, not including c3?
I don't have a use case for this, but if we can specify tow checkpoints
this can be possible.
For example:
<chekpoints from="c1", to="c2">
Or
<checkpoints from="c2">
> + </dd>
> + <dt><code>server</code></dt>
> + <dd>Present only for a pull mode backup. Contains the same
> + attributes as the <code>protocol</code> element of a disk
> + attached via NBD in the domain (such as transport, socket,
> + name, port, or tls), necessary to set up an NBD server that
> + exposes the content of each disk at the time the backup
> + started.
> + </dd>
>
To get the list of changed blocks, we planned to use something like:
qemu-img map nbd+unix:/socket=server.sock
Is this possible now? planned?
To get the actual data, oVirt needs a device to read from. We don't want
to write nbd-client, and we cannot use qemu-img since it does not support
streaming data, and we want to stream data using http to the backup
application.
I guess we will have do this:
qemu-nbd -c /dev/nbd0 nbd+unix:/socket=server.sock
And serve the data from /dev/nbd0.
> + <dt><code>disks</code></dt>
> + <dd>This is an optional listing of instructions for disks
> + participating in the backup (if omitted, all disks
> + participate, and libvirt attempts to generate filenames by
> + appending the current timestamp as a suffix). When provided on
> + input, disks omitted from the list do not participate in the
> + backup. On output, the list is present but contains only the
> + disks participating in the backup job. This element has a
> + list of <code>disk</code> sub-elements, describing anywhere
> + from one to all of the disks associated with the domain.
> + <dl>
> + <dt><code>disk</code></dt>
> + <dd>This sub-element describes the checkpoint properties of
> + a specific disk. The attribute <code>name</code> is
> + mandatory, and must match either the <code><target
> + dev='name'/></code> or an unambiguous <code><source
> + file='name'/></code> of one of
> + the <a href="formatdomain.html#elementsDisks">disk
> + devices</a> specified for the domain at the time of the
> + checkpoint. The optional attribute <code>type</code> can
> + be <code>file</code>, <code>block</code>,
> + or <code>networks</code>, similar to a disk declaration
> + for a domain, controls what additional sub-elements are
> + needed to describe the destination (such
> + as <code>protocol</code> for a network destination). In
> + push mode backups, the primary subelement
> + is <code>target</code>; in pull mode, the primary sublement
> + is <code>scratch</code>; but either way,
> + the primary sub-element describes the file name to be used
> + during the backup operation, similar to
> + the <code>source</code> sub-element of a domain disk. An
> + optional sublement <code>driver</code> can also be used to
> + specify a destination format different from qcow2.
>
This should be similar to the way we specify disks for vm, right?
Anything that works as vm disk will work for pushing backups?
I will continue with the rest of the patch later.
Nir
+ </dd>
> + </dl>
> + </dd>
> + </dl>
> +
> + <h2><a id="example">Examples</a></h2>
> +
> + <p>Using this XML to create a checkpoint of just vda on a qemu
> + domain with two disks and a prior checkpoint:</p>
> + <pre>
> +<domaincheckpoint>
> + <description>Completion of updates after OS
> install</description>
> + <disks>
> + <disk name='vda' checkpoint='bitmap'/>
> + <disk name='vdb' checkpoint='no'/>
> + </disks>
> +</domaincheckpoint></pre>
> +
> + <p>will result in XML similar to this from
> + <code>virDomainCheckpointGetXMLDesc()</code>:</p>
> + <pre>
> +<domaincheckpoint>
> + <name>1525889631</name>
> + <description>Completion of updates after OS
> install</description>
> + <creationTime>1525889631</creationTime>
> + <parent>
> + <name>1525111885</name>
> + </parent>
> + <disks>
> + <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
> + <disk name='vdb' checkpoint='no'/>
> + </disks>
> + <domain>
> + <name>fedora</name>
> + <uuid>93a5c045-6457-2c09-e56c-927cdf34e178</uuid>
> + <memory>1048576</memory>
> + ...
> + <devices>
> + <disk type='file' device='disk'>
> + <driver name='qemu' type='qcow2'/>
> + <source file='/path/to/file1'/>
> + <target dev='vda' bus='virtio'/>
> + </disk>
> + <disk type='file' device='disk' snapshot='external'>
> + <driver name='qemu' type='raw'/>
> + <source file='/path/to/file2'/>
> + <target dev='vdb' bus='virtio'/>
> + </disk>
> + ...
> + </devices>
> + </domain>
> +</domaincheckpoint></pre>
> +
> + <p>With that checkpoint created, the qcow2 image is now tracking
> + all changes that occur in the image since the checkpoint via
> + the persistent bitmap named <code>1525889631</code>. Now, we
> + can make a subsequent call
> + to <code>virDomainBackupBegin()</code> to perform an incremental
> + backup of just this data, using the following XML to start a
> + pull model NBD export of the vda disk:
> + </p>
> + <pre>
> +<domainbackup mode="pull">
> + <incremental>1525889631</incremental>
> + <server transport="unix" socket="/path/to/server"/>
> + <disks/>
> + <disk name='vda' type='file'/>
> + <scratch file=/path/to/file1.scratch'/>
> + </disk>
> + </disks/>
> +</domainbackup>
> + </pre>
> + </body>
> +</html>
> diff --git a/docs/schemas/domaincheckpoint.rng
> b/docs/schemas/domaincheckpoint.rng
> new file mode 100644
> index 0000000000..1e2c16e035
> --- /dev/null
> +++ b/docs/schemas/domaincheckpoint.rng
> @@ -0,0 +1,89 @@
> +<?xml version="1.0"?>
> +<!-- A Relax NG schema for the libvirt domain checkpoint properties XML
> format -->
> +<grammar xmlns="http://relaxng.org/ns/structure/1.0">
> + <start>
> + <ref name='domaincheckpoint'/>
> + </start>
> +
> + <include href='domaincommon.rng'/>
> +
> + <define name='domaincheckpoint'>
> + <element name='domaincheckpoint'>
> + <interleave>
> + <optional>
> + <element name='name'>
> + <text/>
> + </element>
> + </optional>
> + <optional>
> + <element name='description'>
> + <text/>
> + </element>
> + </optional>
> + <optional>
> + <element name='creationTime'>
> + <text/>
> + </element>
> + </optional>
> + <optional>
> + <element name='disks'>
> + <zeroOrMore>
> + <ref name='diskcheckpoint'/>
> + </zeroOrMore>
> + </element>
> + </optional>
> + <optional>
> + <choice>
> + <element name='domain'>
> + <element name='uuid'>
> + <ref name="UUID"/>
> + </element>
> + </element>
> + <!-- Nested grammar ensures that any of our overrides of
> + storagecommon/domaincommon defines do not conflict
> + with any domain.rng overrides. -->
> + <grammar>
> + <include href='domain.rng'/>
> + </grammar>
> + </choice>
> + </optional>
> + <optional>
> + <element name='parent'>
> + <element name='name'>
> + <text/>
> + </element>
> + </element>
> + </optional>
> + </interleave>
> + </element>
> + </define>
> +
> + <define name='diskcheckpoint'>
> + <element name='disk'>
> + <attribute name='name'>
> + <choice>
> + <ref name='diskTarget'/>
> + <ref name='absFilePath'/>
> + </choice>
> + </attribute>
> + <choice>
> + <attribute name='checkpoint'>
> + <value>no</value>
> + </attribute>
> + <group>
> + <optional>
> + <attribute name='checkpoint'>
> + <value>bitmap</value>
> + </attribute>
> + </optional>
> + <optional>
> + <attribute name='bitmap'>
> + <text/>
> + </attribute>
> + </optional>
> + </group>
> + </choice>
> + </element>
> + </define>
> +
> +</grammar>
> diff --git a/libvirt.spec.in b/libvirt.spec.in
> index ace05820aa..50bd79a7d7 100644
> --- a/libvirt.spec.in
> +++ b/libvirt.spec.in
> @@ -2044,6 +2044,7 @@ exit 0
> %{_datadir}/libvirt/schemas/cputypes.rng
> %{_datadir}/libvirt/schemas/domain.rng
> %{_datadir}/libvirt/schemas/domaincaps.rng
> +%{_datadir}/libvirt/schemas/domaincheckpoint.rng
> %{_datadir}/libvirt/schemas/domaincommon.rng
> %{_datadir}/libvirt/schemas/domainsnapshot.rng
> %{_datadir}/libvirt/schemas/interface.rng
> diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
> index 917d2143d8..6912527cf7 100644
> --- a/mingw-libvirt.spec.in
> +++ b/mingw-libvirt.spec.in
> @@ -241,6 +241,7 @@ rm -rf
> $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
> %{mingw32_datadir}/libvirt/schemas/cputypes.rng
> %{mingw32_datadir}/libvirt/schemas/domain.rng
> %{mingw32_datadir}/libvirt/schemas/domaincaps.rng
> +%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
> %{mingw32_datadir}/libvirt/schemas/domaincommon.rng
> %{mingw32_datadir}/libvirt/schemas/domainsnapshot.rng
> %{mingw32_datadir}/libvirt/schemas/interface.rng
> @@ -326,6 +327,7 @@ rm -rf
> $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
> %{mingw64_datadir}/libvirt/schemas/cputypes.rng
> %{mingw64_datadir}/libvirt/schemas/domain.rng
> %{mingw64_datadir}/libvirt/schemas/domaincaps.rng
> +%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
> %{mingw64_datadir}/libvirt/schemas/domaincommon.rng
> %{mingw64_datadir}/libvirt/schemas/domainsnapshot.rng
> %{mingw64_datadir}/libvirt/schemas/interface.rng
> diff --git a/tests/domaincheckpointxml2xmlin/empty.xml
> b/tests/domaincheckpointxml2xmlin/empty.xml
> new file mode 100644
> index 0000000000..dc36449142
> --- /dev/null
> +++ b/tests/domaincheckpointxml2xmlin/empty.xml
> @@ -0,0 +1 @@
> +<domaincheckpoint/>
> diff --git a/tests/domaincheckpointxml2xmlout/empty.xml
> b/tests/domaincheckpointxml2xmlout/empty.xml
> new file mode 100644
> index 0000000000..a26c7caab0
> --- /dev/null
> +++ b/tests/domaincheckpointxml2xmlout/empty.xml
> @@ -0,0 +1,10 @@
> +<domaincheckpoint>
> + <name>1525889631</name>
> + <creationTime>1525889631</creationTime>
> + <disks>
> + <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
> + </disks>
> + <domain>
> + <uuid>9d37b878-a7cc-9f9a-b78f-49b3abad25a8</uuid>
> + </domain>
> +</domaincheckpoint>
> diff --git a/tests/virschematest.c b/tests/virschematest.c
> index 2d35833919..b866db4326 100644
> --- a/tests/virschematest.c
> +++ b/tests/virschematest.c
> @@ -223,6 +223,8 @@ mymain(void)
> "genericxml2xmloutdata", "xlconfigdata",
> "libxlxml2domconfigdata",
> "qemuhotplugtestdomains");
> DO_TEST_DIR("domaincaps.rng", "domaincapsschemadata");
> + DO_TEST_DIR("domaincheckpoint.rng", "domaincheckpointxml2xmlin",
> + "domaincheckpointxml2xmlout");
> DO_TEST_DIR("domainsnapshot.rng", "domainsnapshotxml2xmlin",
> "domainsnapshotxml2xmlout");
> DO_TEST_DIR("interface.rng", "interfaceschemadata");
> --
> 2.14.4
>
>
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
On 06/26/2018 02:51 PM, Nir Soffer wrote: > On Wed, Jun 13, 2018 at 7:42 PM Eric Blake <eblake@redhat.com> wrote: > >> Prepare for new checkpoint and backup APIs by describing the XML >> that will represent a checkpoint. This is modeled heavily after >> the XML for virDomainSnapshotPtr, since both represent a point in >> time of the guest. But while a snapshot exists with the intent >> of rolling back to that state, a checkpoint instead makes it >> possible to create an incremental backup at a later time. >> >> Add testsuite coverage of a minimal use of the XML. >> +++ b/docs/formatcheckpoint.html.in >> @@ -0,0 +1,273 @@ >> +<?xml version="1.0" encoding="UTF-8"?> >> +<!DOCTYPE html> >> +<html xmlns="http://www.w3.org/1999/xhtml"> >> + <body> >> + <h1>Checkpoint and Backup XML format</h1> >> + >> + <ul id="toc"></ul> >> + >> + <h2><a id="CheckpointAttributes">Checkpoint XML</a></h2> >> > > id=CheckpointXML? Matches what the existing formatsnapshot.html.in named its tag. (If you haven't guessed, I'm heavily relying on snapshots as my template for adding this). > > >> + >> + <p> >> + Domain disk backups, including incremental backups, are one form >> + of <a href="domainstatecapture.html">domain state capture</a>. >> + </p> >> + <p> >> + Libvirt is able to facilitate incremental backups by tracking >> + disk checkpoints, or points in time against which it is easy to >> + compute which portion of the disk has changed. Given a full >> + backup (a backup created from the creation of the disk to a >> + given point in time, coupled with the creation of a disk >> + checkpoint at that time), > > > Not clear. > > >> and an incremental backup (a backup >> + created from just the dirty portion of the disk between the >> + first checkpoint and the second backup operation), > > > Also not clear. Okay, I will try to improve these in v2. But (other than answering these good review emails), my current priority is a working demo (to prove the API works) prior to further doc polish. > > >> it is >> + possible to do an offline reconstruction of the state of the >> + disk at the time of the second backup, without having to copy as >> + much data as a second full backup would require. Most disk >> + checkpoints are created in concert with a backup, >> + via <code>virDomainBackupBegin()</code>; however, libvirt also >> + exposes enough support to create disk checkpoints independently >> + from a backup operation, >> + via <code>virDomainCheckpointCreateXML()</code>. >> > > Thanks for the extra context. > > >> + </p> >> + <p> >> + Attributes of libvirt checkpoints are stored as child elements of >> + the <code>domaincheckpoint</code> element. At checkpoint creation >> + time, normally only the <code>name</code>, <code>description</code>, >> + and <code>disks</code> elements are settable; the rest of the >> + fields are ignored on creation, and will be filled in by >> + libvirt in for informational purposes >> > > So the user is responsible for creating checkpoints names? Do we use these > the same name in qcow2? My intent is that if the user does not assign a checkpoint name, then libvirt will default it to the current time in seconds-since-the-Epoch. Then, whatever name is given to the checkpoint (whether chosen by libvirt or assigned by the user) will also be the default name of the bitmap created in each qcow2 volume, but the XML also allows you to name the qcow2 bitmaps something different than the checkpoint name (maybe not a wise idea in the common case, but could come in handy later if you use the _REDEFINE flag to teach libvirt about existing bitmaps that are already present in a qcow2 image rather than placed there by libvirt). >> + <p> >> + Checkpoints are maintained in a hierarchy. A domain can have a >> + current checkpoint, which is the most recent checkpoint compared to >> + the current state of the domain (although a domain might have >> + checkpoints without a current checkpoint, if checkpoints have been >> + deleted in the meantime). Creating or reverting to a checkpoint >> + sets that checkpoint as current, and the prior current checkpoint is >> + the parent of the new checkpoint. Branches in the hierarchy can >> + be formed by reverting to a checkpoint with a child, then creating >> + another checkpoint. >> > > This seems too complex. Why do we need arbitrary trees of checkpoints? Because snapshots had an arbitrary tree, and it was easier to copy from snapshots. Even if we only use a linear tree for now, it is still feasible that in the future, we can facilitate a domain rolling back to the disk state as captured at checkpoint C1, at which point you could then have multiple children C2 (the bitmap created prior to rolling back) and C3 (the bitmap created for tracking changes made after rolling back). Again, for a first cut, I probably will punt and state that snapshots and incremental backups do not play well together yet; but as we get experience and add more code, the API is flexible enough that down the road we really can offer reverting to an arbitrary snapshot and ALSO updating checkpoints to match. > > What is the meaning of reverting a checkpoint? Hmm - right now, you can't (that was one Snapshot API that I intentionally did not copy over to Checkpoint), so I should probably reword that. > > >> + </p> >> + <p> >> + The top-level <code>domaincheckpoint</code> element may contain >> + the following elements: >> + </p> >> + <dl> >> + <dt><code>name</code></dt> >> + <dd>The name for this checkpoint. If the name is specified when >> + initially creating the checkpoint, then the checkpoint will have >> + that particular name. If the name is omitted when initially >> + creating the checkpoint, then libvirt will make up a name for >> + the checkpoint, based on the time when it was created. >> + </dd> >> > > Why not simplify and require the use to provide a name? Because we didn't require the user to provide names for snapshots, and generating a name via the current timestamp is still fairly likely to be usable. > > >> + <dt><code>description</code></dt> >> + <dd>A human-readable description of the checkpoint. If the >> + description is omitted when initially creating the checkpoint, >> + then this field will be empty. >> + </dd> >> + <dt><code>disks</code></dt> >> + <dd>On input, this is an optional listing of specific >> + instructions for disk checkpoints; it is needed when making a >> + checkpoint on only a subset of the disks associated with a >> + domain (in particular, since qemu checkpoints require qcow2 >> + disks, this element may be needed on input for excluding guest >> + disks that are not in qcow2 format); if omitted on input, then >> + all disks participate in the checkpoint. On output, this is >> + fully populated to show the state of each disk in the >> + checkpoint. This element has a list of <code>disk</code> >> + sub-elements, describing anywhere from one to all of the disks >> + associated with the domain. >> > > Why not always specify the disks? Because if your guest uses all qcow2 images, and you don't want to exclude any images from the checkpoint, then not specifying <disks> does the right thing with less typing. Just because libvirt tries to have sane defaults doesn't mean you have to rely on them, though. > > >> + <dl> >> + <dt><code>disk</code></dt> >> + <dd>This sub-element describes the checkpoint properties of >> + a specific disk. The attribute <code>name</code> is >> + mandatory, and must match either the <code><target >> + dev='name'/></code> or an unambiguous <code><source >> + file='name'/></code> of one of >> + the <a href="formatdomain.html#elementsDisks">disk >> + devices</a> specified for the domain at the time of the >> + checkpoint. The attribute <code>checkpoint</code> is >> + optional on input; possible values are <code>no</code> >> + when the disk does not participate in this checkpoint; >> + or <code>bitmap</code> if the disk will track all changes >> + since the creation of this checkpoint via a bitmap, in >> + which case another attribute <code>bitmap</code> will be >> + the name of the tracking bitmap (defaulting to the >> + checkpoint name). >> > > Seems too complicated. Why do we need to support a checkpoint > referencing a bitmap with a different name? For the same reason that you can support an internal snapshot referencing a qcow2 snapshot with a different name. Yeah, it's probably not a common usage, but there are cases (such as when using _REDEFINE) where it can prove invaluable. You're right that most users won't name qcow2 bitmaps differently from the libvirt checkpoint name. > > Instead we can have a list of disk that will participate in the checkpoint. > Anything not specified will not participate in the snapshot. The name of > the checkpoint is always the name of the bitmap. My worry is about future extensibility of the XML. If the XML is too simple, then we may lock ourselves into a corner at not being able to support some other backend implementation of checkpoints (just because qemu implements checkpoints via qcow2 bitmaps does not mean that some other hyperviser won't come along that implements checkpoints via a UUID, so I tried to leave room for <disk checkpoint='uuid' uuid='..-..-...'/> as potential XML for such a hypervisor mapping - and while bitmap names different from checkpoint names is unusual, it is much more likely that UUIDs for multiple disks would have to be different per disk) > > >> + </dd> >> + </dl> >> + </dd> >> + <dt><code>creationTime</code></dt> >> + <dd>The time this checkpoint was created. The time is specified >> + in seconds since the Epoch, UTC (i.e. Unix time). Readonly. >> + </dd> >> + <dt><code>parent</code></dt> >> + <dd>The parent of this checkpoint. If present, this element >> + contains exactly one child element, name. This specifies the >> + name of the parent checkpoint of this one, and is used to >> + represent trees of checkpoints. Readonly. >> + </dd> >> > > I think we are missing here the size of the underlying data in every > disk. This probably means how many dirty bits we have in the bitmaps > referenced by the checkpoint for every disk. That would be an output-only XML element, and only if qemu were even modified to expose that information. But yes, I can see how exposing that could be useful. > > >> + <dt><code>domain</code></dt> >> + <dd>The inactive <a href="formatdomain.html">domain >> + configuration</a> at the time the checkpoint was created. >> + Readonly. >> > > What do you mean by "inactive domain configuration"? Copy-and-paste from snapshots, but in general, what it would take to start a new VM using a restoration of the backup images corresponding to that checkpoint (that is, the XML is the smaller persistent form, rather than the larger running form; my classic example used to be that the 'inactive domain configuration' omits <alias> tags while the 'running configuration' does not - but since libvirt recently added support for user-settable <alias> tags, that no longer holds...). >> + <dl> >> + <dt><code>incremental</code></dt> >> + <dd>Optional. If this element is present, it must name an >> + existing checkpoint of the domain, which will be used to make >> + this backup an incremental one (in the push model, only >> + changes since the checkpoint are written to the destination; >> + in the pull model, the NBD server uses the >> + NBD_OPT_SET_META_CONTEXT extension to advertise to the client >> + which portions of the export contain changes since the >> + checkpoint). If omitted, a full backup is performed. >> > > Just to make it clear: > > For example we start with: > > c1 c2 [c3] > > c3 is the active checkpoint. > > We create a new checkpoint: > > c1 c2 c3 [c4] > > So > - using incremental=c2, we will get data referenced by c2? Your incremental backup would get all changes since the point in time c2 was created (that is, the changes recorded by the merge of bitmaps c2 and c3). > - using incremental=c1, we will get data reference by both c1 and c2? Your incremental backup would get all changes since the point in time c1 was created (that is, the changes recorded by the merge of bitmaps c1, c2, and c3). > > What if we want to backup only data from c1 to c2, not including c3? Qemu can't do that right now, so this API can't do it either. Maybe there's a way to add it into the API (and the fact that we used XML leaves that door wide open), but not right now. > I don't have a use case for this, but if we can specify tow checkpoints > this can be possible. > > For example: > > <chekpoints from="c1", to="c2"> > > Or > > <checkpoints from="c2"> Or the current proposal of <incremental> serves as the 'from', and a new sibling element <limit> becomes the 'to', if it becomes possible to limit a backup to an earlier point in time than the present call to the API. > > >> + </dd> >> + <dt><code>server</code></dt> >> + <dd>Present only for a pull mode backup. Contains the same >> + attributes as the <code>protocol</code> element of a disk >> + attached via NBD in the domain (such as transport, socket, >> + name, port, or tls), necessary to set up an NBD server that >> + exposes the content of each disk at the time the backup >> + started. >> + </dd> >> > > To get the list of changed blocks, we planned to use something like: > > qemu-img map nbd+unix:/socket=server.sock > > Is this possible now? planned? Possible via the x-nbd-server-add-bitmap command added in qemu commit 767f0c7, coupled with a client that knows how to request NBD_OPT_SET_META_CONTEXT "qemu:dirty-bitmap:foo" then read the bitmap with NBD_CMD_BLOCK_STATUS (I have a hack patch sitting on the qemu list that lets qemu-img behave as such a client: https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg05993.html) > > To get the actual data, oVirt needs a device to read from. We don't want > to write nbd-client, and we cannot use qemu-img since it does not support > streaming data, and we want to stream data using http to the backup > application. > > I guess we will have do this: > > qemu-nbd -c /dev/nbd0 nbd+unix:/socket=server.sock > > And serve the data from /dev/nbd0. Yes, except that the kernel NBD client plugin does not have support for NBD_CMD_BLOCK_STATUS, so reading /dev/nbd0 won't be able to find the dirty blocks. But you could always do it in two steps: first, connect a client that only reads the bitmap (such as qemu-img with my hack), then connect the kernel client so that you can stream just the portions of /dev/nbd0 referenced in the map of the first step. (Or, since both clients would be read-only, you can have them both connected to the qemu server at once) > > >> + <dt><code>disks</code></dt> >> + <dd>This is an optional listing of instructions for disks >> + participating in the backup (if omitted, all disks >> + participate, and libvirt attempts to generate filenames by >> + appending the current timestamp as a suffix). When provided on >> + input, disks omitted from the list do not participate in the >> + backup. On output, the list is present but contains only the >> + disks participating in the backup job. This element has a >> + list of <code>disk</code> sub-elements, describing anywhere >> + from one to all of the disks associated with the domain. >> + <dl> >> + <dt><code>disk</code></dt> >> + <dd>This sub-element describes the checkpoint properties of >> + a specific disk. The attribute <code>name</code> is >> + mandatory, and must match either the <code><target >> + dev='name'/></code> or an unambiguous <code><source >> + file='name'/></code> of one of >> + the <a href="formatdomain.html#elementsDisks">disk >> + devices</a> specified for the domain at the time of the >> + checkpoint. The optional attribute <code>type</code> can >> + be <code>file</code>, <code>block</code>, >> + or <code>networks</code>, similar to a disk declaration >> + for a domain, controls what additional sub-elements are >> + needed to describe the destination (such >> + as <code>protocol</code> for a network destination). In >> + push mode backups, the primary subelement >> + is <code>target</code>; in pull mode, the primary sublement >> + is <code>scratch</code>; but either way, >> + the primary sub-element describes the file name to be used >> + during the backup operation, similar to >> + the <code>source</code> sub-element of a domain disk. An >> + optional sublement <code>driver</code> can also be used to >> + specify a destination format different from qcow2. >> > > This should be similar to the way we specify disks for vm, right? > Anything that works as vm disk will work for pushing backups? Ultimately, yes, I'd like to support gluster/NBD/sheepdog/... destinations. My initial implementation is less ambitious, and supports just local files (because those are easier to test and therefore produce a demo with). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 06/13/2018 12:42 PM, Eric Blake wrote:
> Prepare for new checkpoint and backup APIs by describing the XML
> that will represent a checkpoint. This is modeled heavily after
> the XML for virDomainSnapshotPtr, since both represent a point in
> time of the guest. But while a snapshot exists with the intent
> of rolling back to that state, a checkpoint instead makes it
> possible to create an incremental backup at a later time.
>
> Add testsuite coverage of a minimal use of the XML.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
> docs/docs.html.in | 3 +-
> docs/domainstatecapture.html.in | 4 +-
> docs/formatcheckpoint.html.in | 273 +++++++++++++++++++++++++++++
> docs/schemas/domaincheckpoint.rng | 89 ++++++++++
> libvirt.spec.in | 1 +
> mingw-libvirt.spec.in | 2 +
> tests/domaincheckpointxml2xmlin/empty.xml | 1 +
> tests/domaincheckpointxml2xmlout/empty.xml | 10 ++
> tests/virschematest.c | 2 +
> 9 files changed, 382 insertions(+), 3 deletions(-)
> create mode 100644 docs/formatcheckpoint.html.in
> create mode 100644 docs/schemas/domaincheckpoint.rng
> create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
> create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
>
> diff --git a/docs/docs.html.in b/docs/docs.html.in
> index 4c46b74980..11dfd27ba6 100644
> --- a/docs/docs.html.in
> +++ b/docs/docs.html.in
> @@ -79,7 +79,8 @@
> <a href="formatdomaincaps.html">domain capabilities</a>,
> <a href="formatnode.html">node devices</a>,
> <a href="formatsecret.html">secrets</a>,
> - <a href="formatsnapshot.html">snapshots</a></dd>
> + <a href="formatsnapshot.html">snapshots</a>,
> + <a href="formatcheckpoint.html">checkpoints</a></dd>
>
> <dt><a href="uri.html">URI format</a></dt>
> <dd>The URI formats used for connecting to libvirt</dd>
Add a link in the format.html.in and index.html.in pages too.
> diff --git a/docs/domainstatecapture.html.in b/docs/domainstatecapture.html.in
> index 00ab7e8ee1..4de93c87c8 100644
> --- a/docs/domainstatecapture.html.in
> +++ b/docs/domainstatecapture.html.in
> @@ -154,9 +154,9 @@
> time as a new backup, so that the next incremental backup can
> refer to the incremental state since the checkpoint created
> during the current backup. Guest state is then actually
> - captured using <code>virDomainBackupBegin()</code>. <!--See also
> + captured using <code>virDomainBackupBegin()</code>. See also
> the <a href="formatcheckpoint.html">XML details</a> used with
> - this command.--></dd>
> + this command.</dd>
> </dl>
>
> <h2><a id="examples">Examples</a></h2>
> diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in
> new file mode 100644
> index 0000000000..34507a9f68
> --- /dev/null
> +++ b/docs/formatcheckpoint.html.in
> @@ -0,0 +1,273 @@
> +<?xml version="1.0" encoding="UTF-8"?>
> +<!DOCTYPE html>
> +<html xmlns="http://www.w3.org/1999/xhtml">
> + <body>
> + <h1>Checkpoint and Backup XML format</h1>
> +
> + <ul id="toc"></ul>
> +
> + <h2><a id="CheckpointAttributes">Checkpoint XML</a></h2>
> +
> + <p>
> + Domain disk backups, including incremental backups, are one form'
> + of <a href="domainstatecapture.html">domain state capture</a>.
> + </p>
IMO: Strange opening line for something describing checkpoints.
As I've read further, the fact that checkpoint and backup is only
supported for qcow2 domain disks I would think needs to be up at the top
here - front an center. No sense reading any further for raw disks.
Yet another patch 2 "factor" related to choosing a backup plan. What to
do if you have raw devices and how those should be handled.
This would seem to preclude LUKS encrypted devices, true?
What about disks with various levels of backingStore logic? I can only
imagine some of the depth issues causing problems with that logic would
be applicable here too with the hierarchical approach.
> + <p>
> + Libvirt is able to facilitate incremental backups by tracking
> + disk checkpoints, or points in time against which it is easy to
s/, or/ or/
> + compute which portion of the disk has changed. Given a full
> + backup (a backup created from the creation of the disk to a
> + given point in time, coupled with the creation of a disk
s/time,/time)
> + checkpoint at that time), and an incremental backup (a backup
s/time),/time,
> + created from just the dirty portion of the disk between the
> + first checkpoint and the second backup operation), it is
> + possible to do an offline reconstruction of the state of the
> + disk at the time of the second backup, without having to copy as
s/without,/without/
> + much data as a second full backup would require. Most disk
> + checkpoints are created in concert with a backup,
s/backup,/backup/
> + via <code>virDomainBackupBegin()</code>; however, libvirt also
> + exposes enough support to create disk checkpoints independently
> + from a backup operation,
s/operation,/operation/
> + via <code>virDomainCheckpointCreateXML()</code>.
> + </p>
NB: virDomainBackupBegin doesn't exist yet. Still a few patches away.
> + <p>
> + Attributes of libvirt checkpoints are stored as child elements of
> + the <code>domaincheckpoint</code> element. At checkpoint creation
> + time, normally only the <code>name</code>, <code>description</code>,
> + and <code>disks</code> elements are settable; the rest of the
s/; the/. The
> + fields are ignored on creation, and will be filled in by
s/creation,/creation/
> + libvirt in for informational purposes
> + by <code>virDomainCheckpointGetXMLDesc()</code>. However, when
> + redefining a checkpoint,
> + with the <code>VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE</code> flag
> + of <code>virDomainCheckpointCreateXML()</code>, all of the XML
s/XML/XML fields/
> + described here is relevant.
s/is/are/
> + </p>
"All" of them? Even the readonly ones?
> + <p>
> + Checkpoints are maintained in a hierarchy. A domain can have a
First sentence I think should start previous paragraph...
> + current checkpoint, which is the most recent checkpoint compared to
> + the current state of the domain (although a domain might have
> + checkpoints without a current checkpoint, if checkpoints have been
> + deleted in the meantime). Creating or reverting to a checkpoint
What is inside the parenthesis is quite confusing and perhaps "forward
thinking" considering not all checkpoint concepts are described yet.
> + sets that checkpoint as current, and the prior current checkpoint is
> + the parent of the new checkpoint. Branches in the hierarchy can
So you can create a "current" checkpoint or revert to a checkpoint
making it current - so how then does a domain ever have checkpoints
without a current checkpoint. Is this a chicken and egg problem?
> + be formed by reverting to a checkpoint with a child, then creating
> + another checkpoint.
Can one merge two or more checkpoints? Let's say I have A, B, C, D, and
E. I could care less about B, C, and D now and I want to merge them to
create F which is essentially all 3 sets of changes leaving the time
oriented order as A, F, E.
I still don't see how a domain doesn't have a current checkpoint even
though there are checkpoints.
> + </p>
Does the live domain maintain a pointer to the checkpoint? Or are
domaincheckpoint xml files only valid with their own context? How would
the checkpoint code know if a domain was removed? Following events?
> + <p>
> + The top-level <code>domaincheckpoint</code> element may contain
> + the following elements:
s/following/following optional/
> + </p>
> + <dl>
> + <dt><code>name</code></dt>
> + <dd>The name for this checkpoint. If the name is specified when
> + initially creating the checkpoint, then the checkpoint will have
> + that particular name. If the name is omitted when initially
> + creating the checkpoint, then libvirt will make up a name for
> + the checkpoint, based on the time when it was created.
Assuming epoch and that seems to be confirmed by the example.
> + </dd>
> + <dt><code>description</code></dt>
> + <dd>A human-readable description of the checkpoint. If the
> + description is omitted when initially creating the checkpoint,
> + then this field will be empty.
> + </dd>
> + <dt><code>disks</code></dt>
> + <dd>On input, this is an optional listing of specific
> + instructions for disk checkpoints; it is needed when making a
What does specific instructions mean in this context?
> + checkpoint on only a subset of the disks associated with a
> + domain (in particular, since qemu checkpoints require qcow2
> + disks, this element may be needed on input for excluding guest
> + disks that are not in qcow2 format); if omitted on input, then
Ah, well now you know why I added the note at the top... This is buried
inside this description.
> + all disks participate in the checkpoint. On output, this is
I have visions of sugar plum fairies and QE teams using hot (un)plug to
make life absolutely miserable in this regard.
Setting aside hot [un]plug, this makes it's possible to define a subset
of domain qcow2 disks to participate in the checkpoint/backup, fair
statement? If not provided, then all qcow2 disks participate.
> + fully populated to show the state of each disk in the
> + checkpoint. This element has a list of <code>disk</code>
> + sub-elements, describing anywhere from one to all of the disks
s/the/the qcow2 formatted/
> + associated with the domain.
> + <dl>
> + <dt><code>disk</code></dt>
> + <dd>This sub-element describes the checkpoint properties of
> + a specific disk. The attribute <code>name</code> is
> + mandatory, and must match either the <code><target
> + dev='name'/></code> or an unambiguous <code><source
> + file='name'/></code> of one of
> + the <a href="formatdomain.html#elementsDisks">disk
> + devices</a> specified for the domain at the time of the
I honestly think you should just pick one the target dev and be done,
but I assume there's a reason for both which I don't understand yet.
> + checkpoint. The attribute <code>checkpoint</code> is
> + optional on input; possible values are <code>no</code>
> + when the disk does not participate in this checkpoint;
> + or <code>bitmap</code> if the disk will track all changes
> + since the creation of this checkpoint via a bitmap, in
> + which case another attribute <code>bitmap</code> will be
> + the name of the tracking bitmap (defaulting to the
> + checkpoint name).
So now I'm confused again. Initially it seemed as though the <disk>
sub-elements would be the only ones participating, but now it feels like
it's possible to pick and choose to what level something participates.
If nothing is provided, can someone change the 'checkpoint' attribute to
'no' (assuming the default is 'bitmap')? This is all rather mind
boggling - why can stuff never be simple /-|
> + </dd>
> + </dl>
> + </dd>
> + <dt><code>creationTime</code></dt>
> + <dd>The time this checkpoint was created. The time is specified
> + in seconds since the Epoch, UTC (i.e. Unix time). Readonly.
> + </dd>
> + <dt><code>parent</code></dt>
> + <dd>The parent of this checkpoint. If present, this element
> + contains exactly one child element, name. This specifies the
> + name of the parent checkpoint of this one, and is used to
> + represent trees of checkpoints. Readonly.
> + </dd>
> + <dt><code>domain</code></dt>
> + <dd>The inactive <a href="formatdomain.html">domain
> + configuration</a> at the time the checkpoint was created.
> + Readonly.
The whole initial inactive/config XML is loaded here? Or just certain
fields. The example XML output has just a UUID, but the data here has a
lot more and the RNG shows everything.
Did I read something wrong earlier regarding saving domain state? I
though the checkpoint/backup code was purely for the domain disks that
are of type qcow2. I can see saving the UUID since that cannot change
(unlike the name)... But why does the entire "original" config need to
be added here?
> + </dd>
> + </dl>
> +
> + <h2><a id="BackupAttributes">Backup XML</a></h2>
> +
> + <p>
> + Creating a backup, whether full or incremental, is done
> + via <code>virDomainBackupBegin()</code>, which takes an XML
> + description of the actions to perform. There are two general
> + modes for backups: a push mode (where the hypervisor writes out
> + the data to the destination file, which may be local or remote),
> + and a pull mode (where the hypervisor creates an NBD server that
> + a third-party client can then read as needed, and which requires
> + the use of temporary storage, typically local, until the backup
> + is complete).
> + </p>
I think the modes should be described in own bullets rather than part of
a long-ish sentence using parentheses to help define what is meant.
Perhaps even more verbiage regarding general usage expectations of each
model.
> + <p>
> + The instructions for beginning a backup job are provided as
> + attributes and elements of the
> + top-level <code>domainbackup</code> element. This element
> + includes an optional attribute <code>mode</code> which can be
> + either "push" or "pull" (default push). Where elements are
s/Where/Although ???
> + optional on creation, <code>virDomainBackupGetXMLDesc()</code>
> + can be used to see the actual values selected (for example,
> + learning which port the NBD server is using in the pull model,
s/model,/model/
> + or what file names libvirt generated when none were supplied).
> + The following child elements are supported:
> + </p>
> + <dl>
Since all elements are optional, probably don't have to note it again
for each description...
> + <dt><code>incremental</code></dt>
> + <dd>Optional. If this element is present, it must name an
> + existing checkpoint of the domain, which will be used to make
s/domain,/domain/
> + this backup an incremental one (in the push model, only
> + changes since the checkpoint are written to the destination;
> + in the pull model, the NBD server uses the
> + NBD_OPT_SET_META_CONTEXT extension to advertise to the client
> + which portions of the export contain changes since the
> + checkpoint). If omitted, a full backup is performed.
The wording in the parentheses doesn't make complete sense yet.
Perhaps it's better to have each as a bullet describing the action of
the element for the backup "mode" value.
> + </dd>
> + <dt><code>server</code></dt>
> + <dd>Present only for a pull mode backup. Contains the same
> + attributes as the <code>protocol</code> element of a disk
> + attached via NBD in the domain (such as transport, socket,
> + name, port, or tls), necessary to set up an NBD server that
> + exposes the content of each disk at the time the backup
contents ?
> + started.
> + </dd>
This hunk above uses <tab>'s not spaces (syntax-check)
> + <dt><code>disks</code></dt>
> + <dd>This is an optional listing of instructions for disks
> + participating in the backup (if omitted, all disks
> + participate, and libvirt attempts to generate filenames by
> + appending the current timestamp as a suffix). When provided on
> + input, disks omitted from the list do not participate in the
> + backup. On output, the list is present but contains only the
> + disks participating in the backup job. This element has a
> + list of <code>disk</code> sub-elements, describing anywhere
> + from one to all of the disks associated with the domain.
again, qcow2 formatted disks, right? Is that true for both mode's of
backup?
> + <dl>
> + <dt><code>disk</code></dt>
> + <dd>This sub-element describes the checkpoint properties of
> + a specific disk. The attribute <code>name</code> is
> + mandatory, and must match either the <code><target
> + dev='name'/></code> or an unambiguous <code><source
> + file='name'/></code> of one of
> + the <a href="formatdomain.html#elementsDisks">disk
> + devices</a> specified for the domain at the time of the
> + checkpoint. The optional attribute <code>type</code> can
> + be <code>file</code>, <code>block</code>,
> + or <code>networks</code>, similar to a disk declaration
network
> + for a domain, controls what additional sub-elements are
s/,/ and/
> + needed to describe the destination (such
> + as <code>protocol</code> for a network destination). In
> + push mode backups, the primary subelement
sub-element
> + is <code>target</code>; in pull mode, the primary sublement
sub-element
> + is <code>scratch</code>; but either way,
> + the primary sub-element describes the file name to be used
> + during the backup operation, similar to
> + the <code>source</code> sub-element of a domain disk. An
> + optional sublement <code>driver</code> can also be used to
sub-element
> + specify a destination format different from qcow2.
> + </dd>
> + </dl>
> + </dd>
> + </dl>
Well, suffice to say, I'm lost. I guess I at least am getting an idea
why no consensus has been reached yet.
> +
> + <h2><a id="example">Examples</a></h2>
> +
> + <p>Using this XML to create a checkpoint of just vda on a qemu
> + domain with two disks and a prior checkpoint:</p>
> + <pre>
> +<domaincheckpoint>
> + <description>Completion of updates after OS install</description>
> + <disks>
> + <disk name='vda' checkpoint='bitmap'/>
> + <disk name='vdb' checkpoint='no'/>
> + </disks>
> +</domaincheckpoint></pre>
> +
> + <p>will result in XML similar to this from
> + <code>virDomainCheckpointGetXMLDesc()</code>:</p>
> + <pre>
> +<domaincheckpoint>
> + <name>1525889631</name>
> + <description>Completion of updates after OS install</description>
> + <creationTime>1525889631</creationTime>
> + <parent>
> + <name>1525111885</name>
> + </parent>
> + <disks>
> + <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
> + <disk name='vdb' checkpoint='no'/>
> + </disks>
> + <domain>
> + <name>fedora</name>
> + <uuid>93a5c045-6457-2c09-e56c-927cdf34e178</uuid>
> + <memory>1048576</memory>
> + ...
> + <devices>
> + <disk type='file' device='disk'>
> + <driver name='qemu' type='qcow2'/>
> + <source file='/path/to/file1'/>
> + <target dev='vda' bus='virtio'/>
> + </disk>
> + <disk type='file' device='disk' snapshot='external'>
> + <driver name='qemu' type='raw'/>
> + <source file='/path/to/file2'/>
> + <target dev='vdb' bus='virtio'/>
> + </disk>
> + ...
> + </devices>
> + </domain>
> +</domaincheckpoint></pre>
> +
> + <p>With that checkpoint created, the qcow2 image is now tracking
> + all changes that occur in the image since the checkpoint via
> + the persistent bitmap named <code>1525889631</code>. Now, we
> + can make a subsequent call
> + to <code>virDomainBackupBegin()</code> to perform an incremental
> + backup of just this data, using the following XML to start a
> + pull model NBD export of the vda disk:
> + </p>
> + <pre>
> +<domainbackup mode="pull">
> + <incremental>1525889631</incremental>
> + <server transport="unix" socket="/path/to/server"/>
> + <disks/>
> + <disk name='vda' type='file'/>
> + <scratch file=/path/to/file1.scratch'/>
> + </disk>
> + </disks/>
> +</domainbackup>
> + </pre>
> + </body>
> +</html>
> diff --git a/docs/schemas/domaincheckpoint.rng b/docs/schemas/domaincheckpoint.rng
> new file mode 100644
> index 0000000000..1e2c16e035
> --- /dev/null
> +++ b/docs/schemas/domaincheckpoint.rng
> @@ -0,0 +1,89 @@
> +<?xml version="1.0"?>
> +<!-- A Relax NG schema for the libvirt domain checkpoint properties XML format -->
> +<grammar xmlns="http://relaxng.org/ns/structure/1.0">
> + <start>
> + <ref name='domaincheckpoint'/>
> + </start>
> +
> + <include href='domaincommon.rng'/>
> +
> + <define name='domaincheckpoint'>
> + <element name='domaincheckpoint'>
> + <interleave>
> + <optional>
> + <element name='name'>
> + <text/>
> + </element>
> + </optional>
> + <optional>
> + <element name='description'>
> + <text/>
> + </element>
> + </optional>
> + <optional>
> + <element name='creationTime'>
> + <text/>
> + </element>
> + </optional>
> + <optional>
> + <element name='disks'>
> + <zeroOrMore>
> + <ref name='diskcheckpoint'/>
> + </zeroOrMore>
> + </element>
> + </optional>
> + <optional>
> + <choice>
> + <element name='domain'>
> + <element name='uuid'>
> + <ref name="UUID"/>
> + </element>
> + </element>
> + <!-- Nested grammar ensures that any of our overrides of
> + storagecommon/domaincommon defines do not conflict
> + with any domain.rng overrides. -->
> + <grammar>
> + <include href='domain.rng'/>
> + </grammar>
> + </choice>
> + </optional>
> + <optional>
> + <element name='parent'>
> + <element name='name'>
> + <text/>
> + </element>
> + </element>
> + </optional>
> + </interleave>
> + </element>
> + </define>
Since everything is optional, does <optional> have be supplied for each
element or can there be one <optional> before the <interleave>?
I see no <domainbackup> definition yet either - I assume it's coming.
John
> +
> + <define name='diskcheckpoint'>
> + <element name='disk'>
> + <attribute name='name'>
> + <choice>
> + <ref name='diskTarget'/>
> + <ref name='absFilePath'/>
> + </choice>
> + </attribute>
> + <choice>
> + <attribute name='checkpoint'>
> + <value>no</value>
> + </attribute>
> + <group>
> + <optional>
> + <attribute name='checkpoint'>
> + <value>bitmap</value>
> + </attribute>
> + </optional>
> + <optional>
> + <attribute name='bitmap'>
> + <text/>
> + </attribute>
> + </optional>
> + </group>
> + </choice>
> + </element>
> + </define>
> +
> +</grammar>
> diff --git a/libvirt.spec.in b/libvirt.spec.in
> index ace05820aa..50bd79a7d7 100644
> --- a/libvirt.spec.in
> +++ b/libvirt.spec.in
> @@ -2044,6 +2044,7 @@ exit 0
> %{_datadir}/libvirt/schemas/cputypes.rng
> %{_datadir}/libvirt/schemas/domain.rng
> %{_datadir}/libvirt/schemas/domaincaps.rng
> +%{_datadir}/libvirt/schemas/domaincheckpoint.rng
> %{_datadir}/libvirt/schemas/domaincommon.rng
> %{_datadir}/libvirt/schemas/domainsnapshot.rng
> %{_datadir}/libvirt/schemas/interface.rng
> diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
> index 917d2143d8..6912527cf7 100644
> --- a/mingw-libvirt.spec.in
> +++ b/mingw-libvirt.spec.in
> @@ -241,6 +241,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
> %{mingw32_datadir}/libvirt/schemas/cputypes.rng
> %{mingw32_datadir}/libvirt/schemas/domain.rng
> %{mingw32_datadir}/libvirt/schemas/domaincaps.rng
> +%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
> %{mingw32_datadir}/libvirt/schemas/domaincommon.rng
> %{mingw32_datadir}/libvirt/schemas/domainsnapshot.rng
> %{mingw32_datadir}/libvirt/schemas/interface.rng
> @@ -326,6 +327,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
> %{mingw64_datadir}/libvirt/schemas/cputypes.rng
> %{mingw64_datadir}/libvirt/schemas/domain.rng
> %{mingw64_datadir}/libvirt/schemas/domaincaps.rng
> +%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
> %{mingw64_datadir}/libvirt/schemas/domaincommon.rng
> %{mingw64_datadir}/libvirt/schemas/domainsnapshot.rng
> %{mingw64_datadir}/libvirt/schemas/interface.rng
> diff --git a/tests/domaincheckpointxml2xmlin/empty.xml b/tests/domaincheckpointxml2xmlin/empty.xml
> new file mode 100644
> index 0000000000..dc36449142
> --- /dev/null
> +++ b/tests/domaincheckpointxml2xmlin/empty.xml
> @@ -0,0 +1 @@
> +<domaincheckpoint/>
> diff --git a/tests/domaincheckpointxml2xmlout/empty.xml b/tests/domaincheckpointxml2xmlout/empty.xml
> new file mode 100644
> index 0000000000..a26c7caab0
> --- /dev/null
> +++ b/tests/domaincheckpointxml2xmlout/empty.xml
> @@ -0,0 +1,10 @@
> +<domaincheckpoint>
> + <name>1525889631</name>
> + <creationTime>1525889631</creationTime>
> + <disks>
> + <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
> + </disks>
> + <domain>
> + <uuid>9d37b878-a7cc-9f9a-b78f-49b3abad25a8</uuid>
> + </domain>
> +</domaincheckpoint>
> diff --git a/tests/virschematest.c b/tests/virschematest.c
> index 2d35833919..b866db4326 100644
> --- a/tests/virschematest.c
> +++ b/tests/virschematest.c
> @@ -223,6 +223,8 @@ mymain(void)
> "genericxml2xmloutdata", "xlconfigdata", "libxlxml2domconfigdata",
> "qemuhotplugtestdomains");
> DO_TEST_DIR("domaincaps.rng", "domaincapsschemadata");
> + DO_TEST_DIR("domaincheckpoint.rng", "domaincheckpointxml2xmlin",
> + "domaincheckpointxml2xmlout");
> DO_TEST_DIR("domainsnapshot.rng", "domainsnapshotxml2xmlin",
> "domainsnapshotxml2xmlout");
> DO_TEST_DIR("interface.rng", "interfaceschemadata");
>
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Prepare for new checkpoint and backup APIs by describing the XML
that will represent a checkpoint and backup. The checkpoint XML
is modeled heavily after virDomainSnapshotPtr, since both represent
a point in time of the guest (however, a snapshot exists with the
intent to roll back to that point, while a checkpoint exists to
facilitate later incremental backups). Meanwhile, the backup XML
has enough information to represent both push model (the hypervisor
writes the backup file to a location of the user's choice) and the
pull model (the hypervisor needs local temporary storage, and also
creates an NBD server that the user can use to read the backup via
a third-party client).. But while a snapshot exists with the
intent of rolling back to that state, a checkpoint instead makes it
possible to create an incremental backup at a later time.
Add testsuite coverage for some minimal uses of both XML.
Ultimately, I'd love for push model backups to target a network
driver rather than just a local file or block device; but doing
that got hairy (while <domain> uses <source> as the description
of a host or network resource, I picked <target> as the description
of a push model backup target [defaults to qcow2 but can also be
raw or any other format], and <scratch> as the description
of a pull model backup scratch space [must be qcow2]). The ideal
refactoring would be a way to parameterize RNG to accept
<disk type='FOO'>...</disk> so that the name of the subelement
can be <source> for domain, or <target> or <scratch> as needed for
backups. Future patches may improve this area of code.
Signed-off-by: Eric Blake <eblake@redhat.com>
---
An updated version of the XML descriptions, while I still work on
piecing together the pieces needed to demo the XML in action.
Missing changes that were pointed out by John, but also supplies
the <domainbackup> XML that was previously a black box.
docs/docs.html.in | 3 +-
docs/domainstatecapture.html.in | 4 +-
docs/format.html.in | 1 +
docs/formatcheckpoint.html.in | 273 +++++++++++++++++++++++++++
docs/index.html.in | 3 +-
docs/schemas/domainbackup.rng | 180 ++++++++++++++++++
docs/schemas/domaincheckpoint.rng | 89 +++++++++
libvirt.spec.in | 2 +
mingw-libvirt.spec.in | 4 +
tests/domainbackupxml2xmlin/backup-pull.xml | 9 +
tests/domainbackupxml2xmlin/backup-push.xml | 9 +
tests/domainbackupxml2xmlin/empty.xml | 1 +
tests/domainbackupxml2xmlout/backup-pull.xml | 9 +
tests/domainbackupxml2xmlout/backup-push.xml | 9 +
tests/domainbackupxml2xmlout/empty.xml | 7 +
tests/domaincheckpointxml2xmlin/empty.xml | 1 +
tests/domaincheckpointxml2xmlin/sample.xml | 7 +
tests/domaincheckpointxml2xmlout/empty.xml | 10 +
tests/domaincheckpointxml2xmlout/sample.xml | 16 ++
tests/virschematest.c | 4 +
20 files changed, 637 insertions(+), 4 deletions(-)
create mode 100644 docs/formatcheckpoint.html.in
create mode 100644 docs/schemas/domainbackup.rng
create mode 100644 docs/schemas/domaincheckpoint.rng
create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlin/empty.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlout/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml
create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml
diff --git a/docs/docs.html.in b/docs/docs.html.in
index 4c46b74980..4914e7dbed 100644
--- a/docs/docs.html.in
+++ b/docs/docs.html.in
@@ -79,7 +79,8 @@
<a href="formatdomaincaps.html">domain capabilities</a>,
<a href="formatnode.html">node devices</a>,
<a href="formatsecret.html">secrets</a>,
- <a href="formatsnapshot.html">snapshots</a></dd>
+ <a href="formatsnapshot.html">snapshots</a>,
+ <a href="formatcheckpoint.html">backups and checkpoints</a></dd>
<dt><a href="uri.html">URI format</a></dt>
<dd>The URI formats used for connecting to libvirt</dd>
diff --git a/docs/domainstatecapture.html.in b/docs/domainstatecapture.html.in
index 00ab7e8ee1..4de93c87c8 100644
--- a/docs/domainstatecapture.html.in
+++ b/docs/domainstatecapture.html.in
@@ -154,9 +154,9 @@
time as a new backup, so that the next incremental backup can
refer to the incremental state since the checkpoint created
during the current backup. Guest state is then actually
- captured using <code>virDomainBackupBegin()</code>. <!--See also
+ captured using <code>virDomainBackupBegin()</code>. See also
the <a href="formatcheckpoint.html">XML details</a> used with
- this command.--></dd>
+ this command.</dd>
</dl>
<h2><a id="examples">Examples</a></h2>
diff --git a/docs/format.html.in b/docs/format.html.in
index 22b23e3fc7..8c4e15e079 100644
--- a/docs/format.html.in
+++ b/docs/format.html.in
@@ -24,6 +24,7 @@
<li><a href="formatnode.html">Node devices</a></li>
<li><a href="formatsecret.html">Secrets</a></li>
<li><a href="formatsnapshot.html">Snapshots</a></li>
+ <li><a href="formatcheckpoint.html">Backups and checkpoints</a></li>
</ul>
<h2>Command line validation</h2>
diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in
new file mode 100644
index 0000000000..4d65c8e581
--- /dev/null
+++ b/docs/formatcheckpoint.html.in
@@ -0,0 +1,273 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <body>
+ <h1>Checkpoint and Backup XML format</h1>
+
+ <ul id="toc"></ul>
+
+ <h2><a id="CheckpointAttributes">Checkpoint XML</a></h2>
+
+ <p>
+ Domain disk backups, including incremental backups, are one form
+ of <a href="domainstatecapture.html">domain state capture</a>.
+ </p>
+ <p>
+ Libvirt is able to facilitate incremental backups by tracking
+ disk checkpoints, or points in time against which it is easy to
+ compute which portion of the disk has changed. Given a full
+ backup (a backup created from the creation of the disk to a
+ given point in time, coupled with the creation of a disk
+ checkpoint at that time), and an incremental backup (a backup
+ created from just the dirty portion of the disk between the
+ first checkpoint and the second backup operation), it is
+ possible to do an offline reconstruction of the state of the
+ disk at the time of the second backup, without having to copy as
+ much data as a second full backup would require. Most disk
+ checkpoints are created in concert with a backup,
+ via <code>virDomainBackupBegin()</code>; however, libvirt also
+ exposes enough support to create disk checkpoints independently
+ from a backup operation,
+ via <code>virDomainCheckpointCreateXML()</code>.
+ </p>
+ <p>
+ Attributes of libvirt checkpoints are stored as child elements of
+ the <code>domaincheckpoint</code> element. At checkpoint creation
+ time, normally only the <code>name</code>, <code>description</code>,
+ and <code>disks</code> elements are settable; the rest of the
+ fields are ignored on creation, and will be filled in by
+ libvirt in for informational purposes
+ by <code>virDomainCheckpointGetXMLDesc()</code>. However, when
+ redefining a checkpoint,
+ with the <code>VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE</code> flag
+ of <code>virDomainCheckpointCreateXML()</code>, all of the XML
+ described here is relevant.
+ </p>
+ <p>
+ Checkpoints are maintained in a hierarchy. A domain can have a
+ current checkpoint, which is the most recent checkpoint compared to
+ the current state of the domain (although a domain might have
+ checkpoints without a current checkpoint, if checkpoints have been
+ deleted in the meantime). Creating or reverting to a checkpoint
+ sets that checkpoint as current, and the prior current checkpoint is
+ the parent of the new checkpoint. Branches in the hierarchy can
+ be formed by reverting to a checkpoint with a child, then creating
+ another checkpoint.
+ </p>
+ <p>
+ The top-level <code>domaincheckpoint</code> element may contain
+ the following elements:
+ </p>
+ <dl>
+ <dt><code>name</code></dt>
+ <dd>The name for this checkpoint. If the name is specified when
+ initially creating the checkpoint, then the checkpoint will have
+ that particular name. If the name is omitted when initially
+ creating the checkpoint, then libvirt will make up a name for
+ the checkpoint, based on the time when it was created.
+ </dd>
+ <dt><code>description</code></dt>
+ <dd>A human-readable description of the checkpoint. If the
+ description is omitted when initially creating the checkpoint,
+ then this field will be empty.
+ </dd>
+ <dt><code>disks</code></dt>
+ <dd>On input, this is an optional listing of specific
+ instructions for disk checkpoints; it is needed when making a
+ checkpoint on only a subset of the disks associated with a
+ domain (in particular, since qemu checkpoints require qcow2
+ disks, this element may be needed on input for excluding guest
+ disks that are not in qcow2 format); if omitted on input, then
+ all disks participate in the checkpoint. On output, this is
+ fully populated to show the state of each disk in the
+ checkpoint. This element has a list of <code>disk</code>
+ sub-elements, describing anywhere from one to all of the disks
+ associated with the domain.
+ <dl>
+ <dt><code>disk</code></dt>
+ <dd>This sub-element describes the checkpoint properties of
+ a specific disk. The attribute <code>name</code> is
+ mandatory, and must match either the <code><target
+ dev='name'/></code> or an unambiguous <code><source
+ file='name'/></code> of one of
+ the <a href="formatdomain.html#elementsDisks">disk
+ devices</a> specified for the domain at the time of the
+ checkpoint. The attribute <code>checkpoint</code> is
+ optional on input; possible values are <code>no</code>
+ when the disk does not participate in this checkpoint;
+ or <code>bitmap</code> if the disk will track all changes
+ since the creation of this checkpoint via a bitmap, in
+ which case another attribute <code>bitmap</code> will be
+ the name of the tracking bitmap (defaulting to the
+ checkpoint name).
+ </dd>
+ </dl>
+ </dd>
+ <dt><code>creationTime</code></dt>
+ <dd>The time this checkpoint was created. The time is specified
+ in seconds since the Epoch, UTC (i.e. Unix time). Readonly.
+ </dd>
+ <dt><code>parent</code></dt>
+ <dd>The parent of this checkpoint. If present, this element
+ contains exactly one child element, name. This specifies the
+ name of the parent checkpoint of this one, and is used to
+ represent trees of checkpoints. Readonly.
+ </dd>
+ <dt><code>domain</code></dt>
+ <dd>The inactive <a href="formatdomain.html">domain
+ configuration</a> at the time the checkpoint was created.
+ Readonly.
+ </dd>
+ </dl>
+
+ <h2><a id="BackupAttributes">Backup XML</a></h2>
+
+ <p>
+ Creating a backup, whether full or incremental, is done
+ via <code>virDomainBackupBegin()</code>, which takes an XML
+ description of the actions to perform. There are two general
+ modes for backups: a push mode (where the hypervisor writes out
+ the data to the destination file, which may be local or remote),
+ and a pull mode (where the hypervisor creates an NBD server that
+ a third-party client can then read as needed, and which requires
+ the use of temporary storage, typically local, until the backup
+ is complete).
+ </p>
+ <p>
+ The instructions for beginning a backup job are provided as
+ attributes and elements of the
+ top-level <code>domainbackup</code> element. This element
+ includes an optional attribute <code>mode</code> which can be
+ either "push" or "pull" (default push). Where elements are
+ optional on creation, <code>virDomainBackupGetXMLDesc()</code>
+ can be used to see the actual values selected (for example,
+ learning which port the NBD server is using in the pull model,
+ or what file names libvirt generated when none were supplied).
+ The following child elements are supported:
+ </p>
+ <dl>
+ <dt><code>incremental</code></dt>
+ <dd>Optional. If this element is present, it must name an
+ existing checkpoint of the domain, which will be used to make
+ this backup an incremental one (in the push model, only
+ changes since the checkpoint are written to the destination;
+ in the pull model, the NBD server uses the
+ NBD_OPT_SET_META_CONTEXT extension to advertise to the client
+ which portions of the export contain changes since the
+ checkpoint). If omitted, a full backup is performed.
+ </dd>
+ <dt><code>server</code></dt>
+ <dd>Present only for a pull mode backup. Contains the same
+ attributes as the <code>protocol</code> element of a disk
+ attached via NBD in the domain (such as transport, socket,
+ name, port, or tls), necessary to set up an NBD server that
+ exposes the content of each disk at the time the backup
+ started.
+ </dd>
+ <dt><code>disks</code></dt>
+ <dd>This is an optional listing of instructions for disks
+ participating in the backup (if omitted, all disks
+ participate, and libvirt attempts to generate filenames by
+ appending the current timestamp as a suffix). When provided on
+ input, disks omitted from the list do not participate in the
+ backup. On output, the list is present but contains only the
+ disks participating in the backup job. This element has a
+ list of <code>disk</code> sub-elements, describing anywhere
+ from one to all of the disks associated with the domain.
+ <dl>
+ <dt><code>disk</code></dt>
+ <dd>This sub-element describes the checkpoint properties of
+ a specific disk. The attribute <code>name</code> is
+ mandatory, and must match either the <code><target
+ dev='name'/></code> or an unambiguous <code><source
+ file='name'/></code> of one of
+ the <a href="formatdomain.html#elementsDisks">disk
+ devices</a> specified for the domain at the time of the
+ checkpoint. The optional attribute <code>type</code> can
+ be <code>file</code>, <code>block</code>,
+ or <code>network</code>, similar to a disk declaration
+ for a domain, controls what additional sub-elements are
+ needed to describe the destination (such
+ as <code>protocol</code> for a network destination). In
+ push mode backups, the primary subelement
+ is <code>target</code>; in pull mode, the primary sublement
+ is <code>scratch</code>; but either way,
+ the primary sub-element describes the file name to be used
+ during the backup operation, similar to
+ the <code>source</code> sub-element of a domain disk. An
+ optional sublement <code>driver</code> can also be used to
+ specify a destination format different from qcow2.
+ </dd>
+ </dl>
+ </dd>
+ </dl>
+
+ <h2><a id="example">Examples</a></h2>
+
+ <p>Using this XML to create a checkpoint of just vda on a qemu
+ domain with two disks and a prior checkpoint:</p>
+ <pre>
+<domaincheckpoint>
+ <description>Completion of updates after OS install</description>
+ <disks>
+ <disk name='vda' checkpoint='bitmap'/>
+ <disk name='vdb' checkpoint='no'/>
+ </disks>
+</domaincheckpoint></pre>
+
+ <p>will result in XML similar to this from
+ <code>virDomainCheckpointGetXMLDesc()</code>:</p>
+ <pre>
+<domaincheckpoint>
+ <name>1525889631</name>
+ <description>Completion of updates after OS install</description>
+ <creationTime>1525889631</creationTime>
+ <parent>
+ <name>1525111885</name>
+ </parent>
+ <disks>
+ <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
+ <disk name='vdb' checkpoint='no'/>
+ </disks>
+ <domain type='qemu'>
+ <name>fedora</name>
+ <uuid>93a5c045-6457-2c09-e56c-927cdf34e178</uuid>
+ <memory>1048576</memory>
+ ...
+ <devices>
+ <disk type='file' device='disk'>
+ <driver name='qemu' type='qcow2'/>
+ <source file='/path/to/file1'/>
+ <target dev='vda' bus='virtio'/>
+ </disk>
+ <disk type='file' device='disk' snapshot='external'>
+ <driver name='qemu' type='raw'/>
+ <source file='/path/to/file2'/>
+ <target dev='vdb' bus='virtio'/>
+ </disk>
+ ...
+ </devices>
+ </domain>
+</domaincheckpoint></pre>
+
+ <p>With that checkpoint created, the qcow2 image is now tracking
+ all changes that occur in the image since the checkpoint via
+ the persistent bitmap named <code>1525889631</code>. Now, we
+ can make a subsequent call
+ to <code>virDomainBackupBegin()</code> to perform an incremental
+ backup of just this data, using the following XML to start a
+ pull model NBD export of the vda disk:
+ </p>
+ <pre>
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport="unix" socket="/path/to/server"/>
+ <disks/>
+ <disk name='vda' type='file'>
+ <scratch file='/path/to/file1.scratch'/>
+ </disk>
+ </disks/>
+</domainbackup>
+ </pre>
+ </body>
+</html>
diff --git a/docs/index.html.in b/docs/index.html.in
index 1f9f448399..6c5d3a6dc3 100644
--- a/docs/index.html.in
+++ b/docs/index.html.in
@@ -68,7 +68,8 @@
<a href="formatdomaincaps.html">domain capabilities</a>,
<a href="formatnode.html">node devices</a>,
<a href="formatsecret.html">secrets</a>,
- <a href="formatsnapshot.html">snapshots</a></dd>
+ <a href="formatsnapshot.html">snapshots</a>,
+ <a href="formatcheckpoint.html">backups and checkpoints</a></dd>
<dt><a href="http://wiki.libvirt.org">Wiki</a></dt>
<dd>Read further community contributed content</dd>
</dl>
diff --git a/docs/schemas/domainbackup.rng b/docs/schemas/domainbackup.rng
new file mode 100644
index 0000000000..8e6d4b15a2
--- /dev/null
+++ b/docs/schemas/domainbackup.rng
@@ -0,0 +1,180 @@
+<?xml version="1.0"?>
+<!-- A Relax NG schema for the libvirt domain backup properties XML format -->
+<grammar xmlns="http://relaxng.org/ns/structure/1.0">
+ <start>
+ <ref name='domainbackup'/>
+ </start>
+
+ <include href='domaincommon.rng'/>
+
+ <define name='domainbackup'>
+ <element name='domainbackup'>
+ <interleave>
+ <optional>
+ <element name='incremental'>
+ <text/>
+ </element>
+ </optional>
+ <choice>
+ <group>
+ <optional>
+ <attribute name='mode'>
+ <value>push</value>
+ </attribute>
+ </optional>
+ <ref name='backupDisksPush'/>
+ </group>
+ <group>
+ <attribute name='mode'>
+ <value>pull</value>
+ </attribute>
+ <interleave>
+ <element name='server'>
+ <choice>
+ <group>
+ <optional>
+ <attribute name='transport'>
+ <value>tcp</value>
+ </attribute>
+ </optional>
+ <attribute name='name'>
+ <choice>
+ <ref name='dnsName'/>
+ <ref name='ipAddr'/>
+ </choice>
+ </attribute>
+ <optional>
+ <attribute name='port'>
+ <ref name='unsignedInt'/>
+ </attribute>
+ </optional>
+ <!-- add tls? -->
+ </group>
+ <group>
+ <attribute name='transport'>
+ <value>unix</value>
+ </attribute>
+ <attribute name='socket'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </group>
+ </choice>
+ </element>
+ <ref name='backupDisksPull'/>
+ </interleave>
+ </group>
+ </choice>
+ </interleave>
+ </element>
+ </define>
+
+ <define name='backupPushDriver'>
+ <optional>
+ <element name='driver'>
+ <attribute name='type'>
+ <ref name='storageFormat'/>
+ </attribute>
+ </element>
+ </optional>
+ </define>
+
+ <define name='backupDisksPush'>
+ <optional>
+ <element name='disks'>
+ <oneOrMore>
+ <element name='disk'>
+ <attribute name='name'>
+ <choice>
+ <ref name='diskTarget'/>
+ <ref name='absFilePath'/>
+ </choice>
+ </attribute>
+ <choice>
+ <!-- FIXME allow push to a network location, by
+ refactoring 'diskSource' to take element name by a
+ per-grammar ref -->
+ <group>
+ <optional>
+ <attribute name='type'>
+ <value>file</value>
+ </attribute>
+ </optional>
+ <interleave>
+ <optional>
+ <element name='target'>
+ <attribute name='file'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ <ref name='backupPushDriver'/>
+ </interleave>
+ </group>
+ <group>
+ <attribute name='type'>
+ <value>disk</value>
+ </attribute>
+ <interleave>
+ <optional>
+ <element name='target'>
+ <attribute name='dev'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ <ref name='backupPushDriver'/>
+ </interleave>
+ </group>
+ </choice>
+ </element>
+ </oneOrMore>
+ </element>
+ </optional>
+ </define>
+
+ <define name='backupDisksPull'>
+ <optional>
+ <element name='disks'>
+ <oneOrMore>
+ <element name='disk'>
+ <attribute name='name'>
+ <choice>
+ <ref name='diskTarget'/>
+ <ref name='absFilePath'/>
+ </choice>
+ </attribute>
+ <choice>
+ <group>
+ <optional>
+ <attribute name='type'>
+ <value>file</value>
+ </attribute>
+ </optional>
+ <optional>
+ <element name='scratch'>
+ <attribute name='file'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ </group>
+ <group>
+ <attribute name='type'>
+ <value>disk</value>
+ </attribute>
+ <optional>
+ <element name='scratch'>
+ <attribute name='dev'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ </group>
+ </choice>
+ </element>
+ </oneOrMore>
+ </element>
+ </optional>
+ </define>
+
+</grammar>
diff --git a/docs/schemas/domaincheckpoint.rng b/docs/schemas/domaincheckpoint.rng
new file mode 100644
index 0000000000..1e2c16e035
--- /dev/null
+++ b/docs/schemas/domaincheckpoint.rng
@@ -0,0 +1,89 @@
+<?xml version="1.0"?>
+<!-- A Relax NG schema for the libvirt domain checkpoint properties XML format -->
+<grammar xmlns="http://relaxng.org/ns/structure/1.0">
+ <start>
+ <ref name='domaincheckpoint'/>
+ </start>
+
+ <include href='domaincommon.rng'/>
+
+ <define name='domaincheckpoint'>
+ <element name='domaincheckpoint'>
+ <interleave>
+ <optional>
+ <element name='name'>
+ <text/>
+ </element>
+ </optional>
+ <optional>
+ <element name='description'>
+ <text/>
+ </element>
+ </optional>
+ <optional>
+ <element name='creationTime'>
+ <text/>
+ </element>
+ </optional>
+ <optional>
+ <element name='disks'>
+ <zeroOrMore>
+ <ref name='diskcheckpoint'/>
+ </zeroOrMore>
+ </element>
+ </optional>
+ <optional>
+ <choice>
+ <element name='domain'>
+ <element name='uuid'>
+ <ref name="UUID"/>
+ </element>
+ </element>
+ <!-- Nested grammar ensures that any of our overrides of
+ storagecommon/domaincommon defines do not conflict
+ with any domain.rng overrides. -->
+ <grammar>
+ <include href='domain.rng'/>
+ </grammar>
+ </choice>
+ </optional>
+ <optional>
+ <element name='parent'>
+ <element name='name'>
+ <text/>
+ </element>
+ </element>
+ </optional>
+ </interleave>
+ </element>
+ </define>
+
+ <define name='diskcheckpoint'>
+ <element name='disk'>
+ <attribute name='name'>
+ <choice>
+ <ref name='diskTarget'/>
+ <ref name='absFilePath'/>
+ </choice>
+ </attribute>
+ <choice>
+ <attribute name='checkpoint'>
+ <value>no</value>
+ </attribute>
+ <group>
+ <optional>
+ <attribute name='checkpoint'>
+ <value>bitmap</value>
+ </attribute>
+ </optional>
+ <optional>
+ <attribute name='bitmap'>
+ <text/>
+ </attribute>
+ </optional>
+ </group>
+ </choice>
+ </element>
+ </define>
+
+</grammar>
diff --git a/libvirt.spec.in b/libvirt.spec.in
index ace05820aa..e3b3ba19e0 100644
--- a/libvirt.spec.in
+++ b/libvirt.spec.in
@@ -2043,7 +2043,9 @@ exit 0
%{_datadir}/libvirt/schemas/capability.rng
%{_datadir}/libvirt/schemas/cputypes.rng
%{_datadir}/libvirt/schemas/domain.rng
+%{_datadir}/libvirt/schemas/domainbackup.rng
%{_datadir}/libvirt/schemas/domaincaps.rng
+%{_datadir}/libvirt/schemas/domaincheckpoint.rng
%{_datadir}/libvirt/schemas/domaincommon.rng
%{_datadir}/libvirt/schemas/domainsnapshot.rng
%{_datadir}/libvirt/schemas/interface.rng
diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
index 917d2143d8..39d02094f9 100644
--- a/mingw-libvirt.spec.in
+++ b/mingw-libvirt.spec.in
@@ -240,7 +240,9 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
%{mingw32_datadir}/libvirt/schemas/capability.rng
%{mingw32_datadir}/libvirt/schemas/cputypes.rng
%{mingw32_datadir}/libvirt/schemas/domain.rng
+%{mingw32_datadir}/libvirt/schemas/domainbackup.rng
%{mingw32_datadir}/libvirt/schemas/domaincaps.rng
+%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
%{mingw32_datadir}/libvirt/schemas/domaincommon.rng
%{mingw32_datadir}/libvirt/schemas/domainsnapshot.rng
%{mingw32_datadir}/libvirt/schemas/interface.rng
@@ -325,7 +327,9 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
%{mingw64_datadir}/libvirt/schemas/capability.rng
%{mingw64_datadir}/libvirt/schemas/cputypes.rng
%{mingw64_datadir}/libvirt/schemas/domain.rng
+%{mingw64_datadir}/libvirt/schemas/domainbackup.rng
%{mingw64_datadir}/libvirt/schemas/domaincaps.rng
+%{mingw64_datadir}/libvirt/schemas/domaincheckpoint.rng
%{mingw64_datadir}/libvirt/schemas/domaincommon.rng
%{mingw64_datadir}/libvirt/schemas/domainsnapshot.rng
%{mingw64_datadir}/libvirt/schemas/interface.rng
diff --git a/tests/domainbackupxml2xmlin/backup-pull.xml b/tests/domainbackupxml2xmlin/backup-pull.xml
new file mode 100644
index 0000000000..2ce5cd6711
--- /dev/null
+++ b/tests/domainbackupxml2xmlin/backup-pull.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport='tcp' name='localhost' port='10809'/>
+ <disks>
+ <disk name='vda' type='file'>
+ <scratch file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlin/backup-push.xml b/tests/domainbackupxml2xmlin/backup-push.xml
new file mode 100644
index 0000000000..1b7d3061fd
--- /dev/null
+++ b/tests/domainbackupxml2xmlin/backup-push.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="push">
+ <incremental>1525889631</incremental>
+ <disks>
+ <disk name='vda' type='file'>
+ <driver type='raw'/>
+ <target file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlin/empty.xml b/tests/domainbackupxml2xmlin/empty.xml
new file mode 100644
index 0000000000..7ed511f97b
--- /dev/null
+++ b/tests/domainbackupxml2xmlin/empty.xml
@@ -0,0 +1 @@
+<domainbackup/>
diff --git a/tests/domainbackupxml2xmlout/backup-pull.xml b/tests/domainbackupxml2xmlout/backup-pull.xml
new file mode 100644
index 0000000000..2ce5cd6711
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-pull.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport='tcp' name='localhost' port='10809'/>
+ <disks>
+ <disk name='vda' type='file'>
+ <scratch file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlout/backup-push.xml b/tests/domainbackupxml2xmlout/backup-push.xml
new file mode 100644
index 0000000000..1b7d3061fd
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-push.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="push">
+ <incremental>1525889631</incremental>
+ <disks>
+ <disk name='vda' type='file'>
+ <driver type='raw'/>
+ <target file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlout/empty.xml b/tests/domainbackupxml2xmlout/empty.xml
new file mode 100644
index 0000000000..13600fbb1c
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/empty.xml
@@ -0,0 +1,7 @@
+<domainbackup mode="push">
+ <disks>
+ <disk name="vda" type="file">
+ <target file="/path/to/file1.copy"/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domaincheckpointxml2xmlin/empty.xml b/tests/domaincheckpointxml2xmlin/empty.xml
new file mode 100644
index 0000000000..dc36449142
--- /dev/null
+++ b/tests/domaincheckpointxml2xmlin/empty.xml
@@ -0,0 +1 @@
+<domaincheckpoint/>
diff --git a/tests/domaincheckpointxml2xmlin/sample.xml b/tests/domaincheckpointxml2xmlin/sample.xml
new file mode 100644
index 0000000000..70ed964e1e
--- /dev/null
+++ b/tests/domaincheckpointxml2xmlin/sample.xml
@@ -0,0 +1,7 @@
+<domaincheckpoint>
+ <description>Completion of updates after OS install</description>
+ <disks>
+ <disk name='vda' checkpoint='bitmap'/>
+ <disk name='vdb' checkpoint='no'/>
+ </disks>
+</domaincheckpoint>
diff --git a/tests/domaincheckpointxml2xmlout/empty.xml b/tests/domaincheckpointxml2xmlout/empty.xml
new file mode 100644
index 0000000000..a26c7caab0
--- /dev/null
+++ b/tests/domaincheckpointxml2xmlout/empty.xml
@@ -0,0 +1,10 @@
+<domaincheckpoint>
+ <name>1525889631</name>
+ <creationTime>1525889631</creationTime>
+ <disks>
+ <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
+ </disks>
+ <domain>
+ <uuid>9d37b878-a7cc-9f9a-b78f-49b3abad25a8</uuid>
+ </domain>
+</domaincheckpoint>
diff --git a/tests/domaincheckpointxml2xmlout/sample.xml b/tests/domaincheckpointxml2xmlout/sample.xml
new file mode 100644
index 0000000000..559b29c8c1
--- /dev/null
+++ b/tests/domaincheckpointxml2xmlout/sample.xml
@@ -0,0 +1,16 @@
+<domaincheckpoint>
+ <name>1525889631</name>
+ <description>Completion of updates after OS install</description>
+ <creationTime>1525889631</creationTime>
+ <parent>
+ <name>1525111885</name>
+ </parent>
+ <disks>
+ <disk name='vda' checkpoint='bitmap' bitmap='1525889631'/>
+ <disk name='vdb' checkpoint='no'/>
+ </disks>
+ <domain type='qemu'>
+ <name>fedora</name>
+ <uuid>93a5c045-6457-2c09-e56c-927cdf34e178</uuid>
+ </domain>
+</domaincheckpoint>
diff --git a/tests/virschematest.c b/tests/virschematest.c
index aa65a434ff..545301204a 100644
--- a/tests/virschematest.c
+++ b/tests/virschematest.c
@@ -222,7 +222,11 @@ mymain(void)
"lxcxml2xmloutdata", "bhyvexml2argvdata", "genericxml2xmlindata",
"genericxml2xmloutdata", "xlconfigdata", "libxlxml2domconfigdata",
"qemuhotplugtestdomains");
+ DO_TEST_DIR("domainbackup.rng", "domainbackupxml2xmlin",
+ "domainbackupxml2xmlout");
DO_TEST_DIR("domaincaps.rng", "domaincapsschemadata");
+ DO_TEST_DIR("domaincheckpoint.rng", "domaincheckpointxml2xmlin",
+ "domaincheckpointxml2xmlout");
DO_TEST_DIR("domainsnapshot.rng", "domainsnapshotxml2xmlin",
"domainsnapshotxml2xmlout");
DO_TEST_DIR("interface.rng", "interfaceschemadata");
--
2.14.4
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
© 2016 - 2025 Red Hat, Inc.