Libvirt provides QMP passthrough APIs for the QEMU driver and these are
exposed in virsh. It is not especially pleasant, however, using the raw
QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
exposes a human friendly interactive shell. It is not possible to use
this with libvirt managed guest, however, since only one client can
attach to he QMP socket at any point in time.

The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
socket and listens for incoming client connections, speaking QMP on
the connected socket. It will forward any QMP commands received onto
the running libvirt QEMU guest, and forward any replies back to the
QMP client.

  $ virsh start demo
  $ virt-qmp-proxy demo demo.qmp &
  $ qmp-shell demo.qmp
  Welcome to the QMP low-level shell!
  Connected to QEMU 6.2.0

  (QEMU) query-kvm
  {
      "return": {
          "enabled": true,
          "present": true
      }
  }

Note this tool of course has the same risks as the raw libvirt
QMP passthrough. It is safe to run query commands to fetch information
but commands which change the QEMU state risk disrupting libvirt's
management of QEMU, potentially resulting in data loss/corruption in
the worst case.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---

CC'ing QEMU since this is likely of interest to maintainers and users
who work with QEMU and libvirt

Note this impl is fairly crude in that it assumes it is receiving
the QMP commands linewise one at a time. None the less it is good
enough to work with qmp-shell already, so I figured it was worth
exposing to the world. It also lacks support for forwarding events
back to the QMP client.

 docs/manpages/meson.build        |   1 +
 docs/manpages/virt-qmp-proxy.rst | 123 ++++++++++++++++++++++++++++
 tools/meson.build                |   5 ++
 tools/virt-qmp-proxy             | 133 +++++++++++++++++++++++++++++++
 4 files changed, 262 insertions(+)
 create mode 100644 docs/manpages/virt-qmp-proxy.rst
 create mode 100755 tools/virt-qmp-proxy

diff --git a/docs/manpages/meson.build b/docs/manpages/meson.build
index ba673cf472..4162a9969a 100644
--- a/docs/manpages/meson.build
+++ b/docs/manpages/meson.build
@@ -18,6 +18,7 @@ docs_man_files = [
   { 'name': 'virt-pki-query-dn', 'section': '1', 'install': true },
   { 'name': 'virt-pki-validate', 'section': '1', 'install': true },
   { 'name': 'virt-qemu-run', 'section': '1', 'install': conf.has('WITH_QEMU') },
+  { 'name': 'virt-qmp-proxy', 'section': '1', 'install': conf.has('WITH_QEMU') },
   { 'name': 'virt-xml-validate', 'section': '1', 'install': true },
 
   { 'name': 'libvirt-guests', 'section': '8', 'install': conf.has('WITH_LIBVIRTD') },
diff --git a/docs/manpages/virt-qmp-proxy.rst b/docs/manpages/virt-qmp-proxy.rst
new file mode 100644
index 0000000000..94679406ab
--- /dev/null
+++ b/docs/manpages/virt-qmp-proxy.rst
@@ -0,0 +1,123 @@
+==============
+virt-qmp-proxy
+==============
+
+--------------------------------------------------
+Expose a QMP proxy server for a libvirt QEMU guest
+--------------------------------------------------
+
+:Manual section: 1
+:Manual group: Virtualization Support
+
+.. contents::
+
+
+SYNOPSIS
+========
+
+``virt-qmp-proxy`` [*OPTION*]... *DOMAIN* *QMP-SOCKET-PATH*
+
+
+DESCRIPTION
+===========
+
+This tool provides a way to expose a QMP proxy server that communicates
+with a QEMU guest managed by libvirt. This enables standard QMP client
+tools to interact with libvirt managed guests.
+
+**NOTE: use of this tool will result in the running QEMU guest being
+marked as tainted.** It is strongly recommended that this tool *only be
+used to send commands which query information* about the running guest.
+If this tool is used to make changes to the state of the guest, this
+may have negative interactions with the QEMU driver, resulting in an
+inability to manage the guest operation thereafter, and in the worst
+case **potentially lead to data loss or corruption**.
+
+The ``virt-qmp-proxy`` program will listen on a UNIX socket for incoming
+client connections, and run the QMP protocol over the connection. Any
+commands received will be sent to the running libvirt guest, and replies
+sent back.
+
+The ``virt-qemu-proxy`` program may be interrupted (eg Ctrl-C) when it
+is no longer required. The libvirt QEMU guest will continue running.
+
+
+OPTIONS
+=======
+
+*DOMAIN*
+
+The ID or UUID or Name of the libvirt QEMU guest.
+
+*QMP-SOCKET-PATH*
+
+The filesystem path at which to run the QMP server, listening for
+incoming connections.
+
+``-c`` *CONNECTION-URI*
+``--connect``\ =\ *CONNECTION-URI*
+
+The URI for the connection to the libvirt QEMU driver. If omitted,
+a URI will be auto-detected.
+
+``-v``, ``--verbose``
+
+Run in verbose mode, printing all QMP commands and replies that
+are handled.
+
+``-h``, ``--help``
+
+Display the command line help.
+
+
+EXIT STATUS
+===========
+
+Upon successful shutdown, an exit status of 0 will be set. Upon
+failure a non-zero status will be set.
+
+
+AUTHOR
+======
+
+Daniel P. Berrangé
+
+
+BUGS
+====
+
+Please report all bugs you discover.  This should be done via either:
+
+#. the mailing list
+
+   `https://libvirt.org/contact.html <https://libvirt.org/contact.html>`_
+
+#. the bug tracker
+
+   `https://libvirt.org/bugs.html <https://libvirt.org/bugs.html>`_
+
+Alternatively, you may report bugs to your software distributor / vendor.
+
+NOTE: at this time there is no support for forwarding QMP events back
+to the clients
+
+COPYRIGHT
+=========
+
+Copyright (C) 2022 by Red Hat, Inc.
+
+
+LICENSE
+=======
+
+``virt-qemu-proxy`` is distributed under the terms of the GNU LGPL v2+.
+This is free software; see the source for copying conditions. There
+is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
+PURPOSE
+
+
+SEE ALSO
+========
+
+virsh(1), `https://libvirt.org/ <https://libvirt.org/>`_,
+`QMP reference <https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html>`_
diff --git a/tools/meson.build b/tools/meson.build
index bb28a904dc..4e959ecf0b 100644
--- a/tools/meson.build
+++ b/tools/meson.build
@@ -320,6 +320,11 @@ if conf.has('WITH_LIBVIRTD')
   endif
 endif
 
+if conf.has('WITH_QEMU')
+    install_data('virt-qmp-proxy',
+                 install_dir: bindir)
+endif
+
 if bash_completion_dep.found()
   subdir('bash-completion')
 endif
diff --git a/tools/virt-qmp-proxy b/tools/virt-qmp-proxy
new file mode 100755
index 0000000000..57f9759fab
--- /dev/null
+++ b/tools/virt-qmp-proxy
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+
+import argparse
+import libvirt
+import libvirt_qemu
+import os
+import re
+import socket
+import sys
+import json
+
+
+def get_domain(uri, domstr):
+    conn = libvirt.open(uri)
+
+    dom = None
+    if re.match(r'^\d+$', domstr):
+        dom = conn.lookupByID(int(domstr))
+    elif re.match(r'^[+a-f0-9]+$', domstr):
+        dom = conn.lookupByUUIDString(domstr)
+    else:
+        dom = conn.lookupByName(domstr)
+
+    if not dom.isActive():
+        raise Exception(
+            "Domain must be running to validate measurement")
+
+    return conn, dom
+
+
+def qmp_server(conn, dom, client, verbose):
+    ver = conn.getVersion()
+    major = int(ver / 1000000) % 1000
+    minor = int(ver / 1000) % 1000
+    micro = ver % 1000
+
+    greetingobj = {
+        "QMP": {
+            "version": {
+                "qemu": {
+                    "major": major,
+                    "minor": minor,
+                    "micro": micro,
+                },
+                "package": f"qemu-{major}.{minor}.{micro}",
+            },
+            "capabilities": [
+                "oob"
+            ],
+        }
+    }
+    greeting = json.dumps(greetingobj) + "\r\n"
+    if verbose:
+        print(greeting, end='')
+    client.send(greeting.encode("utf-8"))
+
+    while True:
+        # XXX shouldn't blindly assume this one read
+        # will fully capture one-and-only-one cmd
+        cmd = client.recv(1024).decode('utf8')
+        if verbose:
+            print(cmd)
+
+        if cmd == "":
+            break
+
+        if "qmp_capabilities" in cmd:
+            capabilitiesobj = {
+                "return": {},
+            }
+            capabilities = json.dumps(capabilitiesobj) + "\r\n"
+            if verbose:
+                print(capabilities, end='')
+            client.send(capabilities.encode("utf-8"))
+            continue
+
+        id = None
+        if "id" in cmd:
+            id = cmd[id]
+
+        res = libvirt_qemu.qemuMonitorCommand(dom, cmd, 0)
+
+        resobj = json.loads(res)
+        del resobj["id"]
+        if id is not None:
+            resobj["id"] = id
+        res = json.dumps(resobj) + "\r\n"
+        if verbose:
+            print(res, end='')
+
+        client.send(res.encode('utf8'))
+
+
+def parse_commandline():
+    parser = argparse.ArgumentParser(description="Libvirt QMP proxy")
+    parser.add_argument("--connect", "-c",
+                        help="Libvirt QEMU driver connection URI")
+    parser.add_argument("--verbose", "-v", action='store_true',
+                        help="Display QMP traffic")
+    parser.add_argument("domain", metavar="DOMAIN",
+                        help="Libvirt guest domain ID/UUID/Name")
+    parser.add_argument("sockpath", metavar="QMP-SOCK-PATH",
+                        help="UNIX socket path for QMP server")
+
+    return parser.parse_args()
+
+
+def main():
+    args = parse_commandline()
+
+    conn, dom = get_domain(args.connect, args.domain)
+
+    if conn.getType() != "QEMU":
+        raise Exception("QMP proxy requires a QEMU driver connection not %s" %
+                        conn.getType())
+
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    if os.path.exists(args.sockpath):
+        os.unlink(args.sockpath)
+    sock.bind(args.sockpath)
+    sock.listen(1)
+
+    while True:
+        client, peeraddr = sock.accept()
+        qmp_server(conn, dom, client, args.verbose)
+
+
+try:
+    main()
+    sys.exit(0)
+except Exception as e:
+    print("%s: %s" % (sys.argv[0], str(e)))
+    sys.exit(1)
-- 
2.36.1

On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> exposed in virsh. It is not especially pleasant, however, using the raw
> QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> exposes a human friendly interactive shell. It is not possible to use
> this with libvirt managed guest, however, since only one client can
> attach to he QMP socket at any point in time.
> 
> The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> socket and listens for incoming client connections, speaking QMP on
> the connected socket. It will forward any QMP commands received onto
> the running libvirt QEMU guest, and forward any replies back to the
> QMP client.
> 
>   $ virsh start demo
>   $ virt-qmp-proxy demo demo.qmp &
>   $ qmp-shell demo.qmp
>   Welcome to the QMP low-level shell!
>   Connected to QEMU 6.2.0
> 
>   (QEMU) query-kvm
>   {
>       "return": {
>           "enabled": true,
>           "present": true
>       }
>   }
> 
> Note this tool of course has the same risks as the raw libvirt
> QMP passthrough. It is safe to run query commands to fetch information
> but commands which change the QEMU state risk disrupting libvirt's
> management of QEMU, potentially resulting in data loss/corruption in
> the worst case.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> CC'ing QEMU since this is likely of interest to maintainers and users
> who work with QEMU and libvirt
> 
> Note this impl is fairly crude in that it assumes it is receiving
> the QMP commands linewise one at a time. None the less it is good
> enough to work with qmp-shell already, so I figured it was worth
> exposing to the world. It also lacks support for forwarding events
> back to the QMP client.
> 
>  docs/manpages/meson.build        |   1 +
>  docs/manpages/virt-qmp-proxy.rst | 123 ++++++++++++++++++++++++++++
>  tools/meson.build                |   5 ++
>  tools/virt-qmp-proxy             | 133 +++++++++++++++++++++++++++++++
>  4 files changed, 262 insertions(+)
>  create mode 100644 docs/manpages/virt-qmp-proxy.rst
>  create mode 100755 tools/virt-qmp-proxy

[...]

> diff --git a/docs/manpages/virt-qmp-proxy.rst b/docs/manpages/virt-qmp-proxy.rst
> new file mode 100644
> index 0000000000..94679406ab
> --- /dev/null
> +++ b/docs/manpages/virt-qmp-proxy.rst
> @@ -0,0 +1,123 @@
> +==============
> +virt-qmp-proxy
> +==============
> +
> +--------------------------------------------------
> +Expose a QMP proxy server for a libvirt QEMU guest
> +--------------------------------------------------
> +
> +:Manual section: 1
> +:Manual group: Virtualization Support
> +
> +.. contents::
> +
> +
> +SYNOPSIS
> +========
> +
> +``virt-qmp-proxy`` [*OPTION*]... *DOMAIN* *QMP-SOCKET-PATH*
> +
> +
> +DESCRIPTION
> +===========
> +
> +This tool provides a way to expose a QMP proxy server that communicates
> +with a QEMU guest managed by libvirt. This enables standard QMP client
> +tools to interact with libvirt managed guests.
> +
> +**NOTE: use of this tool will result in the running QEMU guest being
> +marked as tainted.** It is strongly recommended that this tool *only be
> +used to send commands which query information* about the running guest.
> +If this tool is used to make changes to the state of the guest, this
> +may have negative interactions with the QEMU driver, resulting in an
> +inability to manage the guest operation thereafter, and in the worst
> +case **potentially lead to data loss or corruption**.
> +
> +The ``virt-qmp-proxy`` program will listen on a UNIX socket for incoming
> +client connections, and run the QMP protocol over the connection. Any
> +commands received will be sent to the running libvirt guest, and replies
> +sent back.
> +
> +The ``virt-qemu-proxy`` program may be interrupted (eg Ctrl-C) when it
> +is no longer required. The libvirt QEMU guest will continue running.
> +
> +
> +OPTIONS
> +=======
> +
> +*DOMAIN*
> +
> +The ID or UUID or Name of the libvirt QEMU guest.
> +
> +*QMP-SOCKET-PATH*
> +
> +The filesystem path at which to run the QMP server, listening for
> +incoming connections.
> +
> +``-c`` *CONNECTION-URI*
> +``--connect``\ =\ *CONNECTION-URI*
> +
> +The URI for the connection to the libvirt QEMU driver. If omitted,
> +a URI will be auto-detected.
> +
> +``-v``, ``--verbose``
> +
> +Run in verbose mode, printing all QMP commands and replies that
> +are handled.
> +
> +``-h``, ``--help``
> +
> +Display the command line help.
> +
> +
> +EXIT STATUS
> +===========
> +
> +Upon successful shutdown, an exit status of 0 will be set. Upon
> +failure a non-zero status will be set.
> +
> +
> +AUTHOR
> +======
> +
> +Daniel P. Berrangé
> +
> +
> +BUGS
> +====
> +
> +Please report all bugs you discover.  This should be done via either:
> +
> +#. the mailing list
> +
> +   `https://libvirt.org/contact.html <https://libvirt.org/contact.html>`_
> +
> +#. the bug tracker
> +
> +   `https://libvirt.org/bugs.html <https://libvirt.org/bugs.html>`_
> +
> +Alternatively, you may report bugs to your software distributor / vendor.
> +
> +NOTE: at this time there is no support for forwarding QMP events back
> +to the clients

Also add caveat about FD passing support.

[...]

> diff --git a/tools/virt-qmp-proxy b/tools/virt-qmp-proxy
> new file mode 100755
> index 0000000000..57f9759fab
> --- /dev/null
> +++ b/tools/virt-qmp-proxy
> @@ -0,0 +1,133 @@
> +#!/usr/bin/env python3
> +
> +import argparse
> +import libvirt
> +import libvirt_qemu
> +import os
> +import re
> +import socket
> +import sys
> +import json
> +
> +
> +def get_domain(uri, domstr):
> +    conn = libvirt.open(uri)
> +
> +    dom = None
> +    if re.match(r'^\d+$', domstr):
> +        dom = conn.lookupByID(int(domstr))
> +    elif re.match(r'^[+a-f0-9]+$', domstr):

This works very poorly if you have a VM named for example 'cd' or any
combination of just letters abcdef.

> +        dom = conn.lookupByUUIDString(domstr)
> +    else:
> +        dom = conn.lookupByName(domstr)
> +
> +    if not dom.isActive():
> +        raise Exception(
> +            "Domain must be running to validate measurement")

This should mention the current usage or a generic error ;)

> +
> +    return conn, dom
> +
> +
> +def qmp_server(conn, dom, client, verbose):
> +    ver = conn.getVersion()

So this gets the version of the "default" emulator version, but if your
VM is using a custom one this will report it wrong.

E.g in my case I have a git qemu for a VM:

 517 2022-05-27 14:01:09.604+0000: 373562: debug : qemuMonitorJSONIOProcessLine:199 : Line [{"QMP": {"version": {"qemu": {"micro": 50, "minor": 0, "major": 7}, "package": "v7.0.0-1253-g2417cbd591"}, "capabilities": ["oob"]}}]



> +    major = int(ver / 1000000) % 1000
> +    minor = int(ver / 1000) % 1000
> +    micro = ver % 1000
> +
> +    greetingobj = {
> +        "QMP": {
> +            "version": {
> +                "qemu": {
> +                    "major": major,
> +                    "minor": minor,
> +                    "micro": micro,
> +                },
> +                "package": f"qemu-{major}.{minor}.{micro}",
> +            },
> +            "capabilities": [
> +                "oob"
> +            ],
> +        }
> +    }

But when I conect I get:

{"QMP": {"version": {"qemu": {"major": 7, "minor": 0, "micro": 0}, "package": "qemu-7.0.0"}, "capabilities": ["oob"]}}


At the very least this should be documented.

> +    greeting = json.dumps(greetingobj) + "\r\n"
> +    if verbose:
> +        print(greeting, end='')
> +    client.send(greeting.encode("utf-8"))
> +
> +    while True:
> +        # XXX shouldn't blindly assume this one read
> +        # will fully capture one-and-only-one cmd
> +        cmd = client.recv(1024).decode('utf8')

IIUC this limits the buffer to 1k max. Libvirt's RPC supports up to 4M.
1k could be limiting with some commands such as blockdev-add.

> +        if verbose:
> +            print(cmd)
> +
> +        if cmd == "":
> +            break
> +
> +        if "qmp_capabilities" in cmd:
> +            capabilitiesobj = {
> +                "return": {},
> +            }
> +            capabilities = json.dumps(capabilitiesobj) + "\r\n"
> +            if verbose:
> +                print(capabilities, end='')
> +            client.send(capabilities.encode("utf-8"))
> +            continue
> +
> +        id = None
> +        if "id" in cmd:
> +            id = cmd[id]
> +
> +        res = libvirt_qemu.qemuMonitorCommand(dom, cmd, 0)

If 'cmd' is not JSON this breaks horribly:

$ tools/virt-qmp-proxy 2 /tmp/asdf
libvirt:  error : internal error: cannot parse json test
: lexical error: invalid string in json text.
                                       test
                     (right here) ------^
tools/virt-qmp-proxy: internal error: cannot parse json test
: lexical error: invalid string in json text.
                                       test
                     (right here) ------^

and stops working, while real qemu behaves differently:

$ qemu-system-x86_64 -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 7}, "package": "qemu-7.0.0-2.fc35"}, "capabilities": ["oob"]}}


help
{"error": {"class": "GenericError", "desc": "JSON parse error, invalid keyword 'help'"}}


Also since it's just a simple loop without event handling from qemu.
E.g. if I destroy the VM while it's running it simply waits. When I
issue another command, then the proxy exits:

$ tools/virt-qmp-proxy 2 /tmp/asdf
libvirt: Domain Config error : Requested operation is not valid: domain is not running
tools/virt-qmp-proxy: Requested operation is not valid: domain is not running


but the client itself just sees a closed socket.

Given the use case it's not a big problem but it should be at least
mentioned in the docs.

> +
> +        resobj = json.loads(res)
> +        del resobj["id"]
> +        if id is not None:
> +            resobj["id"] = id
> +        res = json.dumps(resobj) + "\r\n"
> +        if verbose:
> +            print(res, end='')
> +
> +        client.send(res.encode('utf8'))
> +
> +
> +def parse_commandline():
> +    parser = argparse.ArgumentParser(description="Libvirt QMP proxy")
> +    parser.add_argument("--connect", "-c",
> +                        help="Libvirt QEMU driver connection URI")
> +    parser.add_argument("--verbose", "-v", action='store_true',
> +                        help="Display QMP traffic")
> +    parser.add_argument("domain", metavar="DOMAIN",
> +                        help="Libvirt guest domain ID/UUID/Name")
> +    parser.add_argument("sockpath", metavar="QMP-SOCK-PATH",
> +                        help="UNIX socket path for QMP server")
> +
> +    return parser.parse_args()
> +
> +
> +def main():
> +    args = parse_commandline()
> +
> +    conn, dom = get_domain(args.connect, args.domain)
> +
> +    if conn.getType() != "QEMU":
> +        raise Exception("QMP proxy requires a QEMU driver connection not %s" %
> +                        conn.getType())
> +
> +    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> +    if os.path.exists(args.sockpath):
> +        os.unlink(args.sockpath)
> +    sock.bind(args.sockpath)
> +    sock.listen(1)
> +
> +    while True:
> +        client, peeraddr = sock.accept()
> +        qmp_server(conn, dom, client, args.verbose)
> +
> +
> +try:
> +    main()
> +    sys.exit(0)
> +except Exception as e:
> +    print("%s: %s" % (sys.argv[0], str(e)))
> +    sys.exit(1)
> -- 
> 2.36.1
> 

On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> exposed in virsh. It is not especially pleasant, however, using the raw
> QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> exposes a human friendly interactive shell. It is not possible to use
> this with libvirt managed guest, however, since only one client can
> attach to he QMP socket at any point in time.
> 
> The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> socket and listens for incoming client connections, speaking QMP on
> the connected socket. It will forward any QMP commands received onto
> the running libvirt QEMU guest, and forward any replies back to the
> QMP client.
> 
>   $ virsh start demo
>   $ virt-qmp-proxy demo demo.qmp &
>   $ qmp-shell demo.qmp
>   Welcome to the QMP low-level shell!
>   Connected to QEMU 6.2.0
> 
>   (QEMU) query-kvm
>   {
>       "return": {
>           "enabled": true,
>           "present": true
>       }
>   }
> 
> Note this tool of course has the same risks as the raw libvirt
> QMP passthrough. It is safe to run query commands to fetch information
> but commands which change the QEMU state risk disrupting libvirt's
> management of QEMU, potentially resulting in data loss/corruption in
> the worst case.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> CC'ing QEMU since this is likely of interest to maintainers and users
> who work with QEMU and libvirt
> 
> Note this impl is fairly crude in that it assumes it is receiving
> the QMP commands linewise one at a time. None the less it is good
> enough to work with qmp-shell already, so I figured it was worth
> exposing to the world. It also lacks support for forwarding events
> back to the QMP client.

I originally wanted to teach the qemu tools to work with libvirt
directly similarly how 'scripts/render_block_graph.py' from the qemu
tree already does but I guess this is also an option.

This is an option too albeit a bit more complex to set up, but on the
other hand a bit more universal.

I'll have a look at the code a bit later.

On Fri, May 27, 2022 at 12:20:39PM +0200, Peter Krempa wrote:
> On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> > Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> > exposed in virsh. It is not especially pleasant, however, using the raw
> > QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> > exposes a human friendly interactive shell. It is not possible to use
> > this with libvirt managed guest, however, since only one client can
> > attach to he QMP socket at any point in time.
> > 
> > The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> > socket and listens for incoming client connections, speaking QMP on
> > the connected socket. It will forward any QMP commands received onto
> > the running libvirt QEMU guest, and forward any replies back to the
> > QMP client.
> > 
> >   $ virsh start demo
> >   $ virt-qmp-proxy demo demo.qmp &
> >   $ qmp-shell demo.qmp
> >   Welcome to the QMP low-level shell!
> >   Connected to QEMU 6.2.0
> > 
> >   (QEMU) query-kvm
> >   {
> >       "return": {
> >           "enabled": true,
> >           "present": true
> >       }
> >   }
> > 
> > Note this tool of course has the same risks as the raw libvirt
> > QMP passthrough. It is safe to run query commands to fetch information
> > but commands which change the QEMU state risk disrupting libvirt's
> > management of QEMU, potentially resulting in data loss/corruption in
> > the worst case.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > 
> > CC'ing QEMU since this is likely of interest to maintainers and users
> > who work with QEMU and libvirt
> > 
> > Note this impl is fairly crude in that it assumes it is receiving
> > the QMP commands linewise one at a time. None the less it is good
> > enough to work with qmp-shell already, so I figured it was worth
> > exposing to the world. It also lacks support for forwarding events
> > back to the QMP client.
> 
> I originally wanted to teach the qemu tools to work with libvirt
> directly similarly how 'scripts/render_block_graph.py' from the qemu
> tree already does but I guess this is also an option.

Yes, I do wonder about whether with John's new QMP python APIs,
it would be possible to plug in a livirt transport instead of
the socket transport. I've not spent enough time looking at the
Python QMP code to know if that's viable or not though.

> This is an option too albeit a bit more complex to set up, but on the
> other hand a bit more universal.

The two approaches aren't mutually exclusive either. There's no
reason we can't have both options.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Fri, May 27, 2022, 7:32 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Fri, May 27, 2022 at 12:20:39PM +0200, Peter Krempa wrote:
> > On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> > > Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> > > exposed in virsh. It is not especially pleasant, however, using the raw
> > > QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> > > exposes a human friendly interactive shell. It is not possible to use
> > > this with libvirt managed guest, however, since only one client can
> > > attach to he QMP socket at any point in time.
> > >
> > > The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> > > socket and listens for incoming client connections, speaking QMP on
> > > the connected socket. It will forward any QMP commands received onto
> > > the running libvirt QEMU guest, and forward any replies back to the
> > > QMP client.
> > >
> > >   $ virsh start demo
> > >   $ virt-qmp-proxy demo demo.qmp &
> > >   $ qmp-shell demo.qmp
> > >   Welcome to the QMP low-level shell!
> > >   Connected to QEMU 6.2.0
> > >
> > >   (QEMU) query-kvm
> > >   {
> > >       "return": {
> > >           "enabled": true,
> > >           "present": true
> > >       }
> > >   }
> > >
> > > Note this tool of course has the same risks as the raw libvirt
> > > QMP passthrough. It is safe to run query commands to fetch information
> > > but commands which change the QEMU state risk disrupting libvirt's
> > > management of QEMU, potentially resulting in data loss/corruption in
> > > the worst case.
> > >
> > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > ---
> > >
> > > CC'ing QEMU since this is likely of interest to maintainers and users
> > > who work with QEMU and libvirt
> > >
> > > Note this impl is fairly crude in that it assumes it is receiving
> > > the QMP commands linewise one at a time. None the less it is good
> > > enough to work with qmp-shell already, so I figured it was worth
> > > exposing to the world. It also lacks support for forwarding events
> > > back to the QMP client.
> >
> > I originally wanted to teach the qemu tools to work with libvirt
> > directly similarly how 'scripts/render_block_graph.py' from the qemu
> > tree already does but I guess this is also an option.
>
> Yes, I do wonder about whether with John's new QMP python APIs,
> it would be possible to plug in a livirt transport instead of
> the socket transport. I've not spent enough time looking at the
> Python QMP code to know if that's viable or not though.
>

I can look into it. It looks like render_block_graph works by actually
executing a subprocess. Is there a chance of getting anything socket-like
or stream-like out of libvirt to work with instead?

As long as I can get some kind of stream going it should be easily possible
to just replace the fd(s) the qmp lib uses and talk to libvirt instead.

(possibly with changing some details about the handshake, yadda yadda.)


> > This is an option too albeit a bit more complex to set up, but on the
> > other hand a bit more universal.
>
> The two approaches aren't mutually exclusive either. There's no
> reason we can't have both options.


I've in the past thought about implementing MITM directly in the qmp
library. I wrote a QMPServer() class that I did not check in upstream
because it is only prototype quality, but I used it for testing. I still
have plans to upstream it because it is useful for the test suite.

It has the same limitation as a lot of our Python tools have, though; it
requires newlines because it doesn't have a streaming parser ...

It's been something that I want to improve upon, but since libvirt is
well-behaved and sends newlines, I haven't been properly motivated to.

On Fri, May 27, 2022 at 12:07:55PM -0400, John Snow wrote:
> On Fri, May 27, 2022, 7:32 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Fri, May 27, 2022 at 12:20:39PM +0200, Peter Krempa wrote:
> > > On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> > > > Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> > > > exposed in virsh. It is not especially pleasant, however, using the raw
> > > > QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> > > > exposes a human friendly interactive shell. It is not possible to use
> > > > this with libvirt managed guest, however, since only one client can
> > > > attach to he QMP socket at any point in time.
> > > >
> > > > The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> > > > socket and listens for incoming client connections, speaking QMP on
> > > > the connected socket. It will forward any QMP commands received onto
> > > > the running libvirt QEMU guest, and forward any replies back to the
> > > > QMP client.
> > > >
> > > >   $ virsh start demo
> > > >   $ virt-qmp-proxy demo demo.qmp &
> > > >   $ qmp-shell demo.qmp
> > > >   Welcome to the QMP low-level shell!
> > > >   Connected to QEMU 6.2.0
> > > >
> > > >   (QEMU) query-kvm
> > > >   {
> > > >       "return": {
> > > >           "enabled": true,
> > > >           "present": true
> > > >       }
> > > >   }
> > > >
> > > > Note this tool of course has the same risks as the raw libvirt
> > > > QMP passthrough. It is safe to run query commands to fetch information
> > > > but commands which change the QEMU state risk disrupting libvirt's
> > > > management of QEMU, potentially resulting in data loss/corruption in
> > > > the worst case.
> > > >
> > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > > ---
> > > >
> > > > CC'ing QEMU since this is likely of interest to maintainers and users
> > > > who work with QEMU and libvirt
> > > >
> > > > Note this impl is fairly crude in that it assumes it is receiving
> > > > the QMP commands linewise one at a time. None the less it is good
> > > > enough to work with qmp-shell already, so I figured it was worth
> > > > exposing to the world. It also lacks support for forwarding events
> > > > back to the QMP client.
> > >
> > > I originally wanted to teach the qemu tools to work with libvirt
> > > directly similarly how 'scripts/render_block_graph.py' from the qemu
> > > tree already does but I guess this is also an option.
> >
> > Yes, I do wonder about whether with John's new QMP python APIs,
> > it would be possible to plug in a livirt transport instead of
> > the socket transport. I've not spent enough time looking at the
> > Python QMP code to know if that's viable or not though.
> >
> 
> I can look into it. It looks like render_block_graph works by actually
> executing a subprocess. Is there a chance of getting anything socket-like
> or stream-like out of libvirt to work with instead?
> 
> As long as I can get some kind of stream going it should be easily possible
> to just replace the fd(s) the qmp lib uses and talk to libvirt instead.

Afraid not, any interface that plugs into libvirt would need to be
much higher level - send command and receive events - to map into
the libvirt APIs.

To get something stream oriented requires the proxy that I've proposd
here instead.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 5/27/22 12:20 PM, Peter Krempa wrote:
> On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
>> Libvirt provides QMP passthrough APIs for the QEMU driver and these are
>> exposed in virsh. It is not especially pleasant, however, using the raw
>> QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
>> exposes a human friendly interactive shell. It is not possible to use
>> this with libvirt managed guest, however, since only one client can
>> attach to he QMP socket at any point in time.
>>
>> The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
>> socket and listens for incoming client connections, speaking QMP on
>> the connected socket. It will forward any QMP commands received onto
>> the running libvirt QEMU guest, and forward any replies back to the
>> QMP client.
>>
>>   $ virsh start demo
>>   $ virt-qmp-proxy demo demo.qmp &
>>   $ qmp-shell demo.qmp
>>   Welcome to the QMP low-level shell!
>>   Connected to QEMU 6.2.0
>>
>>   (QEMU) query-kvm
>>   {
>>       "return": {
>>           "enabled": true,
>>           "present": true
>>       }
>>   }
>>
>> Note this tool of course has the same risks as the raw libvirt
>> QMP passthrough. It is safe to run query commands to fetch information
>> but commands which change the QEMU state risk disrupting libvirt's
>> management of QEMU, potentially resulting in data loss/corruption in
>> the worst case.
>>
>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>
>> CC'ing QEMU since this is likely of interest to maintainers and users
>> who work with QEMU and libvirt
>>
>> Note this impl is fairly crude in that it assumes it is receiving
>> the QMP commands linewise one at a time. None the less it is good
>> enough to work with qmp-shell already, so I figured it was worth
>> exposing to the world. It also lacks support for forwarding events
>> back to the QMP client.
> 
> I originally wanted to teach the qemu tools to work with libvirt
> directly similarly how 'scripts/render_block_graph.py' from the qemu
> tree already does but I guess this is also an option.
> 
> This is an option too albeit a bit more complex to set up, but on the
> other hand a bit more universal.
> 
> I'll have a look at the code a bit later.
> 

Would have found it useful, at the time I wrote the multifd save series I ended up just scripting around virsh qemu-monitor-command from either bash or C.

One challenge I had to face was, when doing fd migration doing

"execute": "getfd", "arguments": {"fdname":"migrate"}

in that case we have to use the --pass-fds=N option to pass the FD.

Does the virt-qmp-proxy tool consider the passing of FDs issue?

Thanks,

Claudio

On Fri, May 27, 2022 at 12:35:45PM +0200, Claudio Fontana wrote:
> On 5/27/22 12:20 PM, Peter Krempa wrote:
> > On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> >> Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> >> exposed in virsh. It is not especially pleasant, however, using the raw
> >> QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> >> exposes a human friendly interactive shell. It is not possible to use
> >> this with libvirt managed guest, however, since only one client can
> >> attach to he QMP socket at any point in time.
> >>
> >> The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> >> socket and listens for incoming client connections, speaking QMP on
> >> the connected socket. It will forward any QMP commands received onto
> >> the running libvirt QEMU guest, and forward any replies back to the
> >> QMP client.
> >>
> >>   $ virsh start demo
> >>   $ virt-qmp-proxy demo demo.qmp &
> >>   $ qmp-shell demo.qmp
> >>   Welcome to the QMP low-level shell!
> >>   Connected to QEMU 6.2.0
> >>
> >>   (QEMU) query-kvm
> >>   {
> >>       "return": {
> >>           "enabled": true,
> >>           "present": true
> >>       }
> >>   }
> >>
> >> Note this tool of course has the same risks as the raw libvirt
> >> QMP passthrough. It is safe to run query commands to fetch information
> >> but commands which change the QEMU state risk disrupting libvirt's
> >> management of QEMU, potentially resulting in data loss/corruption in
> >> the worst case.
> >>
> >> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >> ---
> >>
> >> CC'ing QEMU since this is likely of interest to maintainers and users
> >> who work with QEMU and libvirt
> >>
> >> Note this impl is fairly crude in that it assumes it is receiving
> >> the QMP commands linewise one at a time. None the less it is good
> >> enough to work with qmp-shell already, so I figured it was worth
> >> exposing to the world. It also lacks support for forwarding events
> >> back to the QMP client.
> > 
> > I originally wanted to teach the qemu tools to work with libvirt
> > directly similarly how 'scripts/render_block_graph.py' from the qemu
> > tree already does but I guess this is also an option.
> > 
> > This is an option too albeit a bit more complex to set up, but on the
> > other hand a bit more universal.
> > 
> > I'll have a look at the code a bit later.
> > 
> 
> Would have found it useful, at the time I wrote the multifd save series I ended up just scripting around virsh qemu-monitor-command from either bash or C.
> 
> One challenge I had to face was, when doing fd migration doing
> 
> "execute": "getfd", "arguments": {"fdname":"migrate"}
> 
> in that case we have to use the --pass-fds=N option to pass the FD.
> 
> Does the virt-qmp-proxy tool consider the passing of FDs issue?

My impl here doesn't try to support FD passing, but it is conceptually
possible for us to support it given the new libvirt API Peter added
a few months back to allow monitor passthrough with FDs.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Fri, May 27, 2022 at 12:35:45 +0200, Claudio Fontana wrote:
> On 5/27/22 12:20 PM, Peter Krempa wrote:
> > On Fri, May 27, 2022 at 10:47:58 +0100, Daniel P. Berrangé wrote:
> >> Libvirt provides QMP passthrough APIs for the QEMU driver and these are
> >> exposed in virsh. It is not especially pleasant, however, using the raw
> >> QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and
> >> exposes a human friendly interactive shell. It is not possible to use
> >> this with libvirt managed guest, however, since only one client can
> >> attach to he QMP socket at any point in time.
> >>
> >> The virt-qmp-proxy tool aims to solve this problem. It opens a UNIX
> >> socket and listens for incoming client connections, speaking QMP on
> >> the connected socket. It will forward any QMP commands received onto
> >> the running libvirt QEMU guest, and forward any replies back to the
> >> QMP client.
> >>
> >>   $ virsh start demo
> >>   $ virt-qmp-proxy demo demo.qmp &
> >>   $ qmp-shell demo.qmp
> >>   Welcome to the QMP low-level shell!
> >>   Connected to QEMU 6.2.0
> >>
> >>   (QEMU) query-kvm
> >>   {
> >>       "return": {
> >>           "enabled": true,
> >>           "present": true
> >>       }
> >>   }
> >>
> >> Note this tool of course has the same risks as the raw libvirt
> >> QMP passthrough. It is safe to run query commands to fetch information
> >> but commands which change the QEMU state risk disrupting libvirt's
> >> management of QEMU, potentially resulting in data loss/corruption in
> >> the worst case.
> >>
> >> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >> ---
> >>
> >> CC'ing QEMU since this is likely of interest to maintainers and users
> >> who work with QEMU and libvirt
> >>
> >> Note this impl is fairly crude in that it assumes it is receiving
> >> the QMP commands linewise one at a time. None the less it is good
> >> enough to work with qmp-shell already, so I figured it was worth
> >> exposing to the world. It also lacks support for forwarding events
> >> back to the QMP client.
> > 
> > I originally wanted to teach the qemu tools to work with libvirt
> > directly similarly how 'scripts/render_block_graph.py' from the qemu
> > tree already does but I guess this is also an option.
> > 
> > This is an option too albeit a bit more complex to set up, but on the
> > other hand a bit more universal.
> > 
> > I'll have a look at the code a bit later.
> > 
> 
> Would have found it useful, at the time I wrote the multifd save series I ended up just scripting around virsh qemu-monitor-command from either bash or C.

I'd consider this to be just something to achieve compatibility with
tools that already exist and expect monitor socket.

If you are writing new software with libvirt connection in mind you can
use 'virDomainQemuMonitorCommand()' or
'virDomainQemuMonitorCommandWithFiles()' directly or via the language
bindings for your language.

Obviously not for bash though.