The default message recv limit in asyncio is smaller than our needs, so
when opening connections we override it. This was done when opening a
connection using a socket address, but was missed when using a
pre-opened socket file descriptor.

This latent bug was exposed when the QEMUMachine class was enhanced to
use socketpair() when no socket address was passed by:

  commit bd4c0ef409140bd1be393407c04005ac077d4574
  Author: Marc-André Lureau <marcandre.lureau@redhat.com>
  Date:   Wed Jan 11 12:01:01 2023 +0400

    python/qemu/machine: use socketpair() for QMP by default

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 python/qemu/qmp/protocol.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py
index 22e60298d2..2bd341a93e 100644
--- a/python/qemu/qmp/protocol.py
+++ b/python/qemu/qmp/protocol.py
@@ -307,7 +307,10 @@ async def open_with_socket(self, sock: socket.socket) -> None:
 
         :raise StateError: When the `Runstate` is not `IDLE`.
         """
-        self._reader, self._writer = await asyncio.open_connection(sock=sock)
+        self._reader, self._writer = await asyncio.open_connection(
+            sock=sock,
+            limit=self._limit,
+        )
         self._set_state(Runstate.CONNECTING)
 
     @upper_half
-- 
2.39.2

On 20.03.23 13:54, Daniel P. Berrangé wrote:
> The default message recv limit in asyncio is smaller than our needs, so
> when opening connections we override it. This was done when opening a
> connection using a socket address, but was missed when using a
> pre-opened socket file descriptor.
> 
> This latent bug was exposed when the QEMUMachine class was enhanced to
> use socketpair() when no socket address was passed by:
> 
>    commit bd4c0ef409140bd1be393407c04005ac077d4574
>    Author: Marc-André Lureau<marcandre.lureau@redhat.com>
>    Date:   Wed Jan 11 12:01:01 2023 +0400
> 
>      python/qemu/machine: use socketpair() for QMP by default
> 
> Signed-off-by: Daniel P. Berrangé<berrange@redhat.com>

Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>

Thanks!

-- 
Best regards,
Vladimir

On Mon, Mar 20, 2023 at 8:20 AM Vladimir Sementsov-Ogievskiy
<vsementsov@yandex-team.ru> wrote:
>
> On 20.03.23 13:54, Daniel P. Berrangé wrote:
> > The default message recv limit in asyncio is smaller than our needs, so
> > when opening connections we override it. This was done when opening a
> > connection using a socket address, but was missed when using a
> > pre-opened socket file descriptor.
> >
> > This latent bug was exposed when the QEMUMachine class was enhanced to
> > use socketpair() when no socket address was passed by:
> >
> >    commit bd4c0ef409140bd1be393407c04005ac077d4574
> >    Author: Marc-André Lureau<marcandre.lureau@redhat.com>
> >    Date:   Wed Jan 11 12:01:01 2023 +0400
> >
> >      python/qemu/machine: use socketpair() for QMP by default
> >
> > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com>
>
> Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>
> Thanks!
>
> --
> Best regards,
> Vladimir
>

Thanks x3. Will stage in both places.

--js

On Mon, Mar 20, 2023 at 8:27 PM John Snow <jsnow@redhat.com> wrote:
>
> On Mon, Mar 20, 2023 at 8:20 AM Vladimir Sementsov-Ogievskiy
> <vsementsov@yandex-team.ru> wrote:
> >
> > On 20.03.23 13:54, Daniel P. Berrangé wrote:
> > > The default message recv limit in asyncio is smaller than our needs, so
> > > when opening connections we override it. This was done when opening a
> > > connection using a socket address, but was missed when using a
> > > pre-opened socket file descriptor.
> > >
> > > This latent bug was exposed when the QEMUMachine class was enhanced to
> > > use socketpair() when no socket address was passed by:
> > >
> > >    commit bd4c0ef409140bd1be393407c04005ac077d4574
> > >    Author: Marc-André Lureau<marcandre.lureau@redhat.com>
> > >    Date:   Wed Jan 11 12:01:01 2023 +0400
> > >
> > >      python/qemu/machine: use socketpair() for QMP by default
> > >
> > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com>
> >
> > Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> >
> > Thanks!
> >
> > --
> > Best regards,
> > Vladimir
> >
>
> Thanks x3. Will stage in both places.
>

... Realizing I let the two codebases diverge a bit more than I had
meant to allow, and the "open with socket" rewrite I made never got
brought back to qemu.git, tch. I believe this bug only exists in
qemu.git because of that. I need to backport my changes to qemu.git
and ensure that there isn't a problem with the revised code as it
exists in qemu.qmp.

I'll have to fix that *before* I drop qemu.qmp from the qemu.git repo,
which I am hoping will be soon after Paolo takes the new Python
patches that create a venv during configure time.

--js

On Mon, May 15, 2023 at 5:14 PM John Snow <jsnow@redhat.com> wrote:
>
> On Mon, Mar 20, 2023 at 8:27 PM John Snow <jsnow@redhat.com> wrote:
> >
> > On Mon, Mar 20, 2023 at 8:20 AM Vladimir Sementsov-Ogievskiy
> > <vsementsov@yandex-team.ru> wrote:
> > >
> > > On 20.03.23 13:54, Daniel P. Berrangé wrote:
> > > > The default message recv limit in asyncio is smaller than our needs, so
> > > > when opening connections we override it. This was done when opening a
> > > > connection using a socket address, but was missed when using a
> > > > pre-opened socket file descriptor.
> > > >
> > > > This latent bug was exposed when the QEMUMachine class was enhanced to
> > > > use socketpair() when no socket address was passed by:
> > > >
> > > >    commit bd4c0ef409140bd1be393407c04005ac077d4574
> > > >    Author: Marc-André Lureau<marcandre.lureau@redhat.com>
> > > >    Date:   Wed Jan 11 12:01:01 2023 +0400
> > > >
> > > >      python/qemu/machine: use socketpair() for QMP by default
> > > >
> > > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com>
> > >
> > > Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> > >
> > > Thanks!
> > >
> > > --
> > > Best regards,
> > > Vladimir
> > >
> >
> > Thanks x3. Will stage in both places.
> >
>
> ... Realizing I let the two codebases diverge a bit more than I had
> meant to allow, and the "open with socket" rewrite I made never got
> brought back to qemu.git, tch. I believe this bug only exists in
> qemu.git because of that. I need to backport my changes to qemu.git
> and ensure that there isn't a problem with the revised code as it
> exists in qemu.qmp.
>
> I'll have to fix that *before* I drop qemu.qmp from the qemu.git repo,
> which I am hoping will be soon after Paolo takes the new Python
> patches that create a venv during configure time.
>
> --js

OK, I've just sent two series:

(1) A MR for python-qemu-qmp here, which backports a few misc fixes
from qemu.git back over to python-qemu-qmp;
https://gitlab.com/qemu-project/python-qemu-qmp/-/merge_requests/23
This needs approval from at least one person. If you're interested in
becoming a "reviewer" there, let me know and I'll add you if you
aren't already.

(2) A patchset for qemu.git that backports the socket changes that
went into python-qemu-qmp.
https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg04285.html
This should fix the bug described here in this series, and is
necessary to merge before I swap the in-tree qemu.git for a vendored
qemu.qmp package.
Review welcomed, but any changes we need will need to happen on the
python-qemu-qmp repo first.

Dropping the in-tree qemu.qmp package is going to happen after the
large series Paolo just sent a PR for,
https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg04330.html
which adds a configure-time venv to the QEMU build system.

Hoping to drop qemu.qmp *this dev cycle* and release v0.0.3 for
qemu.qmp and vendor it to qemu.git as a .whl file.

Thanks,
--js

On Mon, Mar 20, 2023 at 2:54 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> The default message recv limit in asyncio is smaller than our needs, so
> when opening connections we override it. This was done when opening a
> connection using a socket address, but was missed when using a
> pre-opened socket file descriptor.
>
> This latent bug was exposed when the QEMUMachine class was enhanced to
> use socketpair() when no socket address was passed by:
>
>   commit bd4c0ef409140bd1be393407c04005ac077d4574
>   Author: Marc-André Lureau <marcandre.lureau@redhat.com>
>   Date:   Wed Jan 11 12:01:01 2023 +0400
>
>     python/qemu/machine: use socketpair() for QMP by default
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>


thanks
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>



> ---
>  python/qemu/qmp/protocol.py | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py
> index 22e60298d2..2bd341a93e 100644
> --- a/python/qemu/qmp/protocol.py
> +++ b/python/qemu/qmp/protocol.py
> @@ -307,7 +307,10 @@ async def open_with_socket(self, sock: socket.socket)
> -> None:
>
>          :raise StateError: When the `Runstate` is not `IDLE`.
>          """
> -        self._reader, self._writer = await
> asyncio.open_connection(sock=sock)
> +        self._reader, self._writer = await asyncio.open_connection(
> +            sock=sock,
> +            limit=self._limit,
> +        )
>          self._set_state(Runstate.CONNECTING)
>
>      @upper_half
> --
> 2.39.2
>
>