[PULL 00/44] Python patches

John Snow posted 44 patches 2 years, 11 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/next-importer-push tags/patchew/20210528023220.417057-1-jsnow@redhat.com
Maintainers: "Philippe Mathieu-Daudé" <philmd@redhat.com>, Willian Rampazzo <willianr@redhat.com>, Aurelien Jarno <aurelien@aurel32.net>, Kevin Wolf <kwolf@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Cleber Rosa <crosa@redhat.com>, Max Reitz <mreitz@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, John Snow <jsnow@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, Thomas Huth <thuth@redhat.com>
There is a newer version of this series
docs/devel/testing.rst                      |  26 +++
python/PACKAGE.rst                          |  43 ++++
python/README.rst                           |  58 +++++
python/qemu/README.rst                      |   8 +
python/qemu/machine/README.rst              |   9 +
python/qemu/qmp/README.rst                  |   9 +
python/qemu/utils/README.rst                |   7 +
.gitlab-ci.d/containers.yml                 |   5 +
.gitlab-ci.d/static_checks.yml              |  21 ++
python/.gitignore                           |  16 ++
python/MANIFEST.in                          |   3 +
python/Makefile                             |  48 ++++
python/Pipfile                              |  13 ++
python/Pipfile.lock                         | 231 ++++++++++++++++++++
python/VERSION                              |   1 +
python/avocado.cfg                          |  10 +
python/mypy.ini                             |   4 -
python/qemu/.flake8                         |   2 -
python/qemu/.isort.cfg                      |   7 -
python/qemu/__init__.py                     |  11 -
python/qemu/machine/__init__.py             |  36 +++
python/qemu/{ => machine}/console_socket.py |  11 +-
python/qemu/{ => machine}/machine.py        |  68 ++++--
python/qemu/{ => machine}/qtest.py          |   9 +-
python/qemu/pylintrc                        |  58 -----
python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
python/qemu/utils/__init__.py               |  45 ++++
python/qemu/{ => utils}/accel.py            |   0
python/setup.cfg                            | 102 +++++++++
python/setup.py                             |  23 ++
python/tests/flake8.sh                      |   2 +
python/tests/isort.sh                       |   2 +
python/tests/mypy.sh                        |   2 +
python/tests/pylint.sh                      |   2 +
tests/acceptance/avocado_qemu/__init__.py   |  69 +++++-
tests/acceptance/boot_linux.py              |  18 +-
tests/acceptance/hotplug_cpu.py             |  37 ++++
tests/acceptance/info_usernet.py            |  29 +++
tests/acceptance/linux_ssh_mips_malta.py    |  42 +---
tests/acceptance/virtio-gpu.py              |   2 +-
tests/acceptance/virtiofs_submounts.py      |  71 +-----
tests/docker/dockerfiles/python.docker      |  18 ++
tests/qemu-iotests/297                      |   1 +
tests/qemu-iotests/300                      |   4 +-
tests/qemu-iotests/iotests.py               |   4 +-
tests/requirements.txt                      |   2 +-
tests/vm/aarch64vm.py                       |   2 +-
tests/vm/basevm.py                          |   8 +-
48 files changed, 965 insertions(+), 246 deletions(-)
create mode 100644 python/PACKAGE.rst
create mode 100644 python/README.rst
create mode 100644 python/qemu/README.rst
create mode 100644 python/qemu/machine/README.rst
create mode 100644 python/qemu/qmp/README.rst
create mode 100644 python/qemu/utils/README.rst
create mode 100644 python/.gitignore
create mode 100644 python/MANIFEST.in
create mode 100644 python/Makefile
create mode 100644 python/Pipfile
create mode 100644 python/Pipfile.lock
create mode 100644 python/VERSION
create mode 100644 python/avocado.cfg
delete mode 100644 python/mypy.ini
delete mode 100644 python/qemu/.flake8
delete mode 100644 python/qemu/.isort.cfg
delete mode 100644 python/qemu/__init__.py
create mode 100644 python/qemu/machine/__init__.py
rename python/qemu/{ => machine}/console_socket.py (94%)
rename python/qemu/{ => machine}/machine.py (93%)
rename python/qemu/{ => machine}/qtest.py (95%)
delete mode 100644 python/qemu/pylintrc
rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
create mode 100644 python/qemu/utils/__init__.py
rename python/qemu/{ => utils}/accel.py (100%)
create mode 100644 python/setup.cfg
create mode 100755 python/setup.py
create mode 100755 python/tests/flake8.sh
create mode 100755 python/tests/isort.sh
create mode 100755 python/tests/mypy.sh
create mode 100755 python/tests/pylint.sh
create mode 100644 tests/acceptance/hotplug_cpu.py
create mode 100644 tests/acceptance/info_usernet.py
create mode 100644 tests/docker/dockerfiles/python.docker
[PULL 00/44] Python patches
Posted by John Snow 2 years, 11 months ago
The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a:

  Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into staging (2021-05-27 16:32:57 +0100)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde:

  gitlab: add python linters to CI (2021-05-27 18:11:17 -0400)

----------------------------------------------------------------
Python pull request

Python packaging & CI implementation
Acceptance tests sent w/ Cleber's blessing.
New CI tests send w/ stsquad's RB.

--js

----------------------------------------------------------------

Cleber Rosa (12):
  Python: expose QEMUMachine's temporary directory
  tests/acceptance/virtiofs_submounts.py: add missing accel tag
  tests/acceptance/virtiofs_submounts.py: evaluate string not length
  Python: add utility function for retrieving port redirection
  Acceptance Tests: move useful ssh methods to base class
  Acceptance Tests: add port redirection for ssh by default
  Acceptance Tests: make username/password configurable
  Acceptance Tests: set up SSH connection by default after boot for
    LinuxTest
  tests/acceptance/virtiofs_submounts.py: remove launch_vm()
  Acceptance Tests: add basic documentation on LinuxTest base class
  Acceptance Tests: introduce CPU hotplug test
  tests/acceptance/virtiofs_submounts.py: fix setup of SSH pubkey

John Snow (31):
  python/console_socket: avoid one-letter variable
  python/machine: use subprocess.DEVNULL instead of
    open(os.path.devnull)
  python/machine: use subprocess.run instead of subprocess.Popen
  python/console_socket: Add a pylint ignore
  python/machine: Disable pylint warning for open() in _pre_launch
  python/machine: disable warning for Popen in _launch()
  python/machine: Trim line length to below 80 chars
  iotests/297: add --namespace-packages to mypy arguments
  python: create qemu packages
  python: add qemu package installer
  python: add VERSION file
  python: add directory structure README.rst files
  python: add MANIFEST.in
  python: Add pipenv support
  python: add pylint import exceptions
  python: move pylintrc into setup.cfg
  python: add pylint to pipenv
  python: move flake8 config to setup.cfg
  python: add excluded dirs to flake8 config
  python: Add flake8 to pipenv
  python: move mypy.ini into setup.cfg
  python: add mypy to pipenv
  python: move .isort.cfg into setup.cfg
  python/qemu: add isort to pipenv
  python/qemu: add qemu package itself to pipenv
  python: add devel package requirements to setuptools
  python: add avocado-framework and tests
  python: add Makefile for some common tasks
  python: add .gitignore
  python: add tox support
  gitlab: add python linters to CI

Willian Rampazzo (1):
  acceptance tests: bump Avocado version to 88.1

 docs/devel/testing.rst                      |  26 +++
 python/PACKAGE.rst                          |  43 ++++
 python/README.rst                           |  58 +++++
 python/qemu/README.rst                      |   8 +
 python/qemu/machine/README.rst              |   9 +
 python/qemu/qmp/README.rst                  |   9 +
 python/qemu/utils/README.rst                |   7 +
 .gitlab-ci.d/containers.yml                 |   5 +
 .gitlab-ci.d/static_checks.yml              |  21 ++
 python/.gitignore                           |  16 ++
 python/MANIFEST.in                          |   3 +
 python/Makefile                             |  48 ++++
 python/Pipfile                              |  13 ++
 python/Pipfile.lock                         | 231 ++++++++++++++++++++
 python/VERSION                              |   1 +
 python/avocado.cfg                          |  10 +
 python/mypy.ini                             |   4 -
 python/qemu/.flake8                         |   2 -
 python/qemu/.isort.cfg                      |   7 -
 python/qemu/__init__.py                     |  11 -
 python/qemu/machine/__init__.py             |  36 +++
 python/qemu/{ => machine}/console_socket.py |  11 +-
 python/qemu/{ => machine}/machine.py        |  68 ++++--
 python/qemu/{ => machine}/qtest.py          |   9 +-
 python/qemu/pylintrc                        |  58 -----
 python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
 python/qemu/utils/__init__.py               |  45 ++++
 python/qemu/{ => utils}/accel.py            |   0
 python/setup.cfg                            | 102 +++++++++
 python/setup.py                             |  23 ++
 python/tests/flake8.sh                      |   2 +
 python/tests/isort.sh                       |   2 +
 python/tests/mypy.sh                        |   2 +
 python/tests/pylint.sh                      |   2 +
 tests/acceptance/avocado_qemu/__init__.py   |  69 +++++-
 tests/acceptance/boot_linux.py              |  18 +-
 tests/acceptance/hotplug_cpu.py             |  37 ++++
 tests/acceptance/info_usernet.py            |  29 +++
 tests/acceptance/linux_ssh_mips_malta.py    |  42 +---
 tests/acceptance/virtio-gpu.py              |   2 +-
 tests/acceptance/virtiofs_submounts.py      |  71 +-----
 tests/docker/dockerfiles/python.docker      |  18 ++
 tests/qemu-iotests/297                      |   1 +
 tests/qemu-iotests/300                      |   4 +-
 tests/qemu-iotests/iotests.py               |   4 +-
 tests/requirements.txt                      |   2 +-
 tests/vm/aarch64vm.py                       |   2 +-
 tests/vm/basevm.py                          |   8 +-
 48 files changed, 965 insertions(+), 246 deletions(-)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/machine/README.rst
 create mode 100644 python/qemu/qmp/README.rst
 create mode 100644 python/qemu/utils/README.rst
 create mode 100644 python/.gitignore
 create mode 100644 python/MANIFEST.in
 create mode 100644 python/Makefile
 create mode 100644 python/Pipfile
 create mode 100644 python/Pipfile.lock
 create mode 100644 python/VERSION
 create mode 100644 python/avocado.cfg
 delete mode 100644 python/mypy.ini
 delete mode 100644 python/qemu/.flake8
 delete mode 100644 python/qemu/.isort.cfg
 delete mode 100644 python/qemu/__init__.py
 create mode 100644 python/qemu/machine/__init__.py
 rename python/qemu/{ => machine}/console_socket.py (94%)
 rename python/qemu/{ => machine}/machine.py (93%)
 rename python/qemu/{ => machine}/qtest.py (95%)
 delete mode 100644 python/qemu/pylintrc
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
 create mode 100644 python/qemu/utils/__init__.py
 rename python/qemu/{ => utils}/accel.py (100%)
 create mode 100644 python/setup.cfg
 create mode 100755 python/setup.py
 create mode 100755 python/tests/flake8.sh
 create mode 100755 python/tests/isort.sh
 create mode 100755 python/tests/mypy.sh
 create mode 100755 python/tests/pylint.sh
 create mode 100644 tests/acceptance/hotplug_cpu.py
 create mode 100644 tests/acceptance/info_usernet.py
 create mode 100644 tests/docker/dockerfiles/python.docker

-- 
2.31.1



[PATCH 0/1] VM tests: account for changes in qemu.utils.get_info_usernet_hostfwd_port()
Posted by Cleber Rosa 2 years, 10 months ago
The utility function actually accepts a string, and not the raw
dictionary response from QMP.  Also, it returns (optionally) an
integer, so that must also be accounted for.

The issue can be replicated with (and verified the fix) with `make
vm-build-freebsd`.

Cleber Rosa (1):
  VM tests: account for changes in
    qemu.utils.get_info_usernet_hostfwd_port()

 tests/vm/basevm.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
2.25.4



[PATCH 1/1] VM tests: account for changes in qemu.utils.get_info_usernet_hostfwd_port()
Posted by Cleber Rosa 2 years, 10 months ago
The utility function actually accepts a string, and not the raw
dictionary response from QMP.  Also, it returns (optionally) an
integer, so that must also be accounted for.

Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/vm/basevm.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 6f4f0fc95e..0f2e436ed3 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -227,7 +227,7 @@ def _ssh_do(self, user, cmd, check):
                    "-o", "UserKnownHostsFile=" + os.devnull,
                    "-o",
                    "ConnectTimeout={}".format(self._config["ssh_timeout"]),
-                   "-p", self.ssh_port, "-i", self._ssh_tmp_key_file]
+                   "-p", str(self.ssh_port), "-i", self._ssh_tmp_key_file]
         # If not in debug mode, set ssh to quiet mode to
         # avoid printing the results of commands.
         if not self.debug:
@@ -305,7 +305,7 @@ def boot(self, img, extra_args=[]):
         # Init console so we can start consuming the chars.
         self.console_init()
         usernet_info = guest.qmp("human-monitor-command",
-                                 command_line="info usernet")
+                                 command_line="info usernet").get("return")
         self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)
         if not self.ssh_port:
             raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
-- 
2.25.4


Re: [PATCH 1/1] VM tests: account for changes in qemu.utils.get_info_usernet_hostfwd_port()
Posted by Willian Rampazzo 2 years, 10 months ago
On Tue, Jun 1, 2021 at 12:46 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> The utility function actually accepts a string, and not the raw
> dictionary response from QMP.  Also, it returns (optionally) an
> integer, so that must also be accounted for.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/vm/basevm.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 6f4f0fc95e..0f2e436ed3 100644
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -227,7 +227,7 @@ def _ssh_do(self, user, cmd, check):
>                     "-o", "UserKnownHostsFile=" + os.devnull,
>                     "-o",
>                     "ConnectTimeout={}".format(self._config["ssh_timeout"]),
> -                   "-p", self.ssh_port, "-i", self._ssh_tmp_key_file]
> +                   "-p", str(self.ssh_port), "-i", self._ssh_tmp_key_file]
>          # If not in debug mode, set ssh to quiet mode to
>          # avoid printing the results of commands.
>          if not self.debug:
> @@ -305,7 +305,7 @@ def boot(self, img, extra_args=[]):
>          # Init console so we can start consuming the chars.
>          self.console_init()
>          usernet_info = guest.qmp("human-monitor-command",
> -                                 command_line="info usernet")
> +                                 command_line="info usernet").get("return")
>          self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)

From here, your file differs from the upstream and your patch does not
work. I'm trying on commit 52848929b70dcf92a68aedcfd90207be81ba3274.

>          if not self.ssh_port:
>              raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
> --
> 2.25.4
>


Re: [PATCH 1/1] VM tests: account for changes in qemu.utils.get_info_usernet_hostfwd_port()
Posted by Willian Rampazzo 2 years, 10 months ago
On Tue, Jun 1, 2021 at 2:36 PM Willian Rampazzo <wrampazz@redhat.com> wrote:
>
> On Tue, Jun 1, 2021 at 12:46 PM Cleber Rosa <crosa@redhat.com> wrote:
> >
> > The utility function actually accepts a string, and not the raw
> > dictionary response from QMP.  Also, it returns (optionally) an
> > integer, so that must also be accounted for.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/vm/basevm.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> > index 6f4f0fc95e..0f2e436ed3 100644
> > --- a/tests/vm/basevm.py
> > +++ b/tests/vm/basevm.py
> > @@ -227,7 +227,7 @@ def _ssh_do(self, user, cmd, check):
> >                     "-o", "UserKnownHostsFile=" + os.devnull,
> >                     "-o",
> >                     "ConnectTimeout={}".format(self._config["ssh_timeout"]),
> > -                   "-p", self.ssh_port, "-i", self._ssh_tmp_key_file]
> > +                   "-p", str(self.ssh_port), "-i", self._ssh_tmp_key_file]
> >          # If not in debug mode, set ssh to quiet mode to
> >          # avoid printing the results of commands.
> >          if not self.debug:
> > @@ -305,7 +305,7 @@ def boot(self, img, extra_args=[]):
> >          # Init console so we can start consuming the chars.
> >          self.console_init()
> >          usernet_info = guest.qmp("human-monitor-command",
> > -                                 command_line="info usernet")
> > +                                 command_line="info usernet").get("return")
> >          self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)
>
> From here, your file differs from the upstream and your patch does not
> work. I'm trying on commit 52848929b70dcf92a68aedcfd90207be81ba3274.

Talking to Cleber he mentioned he submitted In-reply-to John's PR, but
Gmail did not show it for me.

On top of John's PR
(https://patchew.org/QEMU/20210528023220.417057-1-jsnow@redhat.com/):

Tested-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>

>
> >          if not self.ssh_port:
> >              raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
> > --
> > 2.25.4
> >


Re: [PATCH 1/1] VM tests: account for changes in qemu.utils.get_info_usernet_hostfwd_port()
Posted by Philippe Mathieu-Daudé 2 years, 10 months ago
On 6/1/21 8:07 PM, Willian Rampazzo wrote:
> On Tue, Jun 1, 2021 at 2:36 PM Willian Rampazzo <wrampazz@redhat.com> wrote:
>>
>> On Tue, Jun 1, 2021 at 12:46 PM Cleber Rosa <crosa@redhat.com> wrote:
>>>
>>> The utility function actually accepts a string, and not the raw
>>> dictionary response from QMP.  Also, it returns (optionally) an
>>> integer, so that must also be accounted for.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/vm/basevm.py | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
>>> index 6f4f0fc95e..0f2e436ed3 100644
>>> --- a/tests/vm/basevm.py
>>> +++ b/tests/vm/basevm.py
>>> @@ -227,7 +227,7 @@ def _ssh_do(self, user, cmd, check):
>>>                     "-o", "UserKnownHostsFile=" + os.devnull,
>>>                     "-o",
>>>                     "ConnectTimeout={}".format(self._config["ssh_timeout"]),
>>> -                   "-p", self.ssh_port, "-i", self._ssh_tmp_key_file]
>>> +                   "-p", str(self.ssh_port), "-i", self._ssh_tmp_key_file]
>>>          # If not in debug mode, set ssh to quiet mode to
>>>          # avoid printing the results of commands.
>>>          if not self.debug:
>>> @@ -305,7 +305,7 @@ def boot(self, img, extra_args=[]):
>>>          # Init console so we can start consuming the chars.
>>>          self.console_init()
>>>          usernet_info = guest.qmp("human-monitor-command",
>>> -                                 command_line="info usernet")
>>> +                                 command_line="info usernet").get("return")
>>>          self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)
>>
>> From here, your file differs from the upstream and your patch does not
>> work. I'm trying on commit 52848929b70dcf92a68aedcfd90207be81ba3274.
> 
> Talking to Cleber he mentioned he submitted In-reply-to

Please don't, as the series end up buried in the other email thread.

> John's PR, but
> Gmail did not show it for me.
> 
> On top of John's PR
> (https://patchew.org/QEMU/20210528023220.417057-1-jsnow@redhat.com/):

The syntax is:

Based-on: 20210528023220.417057-1-jsnow@redhat.com

https://wiki.qemu.org/Contribute/SubmitAPatch#Base_patches_against_current_git_master

> 
> Tested-by: Willian Rampazzo <willianr@redhat.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> 
>>
>>>          if not self.ssh_port:
>>>              raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
>>> --
>>> 2.25.4
>>>
> 


Re: [PATCH 1/1] VM tests: account for changes in qemu.utils.get_info_usernet_hostfwd_port()
Posted by John Snow 2 years, 10 months ago
On 6/1/21 1:36 PM, Willian Rampazzo wrote:
> On Tue, Jun 1, 2021 at 12:46 PM Cleber Rosa <crosa@redhat.com> wrote:
>>
>> The utility function actually accepts a string, and not the raw
>> dictionary response from QMP.  Also, it returns (optionally) an
>> integer, so that must also be accounted for.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   tests/vm/basevm.py | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
>> index 6f4f0fc95e..0f2e436ed3 100644
>> --- a/tests/vm/basevm.py
>> +++ b/tests/vm/basevm.py
>> @@ -227,7 +227,7 @@ def _ssh_do(self, user, cmd, check):
>>                      "-o", "UserKnownHostsFile=" + os.devnull,
>>                      "-o",
>>                      "ConnectTimeout={}".format(self._config["ssh_timeout"]),
>> -                   "-p", self.ssh_port, "-i", self._ssh_tmp_key_file]
>> +                   "-p", str(self.ssh_port), "-i", self._ssh_tmp_key_file]
>>           # If not in debug mode, set ssh to quiet mode to
>>           # avoid printing the results of commands.
>>           if not self.debug:
>> @@ -305,7 +305,7 @@ def boot(self, img, extra_args=[]):
>>           # Init console so we can start consuming the chars.
>>           self.console_init()
>>           usernet_info = guest.qmp("human-monitor-command",
>> -                                 command_line="info usernet")
>> +                                 command_line="info usernet").get("return")
>>           self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)
> 
>  From here, your file differs from the upstream and your patch does not
> work. I'm trying on commit 52848929b70dcf92a68aedcfd90207be81ba3274.
> 

Cleber is patching against my pull request -- I'll be squashing it into 
(from PR tree) 0739780d621f8fd4163ce6969a183e59113245aa -- just waiting 
on some tests to finish running before I move ahead with it.

commit 0739780d621f8fd4163ce6969a183e59113245aa
Author: Cleber Rosa <crosa@redhat.com>
Date:   Mon Apr 12 00:46:36 2021 -0400

     Python: add utility function for retrieving port redirection

     Slightly different versions for the same utility code are currently
     present on different locations.  This unifies them all, giving
     preference to the version from virtiofs_submounts.py, because of the
     last tweaks added to it.

     While at it, this adds a "qemu.utils" module to host the utility
     function and a test.

     Signed-off-by: Cleber Rosa <crosa@redhat.com>
     Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
     Reviewed-by: Eric Auger <eric.auger@redhat.com>
     Reviewed-by: Willian Rampazzo <willianr@redhat.com>
     Message-Id: <20210412044644.55083-4-crosa@redhat.com>
     Signed-off-by: John Snow <jsnow@redhat.com>


>>           if not self.ssh_port:
>>               raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
>> --
>> 2.25.4
>>
> 

thanks,
--js


Re: [PULL 00/44] Python patches
Posted by Peter Maydell 2 years, 11 months ago
On Fri, 28 May 2021 at 03:32, John Snow <jsnow@redhat.com> wrote:
>
> The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a:
>
>   Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into staging (2021-05-27 16:32:57 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde:
>
>   gitlab: add python linters to CI (2021-05-27 18:11:17 -0400)
>
> ----------------------------------------------------------------
> Python pull request
>
> Python packaging & CI implementation
> Acceptance tests sent w/ Cleber's blessing.
> New CI tests send w/ stsquad's RB.
>
> --js
>
> ----------------------------------------------------------------

Fails to build on my machine that runs the BSD VMs, apparently
before it gets to the point of launching the VM:

make: Entering directory '/home/peter.maydell/qemu-freebsd/build'
/usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py
introspect --targets --tests --benchmarks | /usr/bin/python3 -B
scripts/mtest2make.py > Makefile.mtest
{ \
  echo 'ninja-targets = \'; \
  /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
  echo 'build-files = \'; \
  /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
outputs:/q; s/$/ \\/p'; \
} > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
(GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
(GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
/usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd
--debug --genisoimage /usr/bin/genisoimage    --image
"/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force
--build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img
Traceback (most recent call last):
  File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, in <module>
    import basevm
  File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22,
in <module>
    from qemu.machine import QEMUMachine
ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n'
/home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
failed
make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1
make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

thanks
-- PMM

Re: [PULL 00/44] Python patches
Posted by John Snow 2 years, 11 months ago
On 5/30/21 3:09 PM, Peter Maydell wrote:
> On Fri, 28 May 2021 at 03:32, John Snow <jsnow@redhat.com> wrote:
>>
>> The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a:
>>
>>    Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into staging (2021-05-27 16:32:57 +0100)
>>
>> are available in the Git repository at:
>>
>>    https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>>
>> for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde:
>>
>>    gitlab: add python linters to CI (2021-05-27 18:11:17 -0400)
>>
>> ----------------------------------------------------------------
>> Python pull request
>>
>> Python packaging & CI implementation
>> Acceptance tests sent w/ Cleber's blessing.
>> New CI tests send w/ stsquad's RB.
>>
>> --js
>>
>> ----------------------------------------------------------------
> 
> Fails to build on my machine that runs the BSD VMs, apparently
> before it gets to the point of launching the VM:
> 
> make: Entering directory '/home/peter.maydell/qemu-freebsd/build'
> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py
> introspect --targets --tests --benchmarks | /usr/bin/python3 -B
> scripts/mtest2make.py > Makefile.mtest
> { \
>    echo 'ninja-targets = \'; \
>    /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
>    echo 'build-files = \'; \
>    /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
> outputs:/q; s/$/ \\/p'; \
> } > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 dtc capstone slirp)
> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 dtc capstone slirp)
> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd
> --debug --genisoimage /usr/bin/genisoimage    --image
> "/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force
> --build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img
> Traceback (most recent call last):
>    File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, in <module>
>      import basevm
>    File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22,
> in <module>
>      from qemu.machine import QEMUMachine
> ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n'
> /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
> for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
> failed
> make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1
> make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'
> 
> thanks
> -- PMM
> 

Hmm ...

When I have seen this error message in the past, it has been because of 
using a new Python version and it gets confused reading stale cached 
information generated from an older interpreter.

None of these patches *should* have affected what version of python is 
used during the build, only the locations of some of the files. I don't 
immediately know why this PR would affect the QEMU build at all, actually.

Can you do me a favor and delete any __pycache__ folders and/or any 
*.pyc files that might be hiding in your tree and/or build folders and 
try running it again?

(If it helps, I can see about amending 'make clean' to try and clean up 
stale Python cache, which is the best solution I can think of for right 
now.)

In case that doesn't fix it, can you tell me what version of Python your 
BSD setup uses? (And what version of FreeBSD you use?) -- I'll try to 
set up a VM and see if I can reproduce the problem.

--js


Re: [PULL 00/44] Python patches
Posted by Peter Maydell 2 years, 10 months ago
On Sun, 30 May 2021 at 20:22, John Snow <jsnow@redhat.com> wrote:
>
> On 5/30/21 3:09 PM, Peter Maydell wrote:
> > Fails to build on my machine that runs the BSD VMs, apparently
> > before it gets to the point of launching the VM:

> When I have seen this error message in the past, it has been because of
> using a new Python version and it gets confused reading stale cached
> information generated from an older interpreter.

This seems like a bug in Python...

> Can you do me a favor and delete any __pycache__ folders and/or any
> *.pyc files that might be hiding in your tree and/or build folders and
> try running it again?

I tried that, and it does seem to make more progress. It now
fails with:

### Booting installer ...
DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max,sse4.2=off -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm
-drive file=/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.tmp,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -bios
pc-bios/bios-256k.bin -machine graphics=off -device VGA -cdrom
/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.install.iso
DEBUG:qemu.machine.machine:VM launch command: 'qemu-system-x86_64
-display none -vga none -chardev
socket,id=mon,path=/var/tmp/qemu-13084-monitor.sock -mon
chardev=mon,mode=control -machine pc -chardev
socket,id=console,path=/var/tmp/qemu-13084-console.sock,server=on,wait=off
-serial chardev:console -nodefaults -m 4G -cpu max,sse4.2=off -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm
-drive file=/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.tmp,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -bios
pc-bios/bios-256k.bin -machine graphics=off -device VGA -cdrom
/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.install.iso'
DEBUG:QMP.qemu-13084:>>> {'execute': 'qmp_capabilities'}
DEBUG:QMP.qemu-13084:<<< {'return': {}}
DEBUG:QMP.qemu-13084:>>> {'execute': 'human-monitor-command',
'arguments': {'command-line': 'info usernet'}}
DEBUG:QMP.qemu-13084:<<< {'return': 'Hub -1 (vnet):\r\n
Protocol[State]    FD  Source Address  Port   Dest. Address  Port
RecvQ SendQ\r\n  TCP[HOST_FORWARD]  15       127.0.0.1 38577
10.0.2.15    22     0     0\r\n'}
Failed to prepare guest environment
Traceback (most recent call last):
  File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 630, in main
    return vm.build_image(args.image)
  File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 103,
in build_image
    "-cdrom", iso
  File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 309, in boot
    self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)
  File "/home/peter.maydell/qemu-freebsd/tests/vm/../../python/qemu/utils/__init__.py",
line 40, in get_info_usernet_hostfwd_port
    for line in info_usernet_output.split('\r\n'):
AttributeError: 'dict' object has no attribute 'split'
DEBUG:QMP.qemu-13084:>>> {'execute': 'quit'}
DEBUG:QMP.qemu-13084:<<< {'return': {}}
/home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
failed
make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 2
make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

thanks
-- PMM

Re: [PULL 00/44] Python patches
Posted by John Snow 2 years, 10 months ago
On 6/1/21 6:36 AM, Peter Maydell wrote:
> On Sun, 30 May 2021 at 20:22, John Snow <jsnow@redhat.com> wrote:
>>
>> On 5/30/21 3:09 PM, Peter Maydell wrote:
>>> Fails to build on my machine that runs the BSD VMs, apparently
>>> before it gets to the point of launching the VM:
> 
>> When I have seen this error message in the past, it has been because of
>> using a new Python version and it gets confused reading stale cached
>> information generated from an older interpreter.
> 
> This seems like a bug in Python...
> 

Yes. I don't know why it's like that either...

>> Can you do me a favor and delete any __pycache__ folders and/or any
>> *.pyc files that might be hiding in your tree and/or build folders and
>> try running it again?
> 
> I tried that, and it does seem to make more progress. It now
> fails with:
> 

One of the acceptance test patches causes this, I've squashed in a fixup 
and have re-sent the PR.

Thanks,
--js


Re: [PULL 00/44] Python patches
Posted by Philippe Mathieu-Daudé 2 years, 10 months ago
On 5/30/21 9:22 PM, John Snow wrote:
> On 5/30/21 3:09 PM, Peter Maydell wrote:
>> Fails to build on my machine that runs the BSD VMs, apparently
>> before it gets to the point of launching the VM:
>>
>> make: Entering directory '/home/peter.maydell/qemu-freebsd/build'
>> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py
>> introspect --targets --tests --benchmarks | /usr/bin/python3 -B
>> scripts/mtest2make.py > Makefile.mtest
>> { \
>>    echo 'ninja-targets = \'; \
>>    /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
>>    echo 'build-files = \'; \
>>    /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
>> outputs:/q; s/$/ \\/p'; \
>> } > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
>> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
>> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
>> tests/fp/berkeley-softfloat-3 dtc capstone slirp)
>> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
>> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
>> tests/fp/berkeley-softfloat-3 dtc capstone slirp)
>> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd
>> --debug --genisoimage /usr/bin/genisoimage    --image
>> "/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force
>> --build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img
>> Traceback (most recent call last):
>>    File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21,
>> in <module>
>>      import basevm
>>    File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22,
>> in <module>
>>      from qemu.machine import QEMUMachine
>> ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n'
>> /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
>> for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
>> failed
>> make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1
>> make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

> In case that doesn't fix it, can you tell me what version of Python your
> BSD setup uses? (And what version of FreeBSD you use?) -- I'll try to
> set up a VM and see if I can reproduce the problem.

IIUC correctly this is on the host (likely Linux); no BSD VM could be
launched yet.