The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should maybe try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset so
far, but this already proves that many tests only need small modifi-
cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
classes (e.g. based on cloud-init images or using SSH) really depend
on the Avocado framework, so we'd need a solution for those if we
want to continue using them. One solution might be to simply use the
required functions from avocado.utils for these tests, and still run
them via the meson test runner instead, but that needs some further
investigation that will be done later.


Now if you want to try out these patches: Apply the patches, then
recompile and then run:

 make check-functional

You can also run single targets e.g. with:

 make check-functional-ppc

You can also run the tests without any test runner now by
setting the PYTHONPATH environment variable to the "python" folder
of your source tree, and by specifying the build directory via
QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

 export PYTHONPATH=$HOME/qemu/python
 export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
 export PYTHONPATH=$HOME/qemu/build
 ~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under
tests/functional/<testname> - console log and general logs will
be put in separate files there.

Still to be done: Update the documentation for this new test framework.

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so
  many things that I also did not dare to pick up the Reviewed-bys
  from the RFC

Thomas Huth (11):
  tests/functional: Add base classes for the upcoming pytest-based tests
  tests/functional: Convert simple avocado tests into standalone python
    tests
  tests/functional: Convert avocado tests that just need a small
    adjustment
  tests/functional: Add python-based tests to the meson build system
  tests/functional: Implement fetch_asset() method for downloading
    assets
  tests/functional: Convert some tests that download files via
    fetch_asset()
  tests/functional: Add a function for extracting files from an archive
  tests/functional: Convert some avocado tests that needed
    avocado.utils.archive
  tests/functional: Set up logging
  tests/functional: Convert the s390x avocado tests into standalone
    tests
  gitlab-ci: Add "check-functional" to the build tests

 MAINTAINERS                                   |  22 +-
 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +--
 pythondeps.toml                               |   3 +-
 tests/Makefile.include                        |  18 +-
 tests/functional/meson.build                  | 112 +++++
 tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  28 ++
 .../test_arm_canona1100.py}                   |  21 +-
 .../test_arm_n8x0.py}                         |  25 +-
 .../test_avr_mega2560.py}                     |  11 +-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  16 +-
 .../test_mem_addr_space.py}                   |  53 +--
 .../test_mips64el_loongson3v.py}              |  26 +-
 .../test_netdev_ethtool.py}                   |  32 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
 .../test_ppc_74xx.py}                         |  74 ++--
 .../test_ppc_bamboo.py}                       |  23 +-
 .../test_s390x_ccw_virtio.py}                 |  32 +-
 .../test_s390x_topology.py}                   |  70 ++--
 .../test_sparc64_sun4u.py}                    |  25 +-
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_version.py}                   |   8 +-
 tests/meson.build                             |   1 +
 28 files changed, 795 insertions(+), 320 deletions(-)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755

-- 
2.45.2

Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com> ha scritto:

> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>

I think we should take another look at that. Avocado 92 should work, though
I am not sure if it's also depending on "imp", and if I recall correctly
the problem with more recent version was more that it conflicted with
distros that didn't have it packaged. That should also not be a problem
anymore with the pythondeps.toml mechanism.

Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>

This is true.

Paolo


> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>
>
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
>
>  make check-functional
>
> You can also run single targets e.g. with:
>
>  make check-functional-ppc
>
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>
>  export PYTHONPATH=$HOME/qemu/python
>  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>  export PYTHONPATH=$HOME/qemu/build
>  ~/qemu/tests/functional/test_virtio_version.py
>
> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.
>
> Still to be done: Update the documentation for this new test framework.
>
> RFC -> v1:
> - Now using pycotap for running the tests instead of "pytest"
> - Change the name from "tests/pytest" to "tests/functional" accordingly
> - Make it possible to run the tests directly
> - Use Python's urllib instead of wget for downloading
> - Lots of makefile / meson integration improvements
> - Converted more tests
> - Update MAINTAINERS file accordingly
> - Added a patch to run check-functional in the gitlab-CI
> - ... lots of other changes I forgot about ... in fact, I changed so
>   many things that I also did not dare to pick up the Reviewed-bys
>   from the RFC
>
> Thomas Huth (11):
>   tests/functional: Add base classes for the upcoming pytest-based tests
>   tests/functional: Convert simple avocado tests into standalone python
>     tests
>   tests/functional: Convert avocado tests that just need a small
>     adjustment
>   tests/functional: Add python-based tests to the meson build system
>   tests/functional: Implement fetch_asset() method for downloading
>     assets
>   tests/functional: Convert some tests that download files via
>     fetch_asset()
>   tests/functional: Add a function for extracting files from an archive
>   tests/functional: Convert some avocado tests that needed
>     avocado.utils.archive
>   tests/functional: Set up logging
>   tests/functional: Convert the s390x avocado tests into standalone
>     tests
>   gitlab-ci: Add "check-functional" to the build tests
>
>  MAINTAINERS                                   |  22 +-
>  .gitlab-ci.d/buildtest-template.yml           |   3 +-
>  .gitlab-ci.d/buildtest.yml                    |  60 +--
>  pythondeps.toml                               |   3 +-
>  tests/Makefile.include                        |  18 +-
>  tests/functional/meson.build                  | 112 +++++
>  tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
>  tests/functional/qemu_test/utils.py           |  28 ++
>  .../test_arm_canona1100.py}                   |  21 +-
>  .../test_arm_n8x0.py}                         |  25 +-
>  .../test_avr_mega2560.py}                     |  11 +-
>  .../test_cpu_queries.py}                      |   7 +-
>  .../test_empty_cpu_model.py}                  |   7 +-
>  .../test_info_usernet.py}                     |  11 +-
>  .../test_loongarch64_virt.py}                 |  16 +-
>  .../test_mem_addr_space.py}                   |  53 +--
>  .../test_mips64el_loongson3v.py}              |  26 +-
>  .../test_netdev_ethtool.py}                   |  32 +-
>  .../test_pc_cpu_hotplug_props.py}             |  11 +-
>  .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
>  .../test_ppc_74xx.py}                         |  74 ++--
>  .../test_ppc_bamboo.py}                       |  23 +-
>  .../test_s390x_ccw_virtio.py}                 |  32 +-
>  .../test_s390x_topology.py}                   |  70 ++--
>  .../test_sparc64_sun4u.py}                    |  25 +-
>  .../version.py => functional/test_version.py} |  13 +-
>  .../test_virtio_version.py}                   |   8 +-
>  tests/meson.build                             |   1 +
>  28 files changed, 795 insertions(+), 320 deletions(-)
>  create mode 100644 tests/functional/meson.build
>  create mode 100644 tests/functional/qemu_test/__init__.py
>  create mode 100644 tests/functional/qemu_test/utils.py
>  rename tests/{avocado/machine_arm_canona1100.py =>
> functional/test_arm_canona1100.py} (71%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py}
> (71%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py}
> (90%)
>  mode change 100644 => 100755
>  rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py}
> (89%)
>  mode change 100644 => 100755
>  rename tests/{avocado/empty_cpu_model.py =>
> functional/test_empty_cpu_model.py} (84%)
>  mode change 100644 => 100755
>  rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py}
> (87%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_loongarch.py =>
> functional/test_loongarch64_virt.py} (89%)
>  mode change 100644 => 100755
>  rename tests/{avocado/mem-addr-space-check.py =>
> functional/test_mem_addr_space.py} (92%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_mips_loongson3v.py =>
> functional/test_mips64el_loongson3v.py} (55%)
>  mode change 100644 => 100755
>  rename tests/{avocado/netdev-ethtool.py =>
> functional/test_netdev_ethtool.py} (81%)
>  mode change 100644 => 100755
>  rename tests/{avocado/pc_cpu_hotplug_props.py =>
> functional/test_pc_cpu_hotplug_props.py} (90%)
>  mode change 100644 => 100755
>  rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
>  mode change 100644 => 100755
>  rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
>  mode change 100644 => 100755
>  rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py}
> (75%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_s390_ccw_virtio.py =>
> functional/test_s390x_ccw_virtio.py} (95%)
>  mode change 100644 => 100755
>  rename tests/{avocado/s390_topology.py =>
> functional/test_s390x_topology.py} (90%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_sparc64_sun4u.py =>
> functional/test_sparc64_sun4u.py} (60%)
>  mode change 100644 => 100755
>  rename tests/{avocado/version.py => functional/test_version.py} (78%)
>  mode change 100644 => 100755
>  rename tests/{avocado/virtio_version.py =>
> functional/test_virtio_version.py} (98%)
>  mode change 100644 => 100755
>
> --
> 2.45.2
>
>

On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
> 
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
> 
> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
> 
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
> 
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
> 
> 
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
> 
>  make check-functional
> 
> You can also run single targets e.g. with:
> 
>  make check-functional-ppc
> 
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
> 
>  export PYTHONPATH=$HOME/qemu/python
>  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>  export PYTHONPATH=$HOME/qemu/build
>  ~/qemu/tests/functional/test_virtio_version.py

For the whole series as is

 Tested-by: Daniel P. Berrangé <berrange@redhat.com>

as it does what you claim it does here when I tried it.

> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.

As an example, one dir name appears to be:

  __main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script
file name - in this case test_mem_addr_space.py, as I don't
want to have to lookup which class names were defined inside
each test script. We could drop the "test_" prefix from the
method name too

IOW, could we make this dir name be:

  test_mem_addr_space.phybits_ok_pentium_pae


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
...
>> So instead of trying to update the python-based test suite in QEMU
>> to a newer version of Avocado, we should maybe try to better integrate
>> it with the meson test runner instead. Indeed most tests work quite
>> nicely without the Avocado framework already, as you can see with
>> this patch series - it does not convert all tests, just a subset so
>> far, but this already proves that many tests only need small modifi-
>> cations to work without Avocado.
...
>> Now if you want to try out these patches: Apply the patches, then
>> recompile and then run:
>>
>>   make check-functional
>>
>> You can also run single targets e.g. with:
>>
>>   make check-functional-ppc
>>
>> You can also run the tests without any test runner now by
>> setting the PYTHONPATH environment variable to the "python" folder
>> of your source tree, and by specifying the build directory via
>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>
>>   export PYTHONPATH=$HOME/qemu/python
>>   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>   export PYTHONPATH=$HOME/qemu/build
>>   ~/qemu/tests/functional/test_virtio_version.py
> 
> For the whole series as is
> 
>   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
> 
> as it does what you claim it does here when I tried it.

Thanks!

>> The logs of the tests can be found in the build directory under
>> tests/functional/<testname> - console log and general logs will
>> be put in separate files there.
> 
> As an example, one dir name appears to be:
> 
>    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
> 
> I'd rather prefer it if the dir name matched the test script
> file name - in this case test_mem_addr_space.py, as I don't
> want to have to lookup which class names were defined inside
> each test script. We could drop the "test_" prefix from the
> method name too
> 
> IOW, could we make this dir name be:
> 
>    test_mem_addr_space.phybits_ok_pentium_pae

I can try to change that, indeed ... but the boilerplate code will increase 
a little bit, I guess, since I cannot simply rely on the unittest.id() 
function in that case anymore...

  Thomas

On 16/07/2024 19.03, Thomas Huth wrote:
> On 16/07/2024 18.51, Daniel P. Berrangé wrote:
>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> ...
>>> So instead of trying to update the python-based test suite in QEMU
>>> to a newer version of Avocado, we should maybe try to better integrate
>>> it with the meson test runner instead. Indeed most tests work quite
>>> nicely without the Avocado framework already, as you can see with
>>> this patch series - it does not convert all tests, just a subset so
>>> far, but this already proves that many tests only need small modifi-
>>> cations to work without Avocado.
> ...
>>> Now if you want to try out these patches: Apply the patches, then
>>> recompile and then run:
>>>
>>>   make check-functional
>>>
>>> You can also run single targets e.g. with:
>>>
>>>   make check-functional-ppc
>>>
>>> You can also run the tests without any test runner now by
>>> setting the PYTHONPATH environment variable to the "python" folder
>>> of your source tree, and by specifying the build directory via
>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>>
>>>   export PYTHONPATH=$HOME/qemu/python
>>>   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>>   export PYTHONPATH=$HOME/qemu/build
>>>   ~/qemu/tests/functional/test_virtio_version.py
>>
>> For the whole series as is
>>
>>   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>>
>> as it does what you claim it does here when I tried it.
> 
> Thanks!
> 
>>> The logs of the tests can be found in the build directory under
>>> tests/functional/<testname> - console log and general logs will
>>> be put in separate files there.
>>
>> As an example, one dir name appears to be:
>>
>>    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>>
>> I'd rather prefer it if the dir name matched the test script
>> file name - in this case test_mem_addr_space.py, as I don't
>> want to have to lookup which class names were defined inside
>> each test script. We could drop the "test_" prefix from the
>> method name too
>>
>> IOW, could we make this dir name be:
>>
>>    test_mem_addr_space.phybits_ok_pentium_pae
> 
> I can try to change that, indeed ... but the boilerplate code will increase 
> a little bit, I guess, since I cannot simply rely on the unittest.id() 
> function in that case anymore...

After looking at this for a while, I think it's maybe best to ditch the idea 
of making the .py files directly runnable and run the tests via a simple 
pycotap runner instead. Then you get proper module names:

$ pyvenv/bin/python3 -m pycotap test_virtio_version
TAP version 13
ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
1..2

  Thomas

On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
> On 16/07/2024 19.03, Thomas Huth wrote:
> > On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> > > On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> > ...
> > > > So instead of trying to update the python-based test suite in QEMU
> > > > to a newer version of Avocado, we should maybe try to better integrate
> > > > it with the meson test runner instead. Indeed most tests work quite
> > > > nicely without the Avocado framework already, as you can see with
> > > > this patch series - it does not convert all tests, just a subset so
> > > > far, but this already proves that many tests only need small modifi-
> > > > cations to work without Avocado.
> > ...
> > > > Now if you want to try out these patches: Apply the patches, then
> > > > recompile and then run:
> > > > 
> > > >   make check-functional
> > > > 
> > > > You can also run single targets e.g. with:
> > > > 
> > > >   make check-functional-ppc
> > > > 
> > > > You can also run the tests without any test runner now by
> > > > setting the PYTHONPATH environment variable to the "python" folder
> > > > of your source tree, and by specifying the build directory via
> > > > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> > > > QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
> > > > 
> > > >   export PYTHONPATH=$HOME/qemu/python
> > > >   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> > > >   export PYTHONPATH=$HOME/qemu/build
> > > >   ~/qemu/tests/functional/test_virtio_version.py
> > > 
> > > For the whole series as is
> > > 
> > >   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > as it does what you claim it does here when I tried it.
> > 
> > Thanks!
> > 
> > > > The logs of the tests can be found in the build directory under
> > > > tests/functional/<testname> - console log and general logs will
> > > > be put in separate files there.
> > > 
> > > As an example, one dir name appears to be:
> > > 
> > >    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
> > > 
> > > I'd rather prefer it if the dir name matched the test script
> > > file name - in this case test_mem_addr_space.py, as I don't
> > > want to have to lookup which class names were defined inside
> > > each test script. We could drop the "test_" prefix from the
> > > method name too
> > > 
> > > IOW, could we make this dir name be:
> > > 
> > >    test_mem_addr_space.phybits_ok_pentium_pae
> > 
> > I can try to change that, indeed ... but the boilerplate code will
> > increase a little bit, I guess, since I cannot simply rely on the
> > unittest.id() function in that case anymore...
> 
> After looking at this for a while, I think it's maybe best to ditch the idea
> of making the .py files directly runnable and run the tests via a simple
> pycotap runner instead. Then you get proper module names:

I'd really not want to loose that. To me, eliminating the test harness
entirely when debugging is the single biggest improvement of this new
approach, especially when I want to 'strace' the test without
extraneous processes.

> $ pyvenv/bin/python3 -m pycotap test_virtio_version
> TAP version 13
> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
> 1..

With the following change, you get the same output with direct
execution, by making argv look the same as you'd get when
running your pycotap example.

diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
index cc49fd4c94..3a3e65252d 100644
--- a/tests/functional/qemu_test/__init__.py
+++ b/tests/functional/qemu_test/__init__.py
@@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        unittest.main(testRunner = tr)
+        import sys
+        import os.path
+        path = os.path.basename(sys.argv[0])[:-3]
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
 
 
 class QemuSystemTest(QemuBaseTest):

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 17/07/2024 10.37, Daniel P. Berrangé wrote:
> On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
>> On 16/07/2024 19.03, Thomas Huth wrote:
>>> On 16/07/2024 18.51, Daniel P. Berrangé wrote:
>>>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
>>> ...
>>>>> So instead of trying to update the python-based test suite in QEMU
>>>>> to a newer version of Avocado, we should maybe try to better integrate
>>>>> it with the meson test runner instead. Indeed most tests work quite
>>>>> nicely without the Avocado framework already, as you can see with
>>>>> this patch series - it does not convert all tests, just a subset so
>>>>> far, but this already proves that many tests only need small modifi-
>>>>> cations to work without Avocado.
>>> ...
>>>>> Now if you want to try out these patches: Apply the patches, then
>>>>> recompile and then run:
>>>>>
>>>>>    make check-functional
>>>>>
>>>>> You can also run single targets e.g. with:
>>>>>
>>>>>    make check-functional-ppc
>>>>>
>>>>> You can also run the tests without any test runner now by
>>>>> setting the PYTHONPATH environment variable to the "python" folder
>>>>> of your source tree, and by specifying the build directory via
>>>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>>>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>>>>
>>>>>    export PYTHONPATH=$HOME/qemu/python
>>>>>    export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>>>>    export PYTHONPATH=$HOME/qemu/build
>>>>>    ~/qemu/tests/functional/test_virtio_version.py
>>>>
>>>> For the whole series as is
>>>>
>>>>    Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>
>>>> as it does what you claim it does here when I tried it.
>>>
>>> Thanks!
>>>
>>>>> The logs of the tests can be found in the build directory under
>>>>> tests/functional/<testname> - console log and general logs will
>>>>> be put in separate files there.
>>>>
>>>> As an example, one dir name appears to be:
>>>>
>>>>     __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>>>>
>>>> I'd rather prefer it if the dir name matched the test script
>>>> file name - in this case test_mem_addr_space.py, as I don't
>>>> want to have to lookup which class names were defined inside
>>>> each test script. We could drop the "test_" prefix from the
>>>> method name too
>>>>
>>>> IOW, could we make this dir name be:
>>>>
>>>>     test_mem_addr_space.phybits_ok_pentium_pae
>>>
>>> I can try to change that, indeed ... but the boilerplate code will
>>> increase a little bit, I guess, since I cannot simply rely on the
>>> unittest.id() function in that case anymore...
>>
>> After looking at this for a while, I think it's maybe best to ditch the idea
>> of making the .py files directly runnable and run the tests via a simple
>> pycotap runner instead. Then you get proper module names:
> 
> I'd really not want to loose that. To me, eliminating the test harness
> entirely when debugging is the single biggest improvement of this new
> approach, especially when I want to 'strace' the test without
> extraneous processes.
> 
>> $ pyvenv/bin/python3 -m pycotap test_virtio_version
>> TAP version 13
>> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
>> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
>> 1..
> 
> With the following change, you get the same output with direct
> execution, by making argv look the same as you'd get when
> running your pycotap example.
> 
> diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
> index cc49fd4c94..3a3e65252d 100644
> --- a/tests/functional/qemu_test/__init__.py
> +++ b/tests/functional/qemu_test/__init__.py
> @@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
>       def main():
>           tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
>                                      test_output_log = pycotap.LogMode.LogToError)
> -        unittest.main(testRunner = tr)
> +        import sys
> +        import os.path
> +        path = os.path.basename(sys.argv[0])[:-3]
> +        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])

Sweet, thank you very much, looks like this will do the job!

  Thomas