Testing 32-bit host OS support takes a lot of precious time during the QEMU
contiguous integration tests, and considering that many OS vendors stopped
shipping 32-bit variants of their OS distributions and most hardware from
the past >10 years is capable of 64-bit, keeping the 32-bit support alive
is an inadequate burden for the QEMU project. Let's mark the 32-bit
support as deprecated so we can drop it after a while - this will help
us to cut down our limited CI minutes in the gitlab CI, for example.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 docs/about/deprecated.rst | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index 9f1bbc495d..ce6463e72b 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -181,9 +181,20 @@ As Debian 10 ("Buster") moved into LTS the big endian 32 bit version of
 MIPS moved out of support making it hard to maintain our
 cross-compilation CI tests of the architecture. As we no longer have
 CI coverage support may bitrot away before the deprecation process
-completes. The little endian variants of MIPS (both 32 and 64 bit) are
+completes. The little endian variants of MIPS are
 still a supported host architecture.
 
+32-bit host operating systems (since 8.0)
+'''''''''''''''''''''''''''''''''''''''''
+
+Testing 32-bit host OS support takes a lot of precious time during the QEMU
+contiguous integration tests, and considering that many OS vendors stopped
+shipping 32-bit variants of their OS distributions and most hardware from
+the past >10 years is capable of 64-bit, keeping the 32-bit support alive
+is an inadequate burden for the QEMU project. Thus QEMU will soon drop the
+support for 32-bit host systems.
+
+
 QEMU API (QAPI) events
 ----------------------
 
-- 
2.31.1

On 1/30/23 12:44, Thomas Huth wrote:
> Testing 32-bit host OS support takes a lot of precious time during the QEMU
> contiguous integration tests, and considering that many OS vendors stopped
> shipping 32-bit variants of their OS distributions and most hardware from
> the past >10 years is capable of 64-bit, keeping the 32-bit support alive
> is an inadequate burden for the QEMU project. Let's mark the 32-bit
> support as deprecated so we can drop it after a while - this will help
> us to cut down our limited CI minutes in the gitlab CI, for example.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  docs/about/deprecated.rst | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
> index 9f1bbc495d..ce6463e72b 100644
> --- a/docs/about/deprecated.rst
> +++ b/docs/about/deprecated.rst
> @@ -181,9 +181,20 @@ As Debian 10 ("Buster") moved into LTS the big endian 32 bit version of
>  MIPS moved out of support making it hard to maintain our
>  cross-compilation CI tests of the architecture. As we no longer have
>  CI coverage support may bitrot away before the deprecation process
> -completes. The little endian variants of MIPS (both 32 and 64 bit) are
> +completes. The little endian variants of MIPS are
>  still a supported host architecture.
>  
> +32-bit host operating systems (since 8.0)
> +'''''''''''''''''''''''''''''''''''''''''
> +
> +Testing 32-bit host OS support takes a lot of precious time during the QEMU
> +contiguous integration tests, and considering that many OS vendors stopped
> +shipping 32-bit variants of their OS distributions and most hardware from
> +the past >10 years is capable of 64-bit, keeping the 32-bit support alive
> +is an inadequate burden for the QEMU project. Thus QEMU will soon drop the
> +support for 32-bit host systems.
> +
> +
>  QEMU API (QAPI) events
>  ----------------------
>  

From our (SUSE) support perspective, we do not support 32bit virtualization hosts downstream,

and in general I am in favor of the change.

Ciao,

Claudio

I feel the discussion petered out without a conclusion.

I don't think letting the status quo win by inertia is a good outcome
here.

Which 32-bit hosts are still useful, and why?

Please note my question is not about the cost of keeping them (or
savings from not keeping them), it's about the value they provide.  When
value rounds to zero, cost is irrelevant, so let's get a firm idea of
value *first*.

On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> I feel the discussion petered out without a conclusion.
> 
> I don't think letting the status quo win by inertia is a good outcome
> here.
> 
> Which 32-bit hosts are still useful, and why?

NetBSD runs on a bunch of 32 bit-only hosts (sparc32, ppc32, armv7, vax,
mips32 etc.) that all run Qemu fine. They are all actively maintained and
released as part of the main releases.

Maintaining 32 bit host support is thus vital for those machines; not everyone
runs a 64 bit system.

with regards,
Reinoud Zandijk,
NetBSD developer.

On 2/17/23 06:06, Reinoud Zandijk wrote:
> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>> I feel the discussion petered out without a conclusion.
>>
>> I don't think letting the status quo win by inertia is a good outcome
>> here.
>>
>> Which 32-bit hosts are still useful, and why?
> 
> NetBSD runs on a bunch of 32 bit-only hosts (sparc32, ppc32, armv7, vax,
> mips32 etc.) that all run Qemu fine. They are all actively maintained and
> released as part of the main releases.

Are you sure about that?  TCG doesn't support sparc32 or vax.
I suppose you could be using TCI, but I can't even imagine how
slow that would be on vax.


r~

On Fri, Feb 17, 2023 at 08:22:43AM -1000, Richard Henderson wrote:
> On 2/17/23 06:06, Reinoud Zandijk wrote:
> > On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> > > I feel the discussion petered out without a conclusion.
> > > 
> > > I don't think letting the status quo win by inertia is a good outcome
> > > here.
> > > 
> > > Which 32-bit hosts are still useful, and why?
> > 
> > NetBSD runs on a bunch of 32 bit-only hosts (sparc32, ppc32, armv7, vax,
> > mips32 etc.) that all run Qemu fine. They are all actively maintained and
> > released as part of the main releases.
> 
> Are you sure about that?  TCG doesn't support sparc32 or vax.
> I suppose you could be using TCI, but I can't even imagine how
> slow that would be on vax.

I've asked around amongst active developers and they steted that they had used
it in the past but not ianymore recently. I doubt normal users would use them
often though since most hardware is 64 bit capable anyways.

Reinoud

On Fri, Feb 17, 2023 at 05:06:42PM +0100, Reinoud Zandijk wrote:
> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> > I feel the discussion petered out without a conclusion.
> > 
> > I don't think letting the status quo win by inertia is a good outcome
> > here.
> > 
> > Which 32-bit hosts are still useful, and why?
> 
> NetBSD runs on a bunch of 32 bit-only hosts (sparc32, ppc32, armv7, vax,
> mips32 etc.) that all run Qemu fine. They are all actively maintained and
> released as part of the main releases.
> 
> Maintaining 32 bit host support is thus vital for those machines; not everyone
> runs a 64 bit system.

In the context of NetBSD, is QEMU a leaf node package, or is it a
dependancy of anything involved in delivery of the distro ?

ie, if QEMU were to drop 32-bit support, if it is merely a leaf node
package, then end users of NetBSD would not have access to QEMU, but
you would still be able to deliver the rest of NetBSD on those platforms
without the QEMU package present ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> I feel the discussion petered out without a conclusion.
> 
> I don't think letting the status quo win by inertia is a good outcome
> here.
> 
> Which 32-bit hosts are still useful, and why?

Which 32-bit hosts does Linux still provide KVM  support for.

If any, is there an EOL date for Linux 32-bit KVM support ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 2/17/23 11:47, Daniel P. Berrangé wrote:
> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>> I feel the discussion petered out without a conclusion.
>>
>> I don't think letting the status quo win by inertia is a good outcome
>> here.
>>
>> Which 32-bit hosts are still useful, and why?
>
> Which 32-bit hosts does Linux still provide KVM  support for.

All except ARM: MIPS, x86, PPC and RISC-V.

I would like to remove x86, but encountered some objections.

MIPS, nobody is really using it I think.

So that leaves PPC and RISC-V.

> If any, is there an EOL date for Linux 32-bit KVM support ?

No, and I don't think there's going to be one.

Paolo

On 17/02/2023 17.38, Paolo Bonzini wrote:
> On 2/17/23 11:47, Daniel P. Berrangé wrote:
>> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>>> I feel the discussion petered out without a conclusion.
>>>
>>> I don't think letting the status quo win by inertia is a good outcome
>>> here.
>>>
>>> Which 32-bit hosts are still useful, and why?
>>
>> Which 32-bit hosts does Linux still provide KVM  support for.
> 
> All except ARM: MIPS, x86, PPC and RISC-V.
> 
> I would like to remove x86, but encountered some objections.

So if I got that right:

 
https://lore.kernel.org/all/b8fa9561295bb6af2b7fcaa8125c6a3b89b305c7.camel@redhat.com/

... the objection is mainly that some kernel developers want to keep the 
code around for easier testing of nested 32-bit guests L1 hypervisors.

If that's the only use case that is still around for the 32-bit KVM x86 
kernel code, I guess it should also be fine to use older versions of QEMU in 
those L1 hypervisor guests (assuming you have to use an older 32-bit Linux 
distro for this anyway).

So unless I got that wrong, there is really nobody around anymore who needs 
an *upstream* QEMU for running 32-bit x86 KVM hosts, is there?

Please correct me if I'm wrong, but I think we can really deprecated at 
least qemu-system-i386 and qemu-system-arm now, can't we?

  Thanks,
   Thomas

(Cc'ing Huacai & Jiaxun).

On 17/2/23 17:38, Paolo Bonzini wrote:
> On 2/17/23 11:47, Daniel P. Berrangé wrote:
>> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>>> I feel the discussion petered out without a conclusion.
>>>
>>> I don't think letting the status quo win by inertia is a good outcome
>>> here.
>>>
>>> Which 32-bit hosts are still useful, and why?
>>
>> Which 32-bit hosts does Linux still provide KVM  support for.
> 
> All except ARM: MIPS, x86, PPC and RISC-V.
> 
> I would like to remove x86, but encountered some objections.
> 
> MIPS, nobody is really using it I think.

32-bit was added in 2014, commit 222e7d11e7 ("target-mips: Enable KVM
support in build system"). I'm not aware of anybody using it (even
testing it). I don't have hardware to test it (neither time).
We are still cross-compiling it although.

64-bit support was added recently (see commit aa2953fd16 "configure:
Add KVM target support for MIPS64") and is used (see commit fbc5884ce2
"meson.build: Re-enable KVM support for MIPS" from 2020), however I
tend to see it more as hobbyist use than production one. Besides it
is listed as 'Odd Fixes' in MAINTAINERS (still 2020, commit 134f7f7da1
"MAINTAINERS: Reactivate MIPS KVM CPUs").

> So that leaves PPC and RISC-V.
> 
>> If any, is there an EOL date for Linux 32-bit KVM support ?
> 
> No, and I don't think there's going to be one.
> 
> Paolo
> 

On 17/02/2023 18.43, Philippe Mathieu-Daudé wrote:
> (Cc'ing Huacai & Jiaxun).
> 
> On 17/2/23 17:38, Paolo Bonzini wrote:
>> On 2/17/23 11:47, Daniel P. Berrangé wrote:
>>> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>>>> I feel the discussion petered out without a conclusion.
>>>>
>>>> I don't think letting the status quo win by inertia is a good outcome
>>>> here.
>>>>
>>>> Which 32-bit hosts are still useful, and why?
>>>
>>> Which 32-bit hosts does Linux still provide KVM  support for.
>>
>> All except ARM: MIPS, x86, PPC and RISC-V.
>>
>> I would like to remove x86, but encountered some objections.
>>
>> MIPS, nobody is really using it I think.
> 
> 32-bit was added in 2014, commit 222e7d11e7 ("target-mips: Enable KVM
> support in build system"). I'm not aware of anybody using it (even
> testing it). I don't have hardware to test it (neither time).

Could you maybe suggest a kernel patch to remove it, to see what happens? 
... if nobody objects to the removal of the 32-bit MIPS KVM kernel support 
and the patch gets merged, that would help us in the long run, I think.

  Thanks,
   Thomas

在2023年2月17日二月 下午6:57，Thomas Huth写道：
> On 17/02/2023 18.43, Philippe Mathieu-Daudé wrote:
>> (Cc'ing Huacai & Jiaxun).
>> 
>> On 17/2/23 17:38, Paolo Bonzini wrote:
>>> On 2/17/23 11:47, Daniel P. Berrangé wrote:
>>>> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>>>>> I feel the discussion petered out without a conclusion.
>>>>>
>>>>> I don't think letting the status quo win by inertia is a good outcome
>>>>> here.
>>>>>
>>>>> Which 32-bit hosts are still useful, and why?
>>>>
>>>> Which 32-bit hosts does Linux still provide KVM  support for.
>>>
>>> All except ARM: MIPS, x86, PPC and RISC-V.
>>>
>>> I would like to remove x86, but encountered some objections.
>>>
>>> MIPS, nobody is really using it I think.
>> 
>> 32-bit was added in 2014, commit 222e7d11e7 ("target-mips: Enable KVM
>> support in build system"). I'm not aware of anybody using it (even
>> testing it). I don't have hardware to test it (neither time).
>
> Could you maybe suggest a kernel patch to remove it, to see what happens? 
> ... if nobody objects to the removal of the 32-bit MIPS KVM kernel support 
> and the patch gets merged, that would help us in the long run, I think.

I’m still occasionally testing 32-bit MIPS KVM support with MIPS P5600.

It works just fine so there is no need for further maintenance work.

I’d be sad to see the support go but I can live with it.

There are commercial users for MIPS 32 KVM but they’re all running customized downstream QEMU so I guess it’s fine.

Thanks
- Jiaxun

>
>   Thanks,
>    Thomas

-- 
- Jiaxun

On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:

> Which 32-bit hosts are still useful, and why?


Citing my previous mail:

    I now checked all downloads of the latests installers since 2022-12-30.

    qemu-w32-setup-20221230.exe – 509 different IP addresses
    qemu-w64-setup-20221230.exe - 5471 different IP addresses

    339 unique IP addresses are common for 32- and 64-bit, either
    crawlers or people who simply got both variants. So there remain 170
    IP addresses which only downloaded the 32-bit variant in the last week.

    I see 437 different strings for the browser type, but surprisingly
    none of them looks like a crawler.

So there still seems to be a certain small need for QEMU installers for 
32-bit Windows: 170 users für 32 bit only, 339 users for both 32 and 64 
bit, 5132 users for 64 bit only.

Stefan

On Fri, Feb 17, 2023 at 12:05:46PM +0100, Stefan Weil wrote:
> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> 
> > Which 32-bit hosts are still useful, and why?
> 
> Citing my previous mail:
> 
>    I now checked all downloads of the latests installers since 2022-12-30.
> 
>    qemu-w32-setup-20221230.exe – 509 different IP addresses
>    qemu-w64-setup-20221230.exe - 5471 different IP addresses
> 
>    339 unique IP addresses are common for 32- and 64-bit, either
>    crawlers or people who simply got both variants. So there remain 170
>    IP addresses which only downloaded the 32-bit variant in the last week.
> 
>    I see 437 different strings for the browser type, but surprisingly
>    none of them looks like a crawler.
> 
> So there still seems to be a certain small need for QEMU installers for
> 32-bit Windows: 170 users für 32 bit only, 339 users for both 32 and 64 bit,
> 5132 users for 64 bit only.

As you seem to have access to download stats could you check generic download
stats too i.e. not only for Windows installers?

Also, how do you account for all 32 bit host downloads from say mirrors or the
source code downloads that are compiled and redistributed by 3rd parties for
32 bit hosts?

Although I am not that happy with dropping support on any platform I don't
think such a decision should be taken that lightly.

Maybe a change in the test setup would be handier? I.e. only test 32 bit hosts
say once a 2 days and the 64 bit hosts each day?

Reinoud

Am 19.02.23 um 12:27 schrieb Reinoud Zandijk:

> On Fri, Feb 17, 2023 at 12:05:46PM +0100, Stefan Weil wrote:
>> So there still seems to be a certain small need for QEMU installers for
>> 32-bit Windows: 170 users für 32 bit only, 339 users for both 32 and 64 bit,
>> 5132 users for 64 bit only.
> As you seem to have access to download stats could you check generic download
> stats too i.e. not only for Windows installers?


Sorry, but I only have statistics for my own server 
https://qemu.weilnetz.de/ which provides Windows installers.

Stefan

On Fri, Feb 17, 2023 at 12:05:46PM +0100, Stefan Weil wrote:
> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> 
> > Which 32-bit hosts are still useful, and why?
> 
> 
> Citing my previous mail:
> 
>    I now checked all downloads of the latests installers since 2022-12-30.
> 
>    qemu-w32-setup-20221230.exe – 509 different IP addresses
>    qemu-w64-setup-20221230.exe - 5471 different IP addresses
> 
>    339 unique IP addresses are common for 32- and 64-bit, either
>    crawlers or people who simply got both variants. So there remain 170
>    IP addresses which only downloaded the 32-bit variant in the last week.
> 
>    I see 437 different strings for the browser type, but surprisingly
>    none of them looks like a crawler.
> 
> So there still seems to be a certain small need for QEMU installers for
> 32-bit Windows: 170 users für 32 bit only, 339 users for both 32 and 64 bit,
> 5132 users for 64 bit only.

The question which is hard/impossible to answer is whether the people
who downloaded the 32-bit build genuinely needed a 32-bit build or
just did so out of habit or confusion.

I know you can't believe everything you see with statistics, but as an
example, the chart at the bottom of this page suggests new deployments
of 32-bit Windows are negligible today:

    https://www.pcbenchmarks.net/os-marketshare.html

there are existing deployments not accounted for there, but that may
still suggest many of the 32-bit downloads of QEMU will end up being
run on 64-bit hosts.

If we were to apply our support platform rule of only targetting the
latest 2 versions of the OS, this limits our targets to Win 10 and
Win 11. Windows 11 dropped 32-bit IIUC, so we're talking about
32-bit installs of Windows 10 only - even in Win10 days all new
physical hardware would have been 64-bit capable.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Stefan Weil <sw@weilnetz.de> writes:

> On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
>
>> Which 32-bit hosts are still useful, and why?
>
>
> Citing my previous mail:
>
>    I now checked all downloads of the latests installers since 2022-12-30.
>
>    qemu-w32-setup-20221230.exe – 509 different IP addresses
>    qemu-w64-setup-20221230.exe - 5471 different IP addresses
>
>    339 unique IP addresses are common for 32- and 64-bit, either
>    crawlers or people who simply got both variants. So there remain 170
>    IP addresses which only downloaded the 32-bit variant in the last week.
>
>    I see 437 different strings for the browser type, but surprisingly
>    none of them looks like a crawler.
>
> So there still seems to be a certain small need for QEMU installers for 32-bit Windows: 170 users für 32 bit only, 339 users for both 32 and 64 bit, 5132 users for 64 bit only.

Actual data is always welcome!

I wonder, though...  given how decrepit 32-bit-only PCs must now be, and
how well Windows 10 (the only version still supporting them) runs on
these, how many downloaders of the w32 version could (and quite probably
should) use the w64 version instead?

On 2/17/23 11:36, Markus Armbruster wrote:
> I feel the discussion petered out without a conclusion.
> 
> I don't think letting the status quo win by inertia is a good outcome
> here.
> 
> Which 32-bit hosts are still useful, and why?

Hi Markus,
 
if the question is very very general, my opinion is that some of the most useful 32-bit systems are things like ARMv8-R capable systems for deeply embedded real time applications.

Ciao,

Claudio

On Fri, Feb 17, 2023 at 11:36:41AM +0100, Markus Armbruster wrote:
> I feel the discussion petered out without a conclusion.
> 
> I don't think letting the status quo win by inertia is a good outcome
> here.
> 
> Which 32-bit hosts are still useful, and why?
> 
> Please note my question is not about the cost of keeping them (or
> savings from not keeping them), it's about the value they provide.  When
> value rounds to zero, cost is irrelevant, so let's get a firm idea of
> value *first*.

With my OS maintainer hat on, 32-bit host support is not relevant
to Fedora[1] or RHEL. Both have dropped all 32-bit host install
support, only maintaining 32-bit builds of some libaries for use
on 64-bit host install, which is not relevant to QEMU's deliverables.

With regards,
Daniel

[1] Arm7 was the last 32-bit host in Fedora, dropped after F36
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
>
> Testing 32-bit host OS support takes a lot of precious time during the QEMU
> contiguous integration tests, and considering that many OS vendors stopped
> shipping 32-bit variants of their OS distributions and most hardware from
> the past >10 years is capable of 64-bit

True for x86, not necessarily true for other architectures.
Are you proposing to deprecate x86 32-bit, or all 32-bit?
I'm not entirely sure about whether we're yet at a point where
I'd want to deprecate-and-drop 32-bit arm host support.

thanks
-- PMM

On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
> >
> > Testing 32-bit host OS support takes a lot of precious time during the QEMU
> > contiguous integration tests, and considering that many OS vendors stopped
> > shipping 32-bit variants of their OS distributions and most hardware from
> > the past >10 years is capable of 64-bit
> 
> True for x86, not necessarily true for other architectures.
> Are you proposing to deprecate x86 32-bit, or all 32-bit?
> I'm not entirely sure about whether we're yet at a point where
> I'd want to deprecate-and-drop 32-bit arm host support.

Do we have a feeling on which aspects of 32-bit cause us the support
burden ? The boring stuff like compiler errors from mismatched integer
sizes is mostly quick & easy to detect simply through a cross compile.

I vaguely recall someone mentioned problems with atomic ops in the past,
or was it 128-bit ints, caused implications for the codebase ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
>> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
>> >
>> > Testing 32-bit host OS support takes a lot of precious time during the QEMU
>> > contiguous integration tests, and considering that many OS vendors stopped
>> > shipping 32-bit variants of their OS distributions and most hardware from
>> > the past >10 years is capable of 64-bit
>> 
>> True for x86, not necessarily true for other architectures.
>> Are you proposing to deprecate x86 32-bit, or all 32-bit?
>> I'm not entirely sure about whether we're yet at a point where
>> I'd want to deprecate-and-drop 32-bit arm host support.
>
> Do we have a feeling on which aspects of 32-bit cause us the support
> burden ? The boring stuff like compiler errors from mismatched integer
> sizes is mostly quick & easy to detect simply through a cross compile.
>
> I vaguely recall someone mentioned problems with atomic ops in the past,
> or was it 128-bit ints, caused implications for the codebase ?

Atomic operations on > TARGET_BIT_SIZE and cputlb when
TCG_OVERSIZED_GUEST is set. Also the core TCG code and a bunch of the
backends have TARGET_LONG_BITS > TCG_TARGET_REG_BITS ifdefs peppered
throughout.

>
> With regards,
> Daniel


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Am 30. Januar 2023 20:45:47 UTC schrieb "Alex Bennée" <alex.bennee@linaro.org>:
>
>Daniel P. Berrangé <berrange@redhat.com> writes:
>
>> On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
>>> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
>>> >
>>> > Testing 32-bit host OS support takes a lot of precious time during the QEMU
>>> > contiguous integration tests, and considering that many OS vendors stopped
>>> > shipping 32-bit variants of their OS distributions and most hardware from
>>> > the past >10 years is capable of 64-bit
>>> 
>>> True for x86, not necessarily true for other architectures.
>>> Are you proposing to deprecate x86 32-bit, or all 32-bit?
>>> I'm not entirely sure about whether we're yet at a point where
>>> I'd want to deprecate-and-drop 32-bit arm host support.
>>
>> Do we have a feeling on which aspects of 32-bit cause us the support
>> burden ? The boring stuff like compiler errors from mismatched integer
>> sizes is mostly quick & easy to detect simply through a cross compile.
>>
>> I vaguely recall someone mentioned problems with atomic ops in the past,
>> or was it 128-bit ints, caused implications for the codebase ?
>
>Atomic operations on > TARGET_BIT_SIZE and cputlb when
>TCG_OVERSIZED_GUEST is set. Also the core TCG code and a bunch of the
>backends have TARGET_LONG_BITS > TCG_TARGET_REG_BITS ifdefs peppered
>throughout.

Are there any plans or ideas to support 128 bit architectures such as CHERI in the future? There is already a QEMU fork implementing CHERI for RISC V [1]. Also ARM has developed an experimental hardware implementation of CHERI within the Morello project where Linaro is involved as well, although the QEMU implementation is performed by the University of Cambridge [2].

I'm asking because once we deeply bake in the assumption that host size >= guest size then adding such architectures will become much harder.

Best regards,
Bernhard

[1] https://github.com/CTSRD-CHERI/qemu
[2] https://git.morello-project.org/university-of-cambridge/mirrors/qemu/-/tree/qemu-morello-rebased

>
>>
>> With regards,
>> Daniel
>
>

On Wed, Feb 22, 2023 at 09:11:13AM +0000, Bernhard Beschow wrote:
> 
> 
> Am 30. Januar 2023 20:45:47 UTC schrieb "Alex Bennée" <alex.bennee@linaro.org>:
> >
> >Daniel P. Berrangé <berrange@redhat.com> writes:
> >
> >> On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
> >>> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
> >>> >
> >>> > Testing 32-bit host OS support takes a lot of precious time during the QEMU
> >>> > contiguous integration tests, and considering that many OS vendors stopped
> >>> > shipping 32-bit variants of their OS distributions and most hardware from
> >>> > the past >10 years is capable of 64-bit
> >>> 
> >>> True for x86, not necessarily true for other architectures.
> >>> Are you proposing to deprecate x86 32-bit, or all 32-bit?
> >>> I'm not entirely sure about whether we're yet at a point where
> >>> I'd want to deprecate-and-drop 32-bit arm host support.
> >>
> >> Do we have a feeling on which aspects of 32-bit cause us the support
> >> burden ? The boring stuff like compiler errors from mismatched integer
> >> sizes is mostly quick & easy to detect simply through a cross compile.
> >>
> >> I vaguely recall someone mentioned problems with atomic ops in the past,
> >> or was it 128-bit ints, caused implications for the codebase ?
> >
> >Atomic operations on > TARGET_BIT_SIZE and cputlb when
> >TCG_OVERSIZED_GUEST is set. Also the core TCG code and a bunch of the
> >backends have TARGET_LONG_BITS > TCG_TARGET_REG_BITS ifdefs peppered
> >throughout.
> 
> Are there any plans or ideas to support 128 bit architectures
> such as CHERI in the future? There is already a QEMU fork
> implementing CHERI for RISC V [1]. Also ARM has developed an
> experimental hardware implementation of CHERI within the Morello
> project where Linaro is involved as well, although the QEMU
> implementation is performed by the University of Cambridge [2].

If 128 bit hardware exists and has real world non-toy usage,
then a request to support it in QEMU is essentially inevitable.

> I'm asking because once we deeply bake in the assumption that
> host size >= guest size then adding such architectures will
> become much harder.

Yep, that is a risk.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Wed, Feb 22, 2023 at 09:51:57AM +0000, Daniel P. Berrangé wrote:
> On Wed, Feb 22, 2023 at 09:11:13AM +0000, Bernhard Beschow wrote:
> > Are there any plans or ideas to support 128 bit architectures
> > such as CHERI in the future? There is already a QEMU fork
> > implementing CHERI for RISC V [1]. Also ARM has developed an
> > experimental hardware implementation of CHERI within the Morello
> > project where Linaro is involved as well, although the QEMU
> > implementation is performed by the University of Cambridge [2].
> 
> If 128 bit hardware exists and has real world non-toy usage,
> then a request to support it in QEMU is essentially inevitable.
> 
> > I'm asking because once we deeply bake in the assumption that
> > host size >= guest size then adding such architectures will
> > become much harder.
> 
> Yep, that is a risk.

I can second that. Better keep it in the code and deal with it. Maybe those
specific parts can be implemented in such a way that it can easily become a
noop on host size >= guest size.

With regards,
Reinoud

Reinoud Zandijk <reinoud@NetBSD.org> writes:

> On Wed, Feb 22, 2023 at 09:51:57AM +0000, Daniel P. Berrangé wrote:
>> On Wed, Feb 22, 2023 at 09:11:13AM +0000, Bernhard Beschow wrote:
>> > Are there any plans or ideas to support 128 bit architectures
>> > such as CHERI in the future? There is already a QEMU fork
>> > implementing CHERI for RISC V [1]. Also ARM has developed an
>> > experimental hardware implementation of CHERI within the Morello
>> > project where Linaro is involved as well, although the QEMU
>> > implementation is performed by the University of Cambridge [2].
>> 
>> If 128 bit hardware exists and has real world non-toy usage,
>> then a request to support it in QEMU is essentially inevitable.
>> 
>> > I'm asking because once we deeply bake in the assumption that
>> > host size >= guest size then adding such architectures will
>> > become much harder.
>> 
>> Yep, that is a risk.
>
> I can second that. Better keep it in the code and deal with it. Maybe those
> specific parts can be implemented in such a way that it can easily become a
> noop on host size >= guest size.

AIUI these don't expose bigger addresses or natural atomic sizes which
is where things get tricky for the TCG. All the extra bits are used for
authentication or permission checks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

On 30/01/2023 20:45, Alex Bennée wrote:

> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
>> On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
>>> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
>>>>
>>>> Testing 32-bit host OS support takes a lot of precious time during the QEMU
>>>> contiguous integration tests, and considering that many OS vendors stopped
>>>> shipping 32-bit variants of their OS distributions and most hardware from
>>>> the past >10 years is capable of 64-bit
>>>
>>> True for x86, not necessarily true for other architectures.
>>> Are you proposing to deprecate x86 32-bit, or all 32-bit?
>>> I'm not entirely sure about whether we're yet at a point where
>>> I'd want to deprecate-and-drop 32-bit arm host support.
>>
>> Do we have a feeling on which aspects of 32-bit cause us the support
>> burden ? The boring stuff like compiler errors from mismatched integer
>> sizes is mostly quick & easy to detect simply through a cross compile.
>>
>> I vaguely recall someone mentioned problems with atomic ops in the past,
>> or was it 128-bit ints, caused implications for the codebase ?
> 
> Atomic operations on > TARGET_BIT_SIZE and cputlb when
> TCG_OVERSIZED_GUEST is set. Also the core TCG code and a bunch of the
> backends have TARGET_LONG_BITS > TCG_TARGET_REG_BITS ifdefs peppered
> throughout.

I am one of an admittedly small group of people still interested in using KVM-PR on 
ppc32 to boot MacOS, although there is some interest on using 64-bit KVM-PR to run 
super-fast MacOS on modern Talos hardware.

 From my perspective losing the ability to run 64-bit guests on 32-bit hardware with 
TCG wouldn't be an issue, as long as it were still possible to use qemu-system-ppc on 
32-bit hardware using both TCG and KVM to help debug the remaining issues.


ATB,

Mark.

On 1/30/23 02:01, Daniel P. Berrangé wrote:
> I vaguely recall someone mentioned problems with atomic ops in the past,
> or was it 128-bit ints, caused implications for the codebase ?

TCG_OVERSIZED_GUEST, where 32-bit host running 64-bit guest cannot run the guest except in 
round-robin mode.  It's not *that* much of a support burden, since we're not going to 
eliminate round-robin mode.

But I do question whether we need to support 64-bit guests on 32-bit hosts at all.
Retaining 32-bit on 32-bit allows arm32 to emulate i686, which I suspect, but have no 
proof, is the limit of what users actually want.


r~

On 30/1/23 20:19, Richard Henderson wrote:

> But I do question whether we need to support 64-bit guests on 32-bit 
> hosts at all.
> Retaining 32-bit on 32-bit allows arm32 to emulate i686, which I 
> suspect, but have no proof, is the limit of what users actually want.

I presume you implicitly restrict that to user emulation, right?

WRT i686, if your example is "i686 useremu on non-x86 embedde router"
then any 32-bit host is potentially interested, not only arm32.

I remember being able to run armhf binaries on armel hosts (and vice
versa) was useful 7 years ago. Today I have no clue, we could poll the 
community and some distributions.

On 1/30/23 13:14, Philippe Mathieu-Daudé wrote:
> On 30/1/23 20:19, Richard Henderson wrote:
> 
>> But I do question whether we need to support 64-bit guests on 32-bit hosts at all.
>> Retaining 32-bit on 32-bit allows arm32 to emulate i686, which I suspect, but have no 
>> proof, is the limit of what users actually want.
> 
> I presume you implicitly restrict that to user emulation, right?

No, there's no specific reason to eliminate e.g. qemu-system-i386. or any other 32-bit 
guest.  Though quite often such hardware doesn't really have enough ram to do a good job, 
that's not a technical argument against.


> WRT i686, if your example is "i686 useremu on non-x86 embedde router"
> then any 32-bit host is potentially interested, not only arm32.

arm32 was merely an example -- the other 32-bit hosts are i686, mips, ppc.  But we don't 
have many of them.

> I remember being able to run armhf binaries on armel hosts (and vice
> versa) was useful 7 years ago.

Fair enough.

> Today I have no clue, we could poll the community and some 
> distributions.

Sure.


r~

On 31/1/23 00:33, Richard Henderson wrote:
> On 1/30/23 13:14, Philippe Mathieu-Daudé wrote:
>> On 30/1/23 20:19, Richard Henderson wrote:
>>
>>> But I do question whether we need to support 64-bit guests on 32-bit 
>>> hosts at all.
>>> Retaining 32-bit on 32-bit allows arm32 to emulate i686, which I 
>>> suspect, but have no proof, is the limit of what users actually want.
>>
>> I presume you implicitly restrict that to user emulation, right?
> 
> No, there's no specific reason to eliminate e.g. qemu-system-i386. or 
> any other 32-bit guest.  Though quite often such hardware doesn't really 
> have enough ram to do a good job, that's not a technical argument against.

I am not concerned by the RAM limit but by the community maintenance
cost. As of 2023, QEMU v7.2.0 is usable on 32-bit. Fixes will be
backported in the v7.2.x stable releases. Maybe this is enough for
32-bit host users; and planning the unavailability of features released
in v8.2 or v9.0 for 32-bit hosts doesn't seem unreasonable.

Stefan Weil once posted stats of Win32 vs Win64 binary downloads from
last year IIRC, and Win32 is still consumed (but maybe the difference
comes from mirrors or users always downloading both versions).

>> WRT i686, if your example is "i686 useremu on non-x86 embedded router"
>> then any 32-bit host is potentially interested, not only arm32.
> 
> arm32 was merely an example -- the other 32-bit hosts are i686, mips, 
> ppc.  But we don't have many of them.
> 
>> I remember being able to run armhf binaries on armel hosts (and vice
>> versa) was useful 7 years ago.
> 
> Fair enough.
> 
>> Today I have no clue, we could poll the community and some distributions.
> 
> Sure.
> 
> 
> r~

On 30/01/2023 13.01, Daniel P. Berrangé wrote:
> On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
>> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
>>>
>>> Testing 32-bit host OS support takes a lot of precious time during the QEMU
>>> contiguous integration tests, and considering that many OS vendors stopped
>>> shipping 32-bit variants of their OS distributions and most hardware from
>>> the past >10 years is capable of 64-bit
>>
>> True for x86, not necessarily true for other architectures.
>> Are you proposing to deprecate x86 32-bit, or all 32-bit?
>> I'm not entirely sure about whether we're yet at a point where
>> I'd want to deprecate-and-drop 32-bit arm host support.

Even mobile phones have 64-bit processors nowadays, Rasberry PIs are 64-bit 
nowadays ... which arm hosts scenarios are still limited to 32-bit ?

Also, as far as I know, 32-bit KVM arm support has been removed from the 
Linux kernel a while ago already, so it's just about TCG now ... is there 
really still that much interest in running emulation on a non-beefy 32-bit host?

Anyway, we could add the deprecation notice now to find out if there are 
still 32-bit users out there who will then start complaining about this.

> Do we have a feeling on which aspects of 32-bit cause us the support
> burden ? The boring stuff like compiler errors from mismatched integer
> sizes is mostly quick & easy to detect simply through a cross compile.

The burden are the CI minutes of the shared CI runners. We've got quite a 
bunch of 32-bit jobs in the CI:

- cross-armel-system
- cross-armel-user
- cross-armhf-system
- cross-armhf-user
- cross-i386-system
- cross-i386-user
- cross-i386-tci
- cross-mipsel-system
- cross-mipsel-user
- cross-win32-system

If we could finally drop supporting 32-bit hosts, that would help with our 
CI minutes problem quite a lot, I think.

  Thomas

On Mon, Jan 30, 2023 at 01:22:22PM +0100, Thomas Huth wrote:
> On 30/01/2023 13.01, Daniel P. Berrangé wrote:
> > On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
> > > On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
> > > > 
> > > > Testing 32-bit host OS support takes a lot of precious time during the QEMU
> > > > contiguous integration tests, and considering that many OS vendors stopped
> > > > shipping 32-bit variants of their OS distributions and most hardware from
> > > > the past >10 years is capable of 64-bit
> > > 
> > > True for x86, not necessarily true for other architectures.
> > > Are you proposing to deprecate x86 32-bit, or all 32-bit?
> > > I'm not entirely sure about whether we're yet at a point where
> > > I'd want to deprecate-and-drop 32-bit arm host support.
> 
> Even mobile phones have 64-bit processors nowadays, Rasberry PIs are 64-bit
> nowadays ... which arm hosts scenarios are still limited to 32-bit ?
> 
> Also, as far as I know, 32-bit KVM arm support has been removed from the
> Linux kernel a while ago already, so it's just about TCG now ... is there
> really still that much interest in running emulation on a non-beefy 32-bit
> host?
> 
> Anyway, we could add the deprecation notice now to find out if there are
> still 32-bit users out there who will then start complaining about this.
> 
> > Do we have a feeling on which aspects of 32-bit cause us the support
> > burden ? The boring stuff like compiler errors from mismatched integer
> > sizes is mostly quick & easy to detect simply through a cross compile.
> 
> The burden are the CI minutes of the shared CI runners. We've got quite a
> bunch of 32-bit jobs in the CI:
> 
> - cross-armel-system
> - cross-armel-user
> - cross-armhf-system
> - cross-armhf-user
> - cross-i386-system
> - cross-i386-user
> - cross-i386-tci
> - cross-mipsel-system
> - cross-mipsel-user
> - cross-win32-system
> 
> If we could finally drop supporting 32-bit hosts, that would help with our
> CI minutes problem quite a lot, I think.

CI is a non-technical constraint, that's more about support level.

If we want to save CI minutes, we can declare that some or all of the
32-bit hosts scenarios are now tier 2, rather than tier 1. So the 32-bit
host support still exists, but we're not doing gating CI on every
combination. eg could declare 32-bit for linux-user is tier 1 and fully
tested but 32-bit machine emul is tier 2 and adhoc tested. Or make it
more nuanced per arch target

We only need to deprecate and delete if we have some wins at the code
level that we can't do with while it exists.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 30/01/2023 13.43, Daniel P. Berrangé wrote:
> On Mon, Jan 30, 2023 at 01:22:22PM +0100, Thomas Huth wrote:
>> On 30/01/2023 13.01, Daniel P. Berrangé wrote:
>>> On Mon, Jan 30, 2023 at 11:47:02AM +0000, Peter Maydell wrote:
>>>> On Mon, 30 Jan 2023 at 11:44, Thomas Huth <thuth@redhat.com> wrote:
>>>>>
>>>>> Testing 32-bit host OS support takes a lot of precious time during the QEMU
>>>>> contiguous integration tests, and considering that many OS vendors stopped
>>>>> shipping 32-bit variants of their OS distributions and most hardware from
>>>>> the past >10 years is capable of 64-bit
...
>>> Do we have a feeling on which aspects of 32-bit cause us the support
>>> burden ? The boring stuff like compiler errors from mismatched integer
>>> sizes is mostly quick & easy to detect simply through a cross compile.
>>
>> The burden are the CI minutes of the shared CI runners. We've got quite a
>> bunch of 32-bit jobs in the CI:
>>
>> - cross-armel-system
>> - cross-armel-user
>> - cross-armhf-system
>> - cross-armhf-user
>> - cross-i386-system
>> - cross-i386-user
>> - cross-i386-tci
>> - cross-mipsel-system
>> - cross-mipsel-user
>> - cross-win32-system
>>
>> If we could finally drop supporting 32-bit hosts, that would help with our
>> CI minutes problem quite a lot, I think.
> 
> CI is a non-technical constraint, that's more about support level.
 >
> If we want to save CI minutes, we can declare that some or all of the
> 32-bit hosts scenarios are now tier 2, rather than tier 1. So the 32-bit
> host support still exists, but we're not doing gating CI on every
> combination. eg could declare 32-bit for linux-user is tier 1 and fully
> tested but 32-bit machine emul is tier 2 and adhoc tested. Or make it
> more nuanced per arch target

Well, while the burden currently mostly comes from the CI minutes (since it 
catches problems immediately, they don't get committed), you would need 
people's time instead who have to look after the problems once they've been 
committed to the repository (which will surely happen if we don't check 
32-bit host support in the CI anymore).

Who's volunteering in spending his time with analyzing (e.g. bisecting) and 
fixing those problems? At least I don't.

> We only need to deprecate and delete if we have some wins at the code
> level that we can't do with while it exists.

We also would have quite some wins at the code level, I think: At least the 
whole "tcg/arm/" folder would go away, and we could simplify all spots that 
are using HOST_LONG_BITS in the code base.

  Thomas