nl_recv() returns the error "No buffer space available"
when using virsh destroy domain with 240 or more
passhthrough network interfaces.
The patch increases libnl sock receive buffer size to 1M,
and nl_recv() doesn't return error when destroying domain
with 512 network interfaces.

Signed-off-by: ZhiPeng Lu <lu.zhipeng@zte.com.cn>
---
 src/util/virnetlink.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/util/virnetlink.c b/src/util/virnetlink.c
index 92ecf77..bb56c54 100644
--- a/src/util/virnetlink.c
+++ b/src/util/virnetlink.c
@@ -189,10 +189,10 @@ virNetlinkCreateSocket(int protocol)
         goto error;
     }
 
-    if (virNetlinkSetBufferSize(nlhandle, 131702, 0) < 0) {
+    if (virNetlinkSetBufferSize(nlhandle, 1048576, 0) < 0) {
         virReportSystemError(errno, "%s",
                              _("cannot set netlink socket buffer "
-                               "size to 128k"));
+                               "size to 1M"));
         goto error;
     }
     nl_socket_enable_msg_peek(nlhandle);
-- 
1.8.3.1


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On 06/29/2017 02:05 PM, ZhiPeng Lu wrote:
> nl_recv() returns the error "No buffer space available"
> when using virsh destroy domain with 240 or more
> passhthrough network interfaces.

pass-through

> The patch increases libnl sock receive buffer size to 1M,
> and nl_recv() doesn't return error when destroying domain
> with 512 network interfaces.
> 
> Signed-off-by: ZhiPeng Lu <lu.zhipeng@zte.com.cn>
> ---
>  src/util/virnetlink.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 

This feels like something that perhaps should be configurable - that is
some /etc/libvirt/libvirtd.conf variable; otherwise, we'll keep hitting
some conflated maximum based on the size of something.

John

There's quite a bit of history in the archives from the original
implementation of this API... Not sure if you read it or not, but since
I was looking through the history, here

v1: https://www.redhat.com/archives/libvir-list/2015-December/msg00407.html

v1 followup:
https://www.redhat.com/archives/libvir-list/2016-January/msg00107.html

v2: https://www.redhat.com/archives/libvir-list/2016-January/msg00339.html

v3: https://www.redhat.com/archives/libvir-list/2016-January/msg00342.html

v4: https://www.redhat.com/archives/libvir-list/2016-January/msg00866.html

> diff --git a/src/util/virnetlink.c b/src/util/virnetlink.c
> index 92ecf77..bb56c54 100644
> --- a/src/util/virnetlink.c
> +++ b/src/util/virnetlink.c
> @@ -189,10 +189,10 @@ virNetlinkCreateSocket(int protocol)
>          goto error;
>      }
>  
> -    if (virNetlinkSetBufferSize(nlhandle, 131702, 0) < 0) {
> +    if (virNetlinkSetBufferSize(nlhandle, 1048576, 0) < 0) {
>          virReportSystemError(errno, "%s",
>                               _("cannot set netlink socket buffer "
> -                               "size to 128k"));
> +                               "size to 1M"));
>          goto error;
>      }
>      nl_socket_enable_msg_peek(nlhandle);
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Mon, Jul 10, 2017 at 02:51:34PM -0400, John Ferlan wrote:
> 
> 
> On 06/29/2017 02:05 PM, ZhiPeng Lu wrote:
> > nl_recv() returns the error "No buffer space available"
> > when using virsh destroy domain with 240 or more
> > passhthrough network interfaces.
> 
> pass-through
> 
> > The patch increases libnl sock receive buffer size to 1M,
> > and nl_recv() doesn't return error when destroying domain
> > with 512 network interfaces.
> > 
> > Signed-off-by: ZhiPeng Lu <lu.zhipeng@zte.com.cn>
> > ---
> >  src/util/virnetlink.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> 
> This feels like something that perhaps should be configurable - that is
> some /etc/libvirt/libvirtd.conf variable; otherwise, we'll keep hitting
> some conflated maximum based on the size of something.

1 MB matches what systemed/udevd uses, so if we hit that limit, then  the
system as a whole is going to struggle already. So I don't think we need
make it configurable.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On 07/11/2017 05:06 AM, Daniel P. Berrange wrote:
> On Mon, Jul 10, 2017 at 02:51:34PM -0400, John Ferlan wrote:
>>
>> On 06/29/2017 02:05 PM, ZhiPeng Lu wrote:
>>> nl_recv() returns the error "No buffer space available"
>>> when using virsh destroy domain with 240 or more
>>> passhthrough network interfaces.
>> pass-through

(Actually, neither is correct. If you're talking about devices that are
assigned with VFIO, then they are "SRIOV VFs assigned with VFIO"
("passthrough" is a misnomer left over from Xen), and if you're talking
about macvtap, then according to the macvtap spec, the proper term is
"passthru", while the libvirt XML calls them "passthrough" (and in
either case, you should stipulate that you're talking about macvtap
network interfaces).)

>>
>>> The patch increases libnl sock receive buffer size to 1M,
>>> and nl_recv() doesn't return error when destroying domain
>>> with 512 network interfaces.
>>>
>>> Signed-off-by: ZhiPeng Lu <lu.zhipeng@zte.com.cn>
>>> ---
>>>  src/util/virnetlink.c | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>> This feels like something that perhaps should be configurable - that is
>> some /etc/libvirt/libvirtd.conf variable; otherwise, we'll keep hitting
>> some conflated maximum based on the size of something.
> 1 MB matches what systemed/udevd uses, so if we hit that limit, then  the
> system as a whole is going to struggle already. So I don't think we need
> make it configurable.

What bothers me about this (now that John's research has finally
reminded me of the complete gory history that I was deeply involved in
but had already totally forgotten) is that turning on netlink message
peeking (which we've done) is supposed to eliminate the need for a large
initial buffer (at the expense of making every read a bit less
efficient) *That* was supposed to be a permanent solution to the
problem, but apparently hasn't helped.

ZhiPeng - what version of libnl and kernel are you using? The message
history John points to reminded me that there are some versions of libnl
where message peeking doesn't work properly; maybe that's the issue and
the proper solution for you is to update your libnl. Or it's possible
that we need to do something else to make message peeking work properly?

(The latter doesn't seem likely - I just looked it up, and MSG_PEEK has
actually been enabled in libnl by default since this commit:

  
https://github.com/thom311/libnl/commit/55ea6e6b6cd805f441b410971c9dd7575e783ef4

which was in libnl 3.2.29 - even if we got it wrong, it should still be
enabled).

I *was* going to suggest that pushing this patch (rather than the one
making the initial buffer size configurable) was the best course of
action. But after reading through the history and remembering
everything, I've changed my mind - I think we need to figure out why
MSG_PEEK isn't working properly on your system. The first step is to
learn the version of libnl in use on the system that's failing.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list