When we split up the daemons, libvirtd will need to forward different
sets of APIs to different daemons. This means libvirtd is going to need
to have multiple virConnectPtr instances open.

This series prepares for that by introducing "separate" connections,
which are actually just an extra reference on the current single
connection. This will facilitate later changes.

Daniel P. Berrangé (7):
  rpc: refactor way connection object is generated for remote dispatch
  remote: use a separate connection for interface APIs
  remote: use a separate connection for network APIs
  remote: use a separate connection for nodedev APIs
  remote: use a separate connection for nwfilter APIs
  remote: use a separate connection for secret APIs
  remote: use a separate connection for storage APIs

 src/remote/remote_daemon.h          |  6 +++
 src/remote/remote_daemon_dispatch.c | 81 +++++++++++++++++++------------
 src/rpc/gendispatch.pl              | 95 ++++++++++++++++++++++++++++---------
 3 files changed, 128 insertions(+), 54 deletions(-)

-- 
2.14.3

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On 03/28/2018 11:18 AM, Daniel P. Berrangé wrote:
> When we split up the daemons, libvirtd will need to forward different
> sets of APIs to different daemons. This means libvirtd is going to need
> to have multiple virConnectPtr instances open.

Have you done any thinking about what will need to be added to the
network driver API (or how)? I'd like to be involved with that.

One part that troubles me is that there will no longer be a parent-child
relationship between the process keeping track of network allocations
and the qemu process - currently if a qemu process dies, libvirtd knows
about it and can recover the resources that had been allocated, but with
the network driver in a separate process, qemu could die and
libvirtd-qemu might not be around to notify libvirtd-network (or
whatever we end up calling the processes).

Likewise, there is the nwfilter driver's complicated callback setup to
reload the iptables rules of all relevant domains when a filter's
definition is changed. I haven't thought about the details, but I think
that will need some "interesting" handling.

Also the network driver and nwfilter driver both use virfirewall.c,
which has a mutex to prevent simultaneous updates; either we'll need to
put that functionality into one of the drivers and have the other one
call it via a public API, or we'll need to rely on iptables -w (but even
that could lead to different results, since it's locking just during
application of a single rule rather than a complete transaction (as
defined by virfirewall.c).

>
> This series prepares for that by introducing "separate" connections,
> which are actually just an extra reference on the current single
> connection. This will facilitate later changes.
>
> Daniel P. Berrangé (7):
>   rpc: refactor way connection object is generated for remote dispatch
>   remote: use a separate connection for interface APIs
>   remote: use a separate connection for network APIs
>   remote: use a separate connection for nodedev APIs
>   remote: use a separate connection for nwfilter APIs
>   remote: use a separate connection for secret APIs
>   remote: use a separate connection for storage APIs
>
>  src/remote/remote_daemon.h          |  6 +++
>  src/remote/remote_daemon_dispatch.c | 81 +++++++++++++++++++------------
>  src/rpc/gendispatch.pl              | 95 ++++++++++++++++++++++++++++---------
>  3 files changed, 128 insertions(+), 54 deletions(-)
>

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Thu, Apr 05, 2018 at 01:06:28PM -0400, Laine Stump wrote:
> On 03/28/2018 11:18 AM, Daniel P. Berrangé wrote:
> > When we split up the daemons, libvirtd will need to forward different
> > sets of APIs to different daemons. This means libvirtd is going to need
> > to have multiple virConnectPtr instances open.
> 
> Have you done any thinking about what will need to be added to the
> network driver API (or how)? I'd like to be involved with that.

I've not thought about it beyond hoping magic ponies will solve the
problems you describe below :)  So far I'm just working on the general
refactoring to let us (optionally) create the various daemons so there
is something concrete to explore the problems with.

> One part that troubles me is that there will no longer be a parent-child
> relationship between the process keeping track of network allocations
> and the qemu process - currently if a qemu process dies, libvirtd knows
> about it and can recover the resources that had been allocated, but with
> the network driver in a separate process, qemu could die and
> libvirtd-qemu might not be around to notify libvirtd-network (or
> whatever we end up calling the processes).

BTW, naming will be virt${DRIVER}d (eg virtqemud, virtnetworkd, etc) but
minor detail. Anyway, if QEMU dies, and virtqemud isn't running, then
when virtqemud starts up again it would notice that the QEMU went away
and do recovery, presumably telling other daemons to release resources
if needed.

> Likewise, there is the nwfilter driver's complicated callback setup to
> reload the iptables rules of all relevant domains when a filter's
> definition is changed. I haven't thought about the details, but I think
> that will need some "interesting" handling.

I was thinking the nwfilter automatic rebuild ought to be possible to
have entirely self-contained, because rebuilding shouldn't need help
from the QEMU driver - it just needs the TAP dev to exist.

> Also the network driver and nwfilter driver both use virfirewall.c,
> which has a mutex to prevent simultaneous updates; either we'll need to
> put that functionality into one of the drivers and have the other one
> call it via a public API, or we'll need to rely on iptables -w (but even
> that could lead to different results, since it's locking just during
> application of a single rule rather than a complete transaction (as
> defined by virfirewall.c).

Two possible options

 - Replace the mutex with a lock file + fcntl() locks, that both
   virnetworkd & virnwfilterd will use
 - Have virnetworkd actually call virnwfilterd to setup the rules
   it needs. I'm not sure if nwfilter can express the rules we
   need though.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On 04/05/2018 07:06 PM, Laine Stump wrote:
> On 03/28/2018 11:18 AM, Daniel P. Berrangé wrote:
>> When we split up the daemons, libvirtd will need to forward different
>> sets of APIs to different daemons. This means libvirtd is going to need
>> to have multiple virConnectPtr instances open.
> 
> Have you done any thinking about what will need to be added to the
> network driver API (or how)? I'd like to be involved with that.
> 
> One part that troubles me is that there will no longer be a parent-child
> relationship between the process keeping track of network allocations
> and the qemu process - currently if a qemu process dies, libvirtd knows
> about it and can recover the resources that had been allocated,

There's no parent-child relationship between qemu and libvirt nowadays.
I mean, qemu is daemonized and the only way libvirt notices dead qemu is
via EOF on monitor socket.

> but with
> the network driver in a separate process, qemu could die and
> libvirtd-qemu might not be around to notify libvirtd-network (or
> whatever we end up calling the processes).

I don't think this is any different to current situation. If qemu dies
and libvirtd is not around no resources are freed. They are freed once
libvirtd is started again. In the new paradigm this would be done once
libvirtd-qemu starts up in which case it would notify libvirtd-network
to free up resources.

One problem though, if no libvirtd-* daemon is running and qemu dies and
then only libvirtd-qemu is started it'll try to notify libvirtd-network
(which is still not running) which will fail so that even if
libvirtd-network is started later it will not free resources. But this
is more generic problem and not specific just to libvirt-qemu and
libvirtd-network.

The solution I see consists of a queue of messages to be delivered to
the other daemons. So whenever libvirtd-qemu wants to notify network
that domain died, the message will go to a queue so it doesn't get lost.
And then once libvirt-network is started all messages from the queue
will be sent.

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list