[libvirt] [libvirt-jenkins-ci PATCH 05/18] ansible: Add libvirt-cim project

Andrea Bolognani posted 18 patches 8 years, 2 months ago
There is a newer version of this series
[libvirt] [libvirt-jenkins-ci PATCH 05/18] ansible: Add libvirt-cim project
Posted by Andrea Bolognani 8 years, 2 months ago
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
---
 ansible/host_vars/libvirt-centos-6/main.yml       |  1 +
 ansible/host_vars/libvirt-centos-7/main.yml       |  1 +
 ansible/host_vars/libvirt-fedora-25/main.yml      |  1 +
 ansible/host_vars/libvirt-fedora-26/main.yml      |  1 +
 ansible/host_vars/libvirt-fedora-rawhide/main.yml |  1 +
 ansible/tasks/build.yml                           | 19 +++++++++++++++----
 ansible/tasks/packages.yml                        | 11 +++++++++++
 ansible/vars/libvirt-cim/CentOS-6.yml             | 11 +++++++++++
 ansible/vars/libvirt-cim/CentOS-7.yml             | 11 +++++++++++
 ansible/vars/libvirt-cim/Fedora-25.yml            | 11 +++++++++++
 ansible/vars/libvirt-cim/Fedora-26.yml            | 11 +++++++++++
 ansible/vars/libvirt-cim/Fedora-Rawhide.yml       | 11 +++++++++++
 12 files changed, 86 insertions(+), 4 deletions(-)
 create mode 100644 ansible/vars/libvirt-cim/CentOS-6.yml
 create mode 100644 ansible/vars/libvirt-cim/CentOS-7.yml
 create mode 100644 ansible/vars/libvirt-cim/Fedora-25.yml
 create mode 100644 ansible/vars/libvirt-cim/Fedora-26.yml
 create mode 100644 ansible/vars/libvirt-cim/Fedora-Rawhide.yml

diff --git a/ansible/host_vars/libvirt-centos-6/main.yml b/ansible/host_vars/libvirt-centos-6/main.yml
index 4d53bb3..f6fe574 100644
--- a/ansible/host_vars/libvirt-centos-6/main.yml
+++ b/ansible/host_vars/libvirt-centos-6/main.yml
@@ -2,3 +2,4 @@
 projects:
   - base
   - libvirt
+  - libvirt-cim
diff --git a/ansible/host_vars/libvirt-centos-7/main.yml b/ansible/host_vars/libvirt-centos-7/main.yml
index 7508f19..ec2e50f 100644
--- a/ansible/host_vars/libvirt-centos-7/main.yml
+++ b/ansible/host_vars/libvirt-centos-7/main.yml
@@ -3,3 +3,4 @@ projects:
   - base
   - libosinfo
   - libvirt
+  - libvirt-cim
diff --git a/ansible/host_vars/libvirt-fedora-25/main.yml b/ansible/host_vars/libvirt-fedora-25/main.yml
index 7508f19..ec2e50f 100644
--- a/ansible/host_vars/libvirt-fedora-25/main.yml
+++ b/ansible/host_vars/libvirt-fedora-25/main.yml
@@ -3,3 +3,4 @@ projects:
   - base
   - libosinfo
   - libvirt
+  - libvirt-cim
diff --git a/ansible/host_vars/libvirt-fedora-26/main.yml b/ansible/host_vars/libvirt-fedora-26/main.yml
index 7508f19..ec2e50f 100644
--- a/ansible/host_vars/libvirt-fedora-26/main.yml
+++ b/ansible/host_vars/libvirt-fedora-26/main.yml
@@ -3,3 +3,4 @@ projects:
   - base
   - libosinfo
   - libvirt
+  - libvirt-cim
diff --git a/ansible/host_vars/libvirt-fedora-rawhide/main.yml b/ansible/host_vars/libvirt-fedora-rawhide/main.yml
index 7508f19..ec2e50f 100644
--- a/ansible/host_vars/libvirt-fedora-rawhide/main.yml
+++ b/ansible/host_vars/libvirt-fedora-rawhide/main.yml
@@ -3,3 +3,4 @@ projects:
   - base
   - libosinfo
   - libvirt
+  - libvirt-cim
diff --git a/ansible/tasks/build.yml b/ansible/tasks/build.yml
index be9ecd3..2a26024 100644
--- a/ansible/tasks/build.yml
+++ b/ansible/tasks/build.yml
@@ -38,7 +38,15 @@
   environment:
   when:
     - ( project == 'libosinfo' or
-        project == 'libvirt' )
+        project == 'libvirt' or
+        project == 'libvirt-cim' )
+
+- name: '{{ project }}: Run configure'
+  command: './configure {{ configure_options }}'
+  args:
+    chdir: '{{ project }}'
+  when:
+   - project == 'libvirt-cim'
 
 - name: '{{ project }}: Build project'
   command: '{{ make }} -j{{ smp }}'
@@ -46,7 +54,8 @@
     chdir: '{{ project }}'
   when:
     - ( project == 'libosinfo' or
-        project == 'libvirt' )
+        project == 'libvirt' or
+        project == 'libvirt-cim' )
 
 - name: '{{ project }}: Run sanity checks'
   command: '{{ make }} -j{{ smp }} syntax-check'
@@ -63,7 +72,8 @@
     chdir: '{{ project }}'
   when:
     - ( ( project == 'libvirt' and
-          os_name != 'FreeBSD' ) )
+          os_name != 'FreeBSD' ) or
+        project == 'libvirt-cim' )
 
 - name: '{{ project }}: Build RPM package'
   command: '{{ make }} -j{{ smp }} rpm'
@@ -72,4 +82,5 @@
   when:
     - ( os_name == 'CentOS' or
         os_name == 'Fedora' )
-    - ( project == 'libvirt' )
+    - ( project == 'libvirt' or
+        project == 'libvirt-cim' )
diff --git a/ansible/tasks/packages.yml b/ansible/tasks/packages.yml
index 630d794..7a66899 100644
--- a/ansible/tasks/packages.yml
+++ b/ansible/tasks/packages.yml
@@ -9,3 +9,14 @@
     state: present
   with_items:
     '{{ packages }}'
+
+- name: '{{ project }}: Install extra packages'
+  package:
+    name: '{{ item }}'
+    state: present
+  with_items:
+    '{{ extra_packages }}'
+  when:
+    - extra_packages is defined
+    # Only extra additional packages if we're going to build later on
+    - build
diff --git a/ansible/vars/libvirt-cim/CentOS-6.yml b/ansible/vars/libvirt-cim/CentOS-6.yml
new file mode 100644
index 0000000..8ff80c1
--- /dev/null
+++ b/ansible/vars/libvirt-cim/CentOS-6.yml
@@ -0,0 +1,11 @@
+---
+packages:
+  - libcmpiutil-devel
+  - libconfig-devel
+  - libuuid-devel
+  - libxml2-devel
+  - libxslt
+  - wget
+
+extra_packages:
+  - libvirt-devel
diff --git a/ansible/vars/libvirt-cim/CentOS-7.yml b/ansible/vars/libvirt-cim/CentOS-7.yml
new file mode 100644
index 0000000..8ff80c1
--- /dev/null
+++ b/ansible/vars/libvirt-cim/CentOS-7.yml
@@ -0,0 +1,11 @@
+---
+packages:
+  - libcmpiutil-devel
+  - libconfig-devel
+  - libuuid-devel
+  - libxml2-devel
+  - libxslt
+  - wget
+
+extra_packages:
+  - libvirt-devel
diff --git a/ansible/vars/libvirt-cim/Fedora-25.yml b/ansible/vars/libvirt-cim/Fedora-25.yml
new file mode 100644
index 0000000..8ff80c1
--- /dev/null
+++ b/ansible/vars/libvirt-cim/Fedora-25.yml
@@ -0,0 +1,11 @@
+---
+packages:
+  - libcmpiutil-devel
+  - libconfig-devel
+  - libuuid-devel
+  - libxml2-devel
+  - libxslt
+  - wget
+
+extra_packages:
+  - libvirt-devel
diff --git a/ansible/vars/libvirt-cim/Fedora-26.yml b/ansible/vars/libvirt-cim/Fedora-26.yml
new file mode 100644
index 0000000..8ff80c1
--- /dev/null
+++ b/ansible/vars/libvirt-cim/Fedora-26.yml
@@ -0,0 +1,11 @@
+---
+packages:
+  - libcmpiutil-devel
+  - libconfig-devel
+  - libuuid-devel
+  - libxml2-devel
+  - libxslt
+  - wget
+
+extra_packages:
+  - libvirt-devel
diff --git a/ansible/vars/libvirt-cim/Fedora-Rawhide.yml b/ansible/vars/libvirt-cim/Fedora-Rawhide.yml
new file mode 100644
index 0000000..8ff80c1
--- /dev/null
+++ b/ansible/vars/libvirt-cim/Fedora-Rawhide.yml
@@ -0,0 +1,11 @@
+---
+packages:
+  - libcmpiutil-devel
+  - libconfig-devel
+  - libuuid-devel
+  - libxml2-devel
+  - libxslt
+  - wget
+
+extra_packages:
+  - libvirt-devel
-- 
2.13.6

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libvirt-jenkins-ci PATCH 05/18] ansible: Add libvirt-cim project
Posted by Daniel P. Berrange 8 years, 2 months ago
On Mon, Oct 02, 2017 at 05:10:43PM +0200, Andrea Bolognani wrote:
> Signed-off-by: Andrea Bolognani <abologna@redhat.com>
> ---
>  ansible/host_vars/libvirt-centos-6/main.yml       |  1 +
>  ansible/host_vars/libvirt-centos-7/main.yml       |  1 +
>  ansible/host_vars/libvirt-fedora-25/main.yml      |  1 +
>  ansible/host_vars/libvirt-fedora-26/main.yml      |  1 +
>  ansible/host_vars/libvirt-fedora-rawhide/main.yml |  1 +
>  ansible/tasks/build.yml                           | 19 +++++++++++++++----
>  ansible/tasks/packages.yml                        | 11 +++++++++++
>  ansible/vars/libvirt-cim/CentOS-6.yml             | 11 +++++++++++
>  ansible/vars/libvirt-cim/CentOS-7.yml             | 11 +++++++++++
>  ansible/vars/libvirt-cim/Fedora-25.yml            | 11 +++++++++++
>  ansible/vars/libvirt-cim/Fedora-26.yml            | 11 +++++++++++
>  ansible/vars/libvirt-cim/Fedora-Rawhide.yml       | 11 +++++++++++
>  12 files changed, 86 insertions(+), 4 deletions(-)
>  create mode 100644 ansible/vars/libvirt-cim/CentOS-6.yml
>  create mode 100644 ansible/vars/libvirt-cim/CentOS-7.yml
>  create mode 100644 ansible/vars/libvirt-cim/Fedora-25.yml
>  create mode 100644 ansible/vars/libvirt-cim/Fedora-26.yml
>  create mode 100644 ansible/vars/libvirt-cim/Fedora-Rawhide.yml
> 
> diff --git a/ansible/host_vars/libvirt-centos-6/main.yml b/ansible/host_vars/libvirt-centos-6/main.yml
> index 4d53bb3..f6fe574 100644
> --- a/ansible/host_vars/libvirt-centos-6/main.yml
> +++ b/ansible/host_vars/libvirt-centos-6/main.yml
> @@ -2,3 +2,4 @@
>  projects:
>    - base
>    - libvirt
> +  - libvirt-cim
> diff --git a/ansible/host_vars/libvirt-centos-7/main.yml b/ansible/host_vars/libvirt-centos-7/main.yml
> index 7508f19..ec2e50f 100644
> --- a/ansible/host_vars/libvirt-centos-7/main.yml
> +++ b/ansible/host_vars/libvirt-centos-7/main.yml
> @@ -3,3 +3,4 @@ projects:
>    - base
>    - libosinfo
>    - libvirt
> +  - libvirt-cim
> diff --git a/ansible/host_vars/libvirt-fedora-25/main.yml b/ansible/host_vars/libvirt-fedora-25/main.yml
> index 7508f19..ec2e50f 100644
> --- a/ansible/host_vars/libvirt-fedora-25/main.yml
> +++ b/ansible/host_vars/libvirt-fedora-25/main.yml
> @@ -3,3 +3,4 @@ projects:
>    - base
>    - libosinfo
>    - libvirt
> +  - libvirt-cim
> diff --git a/ansible/host_vars/libvirt-fedora-26/main.yml b/ansible/host_vars/libvirt-fedora-26/main.yml
> index 7508f19..ec2e50f 100644
> --- a/ansible/host_vars/libvirt-fedora-26/main.yml
> +++ b/ansible/host_vars/libvirt-fedora-26/main.yml
> @@ -3,3 +3,4 @@ projects:
>    - base
>    - libosinfo
>    - libvirt
> +  - libvirt-cim
> diff --git a/ansible/host_vars/libvirt-fedora-rawhide/main.yml b/ansible/host_vars/libvirt-fedora-rawhide/main.yml
> index 7508f19..ec2e50f 100644
> --- a/ansible/host_vars/libvirt-fedora-rawhide/main.yml
> +++ b/ansible/host_vars/libvirt-fedora-rawhide/main.yml
> @@ -3,3 +3,4 @@ projects:
>    - base
>    - libosinfo
>    - libvirt
> +  - libvirt-cim
> diff --git a/ansible/tasks/build.yml b/ansible/tasks/build.yml
> index be9ecd3..2a26024 100644
> --- a/ansible/tasks/build.yml
> +++ b/ansible/tasks/build.yml
> @@ -38,7 +38,15 @@
>    environment:
>    when:
>      - ( project == 'libosinfo' or
> -        project == 'libvirt' )
> +        project == 'libvirt' or
> +        project == 'libvirt-cim' )
> +
> +- name: '{{ project }}: Run configure'
> +  command: './configure {{ configure_options }}'
> +  args:
> +    chdir: '{{ project }}'
> +  when:
> +   - project == 'libvirt-cim'
>  
>  - name: '{{ project }}: Build project'
>    command: '{{ make }} -j{{ smp }}'
> @@ -46,7 +54,8 @@
>      chdir: '{{ project }}'
>    when:
>      - ( project == 'libosinfo' or
> -        project == 'libvirt' )
> +        project == 'libvirt' or
> +        project == 'libvirt-cim' )
>  
>  - name: '{{ project }}: Run sanity checks'
>    command: '{{ make }} -j{{ smp }} syntax-check'
> @@ -63,7 +72,8 @@
>      chdir: '{{ project }}'
>    when:
>      - ( ( project == 'libvirt' and
> -          os_name != 'FreeBSD' ) )
> +          os_name != 'FreeBSD' ) or
> +        project == 'libvirt-cim' )
>  
>  - name: '{{ project }}: Build RPM package'
>    command: '{{ make }} -j{{ smp }} rpm'
> @@ -72,4 +82,5 @@
>    when:
>      - ( os_name == 'CentOS' or
>          os_name == 'Fedora' )
> -    - ( project == 'libvirt' )
> +    - ( project == 'libvirt' or
> +        project == 'libvirt-cim' )
> diff --git a/ansible/tasks/packages.yml b/ansible/tasks/packages.yml
> index 630d794..7a66899 100644
> --- a/ansible/tasks/packages.yml
> +++ b/ansible/tasks/packages.yml
> @@ -9,3 +9,14 @@
>      state: present
>    with_items:
>      '{{ packages }}'
> +
> +- name: '{{ project }}: Install extra packages'
> +  package:
> +    name: '{{ item }}'
> +    state: present
> +  with_items:
> +    '{{ extra_packages }}'
> +  when:
> +    - extra_packages is defined
> +    # Only extra additional packages if we're going to build later on
> +    - build
> diff --git a/ansible/vars/libvirt-cim/CentOS-6.yml b/ansible/vars/libvirt-cim/CentOS-6.yml
> new file mode 100644
> index 0000000..8ff80c1
> --- /dev/null
> +++ b/ansible/vars/libvirt-cim/CentOS-6.yml
> @@ -0,0 +1,11 @@
> +---
> +packages:
> +  - libcmpiutil-devel
> +  - libconfig-devel
> +  - libuuid-devel
> +  - libxml2-devel
> +  - libxslt
> +  - wget
> +
> +extra_packages:
> +  - libvirt-devel

This isn't right - we should never install libvirt-devel on any of the
build hosts. Downstream projects are chained up to build against the
version of libvirt we just built. The same applies for other deps
we build that are used by other downstream pieces.


Is there any way to get inheritance between these 'vars' files, so we don't
copy+paste the same content for every Fedora/CentOS version ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libvirt-jenkins-ci PATCH 05/18] ansible: Add libvirt-cim project
Posted by Andrea Bolognani 8 years, 2 months ago
On Mon, 2017-10-02 at 16:57 +0100, Daniel P. Berrange wrote:
> > +packages:
> > +  - libcmpiutil-devel
> > +  - libconfig-devel
> > +  - libuuid-devel
> > +  - libxml2-devel
> > +  - libxslt
> > +  - wget
> > +
> > +extra_packages:
> > +  - libvirt-devel
> 
> This isn't right - we should never install libvirt-devel on any of the
> build hosts. Downstream projects are chained up to build against the
> version of libvirt we just built. The same applies for other deps
> we build that are used by other downstream pieces.

That's why it's in 'extra_packages' rather than 'packages' ;)

Again, this is only used during development.

See group_vars/all/main.yml:

  # Wether to build software. While this is useful for figuring out
  # build dependencies, it should not be turned on in production
  # because it causes extra packages to be installed, which can
  # interfere with the CI jobs
  build: false

and tasks/packages.yml:

  - name: '{{ project }}: Install extra packages'
    package:
      name: '{{ item }}'
      state: present
    with_items:
      '{{ extra_packages }}'
    when:
      - extra_packages is defined
      # Only extra additional packages if we're going to build
      - build

Yuck, I accidentally that comment. You get the idea though.

> Is there any way to get inheritance between these 'vars' files, so we don't
> copy+paste the same content for every Fedora/CentOS version ?

Mh, I have the feeling it might complicate things rather than
make them nicer. I can look into it, though.

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libvirt-jenkins-ci PATCH 05/18] ansible: Add libvirt-cim project
Posted by Daniel P. Berrange 8 years, 2 months ago
On Tue, Oct 03, 2017 at 09:18:32AM +0200, Andrea Bolognani wrote:
> On Mon, 2017-10-02 at 16:57 +0100, Daniel P. Berrange wrote:
> > > +packages:
> > > +  - libcmpiutil-devel
> > > +  - libconfig-devel
> > > +  - libuuid-devel
> > > +  - libxml2-devel
> > > +  - libxslt
> > > +  - wget
> > > +
> > > +extra_packages:
> > > +  - libvirt-devel
> > 
> > This isn't right - we should never install libvirt-devel on any of the
> > build hosts. Downstream projects are chained up to build against the
> > version of libvirt we just built. The same applies for other deps
> > we build that are used by other downstream pieces.
> 
> That's why it's in 'extra_packages' rather than 'packages' ;)
> 
> Again, this is only used during development.
> 
> See group_vars/all/main.yml:
> 
>   # Wether to build software. While this is useful for figuring out
>   # build dependencies, it should not be turned on in production
>   # because it causes extra packages to be installed, which can
>   # interfere with the CI jobs
>   build: false
> 
> and tasks/packages.yml:
> 
>   - name: '{{ project }}: Install extra packages'
>     package:
>       name: '{{ item }}'
>       state: present
>     with_items:
>       '{{ extra_packages }}'
>     when:
>       - extra_packages is defined
>       # Only extra additional packages if we're going to build
>       - build
> 
> Yuck, I accidentally that comment. You get the idea though.
> 
> > Is there any way to get inheritance between these 'vars' files, so we don't
> > copy+paste the same content for every Fedora/CentOS version ?
> 
> Mh, I have the feeling it might complicate things rather than
> make them nicer. I can look into it, though.

One of the big wins when I moved the configs out of Jenkins native config
format, into the Jenkins  Job Builder config format is that we could
eliminate all the duplication between the configs. So from that POV it
feels unpleasant to be re-adding lots of duplication. 

In the very last patch you add a bunch of files which define aliases for
the various dependancies, and map those to the distro specific package
name eg 

+cyrus-sasl:
+  - cyrus-sasl       # FreeBSD
+  - cyrus-sasl-devel # CentOS, Fedora
+  - libsasl2-dev     # Debian, Ubuntu

Given these data maps, it seems like we ought to be able to define the
build pre-requisites once in terms of the alias names, and then expand
that into the distro specific package lists, thus avoiding a per-distro
list of deps for each module

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libvirt-jenkins-ci PATCH 05/18] ansible: Add libvirt-cim project
Posted by Andrea Bolognani 8 years, 2 months ago
On Tue, 2017-10-03 at 10:02 +0100, Daniel P. Berrange wrote:
> In the very last patch you add a bunch of files which define aliases for
> the various dependancies, and map those to the distro specific package
> name eg
> 
> +cyrus-sasl:
> +  - cyrus-sasl       # FreeBSD
> +  - cyrus-sasl-devel # CentOS, Fedora
> +  - libsasl2-dev     # Debian, Ubuntu
> 
> Given these data maps, it seems like we ought to be able to define the
> build pre-requisites once in terms of the alias names, and then expand
> that into the distro specific package lists, thus avoiding a per-distro
> list of deps for each module

It's a little more complicated than that, though.

Some dependencies have different names based not just on the OS
but also on the specific version, eg.

  gnutls:
    - gnutls          # FreeBSD
    - gnutls-devel    # CentOS, Fedora
    - libgnutls-dev   # Ubuntu <= 14
    - libgnutls28-dev # Debian, Ubuntu 16

Other dependencies might only be availabe on a subset of the OSs
we support, eg.

  libblkid:
    - libblkid-dev   # Debian, Ubuntu
    - libblkid-devel # CentOS, Fedora

I get the argument against redundancy, but in this case I feel like
the current representation is possibly preferable to a more compact,
less redundant one because it's *way simpler*, both in terms of the
variable files themselves and the Ansible code required to use them.

Would having something like

  gnutls:
    CentOS:
      6: gnutls-devel
      7: gnutls-devel
    Debian:
      8: libgnutls28-dev
      9: libgnutls28-dev
    Fedora:
      25: gnutls-devel
      26: gnutls-devel
      Rawhide: gnutls-devel
    FreeBSD:
      11: gnutls
    Ubuntu:
      12: libgnutls-dev
      14: libgnutls-dev
      16: libgnutls28-dev

or

  gnutls:
    gnutls:
      - FreeBSD-11
    gnutls-devel:
      - CentOS-6
      - CentOS-7
      - Fedora-25
      - Fedora-26
      - Fedora-Rawhide
    libgnutls-dev:
      - Ubuntu-12
      - Ubuntu-14
    libgnutls28-dev:
      - Debian-8
      - Debian-9
      - Ubuntu-16

plus Ansible machinery to expand it really be better than what I'm
proposing? I guess we could have a more compact representation, but
that would be at the cost of even more complex Ansible machinery.
The above is still somewhat redundant, and IMHO less usable because
not only the redundancy is staring right at you instead of being
confined to separate files, you also just introduced one extra layer
of indirection.

What if one of the build dependencies is too old to be used by
libvirt but libosinfo can use it just fine? How would you describe
something like that in a machine-readable way without making the
whole thing utterly unusable by humans?

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list