From nobody Fri Mar 29 05:18:57 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of ovirt.org designates 66.187.230.42 as permitted sender) client-ip=66.187.230.42; envelope-from=kimchi-devel-bounces@ovirt.org; helo=lists.ovirt.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of ovirt.org designates 66.187.230.42 as permitted sender) smtp.mailfrom=kimchi-devel-bounces@ovirt.org; Return-Path: Received: from lists.ovirt.org (lists.phx.ovirt.org [66.187.230.42]) by mx.zohomail.com with SMTPS id 1493304864909428.6338417789591; Thu, 27 Apr 2017 07:54:24 -0700 (PDT) Received: from lists.phx.ovirt.org (localhost [127.0.0.1]) by lists.ovirt.org (Postfix) with ESMTP id BE3DE8204CE; Thu, 27 Apr 2017 14:54:23 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lists.ovirt.org (Postfix) with ESMTPS id 9D3888204C0 for ; Thu, 27 Apr 2017 14:54:07 +0000 (UTC) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v3REmdDT070276 for ; Thu, 27 Apr 2017 10:54:07 -0400 Received: from e24smtp01.br.ibm.com (e24smtp01.br.ibm.com [32.104.18.85]) by mx0b-001b2d01.pphosted.com with ESMTP id 2a3a4es0st-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Thu, 27 Apr 2017 10:54:05 -0400 Received: from localhost by e24smtp01.br.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 27 Apr 2017 11:54:03 -0300 Received: from d24relay03.br.ibm.com (9.18.232.225) by e24smtp01.br.ibm.com (10.172.0.143) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 27 Apr 2017 11:54:01 -0300 Received: from d24av04.br.ibm.com (d24av04.br.ibm.com [9.8.31.97]) by d24relay03.br.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v3REs0FD27787318 for ; Thu, 27 Apr 2017 11:54:00 -0300 Received: from d24av04.br.ibm.com (localhost [127.0.0.1]) by d24av04.br.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id v3REs07R003546 for ; Thu, 27 Apr 2017 11:54:00 -0300 Received: from alinefm-TP440.ibmmodules.com ([9.85.170.146]) by d24av04.br.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id v3REruAR003497 for ; Thu, 27 Apr 2017 11:53:58 -0300 X-Original-To: kimchi-devel@ovirt.org From: Aline Manera To: Kimchi Devel Date: Thu, 27 Apr 2017 11:53:50 -0300 X-Mailer: git-send-email 2.9.3 X-TM-AS-MML: disable x-cbid: 17042714-1523-0000-0000-00000299DB41 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17042714-1524-0000-0000-00002A300BB8 Message-Id: <20170427145350.3379-1-alinefm@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-04-27_13:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704270246 Subject: [Kimchi-devel] [PATCH] [Kimchi] Move Kimchi specific functions from gingerbase.disks to Kimchi X-BeenThere: kimchi-devel@ovirt.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: kimchi-devel-bounces@ovirt.org Errors-To: kimchi-devel-bounces@ovirt.org X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The idea behind this patch is to eliminate the Ginger Base dependency. Some functions in gingerbase.disks are for only Kimchi matters. Others are shared with Ginger and can not be rewritten using PyParted, each means they will be place on Kimchi and Ginger Base. As this code is mature enough, further changes will have few impact and the gain to incorporate Ginger Base into Ginger and eliminate a Kimchi dependency is bigger. Signed-off-by: Aline Manera Reviewed-by: Daniel Barboza --- disks.py | 325 ++++++++++++++++++++++++++++++++++++++++++++++++= ++++ i18n.py | 5 + model/host.py | 4 +- tests/test_disks.py | 53 +++++++++ 4 files changed, 385 insertions(+), 2 deletions(-) create mode 100644 disks.py create mode 100644 tests/test_disks.py diff --git a/disks.py b/disks.py new file mode 100644 index 0000000..e86ca2f --- /dev/null +++ b/disks.py @@ -0,0 +1,325 @@ +# +# Project Kimchi +# +# Copyright IBM Corp, 2015-2017 +# +# Code derived from Ginger Base project +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + +import os.path +import re +from parted import Device as PDevice +from parted import Disk as PDisk + +from wok.exception import NotFoundError, OperationFailed +from wok.stringutils import encode_value +from wok.utils import run_command, wok_log + + +def _get_dev_node_path(maj_min): + """ Returns device node path given the device number 'major:min' """ + + dm_name =3D "/sys/dev/block/%s/dm/name" % maj_min + if os.path.exists(dm_name): + with open(dm_name) as dm_f: + content =3D dm_f.read().rstrip('\n') + return "/dev/mapper/" + content + + uevent =3D "/sys/dev/block/%s/uevent" % maj_min + with open(uevent) as ueventf: + content =3D ueventf.read() + + data =3D dict(re.findall(r'(\S+)=3D(".*?"|\S+)', content.replace("\n",= " "))) + + return "/dev/%s" % data["DEVNAME"] + + +def _get_lsblk_devs(keys, devs=3DNone): + if devs is None: + devs =3D [] + out, err, returncode =3D run_command( + ["lsblk", "-Pbo"] + [','.join(keys)] + devs + ) + if returncode !=3D 0: + if 'not a block device' in err: + raise NotFoundError("KCHDISK00002E") + else: + raise OperationFailed("KCHDISK00001E", {'err': err}) + + return _parse_lsblk_output(out, keys) + + +def _get_dev_major_min(name): + maj_min =3D None + + keys =3D ["NAME", "MAJ:MIN"] + try: + dev_list =3D _get_lsblk_devs(keys) + except: + raise + + for dev in dev_list: + if dev['name'].split()[0] =3D=3D name: + maj_min =3D dev['maj:min'] + break + else: + raise NotFoundError("KCHDISK00003E", {'device': name}) + + return maj_min + + +def _is_dev_leaf(devNodePath, name=3DNone, devs=3DNone, devtype=3DNone): + try: + if devs and devtype !=3D 'mpath': + for dev in devs: + if encode_value(name) =3D=3D dev['pkname']: + return False + return True + # By default, lsblk prints a device information followed by childr= en + # device information + childrenCount =3D len( + _get_lsblk_devs(["NAME"], [devNodePath])) - 1 + except OperationFailed as e: + # lsblk is known to fail on multipath devices + # Assume these devices contain children + wok_log.error( + "Error getting device info for %s: %s", devNodePath, e) + return False + + return childrenCount =3D=3D 0 + + +def _is_dev_extended_partition(devType, devNodePath): + if devType !=3D 'part': + return False + + if devNodePath.startswith('/dev/mapper'): + try: + dev_maj_min =3D _get_dev_major_min(devNodePath.split("/")[-1]) + parent_sys_path =3D '/sys/dev/block/' + dev_maj_min + '/slaves' + parent_dm_name =3D os.listdir(parent_sys_path)[0] + parent_maj_min =3D open( + parent_sys_path + + '/' + + parent_dm_name + + '/dev').readline().rstrip() + diskPath =3D _get_dev_node_path(parent_maj_min) + except Exception as e: + wok_log.error( + "Error dealing with dev mapper device: " + devNodePath) + raise OperationFailed("KCHDISK00001E", {'err': e.message}) + else: + diskPath =3D devNodePath.rstrip('0123456789') + + device =3D PDevice(diskPath) + try: + extended_part =3D PDisk(device).getExtendedPartition() + except NotImplementedError as e: + wok_log.warning( + "Error getting extended partition info for dev %s type %s: %s", + devNodePath, devType, e.message) + # Treate disk with unsupported partiton table as if it does not + # contain extended partitions. + return False + if extended_part and extended_part.path =3D=3D devNodePath: + return True + return False + + +def _parse_lsblk_output(output, keys): + # output is on format key=3D"value", + # where key can be NAME, TYPE, FSTYPE, SIZE, MOUNTPOINT, etc + lines =3D output.rstrip("\n").split("\n") + r =3D [] + for line in lines: + d =3D {} + for key in keys: + expression =3D r"%s=3D\".*?\"" % key + match =3D re.search(expression, line) + field =3D match.group() + k, v =3D field.split('=3D', 1) + d[k.lower()] =3D v[1:-1] + r.append(d) + return r + + +def _is_available(name, devtype, fstype, mountpoint, majmin, devs=3DNone): + devNodePath =3D _get_dev_node_path(majmin) + # Only list unmounted and unformated and leaf and (partition or disk) + # leaf means a partition, a disk has no partition, or a disk not held + # by any multipath device. Physical volume belongs to no volume group + # is also listed. Extended partitions should not be listed. + if fstype =3D=3D 'LVM2_member': + has_VG =3D True + else: + has_VG =3D False + if (devtype in ['part', 'disk', 'mpath'] and + fstype in ['', 'LVM2_member'] and + mountpoint =3D=3D "" and + not has_VG and + _is_dev_leaf(devNodePath, name, devs, devtype) and + not _is_dev_extended_partition(devtype, devNodePath)): + return True + return False + + +def get_partitions_names(check=3DFalse): + names =3D set() + keys =3D ["NAME", "TYPE", "FSTYPE", "MOUNTPOINT", "MAJ:MIN"] + # output is on format key=3D"value", + # where key can be NAME, TYPE, FSTYPE, MOUNTPOINT + for dev in _get_lsblk_devs(keys): + # split()[0] to avoid the second part of the name, after the + # whiteline + name =3D dev['name'].split()[0] + if check and not _is_available(name, dev['type'], dev['fstype'], + dev['mountpoint'], dev['maj:min']): + continue + names.add(name) + + return list(names) + + +def get_partition_details(name): + majmin =3D _get_dev_major_min(name) + dev_path =3D _get_dev_node_path(majmin) + + keys =3D ["TYPE", "FSTYPE", "SIZE", "MOUNTPOINT", "MAJ:MIN", "PKNAME"] + try: + dev =3D _get_lsblk_devs(keys, [dev_path])[0] + except: + wok_log.error("Error getting partition info for %s", name) + return {} + + dev['available'] =3D _is_available(name, dev['type'], dev['fstype'], + dev['mountpoint'], majmin) + if dev['mountpoint']: + # Sometimes the mountpoint comes with [SWAP] or other + # info which is not an actual mount point. Filtering it + regexp =3D re.compile(r"\[.*\]") + if regexp.search(dev['mountpoint']) is not None: + dev['mountpoint'] =3D '' + dev['path'] =3D dev_path + dev['name'] =3D name + return dev + + +def vgs(): + """ + lists all volume groups in the system. All size units are in bytes. + + [{'vgname': 'vgtest', 'size': 999653638144L, 'free': 0}] + """ + cmd =3D ['vgs', + '--units', + 'b', + '--nosuffix', + '--noheading', + '--unbuffered', + '--options', + 'vg_name,vg_size,vg_free'] + + out, err, rc =3D run_command(cmd) + if rc !=3D 0: + raise OperationFailed("KCHDISK00004E", {'err': err}) + + if not out: + return [] + + # remove blank spaces and create a list of VGs + vgs =3D map(lambda v: v.strip(), out.strip('\n').split('\n')) + + # create a dict based on data retrieved from vgs + return map(lambda l: {'vgname': l[0], + 'size': long(l[1]), + 'free': long(l[2])}, + [fields.split() for fields in vgs]) + + +def lvs(vgname=3DNone): + """ + lists all logical volumes found in the system. It can be filtered by + the volume group. All size units are in bytes. + + [{'lvname': 'lva', 'path': '/dev/vgtest/lva', 'size': 12345L}, + {'lvname': 'lvb', 'path': '/dev/vgtest/lvb', 'size': 12345L}] + """ + cmd =3D ['lvs', + '--units', + 'b', + '--nosuffix', + '--noheading', + '--unbuffered', + '--options', + 'lv_name,lv_path,lv_size,vg_name'] + + out, err, rc =3D run_command(cmd) + if rc !=3D 0: + raise OperationFailed("KCHDISK00004E", {'err': err}) + + if not out: + return [] + + # remove blank spaces and create a list of LVs filtered by vgname, if + # provided + lvs =3D filter(lambda f: vgname is None or vgname in f, + map(lambda v: v.strip(), out.strip('\n').split('\n'))) + + # create a dict based on data retrieved from lvs + return map(lambda l: {'lvname': l[0], + 'path': l[1], + 'size': long(l[2])}, + [fields.split() for fields in lvs]) + + +def pvs(vgname=3DNone): + """ + lists all physical volumes in the system. It can be filtered by the + volume group. All size units are in bytes. + + [{'pvname': '/dev/sda3', + 'size': 469502001152L, + 'uuid': 'kkon5B-vnFI-eKHn-I5cG-Hj0C-uGx0-xqZrXI'}, + {'pvname': '/dev/sda2', + 'size': 21470642176L, + 'uuid': 'CyBzhK-cQFl-gWqr-fyWC-A50Y-LMxu-iHiJq4'}] + """ + cmd =3D ['pvs', + '--units', + 'b', + '--nosuffix', + '--noheading', + '--unbuffered', + '--options', + 'pv_name,pv_size,pv_uuid,vg_name'] + + out, err, rc =3D run_command(cmd) + if rc !=3D 0: + raise OperationFailed("KCHDISK00004E", {'err': err}) + + if not out: + return [] + + # remove blank spaces and create a list of PVs filtered by vgname, if + # provided + pvs =3D filter(lambda f: vgname is None or vgname in f, + map(lambda v: v.strip(), out.strip('\n').split('\n'))) + + # create a dict based on data retrieved from pvs + return map(lambda l: {'pvname': l[0], + 'size': long(l[1]), + 'uuid': l[2]}, + [fields.split() for fields in pvs]) diff --git a/i18n.py b/i18n.py index 4460ce5..7dc7b05 100644 --- a/i18n.py +++ b/i18n.py @@ -29,6 +29,11 @@ messages =3D { =20 "KCHPART0001E": _("Partition %(name)s does not exist in the host"), =20 + "KCHDISK00001E": _("Error while accessing dev mapper device, %(err)s"), + "KCHDISK00002E": _("Block device not found."), + "KCHDISK00003E": _("Block device %(device)s not found."), + "KCHDISK00004E": _("Unable to retrieve LVM information. Details: %(err= )s"), + "KCHDEVS0001E": _('Unknown "_cap" specified'), "KCHDEVS0002E": _('"_passthrough" should be "true" or "false"'), "KCHDEVS0003E": _('"_passthrough_affected_by" should be a device name = string'), diff --git a/model/host.py b/model/host.py index abf1191..90834e3 100644 --- a/model/host.py +++ b/model/host.py @@ -1,7 +1,7 @@ # # Project Kimchi # -# Copyright IBM Corp, 2015-2016 +# Copyright IBM Corp, 2015-2017 # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -26,7 +26,7 @@ from wok.exception import InvalidParameter from wok.exception import NotFoundError from wok.xmlutils.utils import xpath_get_text =20 -from wok.plugins.gingerbase import disks +from wok.plugins.kimchi import disks from wok.plugins.kimchi.model import hostdev from wok.plugins.kimchi.model.config import CapabilitiesModel from wok.plugins.kimchi.model.vms import VMModel, VMsModel diff --git a/tests/test_disks.py b/tests/test_disks.py new file mode 100644 index 0000000..6782f55 --- /dev/null +++ b/tests/test_disks.py @@ -0,0 +1,53 @@ +# +# Project Kimchi +# +# Copyright IBM Corp, 2017 +# +# Code derived from Ginger Base +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + +import mock +import unittest + +from wok.exception import NotFoundError, OperationFailed +from wok.plugins.kimchi.disks import _get_lsblk_devs + + +class DiskTests(unittest.TestCase): + + @mock.patch('wok.plugins.kimchi.disks.run_command') + def test_lsblk_returns_404_when_device_not_found(self, mock_run_comman= d): + mock_run_command.return_value =3D ["", "not a block device", 32] + fake_dev =3D "/not/a/true/block/dev" + keys =3D ["MOUNTPOINT"] + + with self.assertRaises(NotFoundError): + _get_lsblk_devs(keys, [fake_dev]) + cmd =3D ['lsblk', '-Pbo', 'MOUNTPOINT', fake_dev] + mock_run_command.assert_called_once_with(cmd) + + @mock.patch('wok.plugins.kimchi.disks.run_command') + def test_lsblk_returns_500_when_unknown_error_occurs( + self, mock_run_command): + + mock_run_command.return_value =3D ["", "", 1] + valid_dev =3D "/valid/block/dev" + keys =3D ["MOUNTPOINT"] + + with self.assertRaises(OperationFailed): + _get_lsblk_devs(keys, [valid_dev]) + cmd =3D ['lsblk', '-Pbo', 'MOUNTPOINT', valid_dev] + mock_run_command.assert_called_once_with(cmd) --=20 2.9.3 _______________________________________________ Kimchi-devel mailing list Kimchi-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/kimchi-devel