From nobody Tue May 13 11:21:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=patchew-devel-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1526972288370260.5166504364157; Mon, 21 May 2018 23:58:08 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6AD253179B44; Tue, 22 May 2018 06:58:07 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5BEEF601A1; Tue, 22 May 2018 06:58:07 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 43AC14BB78; Tue, 22 May 2018 06:58:07 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w4M6w6oc028211 for ; Tue, 22 May 2018 02:58:06 -0400 Received: by smtp.corp.redhat.com (Postfix) id 9198F30001EB; Tue, 22 May 2018 06:58:06 +0000 (UTC) Received: from mx1.redhat.com (ext-mx10.extmail.prod.ext.phx2.redhat.com [10.5.110.39]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 85EB43091AA6 for ; Tue, 22 May 2018 06:58:03 +0000 (UTC) Received: from mail-pf0-f169.google.com (mail-pf0-f169.google.com [209.85.192.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D8ABD16B1E9 for ; Tue, 22 May 2018 06:57:51 +0000 (UTC) Received: by mail-pf0-f169.google.com with SMTP id p12-v6so8271124pff.13 for ; Mon, 21 May 2018 23:57:51 -0700 (PDT) Received: from localhost.localdomain ([208.181.63.202]) by smtp.gmail.com with ESMTPSA id y2-v6sm19818676pgp.92.2018.05.21.23.57.49 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 21 May 2018 23:57:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=4wGAsQTEi4Qgp5h+YOog2hQUd2pljSipp7LhsEmoXVc=; b=m9sbt/IHA0FUZZcB5DekVM/7SxIz7dwQhMtaZCiL+e4nwZDCYHLcts41rL8syhK0gJ m9ibyWlFUHEcODMpXOCArLdn2iVaSbf/mQPwikTvFdOlAhj3WQCDQrNzChtN2JLuodaC CvXtVLAh9MhxZYUt628gz2SIErp+mX2MOnqicfgdtI6YjfOCMVDj8y6KfERLTZUTIfs4 lwRHinjvrkC55BB6oJ4jmBEsmCTUIaOdFUJI4ArCxN7gm0oE7FR2KxCEjGCodkrBW8lo IJZMKOIqlVItFi6T05/IlOr4/ZB2JSyRVB+M7plJ8cBpWBTCrrm8KYqkA0C9lg6AHnKx tzbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=4wGAsQTEi4Qgp5h+YOog2hQUd2pljSipp7LhsEmoXVc=; b=ZrtFS3vvpqllFw9yZsTlD2URTj1hygA0wqN9tIwva5NbkjakkxIspAUtP4QB9WHUEC If6etCxWEYjySXclVuMkaRkEl1QO6TDAWrgSw/BEEfjKwLbViCZ2t1cmAqeb2HA4vwUc uBAZlhMrWEk892VcuciJa4xa9oFvWZi2k/kE+N0EIQIzQzjp7YE7B36OaEXtJompGBBb QmTxNgVjdm6+P7V1j3w8/TXJE3FKj8JY7goQJhxiwpmeKY0ugb6QZYRgqL32+YM6FPT6 uNbZdKij86Ek3S8icCncLn8crDd6MsaZRD59n7+/4nmxma0VHrkVbMzsv0UC3By/oNXS S8Kg== X-Gm-Message-State: ALKqPwdfxEEFAKl28St83QLopL9+GCXVLXLHK0G+4LYDHfX4n3xPuTiO cAYgVyHBSrHU4tAvpAkTHcnpC+St X-Google-Smtp-Source: AB8JxZoZMXoG8W0RSxv9BEQHsT8Aszzy2Jpd3ciRB1dRKHKYVy8lJOQ/neZlfnEN8OPatDuhNmkFSQ== X-Received: by 2002:a65:648a:: with SMTP id e10-v6mr18187269pgv.34.1526972270855; Mon, 21 May 2018 23:57:50 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Tue, 22 May 2018 08:57:34 +0200 Message-Id: <20180522065740.9710-7-pbonzini@redhat.com> In-Reply-To: <20180522065740.9710-1-pbonzini@redhat.com> References: <20180522065740.9710-1-pbonzini@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 22 May 2018 06:57:52 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 22 May 2018 06:57:52 +0000 (UTC) for IP:'209.85.192.169' DOMAIN:'mail-pf0-f169.google.com' HELO:'mail-pf0-f169.google.com' FROM:'paolo.bonzini@gmail.com' RCPT:'' X-RedHat-Spam-Score: -1.2 (DKIM_SIGNED, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_PASS, T_DKIM_INVALID) 209.85.192.169 mail-pf0-f169.google.com 209.85.192.169 mail-pf0-f169.google.com X-RedHat-Possible-Forgery: Paolo Bonzini X-Scanned-By: MIMEDefang 2.78 on 10.5.110.39 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.24 X-loop: patchew-devel@redhat.com Subject: [Patchew-devel] [PATCH 06/12] models: create Result model X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Tue, 22 May 2018 06:58:07 +0000 (UTC) X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The Result model has more or less a 1:1 correspondence with the fields in the REST results detail view, and thus with ResultSerializerFull. Until all plugins are converted, everything will still go through namedtuple results, so the namedtuple is kept, renamed to ResultTuple. The duplicated code will disappear at the end of this series. There is an extra field tracking the time of the last update, which will be used by the testing module; for simplicity it is not yet presented by the serializers, though it will be added when ResultTuple is dropped. The external interface of Result and ResultTuple is more or less the same, except that we have to get rid of the two Python objects that the tuple stores, obj and renderer. The renderer is eliminated by convention: the renderer of the new Result objects is always a PatchewModule, and the name of the result up to the first period (if any) identifies the renderer. As for obj, the parent object, the Result object has no clue whether it comes from a project or a message, so it is passed as a new parameter to render() and get_log_url(). The new parameter is easy to integrate in both the REST API SerializerMethodField getter and in the web view. Compared to properties, logs are always stored in the database, independent of the size. However, they are always stored in xz format compared to the JSON string format used by properties. Using xz will complicate migrations a little, since the log property won't be available there, but not as much as handling blobs. Signed-off-by: Paolo Bonzini --- api/migrations/0027_auto_20180521_0152.py | 38 ++++++++ api/models.py | 103 +++++++++++++++++++--- mods/git.py | 4 +- mods/testing.py | 6 +- 4 files changed, 134 insertions(+), 17 deletions(-) create mode 100644 api/migrations/0027_auto_20180521_0152.py diff --git a/api/migrations/0027_auto_20180521_0152.py b/api/migrations/002= 7_auto_20180521_0152.py new file mode 100644 index 0000000..94aa0d8 --- /dev/null +++ b/api/migrations/0027_auto_20180521_0152.py @@ -0,0 +1,38 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.12 on 2018-05-21 20:05 +from __future__ import unicode_literals + +import django.core.validators +from django.db import migrations, models +import jsonfield.fields + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0026_auto_20180426_0829'), + ] + + operations =3D [ + migrations.CreateModel( + name=3D'Result', + fields=3D[ + ('id', models.AutoField(auto_created=3DTrue, primary_key= =3DTrue, serialize=3DFalse, verbose_name=3D'ID')), + ('name', models.CharField(max_length=3D256)), + ('last_update', models.DateTimeField()), + ('status', models.CharField(db_index=3DTrue, max_length=3D= 7, validators=3D[django.core.validators.RegexValidator(code=3D'invalid', me= ssage=3D'status must be one of pending, success, failure, running', regex= =3D'pending|success|failure|running')])), + ('log_xz', models.BinaryField(blank=3DTrue, null=3DTrue)), + ('data', jsonfield.fields.JSONField()), + ], + ), + migrations.AddField( + model_name=3D'message', + name=3D'results', + field=3Dmodels.ManyToManyField(blank=3DTrue, to=3D'api.Result'= ), + ), + migrations.AddField( + model_name=3D'project', + name=3D'results', + field=3Dmodels.ManyToManyField(blank=3DTrue, to=3D'api.Result'= ), + ), + ] diff --git a/api/models.py b/api/models.py index 497e43c..ee54d04 100644 --- a/api/models.py +++ b/api/models.py @@ -14,7 +14,9 @@ import json import datetime import re =20 +from django.core import validators from django.db import models +from django.db.models import Q from django.contrib.auth.models import User from django.urls import reverse import jsonfield @@ -25,6 +27,86 @@ from event import emit_event, declare_event from .blobs import save_blob, load_blob, load_blob_json import mod =20 +class Result(models.Model): + PENDING =3D 'pending' + SUCCESS =3D 'success' + FAILURE =3D 'failure' + RUNNING =3D 'running' + VALID_STATUSES =3D (PENDING, SUCCESS, FAILURE, RUNNING) + VALID_STATUSES_RE =3D '|'.join(VALID_STATUSES) + + name =3D models.CharField(max_length=3D256) + last_update =3D models.DateTimeField() + status =3D models.CharField(max_length=3D7, db_index=3DTrue, validator= s=3D[ + validators.RegexValidator(regex=3DVALID_STATUSES_RE, + message=3D'status must be one of ' + ', = '.join(VALID_STATUSES), + code=3D'invalid')]) + log_xz =3D models.BinaryField(blank=3DTrue, null=3DTrue) + data =3D jsonfield.JSONField() + + def is_success(self): + return self.status =3D=3D self.SUCCESS + + def is_failure(self): + return self.status =3D=3D self.FAILURE + + def is_completed(self): + return self.is_success() or self.is_failure() + + def is_pending(self): + return self.status =3D=3D self.PENDING + + def is_running(self): + return self.status =3D=3D self.RUNNING + + def save(self): + self.last_update =3D datetime.datetime.utcnow() + return super(Result, self).save() + + @property + def renderer(self): + found =3D re.match("^[^.]*", self.name) + return mod.get_module(found[0]) if found else None + + def render(self, obj): + if self.renderer is None: + return None + return self.renderer.render_result(self, obj) + + @property + def log(self): + if hasattr(self, "_log"): + return self._log + if self.log_xz is None: + self._log =3D None + else: + self._log =3D lzma.decompress(self.log_xz).decode("utf-8") + return self._log + + @log.setter + def log(self, value): + self._log =3D value + if value is None: + self.log_xz =3D None + else: + self.log_xz =3D lzma.compress(value.encode("utf-8")) + + def get_log_url(self, obj, request=3DNone): + if not self.is_completed() or self.renderer is None: + return None + log_url =3D self.renderer.get_result_log_url(obj, self.name) + if log_url is not None and request is not None: + log_url =3D request.build_absolute_uri(log_url) + return log_url + + @staticmethod + def get_result_tuples(obj, module, results): + name_filter =3D Q(name=3Dmodule) | Q(name__startswith=3Dmodule + '= .') + renderer =3D mod.get_module(module) + for r in obj.results.filter(name_filter): + results.append(ResultTuple(name=3Dr.name, obj=3Dobj, status=3D= r.status, + log=3Dr.log, data=3Dr.data, rendere= r=3Drenderer)) + class Project(models.Model): name =3D models.CharField(max_length=3D1024, db_index=3DTrue, unique= =3DTrue, help_text=3D"""The name of the project""") @@ -58,6 +140,7 @@ class Project(models.Model): top project which has parent_project=3DNULL""") maintainers =3D models.ManyToManyField(User, blank=3DTrue) + results =3D models.ManyToManyField(Result, blank=3DTrue) =20 def __str__(self): return self.name @@ -319,6 +402,7 @@ class Message(models.Model): num_patches =3D models.IntegerField(null=3DFalse, default=3D-1, blank= =3DTrue) =20 objects =3D MessageManager() + results =3D models.ManyToManyField(Result, blank=3DTrue) =20 def save_mbox(self, mbox_blob): save_blob(mbox_blob, self.message_id) @@ -595,34 +679,29 @@ class Module(models.Model): def __str__(self): return self.name =20 -class Result(namedtuple("Result", "name status log obj data renderer")): +class ResultTuple(namedtuple("ResultTuple", "name status log obj data rend= erer")): __slots__ =3D () - PENDING =3D 'pending' - SUCCESS =3D 'success' - FAILURE =3D 'failure' - RUNNING =3D 'running' - VALID_STATUSES =3D (PENDING, SUCCESS, FAILURE, RUNNING) =20 def __new__(cls, name, status, obj, log=3DNone, data=3DNone, renderer= =3DNone): - if status not in cls.VALID_STATUSES: + if status not in Result.VALID_STATUSES: raise ValueError("invalid value '%s' for status field" % statu= s) - return super(cls, Result).__new__(cls, status=3Dstatus, log=3Dlog, + return super(cls, ResultTuple).__new__(cls, status=3Dstatus, log= =3Dlog, obj=3Dobj, data=3Ddata, name=3Dn= ame, renderer=3Drenderer) =20 def is_success(self): - return self.status =3D=3D self.SUCCESS + return self.status =3D=3D Result.SUCCESS =20 def is_failure(self): - return self.status =3D=3D self.FAILURE + return self.status =3D=3D Result.FAILURE =20 def is_completed(self): return self.is_success() or self.is_failure() =20 def is_pending(self): - return self.status =3D=3D self.PENDING + return self.status =3D=3D Result.PENDING =20 def is_running(self): - return self.status =3D=3D self.RUNNING + return self.status =3D=3D Result.RUNNING =20 def render(self): if self.renderer is None: diff --git a/mods/git.py b/mods/git.py index a216ea2..eae9be6 100644 --- a/mods/git.py +++ b/mods/git.py @@ -18,7 +18,7 @@ from django.core.exceptions import PermissionDenied from django.utils.html import format_html from mod import PatchewModule from event import declare_event, register_handler, emit_event -from api.models import Message, MessageProperty, Result +from api.models import Message, MessageProperty, Result, ResultTuple from api.rest import PluginMethodField from api.views import APILoginRequiredView, prepare_series from patchew.logviewer import LogView @@ -150,7 +150,7 @@ class GitModule(PatchewModule): status =3D Result.SUCCESS else: status =3D Result.PENDING - results.append(Result(name=3D'git', obj=3Dobj, status=3Dstatus, + results.append(ResultTuple(name=3D'git', obj=3Dobj, status=3Dstatu= s, log=3Dlog, data=3Ddata, renderer=3Dself)) =20 def prepare_message_hook(self, request, message, detailed): diff --git a/mods/testing.py b/mods/testing.py index 2d379cb..98732e3 100644 --- a/mods/testing.py +++ b/mods/testing.py @@ -17,7 +17,7 @@ from mod import PatchewModule import time import math from api.views import APILoginRequiredView -from api.models import Message, MessageProperty, Project, Result +from api.models import Message, MessageProperty, Project, Result, ResultTu= ple from api.search import SearchEngine from event import emit_event, declare_event, register_handler from patchew.logviewer import LogView @@ -270,12 +270,12 @@ class TestingModule(PatchewModule): =20 data =3D p.copy() del data['passed'] - results.append(Result(name=3D'testing.' + tn, obj=3Dobj, statu= s=3Dpassed_str, + results.append(ResultTuple(name=3D'testing.' + tn, obj=3Dobj, = status=3Dpassed_str, log=3Dlog, data=3Ddata, renderer=3Dself)) =20 if obj.get_property("testing.ready"): for tn in all_tests: - results.append(Result(name=3D'testing.' + tn, obj=3Dobj, s= tatus=3D'pending')) + results.append(ResultTuple(name=3D'testing.' + tn, obj=3Do= bj, status=3D'pending')) =20 def prepare_message_hook(self, request, message, detailed): if not message.is_series_head: --=20 2.17.0 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://www.redhat.com/mailman/listinfo/patchew-devel