From nobody Tue May 13 08:31:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=patchew-devel-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1527761041160455.4801862042141; Thu, 31 May 2018 03:04:01 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4637CE14BC; Thu, 31 May 2018 10:04:00 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 36C1A2010CA1; Thu, 31 May 2018 10:04:00 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 28EB2180BA80; Thu, 31 May 2018 10:04:00 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w4VA3xuo024942 for ; Thu, 31 May 2018 06:03:59 -0400 Received: by smtp.corp.redhat.com (Postfix) id ADE2A2010CA6; Thu, 31 May 2018 10:03:59 +0000 (UTC) Received: from mx1.redhat.com (ext-mx09.extmail.prod.ext.phx2.redhat.com [10.5.110.38]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A59702010CA1 for ; Thu, 31 May 2018 10:03:57 +0000 (UTC) Received: from mail-wr0-f179.google.com (mail-wr0-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 64BAA4DD7D for ; Thu, 31 May 2018 10:03:46 +0000 (UTC) Received: by mail-wr0-f179.google.com with SMTP id u12-v6so32365701wrn.8 for ; Thu, 31 May 2018 03:03:46 -0700 (PDT) Received: from donizetti.lan (dynamic-adsl-78-12-189-60.clienti.tiscali.it. [78.12.189.60]) by smtp.gmail.com with ESMTPSA id g16-v6sm3520664wro.86.2018.05.31.03.03.43 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 31 May 2018 03:03:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=FhAu9rLyi9YknI0xn7WXDzQCaYVJmJSFsGZJRFmUgt4=; b=E3MRqJ8x+YaU6tNOQ/DSEV8rY8mcLt68sso47fTXEP/D4CfOeThNmQ/yJ5dZ641fMI lnaxlGq+CIW7fMqPrkA0k8yHzWmRfV3sG/CKleuEAc1lkWO9KdIrMPnPxBwRiTbh3j8q 6cFlNQxgyk4a0DJSfbXAJ3gZnf7YwtsMnsp1d4jM/7ZpOELNScGL1wGvaSa4+ss2BoWi +//k9rlW7sXyT0rParOOsieF7dg05t1+wFh1h9GhD2c3HvS58C3pcQ7sGkWTb9LGWCp5 cX8XsiTGUvy0dLeRnFU3zhJmQSSU8noetpInKYVoN8HiLeWfV6rRhI2DRhlVdUjg+p85 dLXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=FhAu9rLyi9YknI0xn7WXDzQCaYVJmJSFsGZJRFmUgt4=; b=cFSom9yMPspp3gY/EY31gQe/Z6XBzlKZPrg0MR2VV8QnPUBvZermwFfhW/DNuTp8In bLw3k2dNV9Ih52hoMwU9KIreWs6C8OPyiBjNduSoy2n3Wj2VVRU8MRGNls6E0jb6+amH 6KugpsCqhgDgze2XXkHl7B0+mNM+auymE0lx3+t7++3jQq/opSbN529x7Y9cEcuUvFEV TC20ZbzS1z8E6oGBPDL9Blzx6/vmU8OECIF9Zv413h7m63sCdbJh3d01a2egP5C/DN8y qMzS7nUlJMO7eJDClye/Oru9BFCHRkKDZRYSEuUwcTuaNtd2nWQ9GrAUSnoUEODqPhNS wsWg== X-Gm-Message-State: ALKqPwf2KINTvRH00bDH+f2GEvZ2xbjHmve/T+MUXIkrjkSzTDrizBgf kkH3iEhjKX2e+5DmanNiL8DTMSpl X-Google-Smtp-Source: ADUXVKJaZ+aH1xk7DNSx3UppUMqHNcKDAVOQGygw0McViU4s2mElC+GhTb6iq2MXwCbvwicbb15FJw== X-Received: by 2002:adf:9d43:: with SMTP id o3-v6mr5110863wre.210.1527761024660; Thu, 31 May 2018 03:03:44 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 31 May 2018 12:03:28 +0200 Message-Id: <20180531100334.2249-5-pbonzini@redhat.com> In-Reply-To: <20180531100334.2249-1-pbonzini@redhat.com> References: <20180531100334.2249-1-pbonzini@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 31 May 2018 10:03:46 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 31 May 2018 10:03:46 +0000 (UTC) for IP:'209.85.128.179' DOMAIN:'mail-wr0-f179.google.com' HELO:'mail-wr0-f179.google.com' FROM:'paolo.bonzini@gmail.com' RCPT:'' X-RedHat-Spam-Score: -1.1 (DKIM_SIGNED, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_PASS, T_DKIM_INVALID) 209.85.128.179 mail-wr0-f179.google.com 209.85.128.179 mail-wr0-f179.google.com X-RedHat-Possible-Forgery: Paolo Bonzini X-Scanned-By: MIMEDefang 2.78 on 10.5.110.38 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.25 X-loop: patchew-devel@redhat.com Subject: [Patchew-devel] [PATCH 04/10] models: create Result model X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.25 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 31 May 2018 10:04:00 +0000 (UTC) X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The Result model has more or less a 1:1 correspondence with the fields in the REST results detail view, and thus with ResultSerializerFull. Until all plugins are converted, everything will still go through namedtuple results, so the namedtuple is kept, renamed to ResultTuple. The duplicated code will disappear at the end of this series. There is an extra field tracking the time of the last update, which will be used by the testing module; for simplicity it is not yet presented by the serializers, though it will be added when ResultTuple is dropped. The external interface of Result and ResultTuple is more or less the same, except that there is no more renderer. The renderer is now discovered by convention: it is always a PatchewModule, and the name of the result up to the first period (if any) identifies it. As for obj, the parent object, the Result object has no clue whether it comes from a project or a message, but the subclasses (ProjectResult and MessageResult) do, so no change is needed. Compared to properties, logs are always stored in the database, independent of the size. However, they are always stored in xz format compared to the JSON string format used by properties. Using xz will complicate migrations a little, since the log property won't be available there, but not as much as handling blobs. Signed-off-by: Paolo Bonzini --- api/migrations/0027_auto_20180521_0152.py | 61 +++++++++ api/models.py | 150 ++++++++++++++++++++-- mods/git.py | 4 +- mods/testing.py | 6 +- 4 files changed, 204 insertions(+), 17 deletions(-) create mode 100644 api/migrations/0027_auto_20180521_0152.py diff --git a/api/migrations/0027_auto_20180521_0152.py b/api/migrations/002= 7_auto_20180521_0152.py new file mode 100644 index 0000000..d6833ef --- /dev/null +++ b/api/migrations/0027_auto_20180521_0152.py @@ -0,0 +1,61 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.13 on 2018-05-31 05:44 +from __future__ import unicode_literals + +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion +import jsonfield.encoder +import jsonfield.fields + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0026_auto_20180426_0829'), + ] + + operations =3D [ + migrations.CreateModel( + name=3D'LogEntry', + fields=3D[ + ('id', models.AutoField(auto_created=3DTrue, primary_key= =3DTrue, serialize=3DFalse, verbose_name=3D'ID')), + ('data_xz', models.BinaryField()), + ], + ), + migrations.CreateModel( + name=3D'Result', + fields=3D[ + ('id', models.AutoField(auto_created=3DTrue, primary_key= =3DTrue, serialize=3DFalse, verbose_name=3D'ID')), + ('name', models.CharField(max_length=3D256)), + ('last_update', models.DateTimeField()), + ('status', models.CharField(max_length=3D7, validators=3D[= django.core.validators.RegexValidator(code=3D'invalid', message=3D'status m= ust be one of pending, success, failure, running', regex=3D'pending|success= |failure|running')])), + ('data', jsonfield.fields.JSONField(dump_kwargs=3D{'cls': = jsonfield.encoder.JSONEncoder, 'separators': (',', ':')}, load_kwargs=3D{})= ), + ], + ), + migrations.CreateModel( + name=3D'MessageResult', + fields=3D[ + ('result_ptr', models.OneToOneField(auto_created=3DTrue, o= n_delete=3Ddjango.db.models.deletion.CASCADE, parent_link=3DTrue, primary_k= ey=3DTrue, serialize=3DFalse, to=3D'api.Result')), + ('message', models.ForeignKey(on_delete=3Ddjango.db.models= .deletion.CASCADE, related_name=3D'results', to=3D'api.Message')), + ], + bases=3D('api.result',), + ), + migrations.CreateModel( + name=3D'ProjectResult', + fields=3D[ + ('result_ptr', models.OneToOneField(auto_created=3DTrue, o= n_delete=3Ddjango.db.models.deletion.CASCADE, parent_link=3DTrue, primary_k= ey=3DTrue, serialize=3DFalse, to=3D'api.Result')), + ('project', models.ForeignKey(on_delete=3Ddjango.db.models= .deletion.CASCADE, related_name=3D'results', to=3D'api.Project')), + ], + bases=3D('api.result',), + ), + migrations.AddField( + model_name=3D'result', + name=3D'log_entry', + field=3Dmodels.OneToOneField(null=3DTrue, on_delete=3Ddjango.d= b.models.deletion.CASCADE, to=3D'api.LogEntry'), + ), + migrations.AlterIndexTogether( + name=3D'result', + index_together=3Dset([('status', 'name')]), + ), + ] diff --git a/api/models.py b/api/models.py index 96798a6..b5f7ebd 100644 --- a/api/models.py +++ b/api/models.py @@ -14,7 +14,9 @@ import json import datetime import re =20 +from django.core import validators from django.db import models +from django.db.models import Q from django.contrib.auth.models import User from django.urls import reverse import jsonfield @@ -25,6 +27,115 @@ from event import emit_event, declare_event from .blobs import save_blob, load_blob, load_blob_json import mod =20 +class LogEntry(models.Model): + data_xz =3D models.BinaryField() + + @property + def data(self): + if not hasattr(self, "_data"): + self._data =3D lzma.decompress(self.data_xz).decode("utf-8") + return self._data + + @data.setter + def data(self, value): + self._data =3D value + self.data_xz =3D lzma.compress(value.encode("utf-8")) + +class Result(models.Model): + PENDING =3D 'pending' + SUCCESS =3D 'success' + FAILURE =3D 'failure' + RUNNING =3D 'running' + VALID_STATUSES =3D (PENDING, SUCCESS, FAILURE, RUNNING) + VALID_STATUSES_RE =3D '|'.join(VALID_STATUSES) + + name =3D models.CharField(max_length=3D256) + last_update =3D models.DateTimeField() + status =3D models.CharField(max_length=3D7, validators=3D[ + validators.RegexValidator(regex=3DVALID_STATUSES_RE, + message=3D'status must be one of ' + ', = '.join(VALID_STATUSES), + code=3D'invalid')]) + log_entry =3D models.OneToOneField(LogEntry, on_delete=3Dmodels.CASCAD= E, + null=3DTrue) + data =3D jsonfield.JSONField() + + class Meta: + index_together =3D [('status', 'name')] + + def is_success(self): + return self.status =3D=3D self.SUCCESS + + def is_failure(self): + return self.status =3D=3D self.FAILURE + + def is_completed(self): + return self.is_success() or self.is_failure() + + def is_pending(self): + return self.status =3D=3D self.PENDING + + def is_running(self): + return self.status =3D=3D self.RUNNING + + def save(self): + self.last_update =3D datetime.datetime.utcnow() + return super(Result, self).save() + + @property + def renderer(self): + found =3D re.match("^[^.]*", self.name) + return mod.get_module(found[0]) if found else None + + @property + def obj(self): + return None + + def render(self): + if self.renderer is None: + return None + return self.renderer.render_result(self) + + @property + def log(self): + if self.log_entry is None: + return None + else: + return self.log_entry.data + + @log.setter + def log(self, value): + entry =3D self.log_entry + if value is None: + if entry is not None: + self.log_entry =3D None + entry.delete() + else: + if entry is None: + entry =3D LogEntry() + entry.data =3D value + entry.save() + if self.log_entry is None: + self.log_entry =3D entry + + def get_log_url(self, request=3DNone): + if not self.is_completed() or self.renderer is None: + return None + log_url =3D self.renderer.get_result_log_url(self) + if log_url is not None and request is not None: + log_url =3D request.build_absolute_uri(log_url) + return log_url + + @staticmethod + def get_result_tuples(obj, module, results): + name_filter =3D Q(name=3Dmodule) | Q(name__startswith=3Dmodule + '= .') + renderer =3D mod.get_module(module) + for r in obj.results.filter(name_filter): + results.append(ResultTuple(name=3Dr.name, obj=3Dobj, status=3D= r.status, + log=3Dr.log, data=3Dr.data, rendere= r=3Drenderer)) + + def __str__(self): + return '%s (%s)' % (self.name, self.status) + class Project(models.Model): name =3D models.CharField(max_length=3D1024, db_index=3DTrue, unique= =3DTrue, help_text=3D"""The name of the project""") @@ -183,6 +294,16 @@ class Project(models.Model): series.save() return len(updated_series) =20 + def create_result(self, **kwargs): + return ProjectResult(project=3Dself, **kwargs) + +class ProjectResult(Result): + project =3D models.ForeignKey(Project, related_name=3D'results') + + @property + def obj(self): + return self.project + class ProjectProperty(models.Model): project =3D models.ForeignKey('Project', on_delete=3Dmodels.CASCADE) name =3D models.CharField(max_length=3D1024, db_index=3DTrue) @@ -592,12 +713,22 @@ class Message(models.Model): self.save() emit_event("SeriesComplete", project=3Dself.project, series=3Dself) =20 + def create_result(self, **kwargs): + return MessageResult(message=3Dself, **kwargs) + def __str__(self): return self.subject =20 class Meta: unique_together =3D ('project', 'message_id',) =20 +class MessageResult(Result): + message =3D models.ForeignKey(Message, related_name=3D'results') + + @property + def obj(self): + return self.message + class MessageProperty(models.Model): message =3D models.ForeignKey('Message', on_delete=3Dmodels.CASCADE, related_name=3D'properties') @@ -626,34 +757,29 @@ class Module(models.Model): def __str__(self): return self.name =20 -class Result(namedtuple("Result", "name status log obj data renderer")): +class ResultTuple(namedtuple("ResultTuple", "name status log obj data rend= erer")): __slots__ =3D () - PENDING =3D 'pending' - SUCCESS =3D 'success' - FAILURE =3D 'failure' - RUNNING =3D 'running' - VALID_STATUSES =3D (PENDING, SUCCESS, FAILURE, RUNNING) =20 def __new__(cls, name, status, obj, log=3DNone, data=3DNone, renderer= =3DNone): - if status not in cls.VALID_STATUSES: + if status not in Result.VALID_STATUSES: raise ValueError("invalid value '%s' for status field" % statu= s) - return super(cls, Result).__new__(cls, status=3Dstatus, log=3Dlog, + return super(cls, ResultTuple).__new__(cls, status=3Dstatus, log= =3Dlog, obj=3Dobj, data=3Ddata, name=3Dn= ame, renderer=3Drenderer) =20 def is_success(self): - return self.status =3D=3D self.SUCCESS + return self.status =3D=3D Result.SUCCESS =20 def is_failure(self): - return self.status =3D=3D self.FAILURE + return self.status =3D=3D Result.FAILURE =20 def is_completed(self): return self.is_success() or self.is_failure() =20 def is_pending(self): - return self.status =3D=3D self.PENDING + return self.status =3D=3D Result.PENDING =20 def is_running(self): - return self.status =3D=3D self.RUNNING + return self.status =3D=3D Result.RUNNING =20 def render(self): if self.renderer is None: diff --git a/mods/git.py b/mods/git.py index 4bfb5c6..fadce4c 100644 --- a/mods/git.py +++ b/mods/git.py @@ -18,7 +18,7 @@ from django.core.exceptions import PermissionDenied from django.utils.html import format_html from mod import PatchewModule from event import declare_event, register_handler, emit_event -from api.models import Message, MessageProperty, Result +from api.models import Message, MessageProperty, Result, ResultTuple from api.rest import PluginMethodField from api.views import APILoginRequiredView, prepare_series from patchew.logviewer import LogView @@ -150,7 +150,7 @@ class GitModule(PatchewModule): status =3D Result.SUCCESS else: status =3D Result.PENDING - results.append(Result(name=3D'git', obj=3Dobj, status=3Dstatus, + results.append(ResultTuple(name=3D'git', obj=3Dobj, status=3Dstatu= s, log=3Dlog, data=3Ddata, renderer=3Dself)) =20 def prepare_message_hook(self, request, message, detailed): diff --git a/mods/testing.py b/mods/testing.py index efa2d82..ca3d60d 100644 --- a/mods/testing.py +++ b/mods/testing.py @@ -17,7 +17,7 @@ from mod import PatchewModule import time import math from api.views import APILoginRequiredView -from api.models import Message, MessageProperty, Project, Result +from api.models import Message, MessageProperty, Project, Result, ResultTu= ple from api.search import SearchEngine from event import emit_event, declare_event, register_handler from patchew.logviewer import LogView @@ -270,12 +270,12 @@ class TestingModule(PatchewModule): =20 data =3D p.copy() del data['passed'] - results.append(Result(name=3D'testing.' + tn, obj=3Dobj, statu= s=3Dpassed_str, + results.append(ResultTuple(name=3D'testing.' + tn, obj=3Dobj, = status=3Dpassed_str, log=3Dlog, data=3Ddata, renderer=3Dself)) =20 if obj.get_property("testing.ready"): for tn in all_tests: - results.append(Result(name=3D'testing.' + tn, obj=3Dobj, s= tatus=3D'pending')) + results.append(ResultTuple(name=3D'testing.' + tn, obj=3Do= bj, status=3D'pending')) =20 def prepare_message_hook(self, request, message, detailed): if not message.is_series_head: --=20 2.17.0 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://www.redhat.com/mailman/listinfo/patchew-devel