From nobody Tue May 13 19:25:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=patchew-devel-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1528886273817733.7685040572759; Wed, 13 Jun 2018 03:37:53 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CDDC981DF5; Wed, 13 Jun 2018 10:37:51 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BA727600C8; Wed, 13 Jun 2018 10:37:51 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id AFF904BB78; Wed, 13 Jun 2018 10:37:51 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w5DAbnUV009756 for ; Wed, 13 Jun 2018 06:37:50 -0400 Received: by smtp.corp.redhat.com (Postfix) id DF7A9210C6DC; Wed, 13 Jun 2018 10:37:49 +0000 (UTC) Received: from donizetti.redhat.com (unknown [10.36.118.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 39B482026987 for ; Wed, 13 Jun 2018 10:37:49 +0000 (UTC) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Wed, 13 Jun 2018 12:37:38 +0200 Message-Id: <20180613103744.30891-5-pbonzini@redhat.com> In-Reply-To: <20180613103744.30891-1-pbonzini@redhat.com> References: <20180613103744.30891-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-loop: patchew-devel@redhat.com Subject: [Patchew-devel] [PATCH 04/10] models: create Result model X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Wed, 13 Jun 2018 10:37:51 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The Result model has more or less a 1:1 correspondence with the fields in the REST results detail view, and thus with ResultSerializerFull. Until all plugins are converted, everything will still go through namedtuple results, so the namedtuple is kept, renamed to ResultTuple. The duplicated code will disappear at the end of this series. There is an extra field tracking the time of the last update, which will be used by the testing module; for simplicity it is not yet presented by the serializers, though it will be added when ResultTuple is dropped. The external interface of Result and ResultTuple is more or less the same, except that there is no more renderer. The renderer is now discovered by convention: it is always a PatchewModule, and the name of the result up to the first period (if any) identifies it. As for obj, the parent object, the Result object has no clue whether it comes from a project or a message, but the subclasses (ProjectResult and MessageResult) do, so no change is needed. Compared to properties, logs are always stored in the database, independent of the size. However, they are always stored in xz format compared to the JSON string format used by properties. Using xz will complicate migrations a little, since the log property won't be available there, but not as much as handling blobs. Signed-off-by: Paolo Bonzini --- api/migrations/0027_auto_20180521_0152.py | 61 +++++++++ api/models.py | 150 ++++++++++++++++++++-- mods/git.py | 4 +- mods/testing.py | 6 +- 4 files changed, 204 insertions(+), 17 deletions(-) create mode 100644 api/migrations/0027_auto_20180521_0152.py diff --git a/api/migrations/0027_auto_20180521_0152.py b/api/migrations/002= 7_auto_20180521_0152.py new file mode 100644 index 0000000..d6833ef --- /dev/null +++ b/api/migrations/0027_auto_20180521_0152.py @@ -0,0 +1,61 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.13 on 2018-05-31 05:44 +from __future__ import unicode_literals + +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion +import jsonfield.encoder +import jsonfield.fields + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0026_auto_20180426_0829'), + ] + + operations =3D [ + migrations.CreateModel( + name=3D'LogEntry', + fields=3D[ + ('id', models.AutoField(auto_created=3DTrue, primary_key= =3DTrue, serialize=3DFalse, verbose_name=3D'ID')), + ('data_xz', models.BinaryField()), + ], + ), + migrations.CreateModel( + name=3D'Result', + fields=3D[ + ('id', models.AutoField(auto_created=3DTrue, primary_key= =3DTrue, serialize=3DFalse, verbose_name=3D'ID')), + ('name', models.CharField(max_length=3D256)), + ('last_update', models.DateTimeField()), + ('status', models.CharField(max_length=3D7, validators=3D[= django.core.validators.RegexValidator(code=3D'invalid', message=3D'status m= ust be one of pending, success, failure, running', regex=3D'pending|success= |failure|running')])), + ('data', jsonfield.fields.JSONField(dump_kwargs=3D{'cls': = jsonfield.encoder.JSONEncoder, 'separators': (',', ':')}, load_kwargs=3D{})= ), + ], + ), + migrations.CreateModel( + name=3D'MessageResult', + fields=3D[ + ('result_ptr', models.OneToOneField(auto_created=3DTrue, o= n_delete=3Ddjango.db.models.deletion.CASCADE, parent_link=3DTrue, primary_k= ey=3DTrue, serialize=3DFalse, to=3D'api.Result')), + ('message', models.ForeignKey(on_delete=3Ddjango.db.models= .deletion.CASCADE, related_name=3D'results', to=3D'api.Message')), + ], + bases=3D('api.result',), + ), + migrations.CreateModel( + name=3D'ProjectResult', + fields=3D[ + ('result_ptr', models.OneToOneField(auto_created=3DTrue, o= n_delete=3Ddjango.db.models.deletion.CASCADE, parent_link=3DTrue, primary_k= ey=3DTrue, serialize=3DFalse, to=3D'api.Result')), + ('project', models.ForeignKey(on_delete=3Ddjango.db.models= .deletion.CASCADE, related_name=3D'results', to=3D'api.Project')), + ], + bases=3D('api.result',), + ), + migrations.AddField( + model_name=3D'result', + name=3D'log_entry', + field=3Dmodels.OneToOneField(null=3DTrue, on_delete=3Ddjango.d= b.models.deletion.CASCADE, to=3D'api.LogEntry'), + ), + migrations.AlterIndexTogether( + name=3D'result', + index_together=3Dset([('status', 'name')]), + ), + ] diff --git a/api/models.py b/api/models.py index 0545b80..6288d8d 100644 --- a/api/models.py +++ b/api/models.py @@ -14,7 +14,9 @@ import json import datetime import re =20 +from django.core import validators from django.db import models +from django.db.models import Q from django.contrib.auth.models import User from django.urls import reverse import jsonfield @@ -25,6 +27,115 @@ from event import emit_event, declare_event from .blobs import save_blob, load_blob, load_blob_json import mod =20 +class LogEntry(models.Model): + data_xz =3D models.BinaryField() + + @property + def data(self): + if not hasattr(self, "_data"): + self._data =3D lzma.decompress(self.data_xz).decode("utf-8") + return self._data + + @data.setter + def data(self, value): + self._data =3D value + self.data_xz =3D lzma.compress(value.encode("utf-8")) + +class Result(models.Model): + PENDING =3D 'pending' + SUCCESS =3D 'success' + FAILURE =3D 'failure' + RUNNING =3D 'running' + VALID_STATUSES =3D (PENDING, SUCCESS, FAILURE, RUNNING) + VALID_STATUSES_RE =3D '|'.join(VALID_STATUSES) + + name =3D models.CharField(max_length=3D256) + last_update =3D models.DateTimeField() + status =3D models.CharField(max_length=3D7, validators=3D[ + validators.RegexValidator(regex=3DVALID_STATUSES_RE, + message=3D'status must be one of ' + ', = '.join(VALID_STATUSES), + code=3D'invalid')]) + log_entry =3D models.OneToOneField(LogEntry, on_delete=3Dmodels.CASCAD= E, + null=3DTrue) + data =3D jsonfield.JSONField() + + class Meta: + index_together =3D [('status', 'name')] + + def is_success(self): + return self.status =3D=3D self.SUCCESS + + def is_failure(self): + return self.status =3D=3D self.FAILURE + + def is_completed(self): + return self.is_success() or self.is_failure() + + def is_pending(self): + return self.status =3D=3D self.PENDING + + def is_running(self): + return self.status =3D=3D self.RUNNING + + def save(self): + self.last_update =3D datetime.datetime.utcnow() + return super(Result, self).save() + + @property + def renderer(self): + found =3D re.match("^[^.]*", self.name) + return mod.get_module(found.group(0)) if found else None + + @property + def obj(self): + return None + + def render(self): + if self.renderer is None: + return None + return self.renderer.render_result(self) + + @property + def log(self): + if self.log_entry is None: + return None + else: + return self.log_entry.data + + @log.setter + def log(self, value): + entry =3D self.log_entry + if value is None: + if entry is not None: + self.log_entry =3D None + entry.delete() + else: + if entry is None: + entry =3D LogEntry() + entry.data =3D value + entry.save() + if self.log_entry is None: + self.log_entry =3D entry + + def get_log_url(self, request=3DNone): + if not self.is_completed() or self.renderer is None: + return None + log_url =3D self.renderer.get_result_log_url(self) + if log_url is not None and request is not None: + log_url =3D request.build_absolute_uri(log_url) + return log_url + + @staticmethod + def get_result_tuples(obj, module, results): + name_filter =3D Q(name=3Dmodule) | Q(name__startswith=3Dmodule + '= .') + renderer =3D mod.get_module(module) + for r in obj.results.filter(name_filter): + results.append(ResultTuple(name=3Dr.name, obj=3Dobj, status=3D= r.status, + log=3Dr.log, data=3Dr.data, rendere= r=3Drenderer)) + + def __str__(self): + return '%s (%s)' % (self.name, self.status) + class Project(models.Model): name =3D models.CharField(max_length=3D1024, db_index=3DTrue, unique= =3DTrue, help_text=3D"""The name of the project""") @@ -183,6 +294,16 @@ class Project(models.Model): series.save() return len(updated_series) =20 + def create_result(self, **kwargs): + return ProjectResult(project=3Dself, **kwargs) + +class ProjectResult(Result): + project =3D models.ForeignKey(Project, related_name=3D'results') + + @property + def obj(self): + return self.project + class ProjectProperty(models.Model): project =3D models.ForeignKey('Project', on_delete=3Dmodels.CASCADE) name =3D models.CharField(max_length=3D1024, db_index=3DTrue) @@ -592,12 +713,22 @@ class Message(models.Model): self.save() emit_event("SeriesComplete", project=3Dself.project, series=3Dself) =20 + def create_result(self, **kwargs): + return MessageResult(message=3Dself, **kwargs) + def __str__(self): return self.subject =20 class Meta: unique_together =3D ('project', 'message_id',) =20 +class MessageResult(Result): + message =3D models.ForeignKey(Message, related_name=3D'results') + + @property + def obj(self): + return self.message + class MessageProperty(models.Model): message =3D models.ForeignKey('Message', on_delete=3Dmodels.CASCADE, related_name=3D'properties') @@ -626,34 +757,29 @@ class Module(models.Model): def __str__(self): return self.name =20 -class Result(namedtuple("Result", "name status log obj data renderer")): +class ResultTuple(namedtuple("ResultTuple", "name status log obj data rend= erer")): __slots__ =3D () - PENDING =3D 'pending' - SUCCESS =3D 'success' - FAILURE =3D 'failure' - RUNNING =3D 'running' - VALID_STATUSES =3D (PENDING, SUCCESS, FAILURE, RUNNING) =20 def __new__(cls, name, status, obj, log=3DNone, data=3DNone, renderer= =3DNone): - if status not in cls.VALID_STATUSES: + if status not in Result.VALID_STATUSES: raise ValueError("invalid value '%s' for status field" % statu= s) - return super(cls, Result).__new__(cls, status=3Dstatus, log=3Dlog, + return super(cls, ResultTuple).__new__(cls, status=3Dstatus, log= =3Dlog, obj=3Dobj, data=3Ddata, name=3Dn= ame, renderer=3Drenderer) =20 def is_success(self): - return self.status =3D=3D self.SUCCESS + return self.status =3D=3D Result.SUCCESS =20 def is_failure(self): - return self.status =3D=3D self.FAILURE + return self.status =3D=3D Result.FAILURE =20 def is_completed(self): return self.is_success() or self.is_failure() =20 def is_pending(self): - return self.status =3D=3D self.PENDING + return self.status =3D=3D Result.PENDING =20 def is_running(self): - return self.status =3D=3D self.RUNNING + return self.status =3D=3D Result.RUNNING =20 def render(self): if self.renderer is None: diff --git a/mods/git.py b/mods/git.py index 4bfb5c6..fadce4c 100644 --- a/mods/git.py +++ b/mods/git.py @@ -18,7 +18,7 @@ from django.core.exceptions import PermissionDenied from django.utils.html import format_html from mod import PatchewModule from event import declare_event, register_handler, emit_event -from api.models import Message, MessageProperty, Result +from api.models import Message, MessageProperty, Result, ResultTuple from api.rest import PluginMethodField from api.views import APILoginRequiredView, prepare_series from patchew.logviewer import LogView @@ -150,7 +150,7 @@ class GitModule(PatchewModule): status =3D Result.SUCCESS else: status =3D Result.PENDING - results.append(Result(name=3D'git', obj=3Dobj, status=3Dstatus, + results.append(ResultTuple(name=3D'git', obj=3Dobj, status=3Dstatu= s, log=3Dlog, data=3Ddata, renderer=3Dself)) =20 def prepare_message_hook(self, request, message, detailed): diff --git a/mods/testing.py b/mods/testing.py index efa2d82..ca3d60d 100644 --- a/mods/testing.py +++ b/mods/testing.py @@ -17,7 +17,7 @@ from mod import PatchewModule import time import math from api.views import APILoginRequiredView -from api.models import Message, MessageProperty, Project, Result +from api.models import Message, MessageProperty, Project, Result, ResultTu= ple from api.search import SearchEngine from event import emit_event, declare_event, register_handler from patchew.logviewer import LogView @@ -270,12 +270,12 @@ class TestingModule(PatchewModule): =20 data =3D p.copy() del data['passed'] - results.append(Result(name=3D'testing.' + tn, obj=3Dobj, statu= s=3Dpassed_str, + results.append(ResultTuple(name=3D'testing.' + tn, obj=3Dobj, = status=3Dpassed_str, log=3Dlog, data=3Ddata, renderer=3Dself)) =20 if obj.get_property("testing.ready"): for tn in all_tests: - results.append(Result(name=3D'testing.' + tn, obj=3Dobj, s= tatus=3D'pending')) + results.append(ResultTuple(name=3D'testing.' + tn, obj=3Do= bj, status=3D'pending')) =20 def prepare_message_hook(self, request, message, detailed): if not message.is_series_head: --=20 2.17.0 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://www.redhat.com/mailman/listinfo/patchew-devel