From nobody Tue May 13 19:49:00 2025 Delivered-To: importer2@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=patchew-devel-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1555604517; cv=none; d=zoho.com; s=zohoarc; b=CNkG7vg4mbryryifrQ4IdYsNXOUrRK3SnTzJtHPkjR54q9i9iYxnuJlaMM3d9lafZPgMKGETx5FZZixz8Z4LxEznURQDKdJJi3bARf6wMi0x8gqVnQHBXQ2toamqiGyepYdAo0QG8sMgd27rnXd8+05biaBFy7213EyDPJROeS0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1555604517; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=Cjx5BnOXbKWs9dSRs2cYRWouHjwQFsZm13iJdzuOmYg=; b=BILhqPu6cxQWItBODHcVZpV7ZuateuPm87810FVUs7EUKbkADmknED0R8EmJw7ehqgZwroaB0g/IRVk71uyzbfbiKMMVuR2L2Knl7vTQYPkhzJom65JuVH1XPGnow1jUDRzb458p8GZf9fDhawMT6JNwr5Y2uvK3qUoqTO8XiUo= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 155560451726833.08123041885085; Thu, 18 Apr 2019 09:21:57 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0385A307CB28; Thu, 18 Apr 2019 16:21:56 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id ED89B60BEC; Thu, 18 Apr 2019 16:21:55 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id E1F1F4EA71; Thu, 18 Apr 2019 16:21:55 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x3IGLtkg009177 for ; Thu, 18 Apr 2019 12:21:55 -0400 Received: by smtp.corp.redhat.com (Postfix) id 02A8260BEC; Thu, 18 Apr 2019 16:21:55 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-112-46.ams2.redhat.com [10.36.112.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4E02260BE5 for ; Thu, 18 Apr 2019 16:21:54 +0000 (UTC) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 18 Apr 2019 18:21:40 +0200 Message-Id: <20190418162146.5695-5-pbonzini@redhat.com> In-Reply-To: <20190418162146.5695-1-pbonzini@redhat.com> References: <20190418162146.5695-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: patchew-devel@redhat.com Subject: [Patchew-devel] [PATCH 04/10] models: introduce flags into Messages X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Thu, 18 Apr 2019 16:21:56 +0000 (UTC) Content-Type: text/plain; charset="utf-8" Flags are used as a replacement for Boolean properties that are used by search queries. They can be searched efficiently using a trigram index. Unlike properties, flags are general and could be set by any plugin. For example, a GitHub integration webhook could set REVIEWED, OBSOLETE and TESTED based on changes made to a GitHub pull request. --- api/migrations/0046_message_flags.py | 20 +++++++++ .../0047_message_flags_postgres_fts.py | 22 ++++++++++ api/migrations/0048_populate_message_flags.py | 42 +++++++++++++++++++ api/models.py | 13 ++++++ api/search.py | 13 +++--- mods/tags.py | 5 ++- mods/testing.py | 22 +++++----- tests/test_testing.py | 9 ++-- 8 files changed, 124 insertions(+), 22 deletions(-) create mode 100644 api/migrations/0046_message_flags.py create mode 100644 api/migrations/0047_message_flags_postgres_fts.py create mode 100644 api/migrations/0048_populate_message_flags.py diff --git a/api/migrations/0046_message_flags.py b/api/migrations/0046_mes= sage_flags.py new file mode 100644 index 0000000..0db9dac --- /dev/null +++ b/api/migrations/0046_message_flags.py @@ -0,0 +1,20 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.20 on 2019-04-18 14:30 +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0045_message_maintainers'), + ] + + operations =3D [ + migrations.AddField( + model_name=3D'message', + name=3D'flags', + field=3Dmodels.CharField(blank=3DTrue, default=3D'', max_lengt= h=3D64), + ), + ] diff --git a/api/migrations/0047_message_flags_postgres_fts.py b/api/migrat= ions/0047_message_flags_postgres_fts.py new file mode 100644 index 0000000..bdfc84e --- /dev/null +++ b/api/migrations/0047_message_flags_postgres_fts.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.16 on 2018-11-07 15:28 +from __future__ import unicode_literals + +from django.db import migrations, models +import django.db.models.deletion +from django.contrib.postgres.operations import TrigramExtension + +from api.migrations import PostgresOnlyMigration + + +class Migration(PostgresOnlyMigration): + + dependencies =3D [ + ('api', '0046_message_flags'), + ] + + operations =3D [ + TrigramExtension(), + migrations.RunSQL("create index api_message_flags_gin on api_messa= ge using gin(flags gin_trgm_ops);", + "drop index api_message_flags_gin"), + ] diff --git a/api/migrations/0048_populate_message_flags.py b/api/migrations= /0048_populate_message_flags.py new file mode 100644 index 0000000..835e220 --- /dev/null +++ b/api/migrations/0048_populate_message_flags.py @@ -0,0 +1,42 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.conf import settings +from django.db import migrations + +from api.search import FLAG_TESTED, FLAG_REVIEWED, FLAG_OBSOLETE + +def property_to_flags(apps, schema_editor): + MessageProperty =3D apps.get_model('api', 'MessageProperty') + for p in MessageProperty.objects.filter(name=3D'obsoleted-by'): + p.message.flags +=3D FLAG_OBSOLETE + p.message.save() + for p in MessageProperty.objects.filter(name=3D'reviewed'): + p.message.flags +=3D FLAG_REVIEWED + p.message.save() + for p in MessageProperty.objects.filter(name=3D'testing.done'): + p.message.flags +=3D FLAG_TESTED + p.message.save() + MessageProperty.objects.filter(name=3D'reviewed').delete() + MessageProperty.objects.filter(name=3D'testing.done').delete() + +def flags_to_property(apps, schema_editor): + Message =3D apps.get_model('api', 'Message') + for m in Message.objects.exclude(flags=3D''): + if '[reviewed]' in m.flags: + new_prop =3D MessageProperty(message=3Dp.message, name=3D'revi= ewed', value=3DTrue) + new_prop.save() + if FLAG_TESTED in m.flags: + new_prop =3D MessageProperty(message=3Dp.message, name=3D'test= ing.done', value=3DTrue) + new_prop.save() + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0047_message_flags_postgres_fts'), + ] + + operations =3D [ + migrations.RunPython(property_to_flags, + reverse_code=3Dflags_to_property), + ] diff --git a/api/models.py b/api/models.py index d2dafde..3176559 100644 --- a/api/models.py +++ b/api/models.py @@ -508,6 +508,7 @@ class Message(models.Model): objects =3D MessageManager() =20 maintainers =3D jsonfield.JSONField(blank=3DTrue, default=3D[]) + flags =3D models.CharField(max_length=3D64, blank=3DTrue, default=3D"") =20 def save_mbox(self, mbox_blob): save_blob(mbox_blob, self.message_id) @@ -596,6 +597,18 @@ class Message(models.Model): self.refresh_num_patches() return self.num_patches =20 + def add_flag(self, flag): + assert flag[0] =3D=3D '[' and flag[-1] =3D=3D ']' + if not flag in self.flags: + self.flags +=3D flag + self.save() + + def remove_flag(self, flag): + assert flag[0] =3D=3D '[' and flag[-1] =3D=3D ']' + if flag in self.flags: + self.flags =3D self.flags.replace(flag, '') + self.save() + def get_property(self, prop, default=3DNone): return self.get_properties().get(prop, default) =20 diff --git a/api/search.py b/api/search.py index baf6893..1c5aaaf 100644 --- a/api/search.py +++ b/api/search.py @@ -18,6 +18,10 @@ from django.contrib.postgres.search import SearchQuery, = SearchVector, SearchVect from django.db.models import Lookup from django.db.models.fields import Field =20 +# These are used by migrations. Do not change them! +FLAG_REVIEWED =3D '[reviewed]' +FLAG_OBSOLETE =3D '[obsolete]' +FLAG_TESTED =3D '[tested]' =20 @Field.register_lookup class NotEqual(Lookup): @@ -254,16 +258,13 @@ Search text keyword in the email message. Example: self._add_to_keywords('PULL') return Q(subject__contains=3D'[PULL') | Q(subject__contains=3D= '[GIT PULL') elif cond =3D=3D "reviewed": - return self._make_filter_subquery(MessageProperty, Q(name=3D"r= eviewed", value=3DTrue)) + return Q(flags__contains=3DFLAG_REVIEWED) elif cond in ("obsoleted", "old"): - return self._make_filter_subquery( - MessageProperty, - Q(name=3D"obsoleted-by", value__isnull=3DFalse) & ~Q(name= =3D"obsoleted-by", value__iexact=3D'') - ) + return Q(flags__contains=3DFLAG_OBSOLETE) elif cond =3D=3D "applied": return self._make_filter_subquery(MessageResult, Q(name=3D"git= ", status=3DResult.SUCCESS)) elif cond =3D=3D "tested": - return self._make_filter_subquery(MessageProperty, Q(name=3D"t= esting.done", value=3DTrue)) + return Q(flags__contains=3DFLAG_TESTED) elif cond =3D=3D "merged": return Q(is_merged=3DTrue) return None diff --git a/mods/tags.py b/mods/tags.py index 02c493e..6b5bcc9 100644 --- a/mods/tags.py +++ b/mods/tags.py @@ -13,6 +13,7 @@ from mbox import addr_db_to_rest, parse_address from event import register_handler, emit_event, declare_event from api.models import Message from api.rest import PluginMethodField +from api.search import FLAG_OBSOLETE, FLAG_REVIEWED import rest_framework =20 REV_BY_PREFIX =3D "Reviewed-by:" @@ -73,8 +74,10 @@ series cover letter, patch mail body and their replies. return m1.version > m2.version and m1.date >=3D m2.date for m in series.get_alternative_revisions(): if newer_than(m, series): + series.add_flag(FLAG_OBSOLETE) series.set_property("obsoleted-by", m.message_id) elif newer_than(series, m): + m.add_flag(FLAG_OBSOLETE) m.set_property("obsoleted-by", series.message_id) =20 updated =3D self.update_tags(series) @@ -100,7 +103,7 @@ series cover letter, patch mail body and their replies. series_reviewers =3D _find_reviewers(series) reviewers =3D reviewers.union(series_reviewers) if num_reviewed =3D=3D series.get_num()[1] or series_reviewers: - series.set_property("reviewed", True) + series.add_flag(FLAG_REVIEWED) series.set_property("reviewers", list(reviewers)) if updated: emit_event("TagsUpdate", series=3Dseries) diff --git a/mods/testing.py b/mods/testing.py index e390e30..6a61507 100644 --- a/mods/testing.py +++ b/mods/testing.py @@ -22,7 +22,7 @@ from api.views import APILoginRequiredView from api.models import (Message, MessageProperty, MessageResult, Project, ProjectResult, Result) from api.rest import PluginMethodField, reverse_detail -from api.search import SearchEngine +from api.search import SearchEngine, FLAG_TESTED from event import emit_event, declare_event, register_handler from patchew.logviewer import LogView from schema import * @@ -137,9 +137,9 @@ class TestingModule(PatchewModule): _instance.tester_check_in(po, result.data['tester']) if not self.get_testing_results(obj, status__in=3D(Result.PENDING, Result.RUNNING)).= exists(): - obj.set_property("testing.done", True) obj.set_property("testing.tested-head", result.data["head"= ]) if isinstance(obj, Message): + obj.add_flag(FLAG_TESTED) obj.set_property("testing.tested-base", self.get_msg_base_tags(obj)) if isinstance(obj, Project): @@ -187,10 +187,11 @@ class TestingModule(PatchewModule): for tn in all_tests: if not tn in done_tests: obj.create_result(name=3D'testing.' + tn, status=3DRes= ult.PENDING).save() - if len(done_tests) < len(all_tests): - obj.set_property("testing.done", None) - return - obj.set_property("testing.done", True) + if isinstance(obj, Message): + if len(all_tests) and len(done_tests) =3D=3D len(all_tests): + obj.add_flag(FLAG_TESTED) + else: + obj.remove_flag(FLAG_TESTED) =20 def project_recalc_pending_tests(self, project): self.recalc_pending_tests(project) @@ -203,10 +204,9 @@ class TestingModule(PatchewModule): self.recalc_pending_tests(obj) =20 def clear_and_start_testing(self, obj, test=3D""): - for k in list(obj.get_properties().keys()): - if k =3D=3D "testing.done" or \ - k =3D=3D "testing.tested-head": - obj.set_property(k, None) + obj.set_property("testing.tested-head", None) + if isinstance(obj, Message): + obj.remove_flag(FLAG_TESTED) if test: r =3D self.get_testing_result(obj, test) if r: @@ -352,7 +352,7 @@ class TestingModule(PatchewModule): "type": "danger", "char": "T", }) - elif message.get_property("testing.done"): + elif FLAG_TESTED in message.flags: message.status_tags.append({ "title": "Testing passed", "url": reverse("series_detail", diff --git a/tests/test_testing.py b/tests/test_testing.py index a5c22e1..630e7f9 100755 --- a/tests/test_testing.py +++ b/tests/test_testing.py @@ -12,6 +12,7 @@ import abc import subprocess =20 from api.models import Message, Result +from api.search import FLAG_TESTED =20 from .patchewtest import PatchewTestCase, main =20 @@ -56,7 +57,6 @@ class TestingTestCase(PatchewTestCase, metaclass=3Dabc.AB= CMeta): if 'status' not in kwargs: kwargs['status'] =3D Result.SUCCESS self.modify_test_result(obj, **kwargs) - obj.set_property("testing.done", True) =20 def do_testing_report(self, **report): self.api_login() @@ -167,6 +167,7 @@ class MessageTestingTest(TestingTestCase): =20 def do_testing_done(self, **kwargs): self._do_testing_done(self.msg, **kwargs) + self.msg.add_flag(FLAG_TESTED) =20 def do_testing_report(self, **report): r =3D super(MessageTestingTest, self).do_testing_report(**report) @@ -353,7 +354,7 @@ class TestingResetTest(PatchewTestCase): "testing.a": Result.SUCCESS, "testing.b": Result.SUCCESS, "testing.c": Result.FAILURE}) - self.assertTrue(msg.get_property("testing.done")) + self.assertIn(FLAG_TESTED, msg.flags) =20 self.api_login() self.client.post('/login/', {'username': self.user, 'password': se= lf.password}) @@ -364,7 +365,7 @@ class TestingResetTest(PatchewTestCase): "testing.a": Result.PENDING, "testing.b": Result.SUCCESS, "testing.c": Result.FAILURE}) - self.assertFalse(msg.get_property("testing.done")) + self.assertNotIn(FLAG_TESTED, msg.flags) =20 self.client.get('/testing-reset/%s/?type=3Dmessage&test=3Db' % msg= .message_id) self.client.get('/testing-reset/%s/?type=3Dmessage&test=3Dc' % msg= .message_id) @@ -372,7 +373,7 @@ class TestingResetTest(PatchewTestCase): "testing.a": Result.PENDING, "testing.b": Result.PENDING, "testing.c": Result.PENDING}) - self.assertFalse(msg.get_property("testing.done")) + self.assertNotIn(FLAG_TESTED, msg.flags) =20 =20 class TestingDisableTest(PatchewTestCase): --=20 2.21.0 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://www.redhat.com/mailman/listinfo/patchew-devel