From nobody Tue May 13 11:30:18 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=patchew-devel-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 152697229165415.941702444240946; Mon, 21 May 2018 23:58:11 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BE9EFC03D462; Tue, 22 May 2018 06:58:10 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AC28A2010CA5; Tue, 22 May 2018 06:58:10 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 9D7544BB78; Tue, 22 May 2018 06:58:10 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w4M6w9AZ028253 for ; Tue, 22 May 2018 02:58:09 -0400 Received: by smtp.corp.redhat.com (Postfix) id 0725B2010CA3; Tue, 22 May 2018 06:58:09 +0000 (UTC) Received: from mx1.redhat.com (ext-mx06.extmail.prod.ext.phx2.redhat.com [10.5.110.30]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F20422010CA5 for ; Tue, 22 May 2018 06:58:08 +0000 (UTC) Received: from mail-pg0-f44.google.com (mail-pg0-f44.google.com [74.125.83.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C6FD6C6575 for ; Tue, 22 May 2018 06:57:57 +0000 (UTC) Received: by mail-pg0-f44.google.com with SMTP id p8-v6so7429254pgq.10 for ; Mon, 21 May 2018 23:57:57 -0700 (PDT) Received: from localhost.localdomain ([208.181.63.202]) by smtp.gmail.com with ESMTPSA id y2-v6sm19818676pgp.92.2018.05.21.23.57.55 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 21 May 2018 23:57:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=zwhqgVc5UMX3FMhW1cxTaLp8sPUvNyCVyq5JEVubxX8=; b=dSNhFTv/4VT/6e4EjWBGs/3aN3MvL/OnXVvpDiWOmmgSivSTsJWQ9NH75vYmWlE+VB cSt99mVqnvGKBVIQIs38oDQ3lR6Og5N6v5+wW1EaAEyTXYcbsqw/Z2DrMhaPbLC0R27G 0Co15P7d5xka2TfbYdOeg2D5OmP/gk1XhnZkVJP+I9nkJSK/YcpGhmIyb9seBDDPHQ1C ZkEx5uP0k88Y1G+X++PetweysaD7/6kjOdzfedmdBTBWifQRLUS0u6iWZ2PQ5pfZcJip PRhxNo4IRxOh6sWVYuU16R6yfvCHphZgEZ5ri2Ua0k8rMeXXpCVRxH2h1eZXhlNr+yYA hyQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=zwhqgVc5UMX3FMhW1cxTaLp8sPUvNyCVyq5JEVubxX8=; b=XMzgFEMSVpI3d54pDouSZ14qlbv67ZJAlUdTO0cDr4rlQo6YNhO/RpSWSMtsjzTXB/ zCAm3vOcGyBYtfRPzKYtMjB6fo8cV3SOzRw6tDnPXWicmqasVUPTTs9H64gvXHRkT1yT C3qW5pN6GsyHtH7c7+cPv45U0udrY4zcZiIADHh++EXdCqSPe4FrdtBZx5uowsgZmP6a K06IYkZ3ALLFHYnIZHcfitLpMEi3oyGckDwIWHNxCRUrFFxGTvl1n9RT5EQqHThwbMPN QS9HygusL/2yyQQ8ZIL/T55QCdUWxokDL53i2eMZR1VCMFIphfBAjmYYiGL281LCqsrD QuvQ== X-Gm-Message-State: ALKqPweF5RGrG/LhhJYSk5o0uXssV+drgy4HpxfkHWYpoXDu+EoVqxuR HhMbuSfBeKP75/16M9WV6tQSgl5H X-Google-Smtp-Source: AB8JxZpTJuFVITrLYDlKat5Ozs0GyFMWWtJ9AFlJo0c99XiSIaQhNDIcgloU0IGwDxKfwSwmYF9oZA== X-Received: by 2002:a65:61c8:: with SMTP id j8-v6mr18041359pgv.370.1526972276826; Mon, 21 May 2018 23:57:56 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Tue, 22 May 2018 08:57:40 +0200 Message-Id: <20180522065740.9710-13-pbonzini@redhat.com> In-Reply-To: <20180522065740.9710-1-pbonzini@redhat.com> References: <20180522065740.9710-1-pbonzini@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Tue, 22 May 2018 06:57:57 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Tue, 22 May 2018 06:57:57 +0000 (UTC) for IP:'74.125.83.44' DOMAIN:'mail-pg0-f44.google.com' HELO:'mail-pg0-f44.google.com' FROM:'paolo.bonzini@gmail.com' RCPT:'' X-RedHat-Spam-Score: -1.2 (DKIM_SIGNED, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_PASS, T_DKIM_INVALID) 74.125.83.44 mail-pg0-f44.google.com 74.125.83.44 mail-pg0-f44.google.com X-RedHat-Possible-Forgery: Paolo Bonzini X-Scanned-By: MIMEDefang 2.78 on 10.5.110.30 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.25 X-loop: patchew-devel@redhat.com Subject: [Patchew-devel] [PATCH 12/12] models: convert MessageProperty and ProjectProperty to JSONField X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.25 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 22 May 2018 06:58:10 +0000 (UTC) X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The huge logs are not stored anymore as properties, this means that the pro= perty tables can just use JSONField and the blob field can be removed. Signed-off-by: Paolo Bonzini --- api/blobs.py | 7 ---- api/migrations/0030_deblob_properties.py | 34 ++++++++++++++++++ api/migrations/0031_auto_20180520_1654.py | 34 ++++++++++++++++++ api/migrations/__init__.py | 7 ++-- api/models.py | 42 ++++------------------- 5 files changed, 79 insertions(+), 45 deletions(-) create mode 100644 api/migrations/0030_deblob_properties.py create mode 100644 api/migrations/0031_auto_20180520_1654.py diff --git a/api/blobs.py b/api/blobs.py index 7171b71..4ef5c96 100644 --- a/api/blobs.py +++ b/api/blobs.py @@ -29,13 +29,6 @@ def load_blob(name): fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") return lzma.open(fn, 'r').read().decode("utf-8") =20 -def load_blob_json(name): - try: - return json.loads(load_blob(name)) - except json.decoder.JSONDecodeError as e: - logging.error('Failed to load blob %s: %s' %(name, e)) - return None - def delete_blob(name): fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") try: diff --git a/api/migrations/0030_deblob_properties.py b/api/migrations/0030= _deblob_properties.py new file mode 100644 index 0000000..801f7a2 --- /dev/null +++ b/api/migrations/0030_deblob_properties.py @@ -0,0 +1,34 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.conf import settings +from django.db import migrations +from django.db.models import Count + +import json +from api import blobs + +def deblob_properties(apps, schema_editor): + def do_deblob_properties(model): + objects =3D model.objects.filter(blob=3DTrue) + for obj in objects: + obj.blob =3D False + if obj.value is not None: + obj.value =3D blobs.load_blob(obj.value) + obj.save() + + # We can't import the models directly as they may be a newer + # version than this migration expects. We use the historical version. + do_deblob_properties(apps.get_model('api', 'MessageProperty')) + do_deblob_properties(apps.get_model('api', 'ProjectProperty')) + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0029_populate_testing_results'), + ] + + operations =3D [ + migrations.RunPython(deblob_properties, + reverse_code=3Dmigrations.RunPython.noop), + ] diff --git a/api/migrations/0031_auto_20180520_1654.py b/api/migrations/003= 1_auto_20180520_1654.py new file mode 100644 index 0000000..edb0c4f --- /dev/null +++ b/api/migrations/0031_auto_20180520_1654.py @@ -0,0 +1,34 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.12 on 2018-05-20 16:54 +from __future__ import unicode_literals + +from django.db import migrations +import jsonfield.fields + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0030_deblob_properties'), + ] + + operations =3D [ + migrations.RemoveField( + model_name=3D'messageproperty', + name=3D'blob', + ), + migrations.RemoveField( + model_name=3D'projectproperty', + name=3D'blob', + ), + migrations.AlterField( + model_name=3D'messageproperty', + name=3D'value', + field=3Djsonfield.fields.JSONField(), + ), + migrations.AlterField( + model_name=3D'projectproperty', + name=3D'value', + field=3Djsonfield.fields.JSONField(), + ), + ] diff --git a/api/migrations/__init__.py b/api/migrations/__init__.py index 9707fb2..d0f2621 100644 --- a/api/migrations/__init__.py +++ b/api/migrations/__init__.py @@ -23,7 +23,7 @@ def get_property_raw(model, name, **kwargs): return mp =20 def load_property(mp): - if mp.blob: + if hasattr(mp, 'blob') and mp.blob: return load_blob_json_safe(mp.value) else: return json.loads(mp.value) @@ -34,7 +34,7 @@ def get_property(model, name, **kwargs): =20 def delete_property_blob(model, name, **kwargs): mp =3D get_property_raw(model, name, **kwargs) - if mp.blob: + if hasattr(mp, 'blob') and mp.blob: blobs.delete_blob(mp.value) =20 def set_property(model, name, value, **kwargs): @@ -42,5 +42,6 @@ def set_property(model, name, value, **kwargs): value =3D json.dumps(value) mp, created =3D model.objects.get_or_create(name=3Dname, **kwargs) mp.value =3D value - mp.blob =3D False + if hasattr(mp, 'blob'): + mp.blob =3D False mp.save() diff --git a/api/models.py b/api/models.py index f7065ca..dd00da1 100644 --- a/api/models.py +++ b/api/models.py @@ -10,7 +10,6 @@ =20 =20 from collections import namedtuple -import json import datetime import re =20 @@ -23,8 +22,9 @@ import lzma =20 from mbox import MboxMessage from event import emit_event, declare_event -from .blobs import save_blob, load_blob, load_blob_json +from .blobs import save_blob, load_blob import mod +import lzma =20 class Result(models.Model): PENDING =3D 'pending' @@ -143,37 +143,23 @@ class Project(models.Model): def get_property(self, prop, default=3DNone): a =3D ProjectProperty.objects.filter(project=3Dself, name=3Dprop).= first() if a: - if a.blob: - return load_blob_json(a.value) - else: - return json.loads(a.value) + return a.value else: return default =20 def get_properties(self): r =3D {} for m in ProjectProperty.objects.filter(project=3Dself): - if m.blob: - r[m.name] =3D load_blob_json(m.value) - else: - r[m.name] =3D json.loads(m.value) + r[m.name] =3D m.value return r =20 def _do_set_property(self, prop, value): if value =3D=3D None: ProjectProperty.objects.filter(project=3Dself, name=3Dprop).de= lete() return - # TODO: drop old blob - json_data =3D json.dumps(value) - blob =3D len(json_data) > 1024 - if blob: - value =3D save_blob(json_data) - else: - value =3D json.dumps(value) pp, created =3D ProjectProperty.objects.get_or_create(project=3Dse= lf, name=3Dprop) pp.value =3D value - pp.blob =3D blob pp.save() =20 def set_property(self, prop, value): @@ -229,8 +215,7 @@ class Project(models.Model): class ProjectProperty(models.Model): project =3D models.ForeignKey('Project', on_delete=3Dmodels.CASCADE) name =3D models.CharField(max_length=3D1024, db_index=3DTrue) - value =3D models.CharField(max_length=3D1024) - blob =3D models.BooleanField(blank=3DTrue, default=3DFalse) + value =3D jsonfield.JSONField() =20 class Meta: unique_together =3D ('project', 'name',) @@ -494,10 +479,7 @@ class Message(models.Model): all_props =3D self.properties.all() r =3D {} for m in all_props: - if m.blob: - r[m.name] =3D load_blob_json(m.value) - else: - r[m.name] =3D json.loads(m.value) + r[m.name] =3D m.value self._properties =3D r return r =20 @@ -505,17 +487,9 @@ class Message(models.Model): if value =3D=3D None: MessageProperty.objects.filter(message=3Dself, name=3Dprop).de= lete() return - json_data =3D json.dumps(value) - blob =3D len(json_data) > 1024 mp, created =3D MessageProperty.objects.get_or_create(message=3Dse= lf, name=3Dprop) - # TODO: drop old blob - if blob: - value =3D save_blob(json_data) - else: - value =3D json_data mp.value =3D value - mp.blob =3D blob mp.save() # Invalidate cache self._properties =3D None @@ -646,9 +620,7 @@ class MessageProperty(models.Model): message =3D models.ForeignKey('Message', on_delete=3Dmodels.CASCADE, related_name=3D'properties') name =3D models.CharField(max_length=3D256) - # JSON encoded value - value =3D models.CharField(max_length=3D1024) - blob =3D models.BooleanField(blank=3DTrue, default=3DFalse) + value =3D jsonfield.JSONField() =20 def __str__(self): if len(self.value) > 30: --=20 2.17.0 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://www.redhat.com/mailman/listinfo/patchew-devel