django-pb-model 0.3.3

Field name mapping

To adapt schema migration, field mapping are expected.

For example, the email field in previous session is altered to username, but we don’t want to break the consistence of protobuf protocol. You may add pb_2_dj_field_map attribute to solve this problem. Such as:

class Account(ProtoBufMixin, models.Model):
    pb_model = account_pb2.Account
    pb_2_dj_field_map = {
        "email": "username",  # protobuf field as key and django field as value
    }

    username = models.CharField(max_length=64)
    nickname = models.CharField(max_length=64)

Foreign Key

Foreign key is a connect to another model in Django. According to this property, the foreign key could and should be converted to nested singular message in Protobuf. For example:

message Relation {
    int32 id = 1;
}

message Main {
    int32 id = 1;
    Relation fk = 2;
}

Django model:

class Relation(ProtoBufMixin, models.Model):
    pb_model = models_pb2.Relation


class Main(ProtoBufMixin, models.Model):
    pb_model = models_pb2.Main

    fk = models.ForeignKey(Relation)

With above settings, pb_model would recursively serialize and de-serialize between Django and ProtoBuf.

>>> m = Main.objects.create(fk=Relation.objects.create())
>>> m.to_pb()
id: 1
fk {
    id: 1
}

>>> m2 = Main()
>>> m2.from_pb(m.to_pb())
>>> m2.fk.id
1

Note that one can specify a reversed relation by assign related_name:

class Relation(ProtoBufMixin, models.Model):
  pb_model = models_pb2.Relation

  num = models.IntegerField(default=0)
  deeper_relation = models.ForeignKey(DeeperRelation,
                                      on_delete=models.DO_NOTHING,
                                      blank=True,
                                      null=True,
                                      related_name='relations')

When the related proto contains the same field of this reversed relation:

message DeeperRelation {
  int32 id = 1;
  int32 num = 2;
  repeated Relation relations = 3;
}

we will skip serializes the relations field.

Many-to-Many field

M2M field is a QuerySet Relation in Django. By default, we assume target message field is “repeated” nested message, ex:

message M2M {
    int32 id = 1;
}

message Main {
    int32 id = 1;

    repeated M2M m2m = 2;
}

Django model would be:

class M2M(models.Model):
    pass

class Main(models.Model):

    m2m = models.ManyToManyField(M2M)

from django.db import transaction

...

class PBCompatibleModel(ProtoBufMixin, models.Model):

    def _repeated_to_m2m(self, dj_field, _pb_repeated_set):
        with transaction.atomic():
            for item in _pb_repeated_set:
                dj_field.get_or_create(pk=item.pk, defaults={....})

    ...

Limit Foreign key or Many-to-Many field conversion depth

By default, when to_pb() method is called, all related message will be also converted recursively.

For example:

message DeeperRelation {
  int32 id = 1;
  int32 num = 2;
}

// Relation model for testing
message Relation {
    int32 id = 1;
    int32 num = 2;
    DeeperRelation deeper_relation = 3;
}

message Main {
  int32 id = 1;
  Relation fk_field = 2;
}

And code:

>>> m = Main.objects.create(fk=Relation.objects.create(
      deeper_relation=DeeperRelation.objects.create()))
>>> m.to_pb()
fk {
  id: 1
  fk_field {
    id: 1,
    deeper_relation {
      id: 1
    }
  }
}

This may not be the behavior wanted. The depth parameter can be used to limit the depth of these conversion.

If the depth is set to 0, no related field will be converted, the fk_field in Main message will left unset.

If the depth is set to any positive number, the level of related field will be limited by the specified number. For example, if depth is set to 1, the fk_field will contain the related Relation message, however the deeper_relation field of the fk_field message will be unset.

Datetime Field

Datetime is a special singular value and able to convert between datetime.datetime (Python) and google.protobuf.timestamp_pb2.Timestamp (ProboBuf) through built-in datetime serializers. Check Custom Fields if you want other coversion rules.

Default conversion works as following example:

ProtoBuf message:

package models;

import "google/protobuf/timestamp.proto";

message WithDatetime {
    int32 id = 1;
    google.protobuf.Timestamp datetime_field = 2;
}

Django Model:

class WithDatetime(ProtoBufMixin, models.Model):
    pb_model = models_pb2.WithDatetime

    datetime_field = models.DatetimeField(default=timezone.now())

>>> WithDatetime.objects.create().to_pb()
datetime_field {
seconds: 1495119614
nanos: 282705000
}

Any

When using Any as a field in a message, the field is by default kept as Any in your Django model and you may save it like it is any other Django Field. Example on using Any:

package models;

import "google/protobuf/any.proto";

message WithAnyMessage {
    google.protobuf.Any any_field = 1;
}

class WithAny(ProtoBufMixin, models.Model):
    pb_model = models_pb2.WithAny

Then when you’re using your Django:

>>> with_any_message = WithAnyMessage(any_field=Any())
>>> WithAny().from_pb(with_any_message).save()

# See that the we successfully saved Any as a Django field.
>>> any_field = WithAny.objects.last().any_field
>>> type(any_field)
google.protobuf.any_pb2.Any

Custom Fields

You can write your own field serializers, to convert between django.contrib.postgres.fields.JSONField (Python) and string (Protobuf) for example:

ProtoBuf message:

package models;

message WithJSONBlob {
    int32 id = 1;
    string json_blob = 2;
}

Django Model:

def json_serializer(pb_obj, pb_field, dj_value):
    setattr(pb_obj, pb_field.name, json.dumps(value))

def json_deserializer(instance, dj_field_name, pb_field, pb_value):
    setattr(instance, dj_field_name, json.loads(pb_value))

class WithJSONField(ProtoBufMixin, models.Model):
    pb_model = models_pb2.WithJSONBlob

    pb_2_dj_field_serializers = {
        'json_blob': (json_serializer, json_deserializer),
    }

    json_blob = JSONField()

_pb_2_dj_default_field_serializers = {
     models.DateTimeField: (fields._datetimefield_to_pb,
                            fields._datetimefield_from_pb),
     models.UUIDField: (fields._uuid_to_pb,
                        fields._uuid_from_pb),
    }