skip to navigation
skip to content

bloop 1.3.1

ORM for DynamoDB

Package Documentation

Bloop is an object mapper for DynamoDB and DynamoDBStreams. py35+

pip install bloop

Usage

First, we need to import all the things:

>>> from bloop import (
...     BaseModel, Column, String, UUID,
...     GlobalSecondaryIndex, Engine
... )

Next we’ll define the account model (with streaming enabled), and create the backing table:

>>> class Account(BaseModel):
...     class Meta:
...         stream = {
...             "include": {"old", "new"}
...         }
...     id = Column(UUID, hash_key=True)
...     name = Column(String)
...     email = Column(String)
...    by_email = GlobalSecondaryIndex(projection='keys', hash_key='email')
...
>>> engine = Engine()
>>> engine.bind(Account)

Let’s make a few users and persist them:

>>> import uuid
>>> admin = Account(id=uuid.uuid4(), email="admin@domain.com")
>>> admin.name = "Admin McAdminFace"
>>> support = Account(name="this-is-fine.jpg", email="help@domain.com")
>>> support.id = uuid.uuid4()
>>> engine.save(admin, support)

And find them again:

>>> q = engine.query(
...     Account.by_email,
...     key=Account.email=="help@domain.com"
... )
>>> q.first()
Account(email='help@domain.com',
        id=UUID('d30e343f-f067-4fe5-bc5e-0b00cdeaf2ba'))
>>> s = engine.scan(
...     Account,
...     filter=Account.name.begins_with("Admin")
... )
>>> s.one()
Account(email='admin@domain.com',
        id=UUID('08da44ac-5ff6-4f70-8a3f-b75cadb4dd79'),
        name='Admin McAdminFace')

Let’s find them in the stream:

>>> stream = engine.stream(Account, "trim_horizon")
>>> next(stream)
{'key': None,
 'meta': {'created_at': datetime.datetime(...),
  'event': {'id': 'cbb9a9b45eb0a98889b7da85913a5c65',
   'type': 'insert',
   'version': '1.1'},
  'sequence_number': '100000000000588052489'},
 'new': Account(
            email='help@domain.com',
            id=UUID('d30e343f-...-0b00cdeaf2ba'),
            name='this-is-fine.jpg'),
 'old': None}
>>> next(stream)
{'key': None,
 'meta': {'created_at': datetime.datetime(...),
  'event': {'id': 'cbdfac5671ea38b99017c4b43a8808ce',
   'type': 'insert',
   'version': '1.1'},
  'sequence_number': '200000000000588052506'},
 'new': Account(
            email='admin@domain.com',
            id=UUID('08da44ac-...-b75cadb4dd79'),
            name='Admin McAdminFace'),
 'old': None}
>>> next(stream)
>>> next(stream)
>>>

What’s Next

Check out the User Guide or Public API Reference to create your own nested types, overlapping models, set up cross-region replication in less than 20 lines, and more!

Changelog

This changelog structure is based on Keep a Changelog v0.3.0. Bloop follows Semantic Versioning 2.0.0 and a draft appendix for its Public API.

[Unreleased]

1.3.0 - 2017-10-08

This release is exclusively to prepare users for the name/model_name/dynamo_name changes coming in 2.0; your 1.2.0 code will continue to work as usual but will raise DeprecationWarning when accessing model_name on a Column or Index, or when specifying the name= kwarg in the __init__ method of Column, GlobalSecondaryIndex, or LocalSecondaryIndex.

Previously it was unclear if Column.model_name was the name of this column in its model, or the name of the model it is attached to (eg. a shortcut for Column.model.__name__). Additionally the name= kwarg actually mapped to the object’s .dynamo_name value, which was not obvious.

Now the Column.name attribute will hold the name of the column in its model, while Column.dynamo_name will hold the name used in DynamoDB, and is passed during initialization as dynamo_name=. Accessing model_name or passing name= during __init__ will raise deprecation warnings, and bloop 2.0.0 will remove the deprecated properties and ignore the deprecated kwargs.

Added

  • Column.name is the new home of the Column.model_name attribute. The same is true for Index, GlobalSecondaryIndex, and LocalSecondaryIndex.
  • The __init__ method of Column, Index, GlobalSecondaryIndex, and LocalSecondaryIndex now takes dynamo_name= in place of name=.

Changed

  • Accessing Column.model_name raises DeprecationWarning, and the same for Index/GSI/LSI.
  • Providing Column(name=) raises DeprecationWarning, and the same for Index/GSI/LSI.

1.2.0 - 2017-09-11

Changed

  • When a Model’s Meta does not explicitly set read_units and write_units, it will only default to 1/1 if the table does not exist and needs to be created. If the table already exists, any throughput will be considered valid. This will still ensure new tables have 1/1 iops as a default, but won’t fail if an existing table has more than one of either.

    There is no behavior change for explicit integer values of read_units and write_units: if the table does not exist it will be created with those values, and if it does exist then validation will fail if the actual values differ from the modeled values.

    An explicit None for either read_units or write_units is equivalent to omitting the value, but allows for a more explicit declaration in the model.

    Because this is a relaxing of a default only within the context of validation (creation has the same semantics) the only users that should be impacted are those that do not declare read_units and write_units and rely on the built-in validation failing to match on values != 1. Users that rely on the validation to succeed on tables with values of 1 will see no change in behavior. This fits within the extended criteria of a minor release since there is a viable and obvious workaround for the current behavior (declare 1/1 and ensure failure on other values).

  • When a Query or Scan has projection type “count”, accessing the count or scanned properties will immediately execute and exhaust the iterator to provide the count or scanned count. This simplifies the previous workaround of calling next(query, None) before using query.count.

Fixed

  • Fixed a bug where a Query or Scan with projection “count” would always raise KeyError (see Issue #95)
  • Fixed a bug where resetting a Query or Scan would cause __next__ to raise botocore.exceptions.ParamValidationError (see Issue #95)

1.1.0 - 2017-04-26

Added

  • Engine.bind takes optional kwarg skip_table_setup to skip CreateTable and DescribeTable calls (see Issue #83)
  • Index validates against a superset of the projection (see Issue #71)

1.0.3 - 2017-03-05

Bug fix.

Fixed

  • Stream orders records on the integer of SequenceNumber, not the lexicographical sorting of its string representation. This is an annoying bug, because as documented we should be using lexicographical sorting on the opaque string. However, without leading 0s that sort fails, and we must assume the string represents an integer to sort on. Particularly annoying, tomorrow the SequenceNumber could start with non-numeric characters and still conform to the spec, but the sorting-as-int assumption breaks. However, we can’t properly sort without making that assumption.

1.0.2 - 2017-03-05

Minor bug fix.

Fixed

  • extension types in ext.arrow, ext.delorean, and ext.pendulum now load and dump None correctly.

1.0.1 - 2017-03-04

Bug fixes.

Changed

  • The arrow, delorean, and pendulum extensions now have a default timezone of "utc" instead of datetime.timezone.utc. There are open issues for both projects to verify if that is the expected behavior.

Fixed

  • DynamoDBStreams return a Timestamp for each record’s ApproximateCreationDateTime, which botocore is translating into a real datetime.datetime object. Previously, the record parser assumed an int was used. While this fix is a breaking change for an internal API, this bug broke the Stream iterator interface entirely, which means no one could have been using it anyway.

1.0.0 - 2016-11-16

1.0.0 is the culmination of just under a year of redesigns, bug fixes, and new features. Over 550 commits, more than 60 issues closed, over 1200 new unit tests. At an extremely high level:

  • The query and scan interfaces have been polished and simplified. Extraneous methods and configuration settings have been cut out, while ambiguous properties and methods have been merged into a single call.
  • A new, simple API exposes DynamoDBStreams with just a few methods; no need to manage individual shards, maintain shard hierarchies and open/closed polling. I believe this is a first since the Kinesis Adapter and KCL, although they serve different purposes. When a single worker can keep up with a model’s stream, Bloop’s interface is immensely easier to use.
  • Engine’s methods are more consistent with each other and across the code base, and all of the configuration settings have been made redundant. This removes the need for EngineView and its associated temporary config changes.
  • Blinker-powered signals make it easy to plug in additional logic when certain events occur: before a table is created; after a model is validated; whenever an object is modified.
  • Types have been pared down while their flexibility has increased significantly. It’s possible to create a type that loads another object as a column’s value, using the engine and context passed into the load and dump functions. Be careful with this; transactions on top of DynamoDB are very hard to get right.

See the Migration Guide above for specific examples of breaking changes and how to fix them, or the User Guide for a tour of the new Bloop. Lastly, the Public and Internal API References are finally available and should cover everything you need to extend or replace whole subsystems in Bloop (if not, please open an issue).

Added

  • bloop.signals exposes Blinker signals which can be used to monitor object changes, when instances are loaded from a query, before models are bound, etc.

    • before_create_table
    • object_loaded
    • object_saved
    • object_deleted
    • object_modified
    • model_bound
    • model_created
    • model_validated
  • Engine.stream can be used to iterate over all records in a stream, with a total ordering over approximate record creation time. Use engine.stream(model, "trim_horizon") to get started. See the User Guide

  • New exceptions RecordsExpired and ShardIteratorExpired for errors in stream state

  • New exceptions Invalid* for bad input subclass BloopException and ValueError

  • DateTime types for the three most common date time libraries:

    • bloop.ext.arrow.DateTime
    • bloop.ext.delorean.DateTime
    • bloop.ext.pendulum.DateTime
  • model.Meta has a new optional attribute stream which can be used to enable a stream on the model’s table. See the User Guide for details

  • model.Meta exposes the same projection attribute as Index so that (index or model.Meta).projection can be used interchangeably

  • New Stream class exposes DynamoDBStreams API as a single iterable with powerful seek/jump options, and simple json-friendly tokens for pausing and resuming iteration. See the User Guide for details

  • Over 1200 unit tests added

  • Initial integration tests added

  • (internal) bloop.conditions.ReferenceTracker handles building #n0, :v1, and associated values. Use any_ref to build a reference to a name/path/value, and pop_refs when backtracking (eg. when a value is actually another column, or when correcting a partially valid condition)

  • (internal) bloop.conditions.render is the preferred entry point for rendering, and handles all permutations of conditions, filters, projections. Use over ConditionRenderer unless you need very specific control over rendering sequencing.

  • (internal) bloop.session.SessionWrapper exposes DynamoDBStreams operations in addition to previous bloop.Client wrappers around DynamoDB client

  • (internal) New supporting classes streams.buffer.RecordBuffer, streams.shard.Shard, and streams.coordinator.Coordinator to encapsulate the hell^Wjoy that is working with DynamoDBStreams

  • (internal) New class util.Sentinel for placeholder values like missing and last_token that provide clearer docstrings, instead of showing func(..., default=object<0x...>) these will show func(..., default=Sentinel<[Missing]>)

Changed

  • bloop.Column emits object_modified on __set__ and __del__

  • Conditions now check if they can be used with a column’s typedef and raise InvalidCondition when they can’t. For example, contains can’t be used on Number, nor > on Set(String)

  • bloop.Engine no longer takes an optional bloop.Client but instead optional dynamodb and dynamodbstreams clients (usually created from boto3.client("dynamodb") etc.)

  • Engine no longer takes **config – its settings have been dispersed to their local touch points

    • atomic is a parameter of save and delete and defaults to False
    • consistent is a parameter of load, query, scan and defaults to False
    • prefetch has no equivalent, and is baked into the new Query/Scan iterator logic
    • strict is a parameter of a LocalSecondaryIndex, defaults to True
  • Engine no longer has a context to create temporary views with different configuration

  • Engine.bind is no longer by keyword arg only: engine.bind(MyBase) is acceptable in addition to engine.bind(base=MyBase)

  • Engine.bind emits new signals before_create_table, model_validated, and model_bound

  • Engine.delete and Engine.save take *objs instead of objs to easily save/delete small multiples of objects (engine.save(user, tweet) instead of engine.save([user, tweet]))

  • Engine guards against loading, saving, querying, etc against abstract models

  • Engine.load raises MissingObjects instead of NotModified (exception rename)

  • Engine.scan and Engine.query take all query and scan arguments immediately, instead of using the builder pattern. For example, engine.scan(model).filter(Model.x==3) has become engine.scan(model, filter=Model.x==3).

  • bloop.exceptions.NotModified renamed to bloop.exceptions.MissingObjects

  • Any code that raised AbstractModelException now raises UnboundModel

  • bloop.types.DateTime is now backed by datetime.datetime instead of arrow. Only supports UTC now, no local timezone. Use the bloop.ext.arrow.DateTime class to continue using arrow.

  • The query and scan interfaces have been entirely refactored: count, consistent, ascending and other properties are part of the Engine.query(...) parameters. all() is no longer needed, as Engine.scan and .query immediately return an iterable object. There is no prefetch setting, or limit.

  • The complete property for Query and Scan have been replaced with exhausted, to be consistent with the Stream module

  • The query and scan iterator no longer cache results

  • The projection parameter is now required for GlobalSecondaryIndex and LocalSecondaryIndex

  • Calling Index.__set__ or Index.__del__ will raise AttributeError. For example, some_user.by_email = 3 raises if User.by_email is a GSI

  • bloop.Number replaces bloop.Float and takes an optional decimal.Context for converting numbers. For a less strict, lossy Float type see the Patterns section of the User Guide

  • bloop.String.dynamo_dump no longer calls str() on the value, which was hiding bugs where a non-string object was passed (eg. some_user.name = object() would save with a name of <object <0x...>)

  • bloop.DateTime is now backed by datetime.datetime and only knows UTC in a fixed format. Adapters for arrow, delorean, and pendulum are available in bloop.ext

  • bloop.DateTime does not support naive datetimes; they must always have a tzinfo

  • docs:

    • use RTD theme
    • rewritten three times
    • now includes public and internal api references
  • (internal) Path lookups on Column (eg. User.profile["name"]["last"]) use simpler proxies

  • (internal) Proxy behavior split out from Column’s base class bloop.conditions.ComparisonMixin for a cleaner namespace

  • (internal) bloop.conditions.ConditionRenderer rewritten, uses a new bloop.conditions.ReferenceTracker with a much clearer api

  • (internal) ConditionRenderer can backtrack references and handles columns as values (eg. User.name.in_([User.email, "literal"]))

  • (internal) _MultiCondition logic rolled into bloop.conditions.BaseCondition, AndCondition and OrCondition no longer have intermediate base class

  • (internal) AttributeExists logic rolled into bloop.conditions.ComparisonCondition

  • (internal) bloop.tracking rolled into bloop.conditions and is hooked into the object_* signals. Methods are no longer called directly (eg. no need for tracking.sync(some_obj, engine))

  • (internal) update condition is built from a set of columns, not a dict of updates to apply

  • (internal) bloop.conditions.BaseCondition is a more comprehensive base class, and handles all manner of out-of-order merges (and(x, y) vs and(y, x) where x is an and condition and y is not)

  • (internal) almost all *Condition classes simply implement __repr__ and render; BaseCondition takes care of everything else

  • (internal) bloop.Client became bloop.session.SessionWrapper

  • (internal) Engine._dump takes an optional context, **kwargs, matching the signature of Engine._load

  • (internal) BaseModel no longer implements __hash__, __eq__, or __ne__ but ModelMetaclass will always ensure a __hash__ function when the subclass is created

  • (internal) Filter and FilterIterator rewritten entirely in the bloop.search module across multiple classes

Removed

  • AbstractModelException has been rolled into UnboundModel
  • The all() method has been removed from the query and scan iterator interface. Simply iterate with next(query) or for result in query:
  • Query.results and Scan.results have been removed and results are no longer cached. You can begin the search again with query.reset()
  • The new_base() function has been removed in favor of subclassing BaseModel directly
  • bloop.Float has been replaced by bloop.Number
  • (internal) bloop.engine.LoadManager logic was rolled into bloop.engine.load(...)
  • EngineView has been removed since engines no longer have a baseline config and don’t need a context to temporarily modify it
  • (internal) Engine._update has been removed in favor of util.unpack_from_dynamodb
  • (internal) Engine._instance has been removed in favor of directly creating instances from model.Meta.init() in unpack_from_dynamodb

Fixed

  • Column.contains(value) now renders value with the column typedef’s inner type. Previously, the container type was used, so Data.some_list.contains("foo")) would render as (contains(some_list, ["f", "o", "o"])) instead of (contains(some_list, "foo"))
  • Set renders correct wire format – previously, it incorrectly sent {"SS": [{"S": "h"}, {"S": "i"}]} instead of the correct {"SS": ["h", "i"]}
  • (internal) Set and List expose an inner_typedef for conditions to force rendering of inner values (currently only used by ContainsCondition)

0.9.13 - 2016-10-31

[Fixed]

  • Set was rendering an invalid wire format, and now renders the correct “SS”, “NS”, or “BS” values.
  • Set and List were rendering contains conditions incorrectly, by trying to dump each value in the value passed to contains. For example, MyModel.strings.contains("foo") would render contains(#n0, :v1) where :v1 was {"SS": [{"S": "f"}, {"S": "o"}, {"S": "o"}]}. Now, non-iterable values are rendered singularly, so :v1 would be {"S": "foo"}. This is a temporary fix, and only works for simple cases. For example, List(List(String)) will still break when performing a contains check. This is fixed correctly in 1.0.0 and you should migrate as soon as possible.

0.9.12 - 2016-06-13

[Added]

  • model.Meta now exposes gsis and lsis, in addition to the existing indexes. This simplifies code that needs to iterate over each type of index and not all indexes.

[Removed]

  • engine_for_profile was no longer necessary, since the client instances could simply be created with a given profile.

0.9.11 - 2016-06-12

[Changed]

  • bloop.Client now takes boto_client, which should be an instance of boto3.client("dynamodb") instead of a boto3.session.Session. This lets you specify endpoints and other configuration only exposed during the client creation process.
  • Engine no longer uses "session" from the config, and instead takes a client param which should be an instance of bloop.Client. bloop.Client will be going away in 1.0.0 and Engine will simply take the boto3 clients directly.

0.9.10 - 2016-06-07

[Added]

  • New exception AbstractModelException is raised when attempting to perform an operation which requires a table, on an abstract model. Raised by all Engine functions as well as bloop.Client operations.

[Changed]

  • Engine operations raise AbstractModelException when attempting to perform operations on abstract models.
  • Previously, models were considered non-abstract if model.Meta.abstract was False, or there was no value. Now, ModelMetaclass will explicitly set abstract to False so that model.Meta.abstract can be used everywhere, instead of getattr(model.Meta, "abstract", False).

0.9.9 - 2016-06-06

[Added]

  • Column has a new attribute model, the model it is bound to. This is set during the model’s creation by the ModelMetaclass.

[Changed]

  • Engine.bind will now skip intermediate models that are abstract. This makes it easier to pass abstract models, or models whose subclasses may be abstract (and have non-abstract grandchildren).

0.9.8 - 2016-06-05

(no public changes)

0.9.7 - 2016-06-05

[Changed]

  • Conditions implement __eq__ for checking if two conditions will evaluate the same. For example:

    >>> large = Blob.size > 1024**2
    >>> small = Blob.size < 1024**2
    >>> large == small
    False
    >>> also_large = Blob.size > 1024**2
    >>> large == also_large
    True
    >>> large is also_large
    False
    

0.9.6 - 2016-06-04

0.9.6 is the first significant change to how Bloop binds models, engines, and tables. There are a few breaking changes, although they should be easy to update.

Where you previously created a model from the Engine’s model:

from bloop import Engine

engine = Engine()

class MyModel(engine.model):
    ...

You’ll now create a base without any relation to an engine, and then bind it to any engines you want:

from bloop import Engine, new_base

BaseModel = new_base()

class MyModel(BaseModel):
    ...

engine = Engine()
engine.bind(base=MyModel)  # or base=BaseModel

[Added]

  • A new function engine_for_profile takes a profile name for the config file and creates an appropriate session. This is a temporary utility, since Engine will eventually take instances of dynamodb and dynamodbstreams clients. This will be going away in 1.0.0.
  • A new base exception BloopException which can be used to catch anything thrown by Bloop.
  • A new function new_base() creates an abstract base for models. This replaces Engine.model now that multiple engines can bind the same model. This will be going away in 1.0.0 which will provide a BaseModel class.

[Changed]

  • The session parameter to Engine is now part of the config kwargs. The underlying bloop.Client is no longer created in Engine.__init__, which provides an opportunity to swap out the client entirely before the first Engine.bind call. The semantics of session and client are unchanged.
  • Engine._load, Engine._dump, and all Type signatures now pass an engine explicitly through the context parameter. This was mentioned in 0.9.2 and context is now required.
  • Engine.bind now binds the given class and all subclasses. This simplifies most workflows, since you can now create a base with MyBase = new_base() and then bind every model you create with engine.bind(base=MyBase).
  • All exceptions now subclass a new base exception BloopException instead of Exception.
  • Vector types Set, List, Map, and TypedMap accept a typedef of None so they can raise a more helpful error message. This will be reverted in 1.0.0 and will once again be a required parameter.

[Removed]

  • Engine no longer has model, unbound_models, or models attributes. Engine.model has been replaced by the new_base() function, and models are bound directly to the underlying type engine without tracking on the Engine instance itself.
  • EngineView dropped the corresponding attributes above.

0.9.5 - 2016-06-01

[Changed]

  • EngineView attributes are now properties, and point to the underlying engine’s attributes; this includes client, model, type_engine, and unbound_models. This fixed an issue when using with engine.context(...) as view: to perform operations on models bound to the engine but not the engine view. EngineView will be going away in 1.0.0.

0.9.4 - 2015-12-31

[Added]

  • Engine functions now take optional config parameters to override the engine’s config. You should update your code to use these values instead of engine.config, since engine.config is going away in 1.0.0. Engine.delete and Engine.save expose the atomic parameter, while Engine.load exposes consistent.
  • Added the TypedMap class, which provides dict mapping for a single typedef over any number of keys. This differs from Map, which must know all keys ahead of time and can use different types. TypedMap only supports a single type, but can have arbitrary keys. This will be going away in 1.0.0.

0.9.2 - 2015-12-11

[Changed]

  • Type functions _load, _dump, dynamo_load, dynamo_dump now take an optional keyword-only arg context. This dict will become required in 0.9.6, and contains the engine instance that should be used for recursive types. If your type currently uses cls.Meta.bloop_engine, you should start using context["engine"] in the next release. The bloop_engine attribute is being removed, since models will be able to bind to multiple engines.

0.9.1 - 2015-12-07

(no public changes)

0.9.0 - 2015-12-07

 
File Type Py Version Uploaded on Size
bloop-1.3.1-py3-none-any.whl (md5) Python Wheel py3 2017-10-09 73KB
bloop-1.3.1.tar.gz (md5) Source 2017-10-09 67KB