aiootp 0.1.0

Important Disclaimer

aiootp is experimental software that works with Python 3.6+. It’s a work in progress. The programming API could change with future updates, and it isn’t bug free. aiootp provides powerful security tools and misc utilities that’re designed to be developer-friendly and privacy preserving. As a security tool, aiootp needs to be tested and reviewed extensively by the programming and cryptography communities to ensure its implementations are sound. We provide no guarantees. This software hasn’t yet been audited by third-party security professionals.

Quick install

pip3 install --user --upgrade aiootp

Some Examples

Users can create and modify transparently encrypted databases:

   import aiootp


   # Make a new user key for encryption / decryption ->

   key = await aiootp.acsprng()    # <- A fast, cryptographically secure pseudo-random number generator


   # Create a database object ->

   db = await aiootp.AsyncDatabase(key)


   # Store protected data by a ``tag`` ->

   tag = "private_account"

   salt = await db.asalt()

   hmac = await db.ahmac("password012345", salt)

   db[tag] = {hmac: "secured data"}


   # Add to existing stored data ->

   db[tag].update({"password_salt": salt})


   # Read from the database with ``aquery`` ->

   (await db.aquery(tag))[hmac]

>>>'secured data'


   # Or use bracketed lookup (it's an async-safe operation) ->

   salt = db[tag]["password_salt"]

   wrong_hmac = await db.ahmac("wrong password attempt", salt)

   db[tag][wrong_hmac]

>>>KeyError:


   # Or, pop the value out of the database ->

   account_data = await db.apop(tag)


   # Create child databases accessible from the parent by a ``metatag`` ->

   metatag = "child"

   molly = await db.ametatag(metatag)

   molly["hobbies"] = ["skipping", "punching"]

   molly["hobbies"].append("reading")

   molly["hobbies"] is db.child["hobbies"]

>>>True


   # Write the database changes to disk ->

   await db.asave()


   # Delete a child database from the filesystem ->

   await db.adelete_metatag("child")

   db.child["hobbies"]

>>>AttributeError: 'AsyncDatabase' object has no attribute 'child'


   # Automate the write to disk logic with a context manager ->

   async with aiootp.AsyncDatabase(key) as db:

       db["tag"] = {"data": "can be any json serializable object"}

       db["bitcoin"] = "0bb6eee10d2f8f45f8a"

       db["lawyer"] = {"#": "555-555-1000", "$": 13000.50}

       db["safehouses"] = ["Dublin Forgery", "NY Insurrection"]


   # Make mirrors of databases ->

   new_key = await aiootp.acsprng()

   new_db = await aiootp.AsyncDatabase(new_key)

   await new_db.amirror_database(db)

   assert new_db["lawyer"] is db["lawyer"]


   # Delete a database from the filesystem ->

   await db.adelete_database()


   # Initialization of a database object is more computationally expensive than entering its

   # context manager. So keeping a reference to a preloaded database is a great idea, either

   # call ``asave`` / ``save`` periodically, or open a context with the reference whenever

   # wanting to capture changes to the filesystem ->

   async with new_db as db:

       print("Saving to disk...")


   #

What other tools are available to users?:

import aiootp


# Async & synchronous versions of almost everything in the library ->

assert await aiootp.asha_512("data") == aiootp.sha_512("data")

key = aiootp.csprng()

assert aiootp.Database(key).root_filename == (await aiootp.AsyncDatabase(key)).root_filename


# Precomputed & organized values that can aid users, like:

# A dictionary of prime numbers grouped by their bit-size ->

aiootp.primes[512][0]    # <- The first prime greater than 512-bits

aiootp.primes[2048][-1]    # <- The last prime less than 2049-bits


# Symmetric one-time-pad encryption of json data ->

key = aiootp.csprng()

plaintext = {"account": 3311149, "titles": ["queen b"]}

encrypted = aiootp.json_encrypt(plaintext, key=key)

decrypted = aiootp.json_decrypt(encrypted, key=key)

assert decrypted == plaintext


# Generators under-pin most procedures in the library ->

from aiootp import json_encode   # <- A simple generator

from aiootp.ciphers import cipher, decipher    # <- Also simple generators

plaintext_generator = json_encode(plaintext)    # <- Yields plaintext json string in chunks

keystream = aiootp.keys(key)    # <- An endless stream of forward + semi-future secure hashes

with aiootp.cipher(plaintext_generator, keystream) as encrypting:    # <- xor's the plaintext chunks with key chunks

    ciphertext = encrypting.list()    # <- ``list`` returns all results in a list

ciphertext_seed_entropy = keystream.result(exit=True)    # <- Get the auto generated random salt back
                                                         #    This salt is needed for decryption.


# This example was a low-level look at the encryption algorithm. And it was

# seven lines of code. The Comprende class makes working with generators a breeze,

# and working with generators makes solving problems in bite-sized chunks a breeze.

# Here's the two-liner ->

ciphertext = aiootp.json_encode(plaintext).encrypt(key).list()    # <- The salt is encrypted with ``key``
                                                                  # and attached to the ciphertext
plaintext_json = aiootp.unpack(ciphertext).decrypt(key).join()


# We just used the ``list`` & ``join`` end-points to get the full series of results from the

# underlying generators. These results are lru-cached to facilitate their efficient reuse

# for alternate computations. The ``Comprende`` context manager clears the opened instance's

# cache on exit, this clears every instance's cache ->

aiootp.Comprende.clear_class()


# The other end-points can be found under ``aiootp.Comprende.eager_methods`` ->

{
    'adeque',
    'adict',
    'aexhaust',    # <- Doesn't cache results, only returns the last element
    'ajoin',
    'alist',
    'aset',
    'deque',
    'dict',
    'exhaust',    # <- Doesn't cache results, only returns the last element
    'join',
    'list',
    'set',
}


# A lot of this magic with generators is made possible with a sweet little

# ``comprehension`` decorator. It reimagines the generator interface by

# wrapping generators in the innovative ``Comprende`` class, giving every

# generator access to a plethora of data processing & cryptographic utilities

# right out of the box ->

@aiootp.comprehension()

def gen(x=None, y=None):

    z = yield x + y

    return x * y * z


# Drive the generator forward with a context manager ->

with gen(x=1, y=2) as example:

    z = 3


    # Calling the object will send ``None`` into the coroutine by default ->

    sum_of_x_y = example()

    assert sum_of_x_y == 3


    # Passing ``z`` will send it into the coroutine, cause it to reach the

    # return statement & exit the context manager ->

    example(z)


# The result returned from the generator is now available ->

product_of_x_y_z = example.result()

assert product_of_x_y_z == 6


# The ``example`` variable is actually the ``Comprende`` object,

# which redirects values to the wrapped generator's ``send()``

# method using the instance's ``__call__()`` method.


# Here's another example ->

@aiootp.comprehension()

def squares(numbers=20):

    for number in range(numbers):

        yield number ** 2


for hashed_square in squares().sha_256():

    # This is an example chained generator that hashes then yields each output.

    print(hashed_square)


# Chained ``Comprende`` generators are excellent inline data processors ->

base64_data = []

for result in squares().str().to_base64():

    # This will stringify each output of the generator, then base64 encode them ->

    base64_data.append(result)


# Async ``Comprende`` coroutines have almost exactly the same interface as

# synchronous ones ->

@aiootp.comprehesion()

async def gen(x=None, y=None):

    # Because having a return statement in an async generator is a

    # SyntaxError, the return value is expected to be passed into

    # UserWarning, and then raised to propagate upstream. It's then

    # available from the instance's ``aresult`` method ->

    z = yield x + y

    result = x * y * z

    raise UserWarning(result)


# Drive the generator forward.

async with gen(x=1, y=2) as example:

    z = 3

    # Awaiting the ``__call__`` method will send ``None`` into the coroutine by default ->

    sum_of_x_y = await example()

    assert sum_of_x_y == 3


    # Passing ``z`` will send it into the coroutine, cause it to reach the

    # raise statement which will exit the context manager gracefully ->

    await example(z)


# The result returned from the generator is now available ->

product_of_x_y_z = await example.aresult()

assert product_of_x_y_z == 6


# Let's see some other ways async generators mirror synchronous ones ->

@aiootp.comprehension()

async def squares():

    number = 0

    while True:

        yield number ** 2

        number += 1


# This is a chained async generator that salts then hashes then yields each output.

salt = await aiootp.acsprng()

hashed_squares = squares().asha_512(salt)


# Want only the first twenty results? ->

async for hashed_square in hashed_squares[:20]:

    # Then you can slice the generator.

    print(hashed_square)


# Users can slice generators to receive more complex output rules, like:

# Getting every second result starting from the third result to the 50th ->

async for result in hashed_squares[3:50:2]:

    print(result)


# ``Comprende`` generators have loads of tooling for users to explore. Play

# around with it and take a look at the other chainable generator methods

# in ``aiootp.Comprende.lazy_generators``.

{
    '_agetitem',
    '_getitem',
    'aascii_to_int',
    'abin',
    'abytes',
    'adecode',
    'adecrypt',
    'aencode',
    'aencrypt',
    'afeed',
    'afeed_self',
    'afrom_base',
    'afrom_base64',
    'ahalt',
    'ahex',
    'aint',
    'aint_to_ascii',
    "ajson_dumps",
    "ajson_loads",
    'amap_decrypt',
    'amap_encrypt',
    'aresize',
    'ascii_to_int',
    'asha_256',
    'asha_256_hmac',
    'asha_512',
    'asha_512_hmac',
    'aslice',
    'astr',
    'atag',
    'atimeout',
    'ato_base',
    'ato_base64',
    'azfill',
    'bin',
    'bytes',
    'decode',
    'decrypt',
    'encode',
    'encrypt',
    'feed',
    'feed_self',
    'from_base',
    'from_base64',
    'halt',
    'hex',
    'int',
    'int_to_ascii',
    "json_dumps",
    "json_loads",
    'map_decrypt',
    'map_encrypt',
    'resize',
    'sha_256',
    'sha_256_hmac',
    'sha_512',
    'sha_512_hmac',
    'slice',
    'str',
    'tag',
    'timeout',
    'to_base',
    'to_base64',
    'zfill',
}


# Let's look at a more complicated example with the one-time pad

# keystreams. There are many uses for endless streams of deterministic

# key material outside of one-time pad cipher keys. They can, for instance,

# give hash tables order that's cryptographically determined & obscured ->

ordered_entries = {}

salt = await aiootp.acsprng()

names = aiootp.akeys(key, salt)


# Resize each output of ``names`` to 32 characters, tag each output with

# an incrementing number, & stop the stream after 0.1 seconds ->

async for index, name in names.aresize(32).atag().atimeout(0.1):

    ordered_entries[name] = f"{index} data organized by the stream of hashes"


# Retrieving items in the correct order requires knowing both ``key`` & ``salt``

async for index, name in aiootp.akeys(key, salt).aresize(32).atag():

    try:

        assert ordered_entries[name] == f"{index} data organized by the stream of hashes"

    except KeyError:

        print(f"There are no more entries after {index} iterations.")

        assert index == len(ordered_entries) + 1


# There's a prepackaged ``Comprende`` generator function that does encryption / decryption

# of key ordered hash maps. First let's make an actual encryption key stream different from

# ``names`` ->

key_stream = aiootp.akeys(key, salt, pid=aiootp.sha_256(key, salt))


# And example plaintext ->

plaintext = 100 * "Some kinda message..."


# And let's make sure to clean up after ourselves with a context manager ->

async with aiootp.adata(plaintext).amap_encrypt(names, key_stream) as encrypting:

    # ``data`` takes a sequence, & ``amap_encrypt`` takes two iterables, a stream of

    # names for the hash map, & the stream of key material.

    ciphertext_hashmap = await encrypting.adict()


# Now we'll pick the chunks out in the order produced by ``names`` to decrypt them ->

async with aiootp.apick(names, ciphertext_hashmap).amap_decrypt(key_stream) as decrypting:

    decrypted = await decrypting.ajoin()

assert decrypted == plaintext


# This is really neat, & makes sharding encrypted data incredibly easy.


#

Here’s a quick overview of this package’s modules:

import aiootp


# Commonly used constants, datasets & functionality across all modules ->

aiootp.commons


# The basic utilities & abstractions of the package's architechture ->

aiootp.generics


# This module is responsible for providing entropy to the package ->

aiootp.randoms


# The higher-level abstractions used to implement the one-time pad ->

aiootp.ciphers


# The higher-level abstractions used to create / manage key material ->

aiootp.keygens


# Common system paths & the ``pathlib.Path`` utility ->

aiootp.paths


# Global async functionalities & abstractions ->

aiootp.asynchs


# Decorators & classes able to benchmark async/sync functions & generators ->

aiootp.debuggers


#

FAQ

Q: What is the one-time-pad?

A: It’s a provably unbreakable cipher. It’s typically thought to be too cumbersome a cipher because it has strict requirements. Key size is one requirement, since keys must be at least as large as the plaintext in order to ensure this unbreakability. We’ve simplified this requirement by using a forward secret and semi-future secret key ratchet algorithm, with ephemeral salts for each stream, allowing users to securely produce endless streams of key material as needed from a single finite size 512-bit long-term key. This algorithmic approach lends itself to great optimizations, since hash processing hardware/sorftware is continually pushed to the edges of efficiency.

Q: What do you mean the ``aiootp.keys`` generator produces forward & semi-future secure key material?

A: The infinite stream of key material produced by that generator has amazing properties. Under the hood it’s a hashlib.sha3_512 key ratchet algorithm. It’s internal state consists of a seed hash, & three hashlib.sha3_512 objects primed iteratively with the one prior and the seed hash. The first object is updated with the seed, its prior output, and the entropy that may be sent into the generator as a coroutine. This first object is then used to update the last two objects before yielding the last two’s concatenated results. The seed is the hash of a primer seed, which itself is the hash of the input key material, a random salt, and a user-defined ID value which can safely distinguish streams with the same key material. This algorithm is forward secure because compromising a future key will not compromise past keys since these hashes are irreversibly constructed. It’s also semi-future secure since having a past key doesn’t allow you to compute future keys without also compromising the seed hash, and the first ratcheting hashlib object. Since those two states are never disclosed or used for encryption, the key material produced is future secure with respect to itself only. Full future-security would allow for the same property even if the seed & ratchet object’s state were compromised. This feature can, however, be added to the algorithm since the generator itself can receive entropy externally from a user at any arbitrary point in its execution, say, after computing a shared diffie-hellman exchange key.

Q: How fast is this implementation of the one-time pad cipher?

A: Well, because it relies on hashlib.sha3_512 hashing to build key material streams, it’s rather efficient, encrypting & decrypting about 8 MB/s on a ~1.5 GHz core.

Q: Why make a new cipher when AES is strong enough?

A: Although primatives like AES are strong enough for now, there’s no guarantee that future hardware or algorithms won’t be developed that break them. In fact, AES’s theoretical bit-strength has dropped over the years because of hardware and algorithmic developments. It’s still considered a secure cipher, but the one-time pad isn’t considered theoretically “strong enough”, instead it’s mathematically proven to be unbreakable. Such a cryptographic guarantee is too profound not to develop further into an efficient, accessible standard.

Q: What size keys does this one-time pad cipher use?

A: It’s been designed to work with 512-bit hexidecimal or 128 arbitrary character keys.

Q: What’s up with the ``AsyncDatabase`` / ``Database``?

A: The idea is to create an intuitive, pythonic interface to a transparently encrypted and decrypted persistence tool that also cryptographically obscures metadata. It’s designed to work with json serializable data, which gives it native support for some basic python datatypes. It needs improvement with regard to disk memory efficiency. So, it’s still a work in progress, albeit a very nifty one.

Q: Why are the modules transformed into ``Namespace`` objects?

A: We overwrite our modules in this package to have a more fine-grained control over what part of the package’s internal state is exposed to users and applications. The goal is make it more difficult for users to inadvertently jeopardize their security tools, and minimize the attack surface available to adversaries. The aiootp.Namespace class also makes it easier to coordinate and decide the library’s UI/UX across the package.

Minor Changes

• Initial version.

Major Changes

• Initial version.