mpacklog

Log dictionaries to MessagePack files in C++ and Python.

Installation

C++ with Bazel

Add a git repository rule to your Bazel WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "mpacklog",
    remote = "https://github.com/stephane-caron/mpacklog.git",
    tag = "v2.0.0",
)

You can then use the @mpacklog dependency for C++ targets, or the @mpacklog//:python dependency for Python targets.

Python

$ pip install mpacklog

Usage

C++

The C++ implementation uses multi-threading. Add messages to the log using the put function, they will be written to file in the background.

#include <mpacklog/Logger.h>
#include <palimpsest/Dictionary.h>

int main() {
    mpacklog::Logger logger("output.mpack");

    for (unsigned bar = 0; bar < 1000u; ++bar) {
        palimpsest::Dictionary dict;
        dict("foo") = bar;
        dict("something") = "else";
        logger.put(dict):
    }
}

Python

The Python implementation provides both a synchronous and an asynchronous API.

Asynchronous API

Add messages to the log using the put function, have them written to file in the separate write coroutine.

import asyncio
import mpacklog

async def main():
    logger = mpacklog.AsyncLogger("output.mpack")
    await asyncio.gather(main_loop(logger), logger.write())

async def main_loop(logger):
    for bar in range(1000):
        await asyncio.sleep(1e-3)
        await logger.put({"foo": bar, "something": "else"})
    await logger.stop()


if __name__ == "__main__":
    asyncio.run(main())

Synchronous API

The Synchronous API is very similar to the Asynchronous API, except it doesn't provide a stop method and the put and write methods are blocking.

import mpacklog

logger = mpacklog.SyncLogger("output.mpack")

for bar in range(1000):
    logger.put({"foo": bar, "something": "else"})

# Flush all messages to the file
logger.write()

Command-line

If you pip-installed mpacklog, you can use the mpacklog command to dump logs to JSON:

mpacklog dump my_log.mpack

Alternatively and more generally, two great tools to manipulate logs from the command line are:

• rq: transform from/to MessagePack, JSON, YAML, TOML, ...
• jq: manipulate JSON series to add, remove or extend fields

For instance, mpacklog dump is equivalent to:

rq -mJ < my_log.mpack