es2loki

es2loki is a migration library that helps to transfer logs from Elasticsearch to Grafana Loki.

To use es2loki currently you need to define your own mapping of elasticsearch documents to labels for Grafana Loki.

Demo

You may find helpful a demo folder which contains a fully-sufficient demo stand that demonstrates transferring logs using es2loki.

Usage

In the simplest form you don't need to write any Python code at all, Loki will receive no meaningful labels, but nevertheless - let's see how it works.

$ pip install -U es2loki
$ ELASTIC_HOSTS=http://localhost:9200 \
  ELASTIC_INDEX="filebeat-*" \
  LOKI_URL=http://localhost:3100 \
  python -m es2loki

In order to override default es2loki behaviour you need to subclass a es2loki.BaseTransfer class.

To declare how documents map to Loki labels you have to override a extract_doc_labels method (see demo/example.py):

from es2loki import BaseTransfer


class TransferLogs(BaseTransfer):
    def extract_doc_labels(self, source: dict) -> Optional[MutableMapping[str, str]]:
        return dict(
            app=source.get("fields", {}).get("service_name"),
            job="logs",
            level=source.get("level"),
            node_name=source.get("host", {}).get("name"),
            logger_name=source.get("logger_name"),
        )

You can run this using the following code:

import sys
from es2loki import run_transfer

if __name__ == "__main__":
    sys.exit(run_transfer(TransferLogs()))

You can find more examples in the demo folder.

Sorting

By default es2loki assumes that in the documents returned from Elasticsearch there are fields @timestamp (you can change the name - see below) and log.offset. Using these 2 fields we can be sure that we will not reread the same lines multiple times. But if you have your fields that could guarantee such a behaviour - please override a make_es_sort and make_es_search_after methods.

• make_es_sort defines by which fields the sorting will happen.
• make_es_search_after defines an initial "offset". It is needed to resume es2loki after a shutdown.

Configuration