Skip to main content

Tail your log, extract the data, and send it to Fluentd

Project description

Flosculus is a work-in-progress script to tail rotated log file, parse each line, and send it to Fluentd. It is intended to replace a subset of Fluentd in_tail features.

Crash Course

First things first, install Flosculus into your Python path.

$ git clone git://github.com/iromli/flosculus.git
$ cd flosculus
$ python setup.py install

Afterwards you need to create a configuration file to run the flosculusd event loop. The convenient way to do this boring thing is by invoking the flosculusd --init > flosculus.ini command. Here’s an example of flosculus.ini (you can call it whatever you like):

[flosculus]
; the IP address (or host name) of the remote server
remote_host = 127.0.0.1

; the TCP port of the remote server
remote_port = 24224


; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = nginx

By default, the format option is matched against nginx default access log format. You may change the format as long as using a valid Python regex.

[log:/var/log/nginx/access.log]

format = (?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?

And did I already mentioned that you can have multiple log files? Simply copy the whole log section.

; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = nginx

; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/timed-combined.access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = (?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?

Assuming the configuration is written properly, run the event loop to see its magic (well, not really):

$ flosculusd -c flosculus.ini

If you have any further question, poke @iromli on Twitter.

Credits

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

flosculus-0.1.tar.gz (6.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page