Ingests log files into Graphite where they can be visualized.
Project description
Overview
Graphite Log Feeder (GLF) is a generic tool that can parse application log files, extract meaningful data, and forward it to Graphite for visualizing. There is a parser included for ingesting data from The Grinder. This parser can be used as the basis for creating other parsers (for Apache, Scribe, your own application, or whatever.)
There is a similar tool – Logster – which also feeds log data to Graphite. However, Logster only supports realtime log analysis, whereas glf only supports non-realtime analysis, making the two tools complimentary.
Installation
GLF is installed via pip.
pip install graphite_log_feeder
Once pip has installed glf, you will need to generate a config file, and edit it to be suitable for your environment. Glf comes with a command-line option to generate a sample config file
graphite_log_feeder.py -e
This will generate a file named ‘glf.sample.conf’ which you can use as the basis for creating your own configuration.
There must be a running installation of Graphite on your network for GLF to forward data to. See the Graphite web site for details on setting up and configuring Graphite
Usage
(after adjusting the values in your sample config file to be appropriate for your environment)
graphite_log_feeder.py <config_file>
Additional Resources
Java/JMX counters and application-level metrics can be fed to Graphite using the JMXTrans tool:
http://code.google.com/p/jmxtrans/
OS-level metrics (CPU, mem, etc.) can be fed to Graphite via quickstatd or collectd (with graphite plugin)
