Stat, metric, and diagnostic publishing and consuming tools
Project description
Documentation | Package | Code
About
Mmstats is a way to expose and read diagnostic values and metrics for applications.
Think of mmstats as /proc for your application and the readers as procps utilities.
This project is a Python implementation, but compatible implementations can be made in any language (see Goals).
Discuss at https://groups.google.com/group/python-introspection
Goals
Separate publishing/writing from consuming/reading tools
Platform/language independent (a Java writer can be read by a Python tool)
Predictable performance impact for writers via:
No locks (1 writer per thread)
No syscalls (after instantiation)
All in userspace
Reading has no impact on writers
Optional persistent (writer can sync anytime)
1-way (Publish/consume only; mmstats are not management extensions)
Usage
Requirements
CPython 2.6 or 2.7 (Windows is untested)
PyPy (only tested in 1.7, should be faster in 1.8)
Using
python setup.py install
import mmstats
Create a subclass of mmstats.MmStats like:
class WebStats(mmstats.MmStats):
status2xx = mmstats.CounterField(label='status.2XX')
status3xx = mmstats.CounterField(label='status.3XX')
status4xx = mmstats.CounterField(label='status.4XX')
status5xx = mmstats.CounterField(label='status.5XX')
last_hit = mmstats.DoubleField(label='timers.last_hit')
Instantiate it once per process: (instances are automatically thread local)
webstats = WebStats(label_prefix='web.stats.')
Record some data:
if response.status_code == 200:
webstats.status2xx.inc()
webstats.last_hit = time.time()
Run slurpstats to read it
Run mmash to create a web interface for stats
Run pollstats -p web.stats.status 2XX,3XX,4XX,5XX /tmp/mmstats-* for a vmstat/dstat like view.
Did a process die unexpectedly and leave around a stale mmstat file? cleanstats /path/to/mmstat/files will check to see which files are stale and remove them.
History
0.7.0 “Local Artisanal Stats” released 2012-10-02
Per-thread model instances are created automatically - no need to manually create one per thread
Backward incompatible change to naming templates; they now use str.format style substitutions:
New: {CMD} - Current process name
%PID% -> {PID}
%TID% -> {TID}
0.6.2 “Graphtastic” released 2012-03-23
Added live graphing of numeric metrics thanks to @haard’s work at PyCon
Documentation improvements
0.6.1 “MANIFEST.out” released 2012-03-08
Fix packaging issue
0.6.0 “PyCon 2012” released 2012-03-08
[API CHANGE] - MovingAverageField’s kwarg changed from window_size => size
Refactored __init__.py into fields, models, and default (and imported public bits into __init__)
Added TimerField (MovingAverageField + context manager)
Added docs (don’t get too excited, just a start)
0.5.0 “100% More Average” released 2012-02-25
[API CHANGE] - RunningAverage field is now AverageField
Added MovingAverageField with window_size=100 parameter
Tests can now be run via “python setup.py test”
0.4.1 “Derpstats” released 2012-01-31
Fixed pollstats
Updated README slightly
0.4.0 “On the Road to Pycon” released 2012-01-17
Added clean module and cleanstats script to clean stale mmstat files
Added path kwarg to MmStats class to allow easy path overriding
Added StringField for UTF-8 encoded strings
Added StaticFloatField & StaticDoubleField
Added created UNIX timestamp (sys.created) to default MmStats class
Moved all modules into mmstats package
Fixed mmash template packaging
Fixed test mmstat file cleanup
Refactored reading code into mmstats.reader module
0.3.12 “Meow” released 2011-11-29
Use ctypes.get_errno() instead of Linux specific wrapper
0.3.11 “Rawr” released 2011-11-29
Fix libc loading on OSX
0.3.10 “π²” released 2011-11-28
PyPy support (switched from ctypes._CData.from_buffer to .from_address)
Multiple calls to MmStats().remove() no longer error (makes testing easier)
0.3.9 “MLIT” released 1970-01-01
Mistag of 0.3.8
0.3.8 “Hapiness” released 2011-11-20
Allow filename templating with %PID% and %TID% placeholders
Allow setting filename template via MMSTATS_FILES environment variable
Improved docs slightly
Fixed Ctrl-Cing run_flask_example script
Fixed 64 bit integer fields on 32 bit platforms
Fixed StaticInt64Field (was actually a uint64 field before)
Fixed slurpstats debug output (always showed first 40 bytes of file)
Strip newlines from org.python.version
0.3.7 “Depressive Realism is for Winners” released 2011-11-17
Add pollstats utility (similar to dstat/vmstat)
Cleanup development/testing section of the README
Slight improvements to basic flask integration example
0.3.6 “The M is for Mongo” released 2011-11-09
Allow setting the value of CounterFields
0.3.5 “Ornery Orangutan” released 2011-10-20
Added a running average field
Made mmash more configurable and added a console entry point
Updated TODO
TODO
There’s always bugs to fix: https://github.com/schmichael/mmstats/issues/
Add API to dynamically add fields to MmStat classes
Percentiles
Time based windows for moving averages (eg last 60 seconds)
Multiple exposed fields (average, mean, and percentiles) from 1 model field
Add alternative procedural writer API (vs existing declarative models)
Test severity of race conditions (especially: byte value indicating write buffer)
Test performance
Vary filename based on class name
Improve mmash (better live graphing, read from multiple paths, etc)
Include semantic metadata with field types (eg to differentiate an int that’s a datetime from an int that’s a counter)
Logo
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
