skip to navigation
skip to content

Not Logged In

datagrepper 0.3.0

A webapp to query fedmsg history

Latest Version: 0.4.0


A webapp to retrieve historical information about messages on the `fedmsg
<>`_ bus.  It is a JSON api for the `datanommer
<>`_ message store.

Production Instance

Hacking on datagrepper

Setting up the stack

Use a virtualenv::

    $ mkvirtualenv datagrepper
    $ workon datagrepper

Install dependencies::

    $ pip install -r requirements.txt
    $ pip install psycopg2

Set up the fedmsg consumer for the job runner::

    $ python develop

Configuring Postgresql (and getting some data)

In normal operations, the `datanommer
<>`_ consumer daemon will be
running somewhere and continuously stuff each new `fedmsg
<>`_ message that it sees into a postgres DB.  If you're
just sitting down to hack on datagrepper, that won't be your situation
so you'll need a dump of the database.

.. note:: If you've tried installing postgres before and think you've
   messed it up, you'll need to blow away the old databases with
   ``$ rm -rf /var/lib/pgsql``

Install postgres (and fedmsg, while we're at it)::

    $ sudo yum install -y postgresql-server fedmsg
    $ sudo postgresql-setup initdb

Make sure postgres is set to allow connections over tcp/ip using password
authentication.  Edit the ``/var/lib/pgsql/data/pg_hba.conf``.  You might
find a line like this::

    host    all             all               ident

Instead of that line, you need one that looks like this::

    host    all             all               md5


Become yourself again (not the ``postgres`` user) and start up postgres::

    $ sudo systemctl restart postgresql.service

Become the postgres user (again) and run the psql command.  Use that psql
shell to setup the DB, the user, and privileges::

    $ sudo su - postgres
    $ psql
    # create database datanommer;
    # create user datanommer with password 'bunbunbun';
    # grant all privileges on database datanommer to datanommer;
    # \q

Back in the bash shell (but still as the `postgres` user), grab a DB dump and
restore it::

    $ wget
    $ xzcat datanommer-2013-11-11.dump.xz | psql datanommer

Last step, run datagrepper

You have to configure your development datagrepper instance to talk to
postgres (by default, it looks for a sqlite database).  Edit
``fedmsg.d/`` and give it these contents:

.. code-block:: python

    config = {
        'datanommer.enabled': False,
        'datanommer.sqlalchemy.url': 'postgresql+psycopg2://datanommer:bunbunbun@localhost:5432/datanommer',
        'fedmsg.consumers.datagrepper-runner.enabled': True,

As your normal old user self, run the development server::

    $ workon datagrepper
    $ python

In a browser, visit http://localhost:5000 to see the docs.

You can quick test that you can get data by running::

    $ sudo yum install -y httpie
    $ http get localhost:5000/raw/ delta==1000000 rows_per_page==1

Running the job runner

Without starting ``fedmsg-hub``, the job runner won't actually run jobs::

    $ workon datagrepper
    $ fedmsg-hub
File Type Py Version Uploaded on Size
datagrepper-0.3.0.tar.gz (md5, pgp) Source 2014-01-10 79KB
  • Downloads (All Versions):
  • 56 downloads in the last day
  • 400 downloads in the last week
  • 1385 downloads in the last month