Skip to main content

performanceplatform-collector: tools for sendingdata to the Performance Platform

Project description

A python command line tool that aggregates data from third party sources and pushes the result through to the Performance Platform (see http://alphagov.github.io/performanceplatform-documentation/ for more details). This tool will currently interface with Google Analytics and Pingdom to retrieve data.

https://travis-ci.org/alphagov/performanceplatform-collector.svg

Installation

Using pip

$ pip install performanceplatform-collector

From source

$ git clone https://github.com/alphagov/performanceplatform-collector.git
$ cd performanceplatform-collector
$ virtualenv venv
$ source venv/bin/activate
$ python setup.py install

Usage

pp-collector takes paths to various JSON files as arguments:

$ pp-collector -q [query file] -c [credentials file] -t [token file] -b [backdrop file]

There are also some optional command line arguments you can provide pp-collector:

--console-logging
Rather than logging out to log/collector.log it will output all logs to stdout/err

--dry-run
When it comes to submitting the gathered data to the Performance Platform it will skip
making the POST requests and instead log out the url, headers and body to your terminal.

Configuration

There are four configuration files that get injected into pp-collector and are the four required parameters:

  • Query, contains everything about what the collector will do during execution. It provides an entrypoint that pp-collector will execute and provide the query and options k-v pairs:

    {
      "entrypoint": "performanceplatform.collector.pingdom",
      "query": {
        "name": "govuk"
      },
      "options": { },
      "data-set": {
        "data-group": "my-data-group",
        "data-set": "my-data-set"
      }
    }
  • Token, this holds the bearer token to be used by this collector when POSTing to the Performance Platform:

    {
      "token": "some long hex value"
    }
  • Credentials, passes through any usernames, passwords, API keys etc… that are required to communicate to the third party service you desire.

  • Backdrop, where backdrop lives:

    {
      "url": "https://www.performance.service.gov.uk/data"
    }

For our deployment of performanceplatform-collector we pull in deployment from performanceplatform-collector-config repo. You will find much more indepth information about the collector configuration in the repos README.

Entrypoints

The following entrypoints are currently available:

performanceplatform.collector.ga
performanceplatform.collector.ga.trending
performanceplatform.collector.ga.realtime
performanceplatform.collector.pingdom

Extending performanceplatform-collector

performanceplatform-collector can be extended to support new types of collector. To do so you’ll need to add new entrypoints. For each new type of collector create a file at:

performanceplatform/collector/mycollectortype/__init__.py

Inside that file add a main function which has the following signature:

main(credentials, data_set_config, query, options, start_at, end_at)

These arguments are all strings which are forwarded from the command line.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

performanceplatform-collector-0.1.2.tar.gz (28.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page