performanceplatform-collector 0.2.1

Using pip

pip install performanceplatform-collector

From source

git clone https://github.com/alphagov/performanceplatform-collector.git
cd performanceplatform-collector
virtualenv venv
source venv/bin/activate
python setup.py install

Configuration

Note on our configuation

For our deployment of the performanceplatform-collector we pull in configuration files from the performanceplatform-collector-config repo. The structure of our deployment configuration can be found there if more detailed examples are required.

There are four configuration files that get injected into pp-collector, each file is a required parameter.

# pingdom example
{
  "entrypoint": "performanceplatform.collector.pingdom",
  "query": {
    "name": "govuk"
  },
  "options": {
    "additionalFields": {
      # Every record sent to backdrop will have these additional fields
      "foo": "bar",
      "sentAt": "specific-formatted-time-value"
    }
  },
  "plugins": [
    "Comment('Pingdom stats are aggregated using mycustomtemplate.py')
  ],
  "data-set": {
    "data-group": "my-data-group",
    "data-set": "my-data-set"
  }
}

`performanceplatform.collector.ga`_
`performanceplatform.collector.ga.trending`_
`performanceplatform.collector.ga.realtime`_
`performanceplatform.collector.pingdom`_

{
  "url": "https://www.performance.service.gov.uk/data"
}

{
  "token": "some long hex value"
}

# Google analytics Specific example
credentials = {
    "CLIENT_SECRETS": path/to/client_secret.json,
    "STORAGE_PATH": path/to/oauth/db,
}

# Piwik example
{
  "token_auth": "your Piwik secret token",
  "url": "your Piwik API url"
}

You can get your Piwik secret token from the Manage Users
admin area in your Piwik account.

Example Piwik configuration

Here is an example Piwik query file:

{
  "data-set": {
     "data-group": "consular-appointment-booking-service",
     "data-type": "journey-by-goal"
   },
   "entrypoint": "performanceplatform.collector.piwik.core",
   "query": {
     "site_id": "9",
     "api_method": "Goals.get",
     "frequency": "daily",
     "api_method_arguments": {
        "idGoal": "3"
     }
   },
   "options": {
     "mappings": {
       "nb_visits_converted": "converted",
       "nb_conversions": "sessions"
     },
     "idMapping": ["dataType","_timestamp","timeSpan"]
     },
   "token": "piwik_fco"
}

The above configuration will instruct the Piwik collector to fetch data via the Goals.get method of your Piwik Reporting API endpoint. The endpoint is specified via the ‘url’ setting in your credentials file.

The ‘site_id’ and ‘frequency’ settings map to the standard Piwik Reporting API method arguments of ‘idSite’ and ‘period’ respectively.

• site_id - a number representing your website

• frequency - how statistics should be reported (daily, weekly, monthly)

If not specified, the ‘frequency’ setting defaults to ‘weekly’.

You can specify API method-specific arguments using the ‘api_method_arguments’ key in your query file as shown in the example. For a full list of methods available in the Piwik Reporting API, see http://developer.piwik.org/api-reference/reporting-api.

The Piwik collector uses the ‘mappings’ settings in your query file to determine which data items to extract from an API response and how to map their keys. The above query file, for example, will configure the collector to extract the ‘nb_visits_converted’ and ‘nb_conversions’ data items from the following example API response:

{
  "From 2015-05-25 to 2015-05-31": {
    "nb_visits_converted": 791,
    "nb_conversions": 791,
    "conversion_rate": 18.09,
    "revenue": 0 }
}

The keys of these data items will be replaced with ‘converted’ and ‘sessions’ respectively, ready for storage in the Performance Platform’s data application, Backdrop.

Running the Piwik collector

The Piwik collector is run from the command line in the normal way - see the Usage section above.

If you want to collect data by day, week or month over a period of time, specify an appropriate value for the ‘frequency’ setting in your query file and a start and end date in your run command using the ‘–start’ and ‘–end’ optional arguments. The dates are passed to the Piwik API via a ‘date’ argument of the form ‘YYYY-MM-DD,YYYY-MM-DD’.

If date arguments are not provided, a value of ‘previous1’ is passed for the Piwik ‘date’ argument which will return data for the previous day, week or month (according to the value of your ‘frequency’ setting).