Skip to main content

Data management platform for robot logs

Project description

MARV Robotics is an extensible data management platform for robot logs. New robot logs are found by a scanner and configured nodes are run to extract, filter and process data from them.

The robot logs are visualized in a web-based application that features a listing view with filters and summary, and detail views of individual log files.

So far MARV Robotics supports the Robot Operating System (ROS) Bag format used by systems running ROS to record sensor data and system state.

Robot Operating System (ROS)

MARV Robotics is meant to support all alive ROS releases. Currently these are indigo, jade, and kinetic – if you encounter difficulties, please report back!

MARV Robotics is implemented in Python using the MARV framework. Most of its Python dependencies will be installed by pip automatically. Apart from these on an Ubuntu system the following are needed (as root). In case you succeeded installing ROS on anything else than Ubuntu, we trust you’ll also succeed to transfer the following lines to the system you are installing to.

Indigo

For indigo on Ubuntu 14.04 (trusty), first make sure you have the latest Python 2.7 version installed:

# apt-get install software-properties-common
# add-apt-repository ppa:fkrull/deadsnakes-python2.7
# apt-get update
# apt-get upgrade

Then install the requirements:

# apt-get install curl \
                  ros-indigo-ros-base \
                  ros-indigo-cv-bridge \
                  libffi-dev \
                  python2.7-dev \
                  python-opencv \
                  python-virtualenv \
                  libjpeg-dev \
                  libz-dev
...
# rosdep init

Make sure ROS is set-up correctly for your user (not as root):

$ rosdep update
$ source /opt/ros/indigo/setup.bash

Kinetic

# apt-get install curl \
                  ros-kinetic-ros-base \
                  python-cv-bridge \
                  python2.7-dev \
                  python-opencv \
                  python-virtualenv \
                  libjpeg-dev \
                  libz-dev
...
# rosdep init

Make sure ROS is set-up correctly for your user (not as root):

$ rosdep update
$ source /opt/ros/kinetic/setup.bash

Installation

While strictly speaking a virtual Python environment is not necessary and MARV Robotics installs fine without, we highly recommend it. For instructions to install into your user’s home instead, see below.

Option 2: User install

An alternative to install into an isolated virtualenv (see above) is to install into the user’s $HOME/.local folder where MARV Robotics might interfere with other installed programs. First check whether $HOME/.local/bin is in $PATH and $HOME/.local/lib/python2.7/site-packages in $PYTHONPATH:

$ echo $PATH
$ echo $PYTHONPATH

and add if needed:

$ export PATH="$HOME/.local/bin:$PATH"
$ export PYTHONPATH="$HOME/.local/lib/python2.7/site-packages:$PYTHONPATH"

Verify that ROS is still available:

$ python -c 'import rosbag; print("rosbag available")'
rosbag available

Install MARV Robotics and verify it is available by printing its usage instructions:

$ pip install --user marv-robotics
...
$ marv --help
...

By now MARV Robotics should be successfully installed.

Configuration

After having successfully installed MARV Robotics, we can initialize a site that is going to hold configuration files and databases:

(venv) $ marv init ./site
Path to scanroot containing bags: /scanroot
Application root for MARV, if not at root of domain []:
Wrote config file site/marv.conf
Wrote site/wsgi.py
Wrote site/uwsgi.conf

First marv will prompt for the scanroot, the location of your bag files. By default MARV Robotics is expected to be served at the root of a domain (e.g. http://localhost:8000/), specify an application root to server e.g. at /marv-robotics instead.

Take a look at the generated marv.conf, it contains many comments and aims to compensate for the current lack of documentation.

If your bags are dear to you, you should keep them read-only for the user running marv, write-access to anything within the scanroot is not needed and discouraged.

If not specified otherwise, marv looks for the config file in the current directory, a bit like git. Let’s changed into the site:

(venv) $ cd site
(venv) ~/site $

User management

In order to comment and tag through the web, marv user accounts are needed. The password will be prompted for:

(venv) ~/site $ marv user add test
Password:
Repeat for confirmation:

The currently ensuing gibberish can be safely ignored.

Passwords are changed with:

(venv) ~/site $ marv user pw test
Password:
Repeat for confirmation:

Start webserver

MARV Robotics is implemented as a Python WSGI application. To serve it a WSGI server is needed, e.g. uWSGI. A configuration file was created previously by marv init within the marv site:

(venv) ~/site $ pip install uwsgi
...
(venv) ~/site $ uwsgi --ini uwsgi.conf

By now you should be able to visit your MARV Robotics installation with a web browser at http://localhost:8000/. So far no filesets have been scanned.

Scanning filesets and running nodes

Through scanning the scanroot marv detects new and changed filesets:

(venv) ~/site $ marv fileset scan
...

Reloading your browser page should give you a list with detected filesets. Instead of proper names, UUIDs are displayed and there is not much to see yet in the detail view. However, when signed-in you could start to tag and comment already.

In general all *.bag files found in the scanroot are added. If you want ignore certain subtrees, add a .marvignore file:

# touch /scanroot/ignored/.marvignore

In order to get more catchy names to be displayed for your bag sets, run the bagset_name node:

(venv) ~/site $ marv node run --all-filesets --node bagset_name

and reload your browser. A list of nodes is produced by:

(venv) ~/site $ marv node list

or simply run all configured nodes:

(venv) ~/site $ marv node run --all-filesets --all-nodes

Integration with other tools

MARV Robotics integrates well with other tools through its json api. To query all filesets having “gps” in their name use:

$ curl -G -H "Accept: application/vnd.api+json" \
    http://localhost:8000/marv/api/2/listing \
    -d 'filter[objects]=[{"name":"name","op":"like","val":"%gps%"}]'

or with python requests:

import requests
import json

url = 'http://127.0.0.1:8000/marv/api/2/listing'
headers = {'Accept': 'application/vnd.api+json'}

filters = [dict(name='name', op='like', val='%gps%')]
params = {'filter[objects]': json.dumps(filters)}

response = requests.get(url, params=params, headers=headers)
assert response.status_code == 200
print(response.json())

For more information see flask-restless docs, which is also the base for these examples.

Run tests and coverage

So far MARV ships with some tests and more will follow for MARV and MARV Robotics before the final 2.0.0. release. You can simply run the tests:

(venv) $ nosetests venv/lib/python2.7/site-packages/{marv,marv_robotics}

or produce a coverage report:

(venv) $ nosetests --with-coverage \
                   --cover-branches \
                   --cover-erase \
                   --cover-html \
                   --cover-package marv \
                   --cover-package marv-robotics \
                   --cover-tests \
                   venv/lib/python2.7/site-packages/{marv,marv_robotics}
(venv) $ chromium cover/index.html

You should not run these within your site directory

Licensing

MARV Robotics is built using the MARV framework. MARV Robotics is licensed under Apache License 2.0. MARV is available as Free and Open Source Software under AGPLv3 and as part of MARV Robotics Enterprise Edition with custom licenses.

Changelog

2.0.0b2 (2016-10-31)

  • documentation update for indigo

  • update requirements

  • move marv-robotics distribtion top-level

2.0.0b1 (2016-10-07)

  • initial release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

marv-robotics-2.0.0b2.tar.gz (24.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page