HOBO-request

A Kubernetes-ready container application for integrating HOBOlink loggers with HSDS. This application was developed in partnership with the HDF Group with NSF support for the UVA-ARC project (grant #2022639).

Getting started

HOBO-request provides an interface to HOBOlink loggers via a public API that is provided by Onset. It can be used with HSDS for the purposes of large-scale environmental data analysis, or as a stand-alone tool for the purposes of testing. It can also be easily extended to be used with different APIs and sensor networks.

You can install the HOBO-request package or git clone this repository. For installing the pip package, just create a virtual environmental and run:

$ python3 -m venv hobo-request
$ source hobo-request/bin/activate
$ pip install hoborequest

You can also take the development path way and work directly with the source code.

In order to integrate HOBO-request with data loggers, a few libraries / modules need to be installed. You can proceed by installing the dependencies described in requirements.txt.

$ python3 -m venv hobo-request 
$ source hobo-request/bin/activate
$ pip install -r requirements.txt

HOBO-request is now ready to be used, but first it must be configured!

Configuration

First, copy conf/hobo-connect.conf-SAMPLE to conf/hobo-connect.conf. Then, edit 'conf/hobo-connect.conf' and set the following parameters:

• user_id: your Onset user ID
• client_id: your Onset client ID
• client_secret: API secret key obtained from Onset
• start_date_time: for precise time slicing, the start time in UTC, format: 2012-01-01 23:00:00
• end_date_time: same as above, the end time in UTC, format: 2021-01-12 11:00:00
• polling_interval_minutes: if you want to set a regular polling interval, leave start & end time fields empty. Alternatively you can also set a start time, but leave end time empty

Please observe that timestamps are always assumed to be in UTC. For the purposes of data storage, it is more convenient to avoid timezones and daylight-saving differences, and leave to the analyst to manipulate timestamps as he/she/they see fit.

In addition to the configuration of your Onset credentials, you also need to set your HSDS parameters:

• hsds_filename: your HDF data store name
• hsds_endpoint: your HSDS endpoint

Please observe that you can also use HOBO-request locally without HSDS for testing purposes. In that case, you only need to set hsds_filename to point to the h5 file in your local filesystem.

Finally, you must set the repository where your logger and sensor metadata are registered. This is a crucial step because HOBO-request depends on this config to update your metadata attributes for you. You must set the following fields:

• meta_repo: git repository that contains your deployment metadata files
• meta_local_dir: local directory where you want to store the repo temporarily
• meta_root_path: repository directory where the HDF root file metadata file is
• meta_loggers_dir: repository directory where the loggers metadata files are
• meta_sensors_dir: repository directory where the sensors metadata files are

Once this is done, we can move on to the generation of your data store.

Creating your HDF data store with metadata

HOBO-request reads YAML files with metadata organized in three levels:

• HDF root metadata (attributes)
• HOBO loggers (attributes)
• HOBO sensors (attributes)

It is necessary, therefore, to create YAML files following the scheme we describe in the conf/ and conf/sensors/templates directories. For example:

• uva-arc.yaml: provides an example of how to set root attributes in your HDF file
• loggers/: provides examples for the configuration of HOBOLink loggers / stations
• sensors/templates: provides examples for the configuration of HOBOLink sensors

Once you are done creating your YAML files, you can load them in your newly created HDF file. Change directory to src/, then run hobo-config.py:

To generate your HDF file with root attributes:

./hobo-config.py test.h5 ../conf/uva-arc.yaml

Or, if you are using the pip package, you can just run:

$ hoboconfig test.h5 ../conf/uva-arc.yaml

Please note that you have to change the PATH for the YAML file in the command above. You can find real examples for stations and sensors in this repository.

To load your logger and sensor metadata, you can type:

./hobo-config.py test.h5 /tmp/deployment-metadata/alaska/loggers/*
./hobo-config.py test.h5 /tmp/deployment-metadata/alaska/sensors/*

Or... if you are using the pip package hoborequest:

$ hoboconfig test.h5 /tmp/deployment-metadata/alaska/loggers/*
$ hoboconfig test.h5 /tmp/deployment-metadata/alaska/sensors/*

Please note that it is necessary to run this step only once when creating the HDF data store. After you load the HDF file onto HSDS, HOBO-request will automatically pull all the changes from the git repository containing your YAML files and load all the metadata information for you. There is no need to use hobo-config manually again.

Running HOBO-request to consume sensor data

Now that everything is in place, you have the choice of running HOBO-request locally for tests using a regular HDF file (and h5py) or run the HDF data store on HSDS (with h5pyd). This is done automagically for you, so you just have to point in your config file if you want to use a regular file or interface with HSDS to feed data to your HDF data store.

You can start HOBO-request with a simple command:

$ ./hobo-connect.py

Or, if you are running the pip package:

$ hoboconnect

Et... voilà! The application will return the following message as soon as it is done processing the data:

[HOBO-connect]: data processing completed

Generating a container for HOBO-request

We have provided scripts and deployment files to generate a container for HOBO-request, plus documentation on how to run HSDS on k3s (or microk8s) for local testing.

You can run the following commands to build your container and run it:

$ ./build.sh
$ ./docker_run.sh

The directory k8s/ contains the deployment files and scripts for running HOBO-request on k8s or microk8s.

May the source be with you in your environmental studies!

Licensing

See LICENSE and AUTHORS files for details.