Process data from live API of luftdaten.info
Project description
Luftdatenpumpe
About
Luftdatenpumpe acquires the current window of measurement readings from the livedata API of luftdaten.info.
While iterating the readings, it collects information about all stations and sensors they are originating from.
Then, each stations location information gets enhanced by
attaching its geospatial position as a Geohash.
attaching a synthetic real-world address resolved using the reverse geocoding service Nominatim by OpenStreetMap.
Information about stations can be
displayed on STDOUT or STDERR in JSON format.
filtered and transformed interactively through jq, the swiss army knife of JSON manipulation.
stored into RDBMS databases using the fine dataset package. Being built on top of SQLAlchemy, this supports all major databases.
Measurement readings can be
Demo
Live Data
List of stations
Synopsis
Overview
# List stations luftdatenpumpe stations --station=28,297 --reverse-geocode # Write list of stations and metadata to PostgreSQL database luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql:///weatherbase # Forward readings to MQTT luftdatenpumpe readings --station=28,1071 --target=mqtt://mqtt.example.org/luftdaten.info
Details
$ luftdatenpumpe --help
Usage:
luftdatenpumpe stations [options] [--target=<target>]...
luftdatenpumpe readings [options] [--target=<target>]...
luftdatenpumpe --version
luftdatenpumpe (-h | --help)
Options:
--station=<stations> Filter data by given location ids, comma-separated.
--sensor=<sensors> Filter data by given sensor ids, comma-separated.
--reverse-geocode Compute geographical address using the Nominatim reverse geocoder and add to MQTT message
--target=<target> Data output target
--progress Show progress bar
--version Show version information
--dry-run Run data acquisition and postprocessing but skip publishing to MQTT bus
--debug Enable debug messages
-h --help Show this screen
Station list examples:
# Display metadata for given stations in JSON format
luftdatenpumpe stations --station=28,1071 --reverse-geocode
# Display metadata for given sensors in JSON format
luftdatenpumpe stations --sensor=657,2130 --reverse-geocode
# Display list of stations in JSON format, suitable for integrating with Grafana
luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=json.grafana+stream://sys.stdout
# Write list of stations and metadata to PostgreSQL database, also display on STDERR
luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql:///weatherbase --target=json+stream://sys.stderr
Data examples:
# Publish data to topic "luftdaten.info" at MQTT broker "mqtt.example.org"
luftdatenpumpe readings --station=28,1071 --target=json+stream://sys.stderr --target=mqtt://mqtt.example.org/luftdaten.info
# MQTT publishing, with authentication
luftdatenpumpe readings --station=28,1071 --target=mqtt://username:password@mqtt.example.org/luftdaten.info
Setup
Prerequisites
Debian packages:
apt install postgis redis-server redis-tools
Postgres database
Create database:
createuser --no-createdb --pwprompt hiveeyes createdb --owner hiveeyes weatherbase
Create read-only user:
su - postgres psql postgres=# \c weatherbase weatherbase=# CREATE ROLE readonly WITH LOGIN PASSWORD 'XXX'; weatherbase=# GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO readonly; weatherbase=# GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
Redis cache
This program extensively uses a runtime cache based on Redis. To make this work best, you should enable data durability with your Redis instance.
The append-only file is an alternative, fully-durable strategy for Redis. It became available in version 1.1. You can turn on the AOF in your Redis configuration file (e.g. /etc/redis/redis.conf):
appendonly yes
Python module
# Not published yet, please install from repository pip install luftdatenpumpe
References
Upstream luftdaten.info
Standing on the shoulders of giants
Development
License
This program is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program; if not, see: <http://www.gnu.org/licenses/agpl-3.0.txt>, or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Content attributions
The copyright of particular images and pictograms are held by their respective owners, unless otherwise noted.
Icons and pictograms
Water Pump Free Icon from Icon Fonts is licensed by CC BY 3.0.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
