dtool lookup server plugin for receiving elastic-search update notifications
Project description
GitHub: https://github.com/IMTEK-Simulation/dtool-lookup-server-notification-plugin
PyPI: https://pypi.python.org/pypi/dtool-lookup-server-notification-plugin
Free software: MIT License
Features
Listen to elastic search notifications from an S3-compatible storage backend
Introduction
dtool is a command line tool for packaging data and metadata into a dataset. A dtool dataset manages data and metadata without the need for a central database.
However, if one has to manage more than a hundred datasets it can be helpful to have the datasets’ metadata stored in a central server to enable one to quickly find datasets of interest.
The dtool-lookup-server provides a web API for registering datasets’ metadata and provides functionality to lookup, list and search for datasets.
This plugin enables the dtool-lookup-server to listen to elastic search notifications for the registration and deregistration of datasets.
Installation
Install the dtool lookup server dependency graph plugin
$ pip install dtool-lookup-server-notification-pluginSetup and configuration
Configure plugin behavior
The plugin needs to know how to convert a bucket name into a base URI. The environment variable DTOOL_LOOKUP_SERVER_NOTIFY_BUCKET_TO_BASE_URI is used to specify that conversion, e.g.:
DTOOL_LOOKUP_SERVER_NOTIFY_BUCKET_TO_BASE_URI={"bucket": "ecs://bucket"}
It is also advisable to limit access to the notification listener to a certain IP range. Use:
DTOOL_LOOKUP_SERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.0.0/16
to specify the allowed remote network. To specify a single IP, use:
DTOOL_LOOKUP_SERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.1.1/32
Configure elastic search integration in NetApp StorageGRID
Create a new endpoint with URI ` https://myserver:myport/elastic-search ` and URN ` arn:<mysite>:es:::<domain-name>/notify/all ` Note that <mysite> and <domain-name> can be chose arbitrarily. notify/all is appended to the URI and must point to the route of the notify function.
The bucket needs to be configured to support search integration. Use the following XML template
<MetadataNotificationConfiguration>
<Rule>
<ID>dtool</ID>
<Status>Enabled</Status>
<Prefix></Prefix>
<Destination>
<Urn>urn:mysite:es:::domain-name/notify/all</Urn>
</Destination>
</Rule>
</MetadataNotificationConfiguration>Querying server plugin configuration
The request
$ curl -H "$HEADER" http://localhost:5000/elastic-search/configwill return the current notification plugin configuration with all keys in lowercase
{
"allow_access_from": "0.0.0.0/0",
"bucket_to_base_uri": {"bucket": "s3://bucket"},
"version": "0.1.0"
}See dtool_lookup_server_dependency_graph_plugin.config.Config for more information.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dtool-lookup-server-notification-plugin-0.1.1.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a081f704be652372e560cb5ca099ea8dc1c8c78ac144156b124d826be186e73d |
|
| MD5 | 035c38aab57cb7011960d89aae0875b2 |
|
| BLAKE2b-256 | 8df2a524230914b3af40cdd46e259b024fa72067617ced6979504b64cc508698 |
Hashes for dtool_lookup_server_notification_plugin-0.1.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 60650159a752a2ba62ddda53b024a7ff6b6b367a158f3f1697b353836dcc85d6 |
|
| MD5 | 0ea5a8bdbd2d2c6fa3f67139a6f6bf8f |
|
| BLAKE2b-256 | 9e1b10ba23dbfe4ab76506c3754298fbabf61c793cf13f635c564256a3b85601 |
