Rclone wrapper to manage multiple datasets in a project
Project description
rkale
Install
Install rkale in your project using poetry:
poetry add rkale
Use pip if you want a global installation:
pip install rkale
Configuration
Global
~/.config/rkale/rkale.conf:
[data]
root = "path to data folder where datasets are stored"
[aliases]
wasabi = "optional alias for remote in rclone.conf"
[rclone] # global flags for rclone
flags = ["--transfers 32", "--checkers 32"]
If aliases are empty the remote name from the project config is used in the rclone lookup.
Project
Configure project datasets in the pyproject.toml file:
<project path>/pyproject.toml:
[[tool.rkale.dataset]]
name = "dataset_1"
remote = "remote_1"
[[tool.rkale.dataset]]
name = "dataset_2"
remote = "remote_2"
The remote specified for the dataset must match a remote in the rclone.conf
or an alias in the global rkale configuration.
Usage
Python interface
from rkale.config import dataset_paths
def dataset_path():
return dataset_paths()["dataset_1"]
Syncing datasets
Syncs the local datasets to be identical to the remote
rkale psync
Syncs the remote datasets to be identical to the local
rkale psync --upstream
Same as rclone sync but checks differences first and asks for confirmation
rkale sync <source> <destination>
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
