Skip to main content

File backend component of Cumulus

Project description

PyPI Python License

Stratocumulus is the backend component of Cumulus, which aims to providing a unified backend interface for different cloud environments.

Installation

Stratocumulus does not include any Cloud SDK. You need to install the corresponding SDK separately, depending on which Cloud platform you use:

  • Google Cloud: Install Google Cloud SDK.

  • Amazon AWS: Install AWS CLI.

  • Local machine: Works for Linux and macOS. Please make sure rsync is installed, as some commands will use it.

After that, install Stratocumulus in your Python environment by command:

pip install stratocumulus

Usage

Stratocumulus has 4 commands:

  • cp: Copy file(s)/folder(s).

  • sync: Synchronize two folders, usually used for copying one folder.

  • rm: Remove file(s)/folder(s).

  • exist: Check if a (remote) file/folder exists.

These commands have options specific to backend. For now, Stratocumulus supports the following backends:

  • aws: Amazon AWS.

  • gcp: Google Cloud.

  • local: Local machine. In specific, macOS or Linux.

cp

This command copies file(s) or folder(s). See examples below:

# AWS upload
strato cp file1 folder2 s3://my-bucket/target_folder/
# AWS download
strato cp s3://my-bucket/source_folder/*.zip /target_folder/

# GCP upload
strato cp -m -r --ionice file1 folder2 gs://my-bucket/target_folder/
# GCP download
mkdir /target_folder
strato cp -m gs://my-bucket/source_folder/*.zip /target_folder/

# Local Machine
strato cp -r file1 folder2 /target_folder/

Notice that:

  • For AWS backend, you must explicitly attach a trailing slash for source folder.

  • For Google Cloud download, you’ll have to explicitly create the target folder, and then run strato cp command.

  • Wildcards are acceptable.

  • -r option is not needed for AWS, and copying folders is always recursive.

  • -m and --ionice options only work for Google Cloud.

For details on the options, try command strato cp -h.

sync

This command synchronizes two folders. Notice that this synchronization will delete content in the target folder not existing in the source folder.

See examples below:

# AWS
strato sync source_folder s3://my-bucket/target_folder
# GCP
strato sync -m --ionice source_folder gs://my-bucket/target_folder
# Local Machine
strato sync source_folder target_folder

Notice that:

  • -m and --ionice options only work for Google Cloud.

For details on the options, try command strato sync -h.

rm

This command deletes file(s) or folder(s). See examples below:

# AWS
strato rm s3://my-bucket/file1 s3://my-bucket/folder2/
# GCP
strato rm -m gs://my-bucket/file1 gs://my-bucket/folder2 gs://my-bucket/folder3/*.zip
# Local Machine
strato rm file1 folder2

Notice that:

  • -m option only works for Google Cloud.

  • For AWS backend, wildcards are not yet accepted.

For details on the options, try command strato rm -h.

exists

This command checks if a (remote) file or folder exists. If it exists, the command terminates without any output. Otherwise, the command will break with Python subprocess.CalledProcessError exception.

See examples below:

# AWS
strato exists s3://my-bucket/file1
# GCP
strato exists gs://my-bucket/folder2/
# Local Machine
strato exists folder2/

Notice that this command works for both file and folder, regardless of backend.

For details on the options, try command strato exists -h.

help

Type strato -h or strato --help to check available Stratocumulus commands.

Check version

Type strato -v or strato --version to check the version of Stratocumulus currently installed on your machine.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

stratocumulus-0.2.3.tar.gz (13.5 kB view hashes)

Uploaded Source

Built Distribution

stratocumulus-0.2.3-py3-none-any.whl (16.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page