No project description provided
Project description
Circadian SCP Sync
Resumable, interruptible, SCP upload client for any files or directories generated day by day.
Use Case
You have a local directory that generates data every day on your local machine. The directory looks like this:
๐ data-directory-2
โโโ ๐ 20190101
โย ย โโโ ๐ file1.txt
โย ย โโโ ๐ file2.txt
โย ย โโโ ๐ file3.txt
โโโ ๐ 20190102
ย ย โโโ ๐ file1.txt
ย ย โโโ ๐ file2.txt
ย ย โโโ ๐ file3.txt
... or like this:
๐ data-directory-1
โโโ ๐ 20190101.txt
โโโ ๐ 20190102-a.txt
โโโ ๐ 20190102-b.txt
โโโ ๐ 20190103.txt
This tool can sync the local data to a remote server via SCP for these two directory structures.
- It will only upload directories or files one hour after midnight of the day they were generated (files directories labeled with
20190101
will only be considered after2019-01-02 01:00:00
) - It is resumable, meaning if the upload is interrupted, it will continue where it left off
- It is interruptible without running it in a separate thread (
should_abort_upload
callback) - After a file or a directory has been uploaded, this tool will compute the checksum of the respective file(s) and delete the local file(s) only if the checksums match and if specified by the user
- You can specify how dated directories or files are formatted with regexes and
%Y
/%m
/... expressions (see below)
Usage
import circadian_scp_upload
# Use the callbacks to customize the upload process
# and integrate it into your own application. All callbacks
# are optional and the callback object does not need to be
# passed to the upload client. The lambda functions below
# are the default values.
upload_client_callbacks = circadian_scp_upload.UploadClientCallbacks(
# which directories to consider in the upload process; only supports
# %Y/%y/%m/%d - does not support parentheses in the string
dated_directory_regex=r"^" + "%Y%m%d" + r"$",
# which files to consider in the upload process; only supports
# %Y/%y/%m/%d - does not support parentheses in the string
dated_file_regex=r"^[\.].*" + "%Y%m%d" + r".*$",
# use your own logger instead of print statements
log_info=lambda message: print(f"INFO - {message}"),
log_error=lambda message: print(f"ERROR - {message}"),
# callback that is called periodically during the upload
# process to check if the upload should be aborted
should_abort_upload=lambda: False,
)
# teardown happens automatically when leaving the "with"-block
with circadian_scp_upload.RemoteConnection(
"1.2.3.4", "someusername", "somepassword"
) as remote_connection:
# upload a directory full of directories "YYYYMMDD/"
circadian_scp_upload.DailyTransferClient(
remote_connection=remote_connection,
src_path="/path/to/local/data-directory-1",
dst_path="/path/to/remote/data-directory-1",
remove_files_after_upload=True,
variant="directories",
callbacks=upload_client_callbacks,
).run()
# upload a directory full of files "YYYYMMDD.txt"
circadian_scp_upload.DailyTransferClient(
remote_connection=remote_connection,
src_path="/path/to/local/data-directory-2",
dst_path="/path/to/remote/data-directory-2",
remove_files_after_upload=True,
variant="files",
callbacks=upload_client_callbacks,
).run()
You will get an informational output whereever you direct the log output to - the progress is only logged at steps of 10%:
INFO - 2015-01-16: starting
INFO - 2015-01-16: 5 files missing in dst
INFO - 2015-01-16: created remote directory at /tmp/circadian_scp_upload_test_1692833117/20150116
INFO - 2015-01-16: 0.00 % (1/5) uploaded
INFO - 2015-01-16: 20.00 % (2/5) uploaded
INFO - 2015-01-16: 40.00 % (3/5) uploaded
INFO - 2015-01-16: 60.00 % (4/5) uploaded
INFO - 2015-01-16: 80.00 % (5/5) uploaded
INFO - 2015-01-16: 100.00 % (5/5) uploaded (finished)
INFO - 2015-01-16: checksums match
INFO - 2015-01-16: finished removing source
INFO - 2015-01-16: successful
INFO - 2010-03-25: starting
INFO - 2010-03-25: 5 files missing in dst
INFO - 2010-03-25: created remote directory at /tmp/circadian_scp_upload_test_1692833117/20100325
INFO - 2010-03-25: 0.00 % (1/5) uploaded
INFO - 2010-03-25: 20.00 % (2/5) uploaded
INFO - 2010-03-25: 40.00 % (3/5) uploaded
INFO - 2010-03-25: 60.00 % (4/5) uploaded
INFO - 2010-03-25: 80.00 % (5/5) uploaded
INFO - 2010-03-25: 100.00 % (5/5) uploaded (finished)
INFO - 2010-03-25: checksums match
INFO - 2010-03-25: finished removing source
INFO - 2010-03-25: successful
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for circadian_scp_upload-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61143ec90cdfba0c7636e957b9e1cb4a485ecd47f276a2dca395554d8f096e97 |
|
MD5 | 18c67d5dbe26fe534f3e68e530550d5d |
|
BLAKE2b-256 | 28f4739661f7047d27576ec123acbcf37f6f31deb02a173ceec2f49edfe6283b |
Close
Hashes for circadian_scp_upload-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9e0e41f3acfcb1cd15b1479c4a619c807ae46609cab95bbba780efeb44b15d8 |
|
MD5 | 36a441db8ab254abc5ea5b73a17cb720 |
|
BLAKE2b-256 | ee50d4e71dfa8f460309758915f43b1723ee042389c076eb49a66717f69f9cae |