swift-commander 1.4.6

swc upload

use swc upload /local_dir/subdir /my_swift_container/subfolder to copy data from a local or networked posix file system to a swift object store. swc upload wraps swift upload of the standard python swift client:

joe@box:~/sc$ swc upload ./testing /test
*** uploading ./test ***
*** to Swift_Account:/test/ ***
executing:swift upload --changed --segment-size=1073741824 --use-slo --segment-container=".segments_test" --header="X-Object-Meta-Uploaded-by:joe" --object-name="" "test" "./test"
*** please wait... ***
/fld11/file12
/fld11/file11
/fld11/fld2/fld3/fld4/file43
/fld11/fld2/fld3/fld4/file42
.

the swc wrapper adds the following features to upload:

• –segment-size ensures that uploads for files > 5GB do not fail. 1073741824 = 1GB

• Uploaded-by metadata keeps track of the operating system user (often Active Directory user) that upload the data

• setting –segment-container ensures that containers that carry the segments for multisegment files are hidden if users access these containers with 3rd. party GUI tools (ExpanDrive, Cyberduck, FileZilla) to avoid end user confusion

• –slo stands for Static Large Object and SLO’s the recommended object type for large objects / files.

as an addional feature you can add multiple metadata tags to each uploaded object, which is great for retrieving archived files later:

joe@box:~/sc$ swc upload ./test /test/example/meta project:grant-xyz collaborators:jill,joe,jim cancer:breast
*** uploading ./test ***
*** to Swift_Account:/test/example/meta ***
executing:swift upload --changed --segment-size=1073741824 --use-slo --segment-container=".segments_test" --header="X-Object-Meta-Uploaded-by:petersen" --header=X-Object-Meta-project:grant-xyz --header=X-Object-Meta-collaborators:jill,joe,jim --header=X-Object-Meta-cancer:breast --object-name="example/meta" "test" "./test"
*** please wait... ***
example/meta/fld11/fld2/file21
example/meta/fld11/file11
.
.
/test/example/meta

These metadata tags stay in the swift object store with the data. They are stored just like other important metadata such as change data and name of the object.

joe@box:~/sc$ swc meta example/meta/fld11/file13
       Meta Cancer: breast
Meta Collaborators: jill,joe,jim
  Meta Uploaded-By: petersen
      Meta Project: grant-xyz
        Meta Mtime: 1420047068.977197

if you store metadata tags you can later use an external search engine such as ElasticSearch to quickly search for metadata you populated while uploading data

alias: you can use swc up instead of swc upload

swc download

use swc download /my_swift_container/subfolder /local/subfolder to copy data from a swift object store to local or network storage. swc downloadwrapsswift download` of the standard python swift client:

joe@box:~/sc$ swc download /test/example/ $TMPDIR/
example/meta/fld11/fld2/file21
example/meta/fld11/file11

alias: you can use swc down instead of swc download

swc arch

swc arch is a variation of swc upload. Instead of uploading the files as is, it creates a tar.gz archive for each directory and uploads the tar.gz archives. swc arch is different from default tar behavior because it does not create a large tar.gz file of an entire directory structure as large tar.gz files are hard to manage (as one cannot easily navigate the directory structure within or get quick access to a spcific file). Instead swc arch creates tar.gz files that do not include sub directories and it creates a separate tar.gz file for each directory and directory level. The benefit of this approach is that the entire directory structure remains intact and you can easily navigate it by using swc cd and swc ls

swc cd, swc, ls, swc mkdir

these commands are simplified versions of the equivalent standard GNU tools and should work very similar to these tools.

swc mtime

use swc mtime /my_swift_container/subfolder/file to see the modification time data from a swift object store to local or network storage. swc download wraps swift download of the standard python swift client: