skip to navigation
skip to content

vsc-zk 0.6.2

Latest Version: 0.9.7

vsc-zkrsync uses zookeeper to distribute rsync across multiple nodes when rsync ing (large) (shared) filesystem.

Large fraction of rsync calls is spend in agthering the required metadata, and then performing the actual data synchronisation. Especially when dealing with “incremental” rsync. Spreading the load over multiple processes can mean a significant speed increase; although other bottlenecks, eg access to the metadata, might show up.

The implementation uses zookeeper to coordinate the distribution of collecting and sync the data across many processes and/or nodes.

= Zookeeper = == Install zookeeper-server == A zookeeper server with proper ACLs to a base znode is required.

If none is available, installation can be performed as follows:

zkversion=3.4.5 basepath=/tmp/$USER # is not permanent chmod 700 $basepath

mkdir -p $basepath/zk/server/data

cd $basepath/zk/server # use another mirror if needed wget$zkversion/zookeeper-$zkversion.tar.gz tar xzf zookeeper-$zkversion.tar.gz

rm -f cat > <<EOF #!/bin/bash ./zookeeper-$zkversion/bin/ $@ EOF chmod +x

cat > zk.conf <<EOF tickTime=2000 dataDir=$PWD/data clientPort=2181 EOF

== start zookeeper-server == ./ start $PWD/zk.conf

File Type Py Version Uploaded on Size
vsc-zk-0.6.2.tar.gz (md5) Source 2014-04-14 12KB