Skip to main content

UNKNOWN

Project description

vsc-zkrsync uses zookeeper to distribute rsync across multiple nodes when rsync ing (large) (shared) filesystem.

Large fraction of rsync calls is spend in agthering the required metadata, and then performing the actual data synchronisation. Especially when dealing with “incremental” rsync. Spreading the load over multiple processes can mean a significant speed increase; although other bottlenecks, eg access to the metadata, might show up.

The implementation uses zookeeper to coordinate the distribution of collecting and sync the data across many processes and/or nodes.

= Zookeeper = == Install zookeeper-server == A zookeeper server with proper ACLs to a base znode is required.

If none is available, installation can be performed as follows:

zkversion=3.4.5 basepath=/tmp/$USER # is not permanent chmod 700 $basepath

mkdir -p $basepath/zk/server/data

cd $basepath/zk/server # use another mirror if needed wget http://apache.belnet.be/zookeeper/zookeeper-$zkversion/zookeeper-$zkversion.tar.gz tar xzf zookeeper-$zkversion.tar.gz

rm -f zkServer.sh cat > zkServer.sh <<EOF #!/bin/bash ./zookeeper-$zkversion/bin/zkServer.sh $@ EOF chmod +x zkServer.sh

cat > zk.conf <<EOF tickTime=2000 dataDir=$PWD/data clientPort=2181 EOF

== start zookeeper-server == ./zkServer.sh start $PWD/zk.conf

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vsc-zk-0.6.2.tar.gz (13.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page