hashfs 0.7.2

Initialization

from hashfs import HashFS

Designate a root folder for HashFS. If the folder doesn’t already exist, it will be created.

# Set the `depth` to the number of subfolders the file's hash should be split when saving.
# Set the `width` to the desired width of each subfolder.
fs = HashFS('temp_hashfs', depth=4, width=1, algorithm='sha256')

# With depth=4 and width=1, files will be saved in the following pattern:
# temp_hashfs/a/b/c/d/efghijklmnopqrstuvwxyz

# With depth=3 and width=2, files will be saved in the following pattern:
# temp_hashfs/ab/cd/ef/ghijklmnopqrstuvwxyz

NOTE: The algorithm value should be a valid string argument to hashlib.new().

Storing Content

Add content to the folder using either readable objects (e.g. StringIO) or file paths (e.g. 'a/path/to/some/file').

from io import StringIO

some_content = StringIO('some content')

address = fs.put(some_content)

# Or if you'd like to save the file with an extension...
address = fs.put(some_content, '.txt')

# The id of the file (i.e. the hexdigest of its contents).
address.id

# The absolute path where the file was saved.
address.abspath

# The path relative to fs.root.
address.relpath

# Whether the file previously existed.
address.is_duplicate

Retrieving File Address

Get a file’s HashAddress by address ID or path. This address would be identical to the address returned by put().

assert fs.get(address.id) == address
assert fs.get(address.relpath) == address
assert fs.get(address.abspath) == address
assert fs.get('invalid') is None

Retrieving Content

Get a BufferedReader handler for an existing file by address ID or path.

fileio = fs.open(address.id)

# Or using the full path...
fileio = fs.open(address.abspath)

# Or using a path relative to fs.root
fileio = fs.open(address.relpath)

NOTE: When getting a file that was saved with an extension, it’s not necessary to supply the extension. Extensions are ignored when looking for a file based on the ID or path.

Removing Content

Delete a file by address ID or path.

fs.delete(address.id)
fs.delete(address.abspath)
fs.delete(address.relpath)

NOTE: When a file is deleted, any parent directories above the file will also be deleted if they are empty directories.

Repairing Files

The HashFS files may not always be in sync with it’s depth, width, or algorithm settings (e.g. if HashFS takes ownership of a directory that wasn’t previously stored using content hashes or if the HashFS settings change). These files can be easily reindexed using repair().

repaired = fs.repair()

# Or if you want to drop file extensions...
repaired = fs.repair(extensions=False)

WARNING: It’s recommended that a backup of the directory be made before repairing just in case something goes wrong.

Walking Corrupted Files

Instead of actually repairing the files, you can iterate over them for custom processing.

for corrupted_path, expected_address in fs.corrupted():
    # do something

WARNING: HashFS.corrupted() is a generator so be aware that modifying the file system while iterating could have unexpected results.

Walking All Files

Iterate over files.

for file in fs.files():
    # do something

# Or using the class' iter method...
for file in fs:
    # do something

Iterate over folders that contain files (i.e. ignore the nested subfolders that only contain folders).

for folder in fs.folders():
    # do something

Computing Size

Compute the size in bytes of all files in the root directory.

total_bytes = fs.size()

Count the total number of files.

total_files = fs.count()

# Or via len()...
total_files = len(fs)

For more details, please see the full documentation at http://hashfs.readthedocs.org.

v0.7.2 (2019-10-24)

• Fix out-of-memory issue when computing file ID hashes of large files.

v0.7.1 (2018-10-13)

• Replace usage of distutils.dir_util.mkpath with os.path.makedirs.

v0.7.0 (2016-04-19)

• Use shutil.move instead of shutil.copy to move temporary file created during put operation to HashFS directory.

v0.6.0 (2015-10-19)

• Add faster scandir package for iterating over files/folders when platform is Python < 3.5. Scandir implementation was added to os module starting with Python 3.5.

v0.5.0 (2015-07-02)

• Rename private method HashFS.copy to HashFS._copy.

• Add is_duplicate attribute to HashAddress.

• Make HashFS.put() return HashAddress with is_duplicate=True when file with same hash already exists on disk.

v0.4.0 (2015-06-03)

• Add HashFS.size() method that returns the size of all files in bytes.

• Add HashFS.count()/HashFS.__len__() methods that return the count of all files.

• Add HashFS.__iter__() method to support iteration. Proxies to HashFS.files().

• Add HashFS.__contains__() method to support in operator. Proxies to HashFS.exists().

• Don’t create the root directory (if it doesn’t exist) until at least one file has been added.

• Fix HashFS.repair() not using extensions argument properly.

v0.3.0 (2015-06-02)

• Rename HashFS.length parameter/property to width. (breaking change)

v0.2.0 (2015-05-29)

• Rename HashFS.get to HashFS.open. (breaking change)

• Add HashFS.get() method that returns a HashAddress or None given a file ID or path.

v0.1.0 (2015-05-28)

• Add HashFS.get() method that retrieves a reader object given a file ID or path.

• Add HashFS.delete() method that deletes a file ID or path.

• Add HashFS.folders() method that returns the folder paths that directly contain files (i.e. subpaths that only contain folders are ignored).

• Add HashFS.detokenize() method that returns the file ID contained in a file path.

• Add HashFS.repair() method that reindexes any files under root directory whose file path doesn’t not match its tokenized file ID.

• Rename Address classs to HashAddress. (breaking change)

• Rename HashAddress.digest to HashAddress.id. (breaking change)

• Rename HashAddress.path to HashAddress.abspath. (breaking change)

• Add HashAddress.relpath which represents path relative to HashFS.root.

v0.0.1 (2015-05-27)

• First release.

• Add HashFS class.

• Add HashFS.put() method that saves a file path or file-like object by content hash.

• Add HashFS.files() method that returns all files under root directory.

• Add HashFS.exists() which checks either a file hash or file path for existence.