svfsc 0.4.0

Introduction

Sometimes you don’t need the whole file. Sometimes you don’t want the whole file. Especially if it is huge and on some remote server. But, you might know what parts of the file that you want and svfsc can help you store them locally so it looks as if you have access to the complete file but with just the pieces of interest.

svfsc is targeted at reading very large binary files such as TIFF, RP66V1, HDF5 where the structure is well known. For example you might want to parse a TIFF file for its metadata or for a particular image tile or strip which is a tiny fraction of the file itself.

svfsc implements a Sparse Virtual File, a specialised in-memory cache where a particular file might not be available but parts of it can be obtained without reading the whole file. A Sparse Virtual File (SVF) is represented internally as a map of blocks of data with the key being their file offsets. Any write to an SVF will coalesce these blocks where possible. There is no cache punting strategy implemented so an SVF always accumulates data. A Sparse Virtual File System (SVFS) is an extension of this to provide a key/value store where the key is a file ID and the value a Sparse Virtual File.

svfsc is written in C++ with a Python interface. It is thread safe in both domains.

A SVF might be used like this:

• The user requests some data (for example TIFF metadata) from a remote file using a Parser that knows the TIFF structure.

• The Parser consults the SVF, if the SVF has the data then the Parser parses it and gives the results to the user.

• If the SVF does not have the data then the Parser consults the SVF for what data is needed, then issues the appropriate GET request(s) to the remote server.

• That data is used to update the SVF, then the parser can use it and give the results to the user.

Here is a conceptual example of a SVF running on a local file system containing data from a single file.

            CLIENT SIDE           |             LOCAL FILE SYSTEM
                                  .
/------\      /--------\          |              /-------------\
| User | <--> | Parser | <-- read(fpos, len) --> | File System |
\------/      \--------/          |              \-------------/
                   |              .
                   |              |
               /-------\          .
               |  SVF  |          |
               \-------/          .

Here is a conceptual example of an SVFS running with a remote file system.

            CLIENT SIDE           |             SERVER SIDE
                                  .
/------\      /--------\          |             /--------\
| User | <--> | Parser | <-- GET(fpos, len) --> | Server |
\------/      \--------/          |             \--------/
                   |              .                  |
                   |              |                  |
               /-------\          .           /-------------\
               |  SVF  |          |           | File System |
               \-------/          .           \-------------/

Example Python Usage

$ pip install svfsc

import svfsc

# Construct a Sparse Virtual File
svf = svfsc.cSVF('Some file ID')

# Write six bytes at file position 14
svf.write(14, b'ABCDEF')

# Read from it
svf.read(16, 2) # Returns b'CD'

# What do I have to do to read 24 bytes from file position 8?
# This returns a tuple of pairs ((file_position, read_length), ...)
svf.need(8, 24) # Returns ((8, 6), (20, 4))
# Go and get the data from those file positions and write it to
# the SVF then you can read directly from the SVF.

if not svf.has_data(file_position, length):
    for read_position, read_length in svf.need(file_position, length):
        # Somehow get the data as a bytes object at (read_position, read_length)...
        # This could be a GET request to a remote file.
        # Then...
        svf.write(read_position, data)
# Now read directly
svf.read(file_position, length)

import svfsc

svfs = svfsc.cSVFS()

# Insert an empty SVF with a corresponding ID
ID = 'abc'
svfs.insert(ID)

# Write six bytes to that SVF at file position 14
svfs.write(ID, 14, b'ABCDEF')

# Read from the SVF
svfs.read(ID, 16, 2) # Returns b'CD'

# What do I have to do to read 24 bytes from file position 8
# from that SVF?
svfs.need(ID, 8, 24) # Returns ((8, 6), (20, 4))

Example C++ Usage

svfsc is written in C++ so can be used directly:

#include "svf.h"

// File modification time of 1672574430.0 (2023-01-01 12:00:30)
SVFS::SparseVirtualFile svf("Some file ID", 1672574430.0);

// Write six char at file position 14
svf.write(14, "ABCDEF", 6);

// Read from it
char read_buffer[2];
svf.read(16, 2, read_buffer);
// read_buffer now contains "CD"

// What do I have to do to read 24 bytes from file position 8?
// This returns a std::vector<std::pair<size_t, size_t>>
// as ((file_position, read_length), ...)
auto need = svf.need(8, 24);

// The following prints ((8, 6), (20, 4),)
std::cout << "(";
for (auto &val: need) {
    std::cout << "(" << val.first << ", " << val.second << "),";
}
std::cout << ")" << std::endl;

Documentation

Build the documentation from the docs directory or find it on readthedocs: https://svfsc.readthedocs.io/

Acknowledgments

Many thanks to my employer Paige.ai for allowing me to release this as FOSS software.

0.4.0 (2024-02-11)

• Add counters for blocks/bytes erased and blocks/bytes punted and then their associated APIs.

• Use the SVFS_SVF_METHOD_SIZE_T_REGISTER macro in CPython to simplify registering CPython methods.

• Fix builds on Linux, mainly compiler flags.

  • Move to -std=c++17 to exploit [[nodiscard]].

  • Better alignment of compiler flags between CMakeLists.txt and setup.py

• Other minor fixes.

• Because of the extensive use of this in various projects this version 0.4 is moved to production status: Development Status :: 5 - Production/Stable

0.3.0 (2024-01-06)

• Add need_many().

• Fix bug in lru_punt().

• Development Status :: 4 - Beta

0.2.2 (2023-12-28)

• Minor fixes.

• Development Status :: 4 - Beta

0.2.1 (2023-12-27)

• Include stub file.

• Development Status :: 4 - Beta

0.2.0 (2023-12-24)

• Add cache punting.

• Make C docstrings type parsable (good for Sphinx) and add a script that can create a mypy stub file.

• Development Status :: 4 - Beta

0.1.2 (2023-10-03)

• First release.

• Development Status :: 3 - Alpha