vdtk: Visual Description Data Evaluation Tools

This tool is designed to allow for a deep investigation of diversity in visual description datasets, and to help users understand their data at a token, n-gram, description, and dataset level.

Installation

To use this tool, you can easily pip install with pip install . from this directory. Note: Some metrics (METEOR) require a working installation of Java. Please follow the directions (here) to install the Java runtime if you do not already have access to a JRE.

Data format

In order to prepare datasets to work with this tool, datasets must be formatted as JSON files with the following schema

// List of samples in the dataset
[
    // JSON object for each sample
    {
        "_id": "string", // A string ID for each sample. This can help keep track of samples during use.
        "split": "string", // A string corresponding to the split of the data. Default splits are "train", "validate" and "test"
        "references": [
            // List of string references
            "reference 1...",
            "reference 2...",
        ],
        "metadata": {} // Any JSON object. This field is not used by the toolkit at this time.
    }
]

Usage

After installation, the basic menu of commands can be accessed with vdtk-cli --help. We make several experiments/tools available for use:

For more details and options, see the --help command for any of the commands above. Note that some tools are relatively compute intensive. This toolkit will make use of a GPU if available and necessary, as well as a large number of CPU cores and RAM depending on the task.

[BETA] See the API Docs for usage as a library.