Dataset API

Dataset API structure

dataset_api
├── conda
│   └── recipes
│       ├── py38_recipe
│       └── py39_recipe
├── src
│   └── dataset_librarian
│       ├── dataset_api
│       ├── scripts
│       ├── __init__.py
│       ├── dataset.py
│       ├── datasets_urls.json
├── MANIFEST.in
├── README.md
├── pyproject.toml
└── requirements.txt

Environment setup

Clone the Model Zoo for Intel® Architecture repository and navigate to the dataset_api directory.

# Step 1 (recommended): Create and activate a virtual environment
## Option 1: Using virtualenv
virtualenv -p python3 venv
. venv/bin/activate
## Option 2: Using conda
conda create -n venv python=<3.8 or 3.9> -c conda-forge
conda activate venv

# Step 2: Installing package
## Option 1: Installing from source code
cd models/datasets/dataset_api
python -m pip install --upgrade pip build setuptools wheel
python -m pip install .
## Option 2: Installing from PyPI
python -m pip install dataset-librarian

PyPI package can be found here.

Datasets

Command-line Interface

Python API

from dataset_librarian.dataset_api.download import download_dataset
from dataset_librarian.dataset_api.preprocess import preprocess_dataset

# Download the datasets
download_dataset('brca', <path to the raw dataset directory>)

# Preprocess the datasets
preprocess_dataset('brca', <path to the raw dataset directory>)