💾 ir-datasets-clueweb22

Extension for accessing the ClueWeb22 via ir_datasets.

Installation

Install the package from PyPI:

pip install ir-datasets-clueweb22

Usage

Using this extension is simple. Just register the additional datasets by calling register(). Then you can load the datasets with ir_datasets as usual:

from ir_datasets import load
from ir_datasets_clueweb22 import register

# Register the ClueWeb22 datasets.
register()
# Use ir_datasets as usual.
dataset = load("clueweb22/b")

If you want to use the CLI, just use the ir_datasets_clueweb22 instead of ir_datasets. All CLI commands will work as usual, e.g., to list the available datasets:

ir_datasets_clueweb22 list

Development

To build this package and contribute to its development you need to install the build, setuptools, and wheel packages (pre-installed on most systems):

pip install build setuptools wheel

Create and activate a virtual environment:

python3.10 -m venv venv/
source venv/bin/activate

Dependencies

Install the package and test dependencies:

pip install -e .[tests]

Testing

Verify your changes against the test suite to verify.

ruff check .                   # Code format and LINT
mypy .                         # Static typing
bandit -c pyproject.toml -r .  # Security
pytest .                       # Unit tests

Please also add tests for your newly developed code.

Build wheels

Wheels for this package can be built with:

python -m build

Support

If you have any problems using this package, please file an issue. We're happy to help!

License

This repository is released under the MIT license.