↕️ ir_axioms

Intuitive axiomatic retrieval experimentation.

ir_axioms is a Python framework for experimenting with axioms in information retrieval in a declarative way. It includes reference implementations of many commonly used retrieval axioms and is well integrated with the PyTerrier framework and the Pyserini toolkit. Re-rank your search results today with ir_axioms and understand your retrieval systems better by analyzing axiomatic preferences!

Usage

Example Notebooks

We include several example notebooks to demonstrate re-ranking and preference evaluation in PyTerrier using ir_axioms. You can find all examples in the examples/ directory.

• Re-ranking top-20 results using KwikSort
• Re-ranking top-20 results using KwikSort learned from ORACLE preferences
• Re-ranking top-20 results using LambdaMART with axiomatic preference features
• Post-Hoc Analysis of Rankings and Relevance Judgments
• 
• Axiomatic prefernces for TREC Deep Learning 2019 runs (passages)
• Axiomatic prefernces for TREC Deep Learning 2019 runs (documents)
• Axiomatic prefernces for TREC Deep Learning 2020 runs (passages)
• Axiomatic prefernces for TREC Deep Learning 2020 runs (documents)

Backends

TODO

Slurm

If you want to play around with ir_axioms in Jupyter Lab, you can use this command to provision a server via Slurm:

scripts/slurm-start-jupyter-lab.sh

Citation

If you use this package or its components in your research, please cite the following paper describing the ir_axioms framework and its use-cases:

Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen. Axiomatic Retrieval Experimentation with ir_axioms. In 45th International ACM Conference on Research and Development in Information Retrieval (SIGIR 2022), July 2022. ACM.

You can use the following BibTeX entry for citation:

@InProceedings{bondarenko:2022d,
  author =                {Alexander Bondarenko and Maik Fr{\"o}be and {Jan Heinrich} Reimer and Benno Stein and Michael V{\"o}lske and Matthias Hagen},
  booktitle =             {45th International ACM Conference on Research and Development in Information Retrieval (SIGIR 2022)},
  month =                 jul,
  publisher =             {ACM},
  site =                  {Madrid, Spain},
  title =                 {{Axiomatic Retrieval Experimentation with ir_axioms}},
  year =                  2022
}

Development

To build ir_axioms and contribute to its development you need to install the build, and setuptools and wheel packages:

pip install build setuptools wheel

(On most systems, these packages are already pre-installed.)

Installation

Install dependencies for developing the ir_axioms package:

pip install -e .

If you want to develop the Pyserini backend, install dependencies like this:

pip install -e .[pyserini]

If you want to develop the PyTerrier backend, install dependencies like this:

pip install -e .[pyterrier]

Testing

Install test dependencies:

pip install -e .[test]

Verify your changes against our test suite to verify.

flake8 ir_axioms tests
pylint -E ir_axioms tests.unit --ignore-paths=^ir_axioms.backend
pytest ir_axioms/ tests/unit/ --ignore=ir_axioms/backend/

Please also add tests for the axioms or integrations you've added.

Testing backend integrations

Install test dependencies (replace <BACKEND> with either pyserini or pyterrier):

pip install -e .[<BACKEND>]

Verify your changes against our test suite to verify.

pylint -E ir_axioms.backend.<BACKEND> tests.integration.<BACKEND>
pytest tests/integration/<BACKEND>/

Build wheel

A wheel for this package can be built by running:

python -m build

License

This repository is released under the MIT license. If you use ir_axioms in your research, we'd be glad if you'd cite us.