Skip to main content

Facilitates automated and reproducible experimental research

Project description

Every experiment is sacred
Every experiment is great
If an experiment is wasted
God gets quite irate

Current PyPi Version Supported Python Versions MIT licensed ReadTheDocs DOI for this release

Github Actions PyTest Coverage Report Code Scrutinizer Quality Code style: black

Sacred is a tool to help you configure, organize, log and reproduce experiments. It is designed to do all the tedious overhead work that you need to do around your actual experiment in order to:

  • keep track of all the parameters of your experiment

  • easily run your experiment for different settings

  • save configurations for individual runs in a database

  • reproduce your results

Sacred achieves this through the following main mechanisms:

  • Config Scopes A very convenient way of the local variables in a function to define the parameters your experiment uses.

  • Config Injection: You can access all parameters of your configuration from every function. They are automatically injected by name.

  • Command-line interface: You get a powerful command-line interface for each experiment that you can use to change parameters and run different variants.

  • Observers: Sacred provides Observers that log all kinds of information about your experiment, its dependencies, the configuration you used, the machine it is run on, and of course the result. These can be saved to a MongoDB, for easy access later.

  • Automatic seeding helps controlling the randomness in your experiments, such that the results remain reproducible.

Example

Script to train an SVM on the iris dataset

The same script as a Sacred experiment

from numpy.random import permutation
from sklearn import svm, datasets





C = 1.0
gamma = 0.7



iris = datasets.load_iris()
perm = permutation(iris.target.size)
iris.data = iris.data[perm]
iris.target = iris.target[perm]
clf = svm.SVC(C=C, kernel='rbf',
        gamma=gamma)
clf.fit(iris.data[:90],
        iris.target[:90])
print(clf.score(iris.data[90:],
                iris.target[90:]))
from numpy.random import permutation
from sklearn import svm, datasets
from sacred import Experiment
ex = Experiment('iris_rbf_svm')

@ex.config
def cfg():
  C = 1.0
  gamma = 0.7

@ex.automain
def run(C, gamma):
  iris = datasets.load_iris()
  per = permutation(iris.target.size)
  iris.data = iris.data[per]
  iris.target = iris.target[per]
  clf = svm.SVC(C=C, kernel='rbf',
          gamma=gamma)
  clf.fit(iris.data[:90],
          iris.target[:90])
  return clf.score(iris.data[90:],
                   iris.target[90:])

Documentation

The documentation is hosted at ReadTheDocs.

Installing

You can directly install it from the Python Package Index with pip:

pip install sacred

Or if you want to do it manually you can checkout the current version from git and install it yourself:

cd sacred
python setup.py install

You might want to also install the numpy and the pymongo packages. They are optional dependencies but they offer some cool features:

pip install numpy pymongo

Tests

The tests for sacred use the pytest package. You can execute them by running pytest in the sacred directory like this:

pytest

There is also a config file for tox so you can automatically run the tests for various python versions like this:

tox

Update pytest version

If you update or change the pytest version, the following files need to be changed:

  • dev-requirements.txt

  • tox.ini

  • test/test_utils.py

  • setup.py

Contributing

If you find a bug, have a feature request or want to discuss something general you are welcome to open an issue. If you have a specific question related to the usage of sacred, please ask a question on StackOverflow under the python-sacred tag. We value documentation a lot. If you find something that should be included in the documentation please document it or let us know whats missing. If you are using Sacred in one of your projects and want to share your code with others, put your repo in the Projects using Sacred <docs/projects_using_sacred.rst>_ list. Pull requests are highly welcome!

Frontends

At this point there are three frontends to the database entries created by sacred (that I’m aware of). They are developed externally as separate projects.

Omniboard

docs/images/omniboard-table.png docs/images/omniboard-metric-graphs.png

Omniboard is a web dashboard that helps in visualizing the experiments and metrics / logs collected by sacred. Omniboard is written with React, Node.js, Express and Bootstrap.

Incense

docs/images/incense-artifact.png docs/images/incense-metric.png

Incense is a Python library to retrieve runs stored in a MongoDB and interactively display metrics and artifacts in Jupyter notebooks.

Sacredboard

docs/images/sacredboard.png

Sacredboard is a web-based dashboard interface to the sacred runs stored in a MongoDB.

Neptune

docs/images/neptune-compare.png docs/images/neptune-collaboration.png

Neptune is a metadata store for MLOps, built for teams that run a lot of experiments. It gives you a single place to log, store, display, organize, compare, and query all your model-building metadata via API available for both Python and R programming languages:

docs/images/neptune-query-api.png

In order to log your sacred experiments to Neptune, all you need to do is add an observer:

from neptune.new.integrations.sacred import NeptuneObserver
ex.observers.append(NeptuneObserver(api_token='<YOUR_API_TOKEN>',
                                    project='<YOUR_WORKSPACE/YOUR_PROJECT>'))

For more info, check the Neptune + Sacred integration guide.

SacredBrowser

docs/images/sacred_browser.png

SacredBrowser is a PyQt4 application to browse the MongoDB entries created by sacred experiments. Features include custom queries, sorting of the results, access to the stored source-code, and many more. No installation is required and it can connect to a local database or over the network.

Prophet

Prophet is an early prototype of a webinterface to the MongoDB entries created by sacred experiments, that is discontinued. It requires you to run RestHeart to access the database.

License

This project is released under the terms of the MIT license.

Citing Sacred

K. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’, in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sacred-0.8.5.tar.gz (110.1 kB view hashes)

Uploaded Source

Built Distribution

sacred-0.8.5-py2.py3-none-any.whl (107.9 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page