Package for managing experiments and analyses
Project description
xanity is experiment SANITY
Xanity is meant to allow easy and free creation of multiple experiments among which there is a shared codebase or system (conda) environment.
Xanity helps manage the following:
- experiment source code
- analysis source code
- experiment parameters
- experiment data
- run logs
Dependencies:
- conda (from Anaconda or my pref, Miniconda)
Usage:
xanity commands generally assume they're being called from inside a xanity project directory tree. Some commands accept additional arguments to specify the xanity proj path:
xanity init [--with-examples] [new-dir]
< will init a new xanity project directory tree in either the $PWD or
in the specified "new dir" with or without example files>
xanity setup [proj-dir]
< will setup or check dependencies in the environment associated with the
xanity project ($PWD or specified) >
xanity status [proj-dir]
< will print the status of the current xanity project to STDOUT >
xanity run [experiment_names] [-a analyses[...]]
< will run all (or the specified) experiments and optionally, analyses >
xanity analyse [-a analyses] [run_data_root]
< will run all (or the specified) analyses on the most recent (or specified)
data >
Example:
-
install xanity into a new conda env (or if you want, your system python):
conda create -n xanity_test python=3.6 conda activate xanity_test pip install xanityor for system python
pip install --user xanity -
initialize a xanity project and move inside with:
xanity init --with-examples xanity_test_proj cd xanity_test_projthis will create a skeleton directory tree for your experiments and analyses which will be populated with some examples.
-
Open xanity_test_proj/experiments/experiment1.py and have a look. This is a skeletal experiment.
-
Open xanity_test_proj/analyses/analysis1.py and have a look. This is a skeletal analysis.
-
find the conda-environment.yaml file and tweak its contents to suit your requiremnts.
-
resolve these requirements (create/update a conda env) with:
xanity setup -
run everything with:
xanity run -a -
you will find all the experimental data organized under the data/runs directory tree. Source-code snapshots are tarred and kept with the data they produced. Logs are kept too.
-
you can run an analysis script on a completed run:
xanity analyze experiment1this will look for the most recent (or specified) dir of run data, and run the analysis found at analyses/myfaveexp on that data.
-
relax. collect Nobel.
Experiment file skeleton:
(xanity-proj-root/experiments/*.py)
a. Each experiment module must have a main() function defined:
- (xanity will look for and invoke the main() function in each experiment).
- Any parameters to the experiment should be arguments to the main(). (leaving the experiment exposed in the module makes your code more portable.. I think. an alternative would be to define an experiment class inside xanity and make each experiment an instance)
b. The xanity.experiment_parameters() call registers the parameter permutations to use when running the experiment.
c. Include the xanity.metadata_hook() and xanity.run_hook() function calls.
- The metadata_hook will return relevant xanity metadata (loggers, data paths, etc)
- The run_hook will run the experiment if it's invoked directly as a script or as a module:
# experiments/example_experiment.py
import xanity
import numpy as np
# flag this experiment for analysis
xanity.analyze_this()
# register parameter sweeps you'd like to do
xanity.experiment_parameters({
'n_trials': [100,150,200],
'train_frac': [0.9, 0.5, 0.1],
'scale': [1,2,3,]
})
# parameters the experiment will accept
def main(n_trials=200, scale =5, main_frac=0.8):
fakevar = scale * np.random.rand(n_trials)**2
xanity.log("here is a print from experiment 1")
xanity.save_variable(fakevar)
if __name__=='__main__':
xanity.run_hook()
Analysis file skeleton:
(xanity-proj-root/analyses/*.py)
a. Each analysis module must have a main() function defined:
- (xanity will look for and invoke the main() function in each analysis).
- The only parameter to the analysis is the path to the root of a run (or set of runs). (leaving the experiment exposed in the module makes your code more portable.. I think. an alternative would be to define an experiment class inside xanity and make each experiment an instance)
b. The call to xanity.analysis.link_experiments() registers the names of experiments to associate with this analysis.
c. Include the xanity.analysis.metadata_hook() and xanity.analysis.run_hook() function calls.
- The metadata_hook will return relevant xanity metadata (loggers, data paths, etc)
- The run_hook will run the analysis if it's invoked directly as a script or as a module
# analyses/example_analysis.py
import xanity
import matplotlib.pyplot as plt
# define which experiments to associate this analysis with
xanity.associated_experiments([
'experiment1',
#'experiment2',
#'experiment3',
])
# the analysis takes a single argument... path of data (xanity will provide)
ef main(data_dir):
data, paths = xanity.load_variable('fakevar')
plt.figure()
for d in data:
d.sort()
plt.plot(d)
plt.show()
if __name__=='__main__':
xanity.run_hook()
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
