libsncompress 0.0.8

Synopsis

jlacompress [-h] [-d DIR] [-t FILE] [-p PREFIX] [-c z1 z2 [...]] [-n] [-v]

The script requires JLA data files to run. See the section “Data Files” for details.

Command-Line Options

• -h, --help: show help message and exit

• -d DIR, --covdir DIR: path to directory of FITS covariance files (default: ./covmat)

• -t FILE, --table FILE: path to JLA data table file (default: ./jla_lcparams.txt)

• -p PREFIX, --output-prefix PREFIX: prefix to output file names

• -c z1 z2 [...], --controls z1 z2 [...]: locations of control points (as redshift). At least two control points are required. If unspecified, use the default control points in the JLA paper.

• -n, --no-logdet: don’t use the correct conditional probability (default: off; warning: use at your own risk)

• -v, --verbose: turn on verbose output (default: off)

Output

The script writes three output files when it solves the optimization problem successfully:

• Mean (approximate, actually posterior-maximizing) compression parameters in the order of (alpha, beta, delta_M, mu1, mu2, …), where muN is the value of distance modulus at the N-th control point. It is therefore a list of N + 3 numbers

• Covariance matrix (symmetric, and with all elements filled) of the parameters. It is of shape N + 3 × N + 3, and the first 3 rows and columns correspond to the three standardization parameters alpha, beta, and delta_M.

• Redshifts of the N control points.

The default output file names are mean.txt, cov.txt and redshift.txt respectively, but they can be prefixed by arbitrary strings specified by the user with the -p/--prefix option.

The slash (/) character in the prefix will be understood as directory separators. If the directory part of a resulting path does not exist, it will be created if possible, and nested directories may be created by this process.

The output files will have suffix -no-logdet appended to the path but before the .txt extension, if -n or --no-logdet is specified.

When verbose output is enabled by -v or --verbose, additional text will be written to the standard error.

Exit Status

The script exits with 0 for success. Any other value indicates error.

Example Usage

Assuming the data files are in their default locations, the following command reproduces the default compression results in the JLA paper.

jlacompress -n

Using libsncompress in Your Project

To use the package directly in your own Python project, simply

import libsncompress

This will import three classes from its sub-modules into the libsncompress namespace:

• BinnedSN: data-file loader and pre-processor

• BinCollection: redshift binning and sanitizer; not very useful on its own

• CovEvaluator: the actual compressor

The first thing you need to do is to specify a list (or numpy array) of control points, by their base-10 logarithm values. Currently, due to development legacy, the “binning” class and methods are not particularly efficient. This is usually not a problem because it will be used only once.

This list or array of control points must be encapsulate in another container (list, array, or tuple, etc.) before passing to the initializer of libsncompress.BinnedSN class. The instance can be initialized by

binned_sn = libsncompress.BinnedSN(basedirpath,
                                   tablepath,
                                   logbins=control_points)

Here basedirpath is the path to the directory containing the FITS covariance data files, tablepath the path to the text file containing the JLA dataset table, and logbins is the nested list of control points just obtained.

After this, we can initialize the evaluator libsncompress.CovEvaluator class, which implements the evaluation of probability log-density functions and their first 2 derivatives, like this:

ev = libsncompress.CovEvaluator(binned_sn, withlogdet=True)

The optional argument withlogdet controls whether the full effect of parameter-dependent covariance matrix is taken into account. It is so named due to the ubiquitous presence of “ln det Cov” term. It defaults to True but can be set to False, which will evaluate the functions as if the customary chi-squared method were used.

The CovEvaluator instance, ev, provides a method minimize, which is a wrapper of scipy.optimize.minimize. Additional positional and keyword arguments are passed over to that function. The recommended optimization algorithm is trust-ncg which fully utilizes the Hessian matrix. This is the default minimization algorithm if left unspecified, and other algorithms supported by ☞scipy.optimize.minimize can be passed as the optional keyword parameter method.

The return value of CovEvaluator.minimize method is simply that of the underlying scipy function, but with results suitably scaled.

The Hessian of log-PDF function can be obtained, then, at the minimizing point in the parameter space. This can be used for constructing the approximate covariance of compression parameters.

Please notice that this implementation here is not a general, abstract implementation of the linear compression method detailed in our paper. It specifically implements the sawtooth-basis compression, which is compatible with the original JLA one. The implementation details, as well as the exposed API, are likely to see significant revisions in the future.

Setting Up the Testing Environment

To run the tests (including the reproducibility tests), it is necessary to set up the environment with supporting packages and data.

As described in the preceding section, “Data Files”, the recommended way is to clone the Git repository and populate the testdata directory in the repository with the necessary files, which can be done using the download_jla.sh script.

After obtaining the data files, it is recommended to use the recent version of ☞tox to manage the testing environments.

pip install 'tox >= 2.8.0'

Although not strictly necessary for running the tests themselves per se, it is recommended to install the ☞pandoc program (please consult your operating system documentation) and the ☞pypandoc Python package.

Running the Tests

If you have both Python 2.7 and 3.6/3.7 installed, simply invoking

tox

will create the source distribution and run the tests under both Python variants. The default configuration will pull the latest supporting packages from PyPI specified in the file devel-requirements.txt.

If you have only one working variant of Python, for example Python 2.7, you can run

tox -e py2,coverage-report

and skip the unavailable test environment setting.

Reproducibility Tests

One important goal of the test suits in this repository is to ensure that the results of JLA SNIa compression are always reproducible.

First, as we have shown in M16, the JLA compression results (their Tables F.1 and F.2), especially the covariance matrix, are “very close” to the ones obtained using this program on the JLA data release, but with the (highly discouraged) withlogdet=False option enabled for libsncompress.CovEvaluator.

Second, the compression results produced by this program on the released JLA data must match those presented in M16, Tables A1 and A2.

The reproducibility tests check that these constraints are satisfied by all revisions to the codebase. These tests are included in the tests/test_reprod.py script and are run by tox by default.