mmf-setup 0.3.1

mmf_setup

This meta-project provides an easy way to install all of the python tools I typically use. It also serves as a fairly minimal example of setting up a package the pip can install, and specifying dependencies.

In particular, I structure it for the following use-cases:

1. Rapid installation and configuration of the tools I need. For example, I often use [CoCalc](cocalc.com). Whenever I create a new project, I need to perform some initialization. With this project, it is simply a matter of using pip to install this package, and then using some of the tools.

2. Initial setup of a python distribution on a new computer. This is a little more involved since one needs to first install python (I recommend using Miniconda) and then updating the tools.

3. A place to document various aspects of installing and setting up python and related tools. Some of this is old, but kept here for reference.

Quickstart (TL;DR)

1. To get the notebook initialization features without having to install the package, just copy nbinit.py to your project. Importing this will try to execute import mmf_setup;mmf_setup.nbinit() but failing this, will manually run a similar code.

2. Install this package from the source directory, PyPI, etc. with one of the following:

   • Directly from PyPI

     pip install --process-dependency-links --user mmf_setup[nbextensions]

   • From Source

     pip install --process-dependency-links --user hg+https://bitbucket.org/mforbes/mmf_setup[nbextensions]

   • From Local Source (Run this from the source directory after you unpack it.)

     pip install --process-dependency-links --user .[nbextnensions]

   Note: these can be run without the --user flag if you want to install them system-wide rather than into site.USER_BASE.

3. To get the notebook tools for Jupyter (IPython) notebooks, execute the following as a code cell in your notebook and then trust the notebook with File/Trust Notebook:

   import mmf_setup; mmf_setup.nbinit()

   This will download and enable the calico extensions, as well as set the theme which is implemented in the output cell so it is stored for use online such as when rendered through NBViewer. One can specify different themes. (Presently only theme='default' and theme='mmf' are supported.)

4. To use the mercurial notebook cleaning tools, simply source the mmf_setup script:

   . mmf_setup -v

   To do this automatically when you login, add this line to your ~/.bashc or ~/.bash_profile scripts. These can also be enabled manually by adding the following to your ~/.hgrc file:

   [extensions]
   strip=
   mmf_setup.nbclean=$MMF_UTILS/nbclean.py

   where $MMF_UTILS expands to the install location for the package (which can be seen by running mmf_setup -v).

   This will provide commands for committing clean notebooks such as hg cstatus, hg cdiff and hg ccommit.

Installing Tools

The following code will download and install the Calico notebook extensions from here:

import mmf_setup.notebook_configuration
mmf_setup.notebook_configuration.install_extensions()

Mercurial (hg) Tools

If you source the output of the mmf_setup script:

. mmf_setup

then your HGRCPATH will be amended to include this projects hgrc file which does the following:

1. Adds some useful extensions.

2. Adds the following commands:

   • hg lga (or hg lg): provides a nice concise graphical display of the repo.

   • hg cstatus (or hg cst):

   • hg cdiff: same for hg diff

   • hg cediff: same for hg ediff

   • hg crecord: same for hg record. Note, this does not function like commit - it will not record the notebooks with the full output.

   • hg ccommit (or hg ccom): same for hg com but also commits the full version of the notebooks with output as a new node off the present node with the message ..: Automatic commit of output. This command has two behaviours depending on the configuration option nbclean.output_branch. If this is not set:

     [nbclean]
     output_branch =

     then hg ccommit will commit a cleaned copy of your notebooks with the output stripped, and then will commit the full notebook with output (provided that the notebooks have output) as a new head:

     | o  4: test ...: Automatic commit with .ipynb output
     |/
     @  3: test ccommit 3
     |
     | o  2: test ...: Automatic commit with .ipynb output
     |/
     o  1: test ccommit 1
     |
     o  0: test commit 0

     The parent will always be set to the clean node so that the output commits can be safely stripped from your repository if you choose not to keep them.

     The other mode of operation can be enabled by specifying a name for the output branch:

     [nbclean]
     output_branch = auto_output

     This will merge the changes into a branch with the specified name:

     | o  4: test ...: Automatic commit with .ipynb output (...) auto_output
     |/|
     @ |  3: test ccommit 3
     | |
     | o  2: test ...: Automatic commit with .ipynb output (...) auto_output
     |/
     o  1: test ccommit 1
     |
     o  0: test commit 0

     This facilitates stripping the output hg strip 2 for example will remove all output. It also allows you to track the changes in the output.

Developer Notes

There are a couple of subtle points here that should be mentioned.

• I explored both (un)shelve and commit/strip versions of saving the current state. While the former allows for shorter aliases, it can potentially trigger merges, so we use the latter.

• I sometimes write commit hook. These should only be run on the real commit, so we define the alias _commit which will bypass the hooks as discussed here.

• The list of files to strip is generated by hg status -man. This only includes added or modified files. This, if a notebook was commited with output (using hg com) then it will not be stripped.

• Our approach of . mmf_setup sets HGRCPATH but if this was not set before, then this will skip the default search. As such, we insert ~/.hgrc if HGRCPATH was not previously set. This is not ideal, but short of sticking an %include statement in the users ~/.hgrc file, or creating an hg alias, I do not see a robust solution. Note: we only insert ~/.hgrc if HGRCPATH is not defined - I ran into problems when always inserting it since this can change the order of precedence.

• Chain commands with semicolons ; not && so that things are restored even if a command fails. (For example, hg ccom with a notebook that only has output changes used to fail early.)

Changes

0.3.1:

• Added mmf_setup cocalc initialization for CoCalc.com projects (changed from mmf_setup smc).

• Add message to nbinit() call indicating that HGROOT has been added to sys.path.

0.3.0:

• Support python 3 - drop support for python 2.

• Conda installable from mforbes channel on anaconda cloud.

• Add missing default.tex theme so that nbinit() brings in math definitions even if no theme is specified.

• Fixed KeyError: ‘asyncio’ error on failed IPython import.

0.1.13:

• Incomplete version… everything here is rolled into 0.3.0

0.1.12:

• Made mmf_initial_setup python 3 compliant.

• Added logging to nbinit() and made some slight tweaks to HGROOT.

• Added D, sn, cn, dn commands for mathjax.

0.1.11:

• Resolve issue #20:

  • mmf_setup.set_path.set_path_from_file allows for configuration of path in setup.cfg file.

  • Fix python 3 bug: TypeError: Can’t mix strings and bytes in path components

0.1.10:

• Added better backwards compatibility for previous changes.

• Simplified nbinit theme management and use ‘default’ theme as default.

  • New themes only need to overwrite what they need.

  • Don’t change fonts as default since this does not work well on CoCalc (the code cells change size on clicking which is a pain.)

0.1.9:

• Resolve issues:

  • #17: store mmf_setup.HGROOT

  • #18: safer exception handling with nbclean commands so data is not lost

  • #19: nbclean works with new mercurial API

• Added erfi, sech, Braket

• import mmf_setup.set_path.hgroot will add HGROOT to path without IPython

• Added standalone nbinit.py for use without mmf_setup installs.

0.1.8:

• Resolves issue #15.

• Use $BASH_SOURCE{0} to get BIN_DIR (resolves issue #16)

• Added nbinit(hgroot) option to add hg root to sys.path (issue #13)

0.1.7:

• Changed option to –clean-all (resolves issue #7)

• Updated the notebook style (added Tr, fixed output overflow issue)

• Added pip option mmf_setup[nbextensions]

• Removed ‘EnableNBExtensionApp’ dependence which broke new Jupyter (resolves issue #12)

• Added some files for initializing setup on Sage Mathcloud (SMC) (resolves issue #14).

  • Added mmf_initial_setup script and some init files (.inputrc, .hgrc, .bash_aliases).

  • Run with mmf_setup smc

• Removed old extension manager since nbextensions are quite good now.

0.1.6:

• Added cupdate command (resolves issue #4)

• Fixed bug with ccommit where it did not accept a list of files (issue #6)

• Issue commands in a context to provide a robust mechanism for recovery (issue #5)