mmf-setup

This meta-project provides an easy way to install all of the python tools I typically use. It also serves as a fairly minimal example of setting up a package tate pip can install, and specifying dependencies.

In particular, I structure it for the following use-cases:

1. Rapid installation and configuration of the tools I need. For example, I often use CoCalc. Whenever I create a new project, I need to perform some initialization. With this project, it is simply a matter of using [pipx] to install this package, and then using some of the tools. Specifically:

   pipx install mmf-setup
   mmf_setup cocalc
   

   This does the following, and then links various configuration files to the home directory:

   pipx install mmf-setup
   pipx inject mmf-setup mercurial hg-evolve hg-git black
   curl -sSL https://install.python-poetry.org | python3 -
   

2. Initial setup of a python distribution on a new computer. This is a little more involved since one needs to first install python (I recommend using Miniconda) and then updating the tools.

3. A place to document various aspects of installing and setting up python and related tools. Some of this is old, but kept here for reference.

4. A generic way of setting sys.path for development work using the following (in order of decreasing precedence) by calling mmf_setup.set_path().

   • An entry in a pyproject.toml file somewhere in a higher-level directory.
   • An entry in a setup.cfg file somewhere in a higher-level directory.
   • The first parent directory with either of these files, a setup.py file, or .hg or .git directories.

Quickstart (TL;DR)

1. To get the notebook initialization features without having to install the package, just copy nbinit.py to your project. Importing this will try to execute import mmf_setup;mmf_setup.nbinit() but failing this, will manually run a similar code.

2. Install this package from the source directory, PyPI, etc. with one of the following:

   • Directly from PyPI

     python3 -m pip install --user mmf-setup[nbextensions]
     

   • From Source

     python3 -m pip install --user
     hg+https://alum.mit.edu/www/mforbes/hg/forbes-group/mmf-setup[nbextensions]
     

   • From Local Source (Run this from the source directory after you unpack it.)

     python3 -m pip install --user .[nbextensions]
     

   Note: these includes the nbextensions extra. You and run without the --user flag if you want to install them system-wide rather than into site.USER_BASE.

3. To get the notebook tools for Jupyter (IPython) notebooks, execute the following as a code cell in your notebook and then trust the notebook with File/Trust Notebook:

   import mmf_setup; mmf_setup.nbinit()
   

   This will set the theme which is implemented in the output cell so it is stored for use online such as when rendered through NBViewer. One can specify different themes. (Presently only theme='default' and theme='mmf' are supported.)

4. Mercurial: If you want to install mercurial with the hg-git and Evolve extensions, then you can do that with the hg extra:

   python3 -m pip install --user .[hg]
   

   This essentially runs pip install mercurial hg-git hg-evolve. You can then enable these in your environment by sourcing the mmf_setup script in one of the following ways:

   eval $"(mmf_setup -v)"     # Enable hg lg but not evolve, etc.
   eval $"(mmf_setup -v -H)"  # Enable evolve, etc.
   mmf_setup -v [-H]          # Shows what will be set set (dry run).
   

   To do this automatically when you login, add this line to your ~/.bashc or ~/.bash_profile scripts.

   Warning: the eval $"(mmf_setup -v -H)" option also includes a mercurial update hook which will add %include ../.hgrc to your projects .hg/hgrc file upon hg > update. This allows you to include project-specific mercurial customizations in your repository, but is a potential security risk. See the discussion below.

   These can also be enabled manually by adding the following to your ~/.hgrc file:

   # ~/.hgrc
   ...
   [extensions]
   evolve =
   topics =
   hggit =
   

Setting sys.path

The preferred way to work with python code is to install it, preferably into a virtual environment or a conda environment. By installing it, you will assure that your program or library is available to notebooks, etc. by being properly inserted into sys.path. When developing, code, one can also do a "editable" installs with python -m install -e \<path\> so that code updates are seen.

When developing code, however, this may not meet all use-cases, so we provide mmf_utils.set_path(cwd='.') which will set mmf_setup.ROOT and insert it at the start of sys.path.

The algorithm for determining ROOT is as follows starting from cwd (which is '.' by default, but can be specified in the call to set_path()). Note: in each case, the specified path must be an existing directory or it will be ignored.

• A value specified as mmf_setup.ROOT will override everything.

• A value in the environmental variable MMF_SETUP_ROOT will override everything else.

• An explicit ROOT entry in the first pyproject.toml found:

  # pyproject.toml
  [tool.mmf_setup]
  ROOT = 'src'
  

• An explict entry in the first setup.cfg:

  # setup.cfg
  [mmf_setup]
  ROOT = src
  

• The first parent directory to contain any of the files setup.py, setup.cfg, or pyproject.toml, or directories .git, or .hg.

If none of these yields a valid existing directory for ROOT, then it will not be set.

Mercurial (hg) Tools

If you source the output of the mmf_setup script with one of the following:

eval $"(mmf_setup -v)"
eval $"(mmf_setup -v -H)"

then your HGRCPATH will be amended to include hgrc.lga or hgrc.full respectively. The first adds a useful hg lga (hg lg for short) command which provides a concise graphical display:

$ hg lg
@  200:d michael (64 minutes ago) 0.4.0[0.4.0] tip
|   ENH,DOC,TST,API: Cleaned up set_path...
o  199:d michael (31 hours ago) 0.4.0[0.4.0]
|   WIP,CHK: Cleaning up set_path.
o  198:d michael (20 hours ago) 0.4.0[0.4.0]
|   DOC: Added developer Notes.md
...

The second version with -H adds some useful extensions: hg-git, Evolve, and enables Topics. The latter are required, for example, to interface with Heptapod.

Finally, the -H option enables an hg update hook, which adds %include ../.hgrc to your project's .hg/hgrc file when you call hg update. This is a potential security risk, because an untrusted repo could include dangerous commands in .hgrc. Thus, we require user intervention before including this:

$ eval $"(mmf_setup -v -H)"
$ hg clone https://alum.mit.edu/www/mforbes/hg/forbes-group/mmf-setup
...
updating to branch default
88 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd mmf-setup
$ hg up 0.4.0
Repository .../mmf-setup contains an .hgrc file.
Including this in .hg/hgrc is a potential security risk.
Only do this if you trust the repository.

Include this .hgrc in .hg/hgrc? [yN]y
Adding '%include ../.hgrc' to '.hg/hgrc'
...
$ tail .hg/hgrc
...
%include ../.hgrc

Notes

See Notes.md for developer notes. Other notes about python, IPython, etc. are stored in the docs folder.

Changes

0.4.7

• Recommend and support installing mmf-setup with pipx on CoCalc.
• Use new Poetry install.
• Drop support for python3.6.

0.4.6

• Use python3 for CoCalc setup. (Is a problem on Docker images).
• Ignore matplotlib errors.

0.4.3

• Resolves issue #28: Failure of mmf_initial_setup on Mac OS X if .DS_Store files exist.
• Resolves issue #29: Relative paths in config files resolve to absolute paths
• Added aliases findpy, findf, finda etc. for CoCalc for running find + grep.

0.4.2

• Resolves issue #26: Significant improvements for installation on CoCalc.
  • Moves ~/.bashrc to ~/.bashrc_cocalc so we can insert ~/.local/bin to the PATH for non-login shells. This is important when trying to push with hg over SSH, which runs hg without logging in. Our new workflow needs that hg to have access to evolve etc. Previously this worked by updating these with the system Python 2 version of hg, but that now fails with new projects, so we have removed this.
  • Add .hgrc tweaks to top-level cocalc ~.hgrc (might as well include hgrc.lga here now so students etc. can see these. Don't enable evolve etc. by default though.)
  • Users should now call eval "$(mmf_setup -v -H)" rather than using source . mmf_setup -H. Sourcing mmf_setup now emits a deprecation warning message.
  • Cocalc install is now tested in an isolated manner. (Only the nox -s test_cocalc works... see issue #27.)
• Make hgrc.full POSIX compliant (work with sh). (Was causing GitHub workflow to fail.) Scripts still require bash which is explicitly chosen when we call run-tests.py.
• Working GitHub workflows and cocalc tests.
• Updated release process.

0.4.1

• Make mmf_setup -v output clean so that we can eval "$(mmf_setup -v)" rather than sourcing. In the future we will make this a python file, so eval will be the only option.
• Fixed issues with nbinit() failing with not paths defined.
• Added more tests.

0.4.0

• Resolves issue #23: Drop support of nbclean mercurial extension (to hard to maintain), thereby making issues #1, #3, #8, #9, #11, obsolete.

• Remove installation of old obsolete notebook extensions like Calico tools, RISE etc. These can all be installed now with either the standard extensions, or pip.

  • With pip:

    python -m pip install jupyter_contrib_nbextensions RISE
    jupyter contrib nbextension install --user
    

  • With conda:

    conda install -c conda-forge jupyter_contrib_nbextensions rise
    jupyter contrib nbextension install --user
    

  See also NBExtensions.

• Improved test isolation.

• Expanded options for mmf_setup.

  • Mercurial configuration is now in two stages: hgrc.lga which gives the hg lg command safely using only builtin modules, and hgrc.full which includes this, and also activates the hg-git and evolve extensions. This also applies the update hook for including .hgrc files, but asks the user for confirmation because this is a security risk.

0.3.1

• Added mmf_setup cocalc initialization for CoCalc.com projects (changed from mmf_setup smc).
• Add message to nbinit() call indicating that HGROOT has been added to sys.path.
• Resolves issue #22 with git username classes on CoCalc.

0.3.0

• Support python 3 - drop support for python 2.
• Conda installable from mforbes channel on anaconda cloud.
• Add missing default.tex theme so that nbinit() brings in math definitions even if no theme is specified.
• Fixed KeyError: asyncio error on failed IPython import.

0.1.13

• Incomplete version... everything here is rolled into 0.3.0

0.1.12

• Made mmf_initial_setup python 3 compliant.
• Added logging to nbinit() and made some slight tweaks to HGROOT.
• Added \D, \sn, \cn, \dn commands for MathJaX.

0.1.11

• Resolve issue #20:
  • mmf_setup.set_path.set_path_from_file allows for configuration of path in setup.cfg file.
  • Fix python 3 bug: TypeError: Can't mix strings and bytes in path components

0.1.10

• Added better backwards compatibility for previous changes.
• Simplified nbinit theme management and use 'default' theme as default.
  • New themes only need to overwrite what they need.
  • Don't change fonts as default since this does not work well on CoCalc (the code cells change size on clicking which is a pain.)

0.1.9

• Resolve issues:

  • #17: store mmf_setup.HGROOT
  • #18: safer exception handling with nbclean commands so data is not lost
  • #19: nbclean works with new mercurial API

• Added \erfi, \sech, \Braket

• import mmf_setup.set_path.hgroot will add HGROOT to path without IPython

• Added standalone nbinit.py for use without mmf_setup installs.

0.1.8

• Resolves issue #15.
• Use $BASH_SOURCE{0} to get BIN_DIR (resolves issue #16)
• Added nbinit(hgroot) option to add hg root to sys.path (issue #13)

0.1.7

• Changed option to --clean-all (resolves issue #7)
• Updated the notebook style (added \Tr, fixed output overflow issue)
• Added pip option mmf_setup[nbextensions]
• Removed 'EnableNBExtensionApp' dependence which broke new Jupyter (resolves issue #12)
• Added some files for initializing setup on Sage Mathcloud (SMC) (resolves issue #14).
  • Added mmf_initial_setup script and some init files (.inputrc, .hgrc, .bash_aliases).
  • Run with mmf_setup smc
• Removed old extension manager since nbextensions are quite good now.

0.1.6

• Added cupdate command (resolves issue #4)
• Fixed bug with ccommit where it did not accept a list of files (issue #6)
• Issue commands in a context to provide a robust mechanism for recovery (issue #5)