Python Tools
Project description
mmf-setup
This meta-project provides an easy way to install all of the python tools I typically use. It also serves as a fairly minimal example of setting up a package that pip can install, and specifying dependencies.
In particular, I structure it for the following use-cases:
-
Rapid installation and configuration of the tools I need. For example, I often use CoCalc. Whenever I create a new project, I need to perform some initialization. With this project, it is simply a matter of using pipx to install this package, and then using some of the tools. Specifically:
pipx install mmf-setup mmf_setup cocalc mercurial hg-evolve hg-git black
This does the following, and then links various configuration files to the home directory:
pipx install mmf-setup pipx inject mmf-setup mercurial hg-evolve hg-git black curl -sSL https://install.python-poetry.org | python3 -
-
Initial setup of a python distribution on a new computer. This is a little more involved since one needs to first install python (I recommend using Miniconda) and then updating the tools.
-
A place to document various aspects of installing and setting up python and related tools. Some of this is old, but kept here for reference.
-
A generic way of setting
sys.path
for development work using the following (in order of decreasing precedence) by callingmmf_setup.set_path()
.- An entry in a
pyproject.toml
file somewhere in a higher-level directory. - An entry in a
setup.cfg
file somewhere in a higher-level directory. - The first parent directory with either of these files, a
setup.py
file, or.hg
or.git
directories.
- An entry in a
Quickstart (TL;DR)
-
To get the notebook initialization features without having to install the package, just copy nbinit.py to your project. Importing this will try to execute
import mmf_setup;mmf_setup.nbinit()
but failing this, will manually run a similar code. -
Install this package from the source directory, PyPI, etc. with one of the following:
-
Directly from PyPI
python3 -m pip install --user mmf-setup[nbextensions]
-
From Source
python3 -m pip install --user hg+https://alum.mit.edu/www/mforbes/hg/forbes-group/mmf-setup[nbextensions]
-
From Local Source (Run this from the source directory after you unpack it.)
python3 -m pip install --user .[nbextensions]
Note: these includes the
nbextensions
extra. You and run without the--user
flag if you want to install them system-wide rather than intosite.USER_BASE
. -
-
To get the notebook tools for Jupyter (IPython) notebooks, execute the following as a code cell in your notebook and then trust the notebook with
File/Trust Notebook
:import mmf_setup; mmf_setup.nbinit()
This will set the theme which is implemented in the output cell so it is stored for use online such as when rendered through NBViewer. One can specify different themes. (Presently only
theme='default'
andtheme='mmf'
are supported.) -
Mercurial: If you want to install mercurial with the hg-git and Evolve extensions, then you can do that with the
hg
extra:python3 -m pip install --user .[hg]
This essentially runs
pip install mercurial hg-git hg-evolve
. You can then enable these in your environment by sourcing themmf_setup
script in one of the following ways:eval $"(mmf_setup -v)" # Enable hg lg but not evolve, etc. eval $"(mmf_setup -v -H)" # Enable evolve, etc. mmf_setup -v [-H] # Shows what will be set set (dry run).
To do this automatically when you login, add this line to your
~/.bashc
or~/.bash_profile
scripts.Warning: the
eval $"(mmf_setup -v -H)"
option also includes a mercurial update hook which will add%include ../.hgrc
to your projects.hg/hgrc
file uponhg > update
. This allows you to include project-specific mercurial customizations in your repository, but is a potential security risk. See the discussion below.These can also be enabled manually by adding the following to your
~/.hgrc
file:# ~/.hgrc ... [extensions] evolve = topics = hggit =
Setting sys.path
The preferred way to work with python code is to install it, preferably into a virtual
environment or a conda environment. By installing it, you will assure that your program
or library is available to notebooks, etc. by being properly inserted into
sys.path. When developing, code, one can also do a "editable"
installs
with python -m install -e \<path\>
so that code updates are seen.
When developing code, however, this may not meet all use-cases, so we provide
mmf_utils.set_path(cwd='.')
which will set mmf_setup.ROOT
and insert it at the start
of sys.path
.
The algorithm for determining ROOT
is as follows starting from cwd
(which is '.'
by default, but can be specified in the call to set_path()
). Note: in each case, the
specified path must be an existing directory or it will be ignored.
-
A value specified as
mmf_setup.ROOT
will override everything. -
A value in the environmental variable
MMF_SETUP_ROOT
will override everything else. -
An explicit
ROOT
entry in the firstpyproject.toml
found:# pyproject.toml [tool.mmf_setup] ROOT = 'src'
-
An explict entry in the first
setup.cfg
:# setup.cfg [mmf_setup] ROOT = src
-
The first parent directory to contain any of the files
setup.py
,setup.cfg
, orpyproject.toml
, or directories.git
, or.hg
.
If none of these yields a valid existing directory for ROOT
, then it will not be set.
Mercurial (hg) Tools
If you source the output of the mmf_setup
script with one of the following:
eval $"(mmf_setup -v)"
eval $"(mmf_setup -v -H)"
then your HGRCPATH
will be amended to include
hgrc.lga
or
hgrc.full
respectively. The first adds a useful hg lga
(hg lg
for short) command which provides a concise graphical display:
$ hg lg
@ 200:d michael (64 minutes ago) 0.4.0[0.4.0] tip
| ENH,DOC,TST,API: Cleaned up set_path...
o 199:d michael (31 hours ago) 0.4.0[0.4.0]
| WIP,CHK: Cleaning up set_path.
o 198:d michael (20 hours ago) 0.4.0[0.4.0]
| DOC: Added developer Notes.md
...
The second version with -H
adds some useful extensions: hg-git, Evolve, and
enables Topics. The latter are required, for example, to interface with
Heptapod.
Finally, the -H
option enables an hg update
hook, which adds %include ../.hgrc
to
your project's .hg/hgrc
file when you call hg update
. This is a potential security
risk, because an untrusted repo could include dangerous commands in .hgrc
. Thus, we
require user intervention before including this:
$ eval $"(mmf_setup -v -H)"
$ hg clone https://alum.mit.edu/www/mforbes/hg/forbes-group/mmf-setup
...
updating to branch default
88 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd mmf-setup
$ hg up 0.4.0
Repository .../mmf-setup contains an .hgrc file.
Including this in .hg/hgrc is a potential security risk.
Only do this if you trust the repository.
Include this .hgrc in .hg/hgrc? [yN]y
Adding '%include ../.hgrc' to '.hg/hgrc'
...
$ tail .hg/hgrc
...
%include ../.hgrc
CoCalc
We provide some tools for working on CoCalc. To get started, simply do to the following once you have enabled network access on your project by applying a license:
pipx install mmf-setup
mmf_setup cocalc
To see exactly what this will do, run:
mmf_setup cocalc -v
Once this is done, you should ensure that, whenever you use a terminal on CoCalc, you appropriately set the following variables:
These will identify you as you when you commit your work. (Needed, because all users share the same project. See cocalc#370 for details.)
I recommend the following:
-
Forward these variables in your SSH config file (
~/.ssh/config
):# ~/.ssh/config Host cc-project1 User ff1cb986f... Host cc* HostName ssh.cocalc.com ForwardAgent yes SetEnv LC_HG_USERNAME=Your Full Name <your.email.address+hg@gmail.com> SetEnv LC_GIT_USERNAME=Your Full Name SetEnv LC_GIT_USEREMAIL=your.email.address+git@gmail.com SetEnv LC_EDITOR=vi
The appropriate value for
User
can be found in the project Settings on CoCalc.Optionally, use
SendEnv ...
and set these on your personal computer:# ~/.bashrc or similar LC_HG_USERNAME=Your Full Name <your.email.address+hg@gmail.com> LC_GIT_USEREMAIL=your.email.address+git@gmail.com LC_GIT_USERNAME=Your Full Name
-
If you want to run Git or Mercurial from the CoCalc web interface, then create a named terminal -- i.e.
Michael.term
-- and then set these variables in the terminal startup script. (See https://doc.cocalc.com/terminal.html#startup-files for details.)
Notes
See Notes.md for developer notes. Other notes about python, IPython, etc. are stored in the docs folder.
Changes
0.4.11
- Fix bug with warning on CoCalc.
0.4.10
- Add warning on CoCalc if users have not set
LC_HG_USERNAME
, orLC_GIT_USERNAME
.
0.4.9
- Remove blank lines from
default.tex
so thatmmf_setup.nbinit()
works on CoCalc (workaround for CoCalc issue #6210).
0.4.8
-
Don't install mercurial etc. on CoCalc by default: these are now included in the Default Ubuntu 20.04 image. If you need these, then explicitly specify them:
mmf-setup cocalc mercurial hg-git hg-evolve jupytext black
0.4.7
- Recommend and support installing mmf-setup with pipx on CoCalc.
- Use new Poetry install.
- Drop support for python3.6.
0.4.6
- Use python3 for CoCalc setup. (Is a problem on Docker images).
- Ignore matplotlib errors.
0.4.3
- Resolves issue #28: Failure of
mmf_initial_setup
on Mac OS X if .DS_Store files exist. - Resolves issue #29: Relative paths in config files resolve to absolute paths
- Added aliases
findpy
,findf
,finda
etc. for CoCalc for runningfind
+grep
.
0.4.2
- Resolves issue #26: Significant improvements for installation on CoCalc.
- Moves
~/.bashrc
to~/.bashrc_cocalc
so we can insert~/.local/bin
to thePATH
for non-login shells. This is important when trying to push withhg
over SSH, which runshg
without logging in. Our new workflow needs thathg
to have access toevolve
etc. Previously this worked by updating these with the systemPython 2
version ofhg
, but that now fails with new projects, so we have removed this. - Add
.hgrc
tweaks to top-level cocalc~.hgrc
(might as well includehgrc.lga
here now so students etc. can see these. Don't enableevolve
etc. by default though.) - Users should now call
eval "$(mmf_setup -v -H)"
rather than using source. mmf_setup -H
. Sourcingmmf_setup
now emits a deprecation warning message. - Cocalc install is now tested in an isolated manner. (Only the
nox -s test_cocalc
works... see issue #27.)
- Moves
- Make
hgrc.full
POSIX compliant (work withsh
). (Was causing GitHub workflow to fail.) Scripts still requirebash
which is explicitly chosen when we callrun-tests.py
. - Working GitHub workflows and cocalc tests.
- Updated release process.
0.4.1
- Make
mmf_setup -v
output clean so that we caneval "$(mmf_setup -v)"
rather than sourcing. In the future we will make this a python file, soeval
will be the only option. - Fixed issues with
nbinit()
failing with not paths defined. - Added more tests.
0.4.0
-
Resolves issue #23: Drop support of nbclean mercurial extension (to hard to maintain), thereby making issues #1, #3, #8, #9, #11, obsolete.
-
Remove installation of old obsolete notebook extensions like Calico tools, RISE etc. These can all be installed now with either the standard extensions, or pip.
-
With pip:
python -m pip install jupyter_contrib_nbextensions RISE jupyter contrib nbextension install --user
-
With conda:
conda install -c conda-forge jupyter_contrib_nbextensions rise jupyter contrib nbextension install --user
See also NBExtensions.
-
-
Improved test isolation.
-
Expanded options for
mmf_setup
.- Mercurial configuration is now in two stages:
hgrc.lga
which gives thehg lg
command safely using only builtin modules, andhgrc.full
which includes this, and also activates thehg-git
andevolve
extensions. This also applies theupdate
hook for including.hgrc
files, but asks the user for confirmation because this is a security risk.
- Mercurial configuration is now in two stages:
0.3.1
- Added
mmf_setup cocalc
initialization for CoCalc.com projects (changed frommmf_setup smc
). - Add message to
nbinit()
call indicating that HGROOT has been added tosys.path
. - Resolves issue #22 with git username classes on CoCalc.
0.3.0
- Support python 3 - drop support for python 2.
- Conda installable from mforbes channel on anaconda cloud.
- Add missing default.tex theme so that nbinit() brings in math definitions even if no theme is specified.
- Fixed KeyError:
asyncio
error on failed IPython import.
0.1.13
- Incomplete version... everything here is rolled into 0.3.0
0.1.12
- Made
mmf_initial_setup
python 3 compliant. - Added logging to
nbinit()
and made some slight tweaks toHGROOT
. - Added
\D
,\sn
,\cn
,\dn
commands for MathJaX.
0.1.11
- Resolve issue #20:
mmf_setup.set_path.set_path_from_file
allows for configuration of path insetup.cfg
file.- Fix python 3 bug: TypeError: Can't mix strings and bytes in path components
0.1.10
- Added better backwards compatibility for previous changes.
- Simplified nbinit theme management and use
'default'
theme as default.- New themes only need to overwrite what they need.
- Don't change fonts as default since this does not work well on CoCalc (the code cells change size on clicking which is a pain.)
0.1.9
-
Resolve issues:
- #17: store
mmf_setup.HGROOT
- #18: safer exception handling with
nbclean
commands so data is not lost - #19:
nbclean
works with new mercurial API
- #17: store
-
Added
\erfi
,\sech
,\Braket
-
import mmf_setup.set_path.hgroot
will addHGROOT
to path without IPython -
Added standalone
nbinit.py
for use withoutmmf_setup
installs.
0.1.8
- Resolves issue #15.
- Use
$BASH_SOURCE{0}
to getBIN_DIR
(resolves issue #16) - Added
nbinit(hgroot)
option to addhg root
tosys.path
(issue #13)
0.1.7
- Changed option to
--clean-all
(resolves issue #7) - Updated the notebook style (added
\Tr
, fixed output overflow issue) - Added pip option mmf_setup[nbextensions]
- Removed
'EnableNBExtensionApp'
dependence which broke new Jupyter (resolves issue #12) - Added some files for initializing setup on Sage Mathcloud (SMC) (resolves issue #14).
- Added
mmf_initial_setup
script and some init files (.inputrc
,.hgrc
,.bash_aliases
). - Run with
mmf_setup smc
- Added
- Removed old extension manager since nbextensions are quite good now.
0.1.6
- Added cupdate command (resolves issue #4)
- Fixed bug with ccommit where it did not accept a list of files (issue #6)
- Issue commands in a context to provide a robust mechanism for recovery (issue #5)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mmf_setup-0.4.11-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8afb4ac4606c44e8b88ddeff9f8f839ba34aaa4d25e25fda0c42f24523230365 |
|
MD5 | 4a21827d2f9e56c416aedd3483f49502 |
|
BLAKE2b-256 | 821456b64420bdeed27333d516dd594e6e8e9f79e0593322b36f565f5c3aa6bb |