argcmdr 0.2.0

Inspecting execution

Subprocess commands emitted by Local.prepare are executed in order and, by default, failed execution is interrupted by a raised exception:

class Release(Local):
    """release the package to pypi"""

    def __init__(self, parser):
        parser.add_argument(
            'part',
            choices=('major', 'minor', 'patch'),
            help="part of the version to be bumped",
        )

    def prepare(self, args):
        yield self.local['bumpversion'][args.part]
        yield self.local['python']['setup.py', 'sdist', 'bdist_wheel']
        yield self.local['twine']['upload', 'dist/*']

Should the bumpversion command fail, the deploy command will not proceed.

In some cases, however, we might like to disable this functionality, and proceed regardless of a subprocess’s exit code. We may pass arguments such as retcode to plumbum by setting this attribute on the prepare method:

def prepare(self, args):
    yield self.local['bumpversion'][args.part]
    yield self.local['python']['setup.py', 'sdist', 'bdist_wheel']
    yield self.local['twine']['upload', 'dist/*']

prepare.retcode = None

Subprocess commands emitted by the above method will not raise execution exceptions, regardless of their exit code. (To allow only certain exit code(s), set retcode as appropriate – see plumbum.)

Having disabled execution exceptions – and regardless – we might need to be able to inspect a subprocess command’s exit code, standard output or standard error. As such, (whether we manipulate retcode or not), argcmdr communicates these command results with prepare generator methods:

def prepare(self, args):
    (code, out, err) = yield self.local['bumpversion']['--list', args.part]

    yield self.local['python']['setup.py', 'sdist', 'bdist_wheel']

    if out is None:
        version = 'DRY-RUN'
    else:
        (version_match,) = re.finditer(
            r'^new_version=([\d.]+)$',
            out,
            re.M,
        )
        version = version_match.group(1)

    yield self.local['twine']['upload', f'dist/*{version}*']

In the above, prepare stores the results of bumpversion execution, in order to determine from its standard output the version to be released.

Access to the parsed argument namespace

The command invocation’s parsed arguments are most straight-forwardly accessible as the first argument of the Command invocation signature, either __call__ or prepare. However, in less-than-trivial implementations, wherein command methods are factored for reusability, passing the argument namespace from method to method may become tedious. To support such scenarios, this object is made additionally available via the Command property, args.

Consider a class of commands which require a database password. We don’t want to store this password anywhere in plain text; rather, we expect it to be input, either via (piped) standard input or the TTY:

class DbSync(Command):
    """sync databases"""

    def __init__(self, parser):
        parser.add_argument(
            '-p', '--password',
            action='store_true',
            dest='stdin_password',
            default=False,
            help="read database password from standard input",
        )

    def __call__(self, args):
        engine = self.dbengine(args)
        ...

    def dbcreds(self, args):
        dbcreds = {
            'username': os.getenv('PGUSER'),
            'host': os.getenv('PGHOST'),
            'port': os.getenv('PGPORT'),
            'database': os.getenv('PGDATABASE'),
        }

        missing = [key for (key, value) in dbcreds.items() if not value]
        if missing:
            raise RuntimeError(
                "database connection information missing from "
                "environmental configuration: " + ', '.join(missing)
            )

        if args.stdin_password:
            dbcreds['password'] = sys.stdin.read().rstrip('\n\r')

            # we're done with the (pipe) stdin, so force it back to TTY for
            # any subsequent input()
            sys.stdin = open('/dev/tty')
        else:
            dbcreds['password'] = os.getenv('PGPASSWORD')
            if not dbcreds['password']:
                dbcreds['password'] = getpass.getpass(
                    'enter password for '
                    + ('{username}@{host}:{port}'.format_map(dbcreds) | colors.bold)
                    + ': '
                    | colors.yellow
                )

        return dbcreds

    def dburi(self, args):
        return sqlalchemy.engine.url.URL('postgres', **self.dbcreds(args))

    def dbengine(self, args):
        return sqlalchemy.create_engine(self.dburi(args))

Not only were we forced to verbosely daisy-chain the arguments namespace, args, from method to method; moreover, we were prevented from (trivially) caching the result of dbcreds, to ensure that the password isn’t ever requested more than once.

Now, let’s reimplement the above, making use of the property args:

class DbSync(Command):
    """sync databases"""

    def __init__(self, parser):
        parser.add_argument(
            '-p', '--password',
            action='store_true',
            dest='stdin_password',
            default=False,
            help="read database password from standard input",
        )

    def __call__(self):
        engine = self.dbengine
        ...

    @cachedproperty
    def dbcreds(self):
        dbcreds = {
            'username': os.getenv('PGUSER'),
            'host': os.getenv('PGHOST'),
            'port': os.getenv('PGPORT'),
            'database': os.getenv('PGDATABASE'),
        }

        missing = [key for (key, value) in dbcreds.items() if not value]
        if missing:
            raise RuntimeError(
                "database connection information missing from "
                "environmental configuration: " + ', '.join(missing)
            )

        if self.args.stdin_password:
            dbcreds['password'] = sys.stdin.read().rstrip('\n\r')

            # we're done with the (pipe) stdin, so force it back to TTY for
            # any subsequent input()
            sys.stdin = open('/dev/tty')
        else:
            dbcreds['password'] = os.getenv('PGPASSWORD')
            if not dbcreds['password']:
                dbcreds['password'] = getpass.getpass(
                    'enter password for '
                    + ('{username}@{host}:{port}'.format_map(dbcreds) | colors.bold)
                    + ': '
                    | colors.yellow
                )

        return dbcreds

    @property
    def dburi(self):
        return sqlalchemy.engine.url.URL('postgres', **self.dbcreds)

    @property
    def dbengine(self):
        return sqlalchemy.create_engine(self.dburi)

In this form, args needn’t be passed from method to method; in fact, methods of the DbSync command needn’t worry about arguments which don’t directly interest them at all. And, using cachedproperty from Dickens, the database credentials are trivially cached, ensuring they aren’t needlessly re-requested.

Note that attempting to access the args property before invocation arguments have been parsed – e.g. within __init__ – is not allowed, and will raise RuntimeError.

Nested commands

Rather than procedurally defining subparsers, Command class declarations may simply be nested.

Let’s define an executable file manage for managing a codebase:

#!/usr/bin/env python

import os

from argcmdr import Local, main

class Management(Local):
    """manage deployment"""

    def __init__(self, parser):
        parser.add_argument(
            '-e', '--env',
            choices=('development', 'production'),
            default='development',
            help="target environment",
        )

    class Build(Local):
        """build app"""

        def prepare(self, args):
            req_path = os.path.join('requirements', f'{args.env}.txt')
            yield self.local['pip']['-r', req_path]

    class Deploy(Local):
        """deploy app"""

        def prepare(self, args):
            yield self.local['eb']['deploy', args.env]

if __name__ == '__main__':
    main(Management)

Local command Management, above, defines no functionality of its own. As such, executing manage without arguments prints its autogenerated usage:

usage: manage [-h] [--tb] [-q] [-d] [-s] [--no-show]
              [-e {development,production}]
              {build,deploy} ...

Because Management extends Local, it inherits argumentation controlling whether standard output is printed and offering to run commands in “dry” mode. (Note, however, that it could have omitted these options by extending Command. Moreover, it may override class method base_parser().)

Management adds to the basic interface the optional argument --env. Most important, however, are the related, nested commands Build and Deploy, which define functionality via prepare. Neither nested command extends its subparser – though they could; but rather, they depend upon the common argumentation defined by Management.

Exploring the interface via --help tells us a great deal, for example manage -h:

usage: manage [-h] [--tb] [-q] [-d] [-s] [--no-show]
              [-e {development,production}]
              {build,deploy} ...

manage deployment

optional arguments:
  -h, --help            show this help message and exit
  --tb, --traceback     print error tracebacks
  -q, --quiet           do not print command output
  -d, --dry-run         do not execute commands, but print what they are
                        (unless --no-show is provided)
  -s, --show            print command expressions (by default not printed
                        unless dry-run)
  --no-show             do not print command expressions (by default not
                        printed unless dry-run)
  -e {development,production}, --env {development,production}
                        target environment

management commands:
  {build,deploy}        available commands
    build               build app
    deploy              deploy app

And manage deploy -h:

usage: manage deploy [-h]

deploy app

optional arguments:
  -h, --help  show this help message and exit

As such, a “dry run”:

manage -de production deploy

prints the following:

> /home/user/.local/bin/eb deploy production

and without the dry-run flag the above operating system command is executed.

Decorated commands

There is no artificial limit to the number of levels you may add to your command hierarchy. However, application interfaces are commonly “wider” than they are “deep”. For these reasons, as an alternative to class-nesting, the hierarchical relationship may be defined by class decorator.

Let’s define the executable file git with no particular purpose whatsoever:

#!/usr/bin/env python

from argcmdr import Command, RootCommand, main

class Git(RootCommand):
    """another stupid content tracker"""

    def __init__(self, parser):
        parser.add_argument(
            '-C',
            default='.',
            dest='path',
            help="run as if git was started in <path> instead of the current "
                 "working directory.",
        )

@Git.register
class Stash(Command):
    """stash the changes in a dirty working directory away"""

    def __call__(self, args):
        self['save'](args)

    class Save(Command):
        """save your local modifications to a new stash"""

        def __init__(self, parser):
            parser.add_argument(
                '-p', '--patch',
                dest='interactive',
                action='store_true',
                default=False,
                help="interactively select hunks from the diff between HEAD "
                     "and the working tree to be stashed",
            )

        def __call__(self, args):
            interactive = getattr(args, 'interactive', False)
            print("stash save", f"(interactive: {interactive})")

    class List(Command):
        """list the stashes that you currently have"""

        def __call__(self):
            print("stash list")

if __name__ == '__main__':
    main(Git)

We anticipate adding many subcommands to git beyond stash; and so, rather than nest all of these command classes under Git:

• we’ve defined Git as a RootCommand

• we’ve defined Stash at the module root

• we’ve decorated Stash with Git.register

The RootCommand functions identically to the Command; it only adds this ability to extend the listing of its subcommands by those registered via its decorator. (Notably, LocalRoot composes the functionaliy of Local and RootCommand via multiple inheritance.)

The stash command, on the other hand, contains the entirety of its hierarchical functionality, nesting its own subcommands list and save.

Walking the hierachy

Unlike the base command git in the example above, the command git stash – despite defining its own subcommands – also defines its own functionality, via __call__. This functionality, however, is merely a shortcut to the stash command save. Rather than repeat the definition of this functionality, Stash “walks” its hierarchy to access the instantiation of Save, and invokes this command by reference.

Much of argcmdr is defined at the class level, and as such many Command methods are classmethod. In the static or class context, we might walk the command hierarchy by reference, e.g. to Stash.Save; or, from a class method of Stash, as cls.Save. Moreover, Command defines the class-level “property” subcommands, which returns a list of Command classes immediately “under” it in the hierarchy.

The hierarchy of executable command objects, however, is instantiated at runtime and cached within the Command instance. To facilitate navigation of this hierarchy, the Command object is itself subscriptable. Look-up keys may be:

• strings – descend the hierarchy to the named command

• negative integers – ascend the hierarchy this many levels

• a sequence combining the above – to combine “steps” into a single action

In the above example, Stash may have (redundantly) accessed Save with the look-up key:

(-1, 'stash', 'save')

that is with the full expression:

self[-1, 'stash', 'save']

(The single key 'save', however, was far more to the point.)

Because command look-ups are relative to the current command, Command also offers the property root, which returns the base command. As such, our redundant expression could be rewritten:

self.root['stash', 'save']

Bootstrapping

To ensure that such a friendly – and relatively high-level – project requirement as argcmdr is satisfied, consider the expressly low-level utility install-cli, with which to guide contributors through the process of provisioning your project’s most basic requirements.