Skip to main content

a noble script configgens the smallest app

Project description

description

configgen is a tool and library for generating JSON files given a script and set of options. [1] it’s useful when you have multiple configurations that share some information, but change in different ways in different contexts. define your config info in one place and generate a version specific to your app, cron jobs, or developer utilities, in development, staging, or production, for your dev virtual machine or server environments, in any combination.

this is not intended to replace a full template language.

install it with pip:

pip install configgen

the PyPI page is: https://pypi.python.org/pypi/configgen

the source code lives at bitbucket: https://bitbucket.org/dystedium/configgen

command line usage

usage: configgen [-h] [--config CONFIG] [--module] [--option OPTION]
                 [--printconfig] [--squishee] [--no-replace] [--verbose]
                 [--version]
                 input [output]

generate a config file given a python module and set of options

positional arguments:
  input                 where to get input class structure, by default this
                        expects a path to a file, also see help text for the
                        --module option
  output                optional, file to which output is written, no output
                        by default (see --printconfig)

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
                        if there are multiple Config instances in the class
                        structure, the value of this parameter specifies which
                        one to output
  --module, -m          interpret the input parameter as a module import with
                        dot notation, e.g. my.config.module, must be in python
                        path
  --option OPTION, -o OPTION
                        an option string that will be used to choose which
                        value to output for a given field name (may be passed
                        multiple times)
  --printconfig, -p     print the converted config object to the console
  --squishee, -s        make the converted config output less human-readable
  --no-replace, -n      do not do internal string replacement in the output
  --verbose, -v         increase console output
  --version             show program's version number and exit

command line examples

disclaimer: all of the below examples are for illustrative purposes only. this is not a recommendation to do things like store production usernames and passwords in plaintext files.

basic usage

if hello.py contained the following:

from configgen import Config

cfg = Config(
    _object='world', # keys starting with "_" are not output
    _object__frog='ma baby', # options are separated from the key by "__"
    statement='hello, {_object}', # string replacement
    statement__frog__crowd='ribbit' # if multiple options are present, all must match
    )

configgen --printconfig hello.py would output:

{
    "statement": "hello, world"
}

configgen --printconfig --option frog hello.py would output:

{
    "statement": "hello, ma baby"
}

put the frog in front of a crowd with configgen -p -o frog -o crowd hello.py:

{
    "statement": "ribbit"
}

inheritance and string replacement

contents of inherit.py:

from configgen import Config, KeyValue, make_multi_key as MK

# option strings
production = 'production'

# define these field name strings in one place
projectName = 'projectName'
_db = '_db'
dbType = 'dbType'
user = 'user'
password = 'password'
host = 'host'

# KeyValue objects have most of the same abilities as Config
site1=KeyValue(
    projectName='site1',
    _db={ # KeyValue values aren't restricted to base types
        dbType:'mysql',
        user:'testuser',
        password:'testpass', # use your best judgment
        host:'localserver'
        },
    braces='{{}}' # actual { or } character escaping
    )

# since this is a normal python script, the usual syntax rules and patterns
# apply - parameters to KeyValue can be passed in a keyword argument dictionary
# using the ** syntax.  make_multi_key(), abbreviated here as MK(),
# is a convenience function for combining keys and options - because it just
# returns the concatenated string, it can't be used when using the
# KeyValue(parameter=value) style declaration.
site2 = KeyValue(**{
    projectName:'site2',
    _db:KeyValue(**{ # it is possible to nest KeyValue instances
        dbType:'postgresql',
        user:'testuser',
        password:'testpass',
        MK(user, production):'realuser',
        MK(password, production):'realpass', # use your best judgment
        host:'localserver',
        MK(host, production):'cloudserver'
        })
    })

# note that the string replacement here references fields inherited by
# the parent Config object.  also note the use of attribute-style access
# (via the "." operator) to reference fields in dictionaries/KeyValues
siteCfg = KeyValue(
    rootPath='~/www/{projectName}',
    databaseUrl='{_db.dbType}://{_db.user}:{_db.password}@{_db.host}'
    )

cfgSite1 = Config(inherits=site1, site=siteCfg)
cfgSite2 = Config(inherits=site2, site=siteCfg)

if multiple Config objects are defined in a file, the one to output must be specified with the –config or -c option.

configgen --printconfig --config cfgSite1 inherit.py would output:

{
    "projectName": "site1",
    "site": {
        "databaseUrl": "mysql://testuser:testpass@localserver",
        "rootPath": "~/www/site1"
    },
    "braces": "{}"
}

configgen -p -c cfgSite2 inherit.py would output:

{
    "projectName": "site2",
    "site": {
        "databaseUrl": "postgresql://testuser:testpass@localserver",
        "rootPath": "~/www/site2"
    }
}

configgen -p -c cfgSite2 -o production inherit.py would output:

{
    "projectName": "site2",
    "site": {
        "databaseUrl": "postgresql://realuser:realpass@cloudserver",
        "rootPath": "~/www/site2"
    }
}

notes

there is currently no simple way to emit a key/value pair only for a specific option set. there is a clunky way to do this:

common = KeyValue(key1=value1, key2=value2)
extraField = KeyValue(inherits=common, key3=value3)
output = KeyValue(fields=common, fields__addkey3=extraField)

the JSON generated by output will include key3 only when the option string ‘addkey3’ is present.

library examples

the configgen package can also be used as a library in a larger program. the source of configgen.main is an example of basic usage.

operation

KeyValue string replacement

when resolving named references to other parts of the KeyValue structure during string replacement, the following steps are taken:

  1. define the KeyValue instance that contains the string being resolved as “nearest”

  2. start searching at the following KeyValue instances for the entire reference, continuing to the next one if the reference cannot be resolved:

    1. the nearest instance

    2. the nearest instance’s inherited fields, if present

    3. the outermost KeyValue instance (usually the Config instance)

    4. the outermost instance’s inherited fields, if present

  3. while searching, if a named field is found in a KeyValue instance, define that instance as “nearest” (overwriting any previous value)

  4. if not found, emit an error, otherwise, if the replaced value is a string, use the new nearest instance and begin a new string replacement operation (allowing replaced strings to contain string replacement directives themselves)

TODO: create examples, for now, the test.py script has some barely-commented examples

building MultiValue sets

when adding multiple keys with the same base name and options separated by “__”, those keys are grouped and become a MultiValue. only one value will be emitted into the JSON, selected based on the set of options provided to the conversion function. for now, a MultiValue set must contain a default value, which is output when no options are provided or there is no matching option set. any combination of the default value and keys for specific option sets may be split across a KeyValue instance and the instance it inherits from. some examples:

from configgen import Config, KeyValue, make_multi_key as MK

# keys
multiValue0 = 'multiValue0'
multiValue1 = 'multiValue1'
multiValue2 = 'multiValue2'

# options
one = 'one'
two = 'two'
three = 'three'

# the make_multi_key() convenience function (imported here as "MK") helps
# define MultiValues in a KeyValue by joining all the string parameters with
# the option separator.
defaults = KeyValue(**{
    multiValue0:'multiValue0 inherited, default',
    MK(multiValue0, one):'multiValue0 inherited, options: one',
    })

# MultiValue keys can come from the inherited object.  when this happens,
# all of the relevant keys are copied to the inheriting KeyValue instance
# during construction and combined into a local MultiValue instance, so
# overrides from one KeyValue instance will not affect another instance
# that inherits from the same object.  if a key is defined with the same
# option set in both the inheriting and inherited KeyValue instances, the
# one in the inheriting instance is used.
# note that section0 does not define a default for multiValue0 - it's
# inherited from the defaults object.
section0 = KeyValue(inherits=defaults, **{
    MK(multiValue0, one):'multiValue0 section0, options: one',
    MK(multiValue0, two, three):'multiValue0 section0, options: two, three',
    multiValue1:'multiValue1 section0, default',
    MK(multiValue1, three):'multiValue1 section0, options: three',
    #MK(multiValue2, two):'defining this by itself is an error (no default value)'
    })

# section1 overrides only the default value for multiValue0.  section1.multiValue2
# shows that the value type need not be the same across all of a MultiValue.
# section1.multiValue2 also demonstrates a case where multiple values have an
# option set of the same cardinality.  as long as the options "one" and "two"
# are mutually exclusive, this will not cause an error.
section1 = KeyValue(inherits=defaults, **{
    multiValue0:'multiValue0 section1, default',
    multiValue1:'multiValue1 section1, simple value',
    multiValue2:None,
    MK(multiValue2, one):{'info':'options: one'},
    MK(multiValue2, two):{'info':'options: two'},
    })

cfg = Config(section0=section0, section1=section1)

selecting a specific value in a MultiValue

from the command line, specify the --option/-o option one or more times to build a set of options to use when generating JSON output (the generation set). when using the configgen library directly, a python frozenset of options should be passed to the Config.convertToJson() function as the generation set. configgen uses the following criteria when selecting a value, in order:

  1. a value whose option set matches the generation set exactly

  2. the value whose option set is the largest common subset of the generation set (ignoring any option sets that are disjoint)

  3. the default value is used if no value’s option set is a subset of the generation set

note that because the values are selected based on set intersections, neither the order of the options specified in the MultiValue key nor the command line affect the selection process. if there is a tie for the largest common option subset between values, a configgen.LookupError may be raised. this error can be avoided in at least two ways:

  • the generation set matches one of the values’ option sets exactly

  • one or more of the options in the tied values’ option sets are mutually exclusive

for example, given a MultiValue containing values with these option sets (excluding the default value):

  1. (“server”, “logToFile”, “production”)

  2. (“server”, “verbose”, “staging”)

  3. (“server”, “logToFile”, “staging”)

here are example cases:

  • (“server”, “logToFile”, “production”): exact match for value A, use that one

  • (“server”, “verbose”, “logToFile”, “staging”): error, values B and C both match 3 items in the generation set

  • as long as “production” and “staging” never appear in the same generation set alongside “server” and “logToFile”, values A and C will not conflict

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

configgen-1.0.0.tar.gz (13.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page