Project config files
Project description
What is this?
A config system that doesn't waste your time
- per-machine settings that stay in sync
- a consistent way to handle filepaths (stop hardcoding filepaths as python strings!)
- hierarchical, with inheritable groups of settings (profiles)
- default works along side
argparse
, but also can just replace it entirely for rapid development - all values in the hierarchy can be overridden with CLI args
- select multiple profiles from CLI (ex: GPU & DEV or UNIX & GPU & PROD)
- can combine/import multiple config files
How do I use this?
pip install quik-config
In a config.py
:
from quik_config import find_and_load
info = find_and_load(
"info.yaml", # walks up folders until it finds a file with this name
cd_to_filepath=True, # helpful if using relative paths
fully_parse_args=True, # if you already have argparse, use parse_args=True instead
show_help_for_no_args=False, # change if you want
)
print(info.config) # dictionary
Create info.yaml
with a structure like this:
# names in parentheses are special, all other names are not!
# (e.g. add/extend this with any custom fields)
(project):
# the local_data file will be auto-generated
# (its for machine-specific data)
# so probably git-ignore whatever path you pick
(local_data): ./local_data.ignore.yaml
# example profiles
(profiles):
(default):
blah: "blah blah blah"
mode: development # or production. Same thing really
has_gpu: maybe
constants:
pi: 3 # its 'bout 3
PROFILE1:
constants:
e: 2.7182818285
PROD:
mode: production
constants:
pi: 3.1415926536
problems: true
Then run it:
python ./config.py
Which will print out this config:
{
"blah": "blah blah blah", # from (default)
"mode": "development", # from (default)
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3, # from (default)
},
}
Features
Builtin Help
python ./config.py --help --profiles
available profiles:
- DEV
- GPU
- PROD
as cli argument:
-- --profiles='["DEV"]'
-- --profiles='["GPU"]'
-- --profiles='["PROD"]'
---------------------------------------------------------------------------------
QUIK CONFIG HELP
---------------------------------------------------------------------------------
open the file below and look for "(profiles)" for more information:
$PWD/info.yaml
examples:
python3 ./ur_file.py -- --help --profiles
python3 ./ur_file.py -- --help key1
python3 ./ur_file.py -- --help key1:subKey
python3 ./ur_file.py -- --help key1:subKey key2
python3 ./ur_file.py -- --profiles='[YOUR_PROFILE, YOUR_OTHER_PROFILE]'
python3 ./ur_file.py -- thing1:"Im a new value" part2:"10000"
python3 ./ur_file.py -- thing1:"I : cause errors" part2:10000
python3 ./ur_file.py -- 'thing1:"I : dont cause errors" part2:10000
python3 ./ur_file.py -- 'thing1:["Im in a list"]'
python3 ./ur_file.py -- 'thing1:part_A:"Im nested"'
python3 ./ur_file.py "I get sent to ./ur_file.py" -- part2:"new value"
python3 ./ur_file.py "I get ignored" "me too" -- part2:10000
how it works:
- the "--" is a required argument, quik config only looks after the --
- given "thing1:10", "thing1" is the key, "10" is the value
- All values are parsed as json/yaml
- "true" is boolean true
- "10" is a number
- '"10"' is a string (JSON escaping)
- '"10\n"' is a string with a newline
- '[10,11,hello]' is a list with two numbers and an unquoted string
- '{"thing": 10}' is a map/object
- "blah blah" is an un-quoted string with a space. Yes its valid YAML
- multiline values are valid, you can dump an whole JSON doc as 1 arg
- "thing1:10" overrides the "thing1" in the (profiles) of the info.yaml
- "thing:subThing:10" is shorthand, 10 is the value, the others are keys
it will only override the subThing (and will create it if necessary)
- '{"thing": {"subThing":10} }' is long-hand for "thing:subThing:10"
- '"thing:subThing":10' will currently not work for shorthand (parse error)
options:
--help
--profiles
---------------------------------------------------------------------------------
your default top-level keys:
- mode
- has_gpu
- constants
your local defaults file:
./local_data.ignore.yaml
your default profiles:
- DEV
---------------------------------------------------------------------------------
Select Profiles from CLI
python ./config.py @PROFILE1
prints:
{
"blah": "blah blah blah", # from (default)
"mode": "development", # from (default)
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3.1415926536, # from (default)
"e": 2.7182818285, # from PROFILE1
},
}
python ./config.py @PROFILE1 @PROD
prints:
{
"blah": "blah blah blah", # from (default)
"mode": "production", # from PROD
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3.1415926536, # from (default)
"e": 2.7182818285, # from PROFILE1
"problems": True, # from PROD
},
}
Override Values from CLI
python ./config.py @PROFILE1 mode:custom constants:problems:99
prints:
{
"blah": "blah blah blah", # from (default)
"mode": "custom", # from CLI
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3.1415926536, # from (default)
"e": 2.7182818285, # from PROFILE1
"problems": 99, # from CLI
},
}
Again but with really complicated arguments:
(each argument is parsed as yaml)
python ./run.py arg1 -- mode:my_custom_mode 'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'
prints:
config: {
"mode": "my_custom_mode",
"has_gpu": False,
"constants": {
"pi": 3.1415926,
"tau": 6.2831853072,
"reserved_letters": ["C", "K", "i", ],
},
}
unused_args: ["arg1"]
Working Alongside Argparse (quick)
Remove fully_parse_args
and replace it with just parse_args
info = find_and_load(
"info.yaml",
parse_args=True, # <- will only parse after --
)
Everthing in the CLI is the same, but it waits for --
For example:
# quik_config ignores arg1 --arg2 arg3, so argparse can do its thing with them
python ./config.py arg1 --arg2 arg3 -- @PROD
Working Alongside Argparse (advanced)
Arguments can simply be passed as a list of strings, which can be useful for running many combinations of configs.
info = find_and_load(
"info.yaml",
args=[ "@PROD" ],
)
Relative and absolute paths
Add them to the info.yaml
(project):
(local_data): ./local_data.ignore.yaml
# filepaths (relative to location of info.yaml)
(path_to):
this_file: "./info.yaml"
blah_file: "./data/results.txt"
# example profiles
(profiles):
(default):
blah: "blah blah blah"
Access them in python
info = find_and_load("info.yaml")
info.path_to.blah_file
info.absolute_path_to.blah_file # nice when then PWD != folder of the info file
Import other yaml files
FIXME
Different Profiles For Different Machines
Lets say you've several machines and an info.yaml like this:
(project):
(profiles):
DEV:
cores: 1
database_ip: 192.168.10.10
mode: dev
LAPTOP:
cores: 2
DESKTOP:
cores: 8
UNIX:
line_endings: "\n"
WINDOWS:
line_endings: "\r\n"
PROD:
database_ip: 183.177.10.83
mode: prod
cores: 32
And lets say you have a config.py
like this:
from quik_config import find_and_load
info = find_and_load(
"info.yaml",
defaults_for_local_data=["DEV", ],
# if the ./local_data.ignore.yaml doesnt exist,
# => create it and add DEV as the default no-argument choice
)
Run the code once to get a ./local_data.ignore.yaml
file.
Each machine gets to pick the profiles it defaults to.
So, on your Macbook you can edit the ./local_data.ignore.yaml
to include something like the following:
(selected_profiles):
- LAPTOP # the cores:2 will be used (instead of cores:1 from DEV)
- UNIX # because LAPTOP is higher in the list than DEV
- DEV
On your Windows laptop you can edit it and put:
(selected_profiles):
- LAPTOP
- WINDOWS
- DEV
Command Line Arguments
If you have run.py
like this:
from quik_config import find_and_load
info = find_and_load("info.yaml", parse_args=True)
print("config:", info.config )
print("unused_args:", info.unused_args)
#
# call some other function you've got
#
#from your_code import run
#run(*info.unused_args)
Example 0
Using the python file and config file above
python ./run.py
Running that will output:
config: {
"mode": "development",
"has_gpu": False,
"constants": {
"pi": 3
}
}
unused_args: []
Example 1
Show help. This output can be overridden in the info.yaml by setting (help):
under the (project):
key.
python ./run.py -- --help
Note the --
is needed in front of the help.
You can also add show_help_for_no_args=True
if you want that behavior.
Ex:
from quik_config import find_and_load
info = find_and_load(
"info.yaml",
show_help_for_no_args=True
parse_args=True,
)
Example 2
Again but selecting some profiles
python ./run.py arg1 -- --profiles='[PROD]'
# or
python ./run.py arg1 -- @PROD
Output:
config: {
"mode": "production",
"has_gpu": False,
"constants": {
"pi": 3.1415926536,
"problems": True,
},
}
unused_args: ["arg1"]
Example 3
Again but with custom arguments:
python ./run.py arg1 -- mode:my_custom_mode constants:tau:6.2831853072
config: {
"mode": "my_custom_mode",
"has_gpu": False,
"constants": {
"pi": 3,
"tau": 6.2831853072,
},
}
unused_args: ["arg1"]
Example 4
Again but with really complicated arguments:
(each argument is parsed as yaml)
python ./run.py arg1 -- mode:my_custom_mode 'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'
prints:
config: {
"mode": "my_custom_mode",
"has_gpu": False,
"constants": {
"pi": 3.1415926,
"tau": 6.2831853072,
"reserved_letters": ["C", "K", "i", ],
},
}
unused_args: ["arg1"]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for quik_config-1.7.9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35314f306c825d68cec2dae2ca09d890d91943d1ea1c8c444fcf69d818c9771d |
|
MD5 | 45416773d7837d56a43a4d87d789e1b4 |
|
BLAKE2b-256 | 894cbf24d95648851d2edb189062094ec94a6a9fb0fcf3bc3bbc123a2619796f |