No project description provided
Project description
Config
This package allows you to configure functions explicitly and safely. You will be able to create an intuitive type-checked configuration file that directly sets function arguments, globally.
This is a lightweight package with only two widely used dependencies and only a couple hundred line of code.
Contents
Install
Make sure you have Python 3, then you can install pythonic-config
using pip
:
pip install pythonic-config
Install the latest code via:
pip install git+https://github.com/PetrochukM/Config.git
Usage 🤗
Configuring a function
Any function can be configured, and then used anywhere, see below:
import config
# Define function
def do_something_cool(how_many_times: int):
pass
# Configure function
config.add({do_something_cool: config.Args(how_many_times=5)})
# Use the configured function anywhere! 🎉
do_something_cool(how_many_times=config.get())
This approach is simple but powerful. Now, each configuration can be directly attributed to a documented function argument.
Furthermore, config
incorporates typeguard
💂♀️ so every configuration is type checked at runtime.
Writing a configuration file
The simple example above can be extended to create a configuration file, for example:
import config
import data
import train
config.add({
data.get_data: config.Args(
train_data_path="url_lists/all_train.txt",
val_data_path="url_lists/all_val.txt"
),
data.dataset_reader: config.Args(
type_="cnn_dm",
source_max_tokens=1022,
target_max_tokens=54,
),
train.make_model: config.Args(type_="bart"),
train.Trainer.make_optimizer: config.Args(
type_="huggingface_adamw",
lr=3e-5,
correct_bias=True
)
train.Trainer.__init__: config.Args(
num_epochs=3,
learning_rate_scheduler="polynomial_decay",
grad_norm=1.0,
)
})
With this approach, this configuration file will make it clear which (hyper)parameters are set and where. This improves overall readability of the configuration file.
🐍 Last but not least, the configuration file is written in Python, you can use variables, lambdas, etc to further modularize.
Configuring via the command line
In case you want to change one variable at a time, this package supports configuration from the command line, for example:
python example.py --sorted='Args(reverse=True)'
import sys
import config
config.add(config.parse_cli_args(sys.argv[1:]))
Logging the configuration
Lastly, it's useful to track the configuration file by logging it. This package supports that
via config.log
. In the example below, we log the configuration to
Comet.
from comet_ml import Experiment
import config
experiment = Experiment()
experiment.log_parameters(config.log())
Advanced: Sharing configurations between processes
In multiprocessing, it may be useful to share the configuration file between processes. In this case, the configuration can be exported to another process and then subsequently imported, see below:
from multiprocessing import Process
import config
def handler(configs: config.Config):
config.add(configs)
if __name__ == "__main__":
process = Process(target=handler, args=(config.export(),))
process.start()
process.join()
Advanced: Ensuring the configuration is used
In a large code base, it might be hard to tell if the configuration has been set for every function
call. In this case, we've exposed config.trace
which can double check every function call
against the configuration, see below:
import sys
import config
def configured(a=111):
pass
sys.settrace(config.trace)
config.add({configured: config.Args(a=1)})
configured() # `config.trace` issues a WARNING!
configured(a=config.get())
We also have another option for faster tracing with enable_fast_trace
. Instead of a system wide
trace, this will inject new code into the __code__
of every function in your configuration for
tracing. This has much lower overhead; however, it is still in beta due to the number of edge cases.
How does this work?
Our approach is simple, config
maintains a global configuration mapping each function to its
associated arguments. When a user calls config.get
, it'll attempt to parse the code to determine
the calling function and associated argument. In most cases, this will be successful and it will
fetch the appropriate value. If it's not successful, it'll raise
an error that'll help you fix
the issue.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pythonic_config-0.5.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3a009aec784d1d1225680fe68f52ee26fb412ecf45f06c806bf3962f79b1df5 |
|
MD5 | f3ed10b53180a1b1d9e68f88de16f83a |
|
BLAKE2b-256 | 4c38130e588fb1791f0bd19e74345c8fe4233d9913a9509263c532d3f14ee60f |