optimizer & lr scheduler implementations in PyTorch with clean-code, strict types. Also, including useful optimization ideas.
Project description
Build |
|
Quality |
|
Package |
|
Status |
|
License |
Getting Started
For more, see the documentation.
Installation
$ pip3 install -U pytorch-optimizer
If there’s a version issue when installing the package, try with –no-deps option.
$ pip3 install -U --no-deps pytorch-optimizer
Simple Usage
from pytorch_optimizer import AdamP model = YourModel() optimizer = AdamP(model.parameters()) # or you can use optimizer loader, simply passing a name of the optimizer. from pytorch_optimizer import load_optimizer model = YourModel() opt = load_optimizer(optimizer='adamp') optimizer = opt(model.parameters())
Also, you can load the optimizer via torch.hub
import torch
model = YourModel()
opt = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
optimizer = opt(model.parameters())
If you want to build the optimizer with parameters & configs, there’s create_optimizer() API.
from pytorch_optimizer import create_optimizer
optimizer = create_optimizer(
model,
'adamp',
lr=1e-3,
weight_decay=1e-3,
use_gc=True,
use_lookahead=True,
)
Supported Optimizers
You can check the supported optimizers & lr schedulers.
from pytorch_optimizer import get_supported_optimizers, get_supported_lr_schedulers supported_optimizers = get_supported_optimizers() supported_lr_schedulers = get_supported_lr_schedulers()
Optimizer |
Description |
Official Code |
Paper |
|---|---|---|---|
AdaBelief |
Adapting Step-sizes by the Belief in Observed Gradients |
||
AdaBound |
Adaptive Gradient Methods with Dynamic Bound of Learning Rate |
||
AdaHessian |
An Adaptive Second Order Optimizer for Machine Learning |
||
AdamD |
Improved bias-correction in Adam |
||
AdamP |
Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights |
||
diffGrad |
An Optimization Method for Convolutional Neural Networks |
||
MADGRAD |
A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic |
||
RAdam |
On the Variance of the Adaptive Learning Rate and Beyond |
||
Ranger |
a synergistic optimizer combining RAdam and LookAhead, and now GC in one optimizer |
||
Ranger21 |
a synergistic deep learning optimizer |
||
Lamb |
Large Batch Optimization for Deep Learning |
||
Shampoo |
Preconditioned Stochastic Tensor Optimization |
||
Nero |
Learning by Turning: Neural Architecture Aware Optimisation |
||
Adan |
Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models |
||
Adai |
Disentangling the Effects of Adaptive Learning Rate and Momentum |
||
GSAM |
Surrogate Gap Guided Sharpness-Aware Minimization |
||
D-Adaptation |
Learning-Rate-Free Learning by D-Adaptation |
||
AdaFactor |
Adaptive Learning Rates with Sublinear Memory Cost |
||
Apollo |
An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization |
||
NovoGrad |
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks |
||
Lion |
Symbolic Discovery of Optimization Algorithms |
||
Ali-G |
Adaptive Learning Rates for Interpolation with Gradients |
||
SM3 |
Memory-Efficient Adaptive Optimization |
Useful Resources
Several optimization ideas to regularize & stabilize the training. Most of the ideas are applied in Ranger21 optimizer.
Also, most of the captures are taken from Ranger21 paper.
Adaptive Gradient Clipping
Gradient Centralization
|
Gradient Centralization (GC) operates directly on gradients by centralizing the gradient to have zero mean.
Softplus Transformation
By running the final variance denom through the softplus function, it lifts extremely tiny values to keep them viable.
paper : arXiv
Gradient Normalization
Norm Loss
|
paper : arXiv
Positive-Negative Momentum
|
Linear learning rate warmup
|
paper : arXiv
Stable weight decay
|
Explore-exploit learning rate schedule
|
Lookahead
Chebyshev learning rate schedule
Acceleration via Fractal Learning Rate Schedules
paper : arXiv
(Adaptive) Sharpness-Aware Minimization
On the Convergence of Adam and Beyond
paper : paper
Gradient Surgery for Multi-Task Learning
paper : paper
Citations
Explore-Exploit Learning Rate Schedule
On the adequacy of untuned warmup for adaptive optimization
Stable weight decay regularization
Adaptive Sharpness-aware minimization
On the Convergence of Adam and Beyond
Citation
Please cite original authors of optimization algorithms. If you use this software, please cite it as below. Or you can get from “cite this repository” button.
@software{Kim_pytorch_optimizer_Bunch_of_2022,
author = {Kim, Hyeongchan},
month = {1},
title = {{pytorch_optimizer: optimizer & lr scheduler implementations in PyTorch}},
version = {1.0.0},
year = {2022}
}
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pytorch_optimizer-2.6.0-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7d617627cf90ff011cc2c775c36b6e5d2c6d50a756638bb87cf769b1e35b5b4f |
|
| MD5 | 90033bbf78730e2eb611f20acea99c77 |
|
| BLAKE2b-256 | ea4b2f8774abd6b13de61f3d0443a9383953b2def4aa6c20d5d641f883953379 |
