Schedulers#
PyTorch Network Loader provides the BaseScheduler class along with some common schedulers
for PyTorch safe weights and dynamic parameter updates during training, such as learning rate scheduling.
Custom Schedulers#
NetLoader provides two base scheduler classes:
BaseFuncScheduler- Schedulers that update parameters based on a function of the current step.BaseScheduler- Schedulers with more complex behavior that cannot be expressed by a function.
The BaseScheduler class has the following methods:
- abstractmethod BaseScheduler.step(epoch)[source]
Steps the scheduler.
- BaseScheduler.get_step(epoch)[source]
Returns the current value of the scheduler for the given epoch.
Along with the magic method:
- abstractmethod BaseScheduler.__call__()[source]
Returns the current value of the scheduler.
Schedulers are updated via the step method with the current epoch argument.
The current value can be obtained from calling the scheduler object.
The method get_step combines the functionality of both of these methods by updating the scheduler and returning the
current value for the given epoch.
All custom schedulers must implement the step method, along with the __call__, __getstate__, &
__setstate__ methods if the custom scheduler inherits from BaseScheduler or extra
functionality is required.
Custom Schedulers from BaseFuncScheduler#
The BaseFuncScheduler class provides the base functionality for all schedulers that
update parameters based on a function of the current step.
When creating a custom scheduler from BaseFuncScheduler, the only requirement is to
define the step method, which should define the function.
If extra attributes are required, then the __getstate__ and __setstate__ methods should also be defined with the
additional attributes.
The BaseFuncScheduler class is initialised with the following parameters:
- Network.__init__(name, config, in_shape, out_shape, *, suppress_warning=False, root='', defaults=None)[source]
- Parameters:
- name
str Name of the network configuration file
- config
str|Config Path to the network config directory or configuration dictionary
- in_shape
list[int] |list[list[int]] |tuple[int, …] Shape of the input tensor(s), excluding batch size
- out_shape
list[int] Shape of the output tensor, excluding batch size
- suppress_warningbool,
Optional If output shape mismatch warning should be suppressed, default = False
- root
str,Optional Root directory to prepend to config file path
- defaults
dict[str,Any] |NoneType,Optional Default values for the parameters for each type of layer
- name
The attributes of the BaseFuncScheduler object include:
- class BaseFuncScheduler(max_epochs, *, warmup_epochs=0, final_value=1, initial_value=0)[source]
Bases:
BaseScheduler,ABCAbstract base class for simple function schedulers.
Example implementing a linear scheduler from BaseFuncScheduler:
from netloader.schedulers import BaseFuncScheduler
class LinearScheduler(BaseFuncScheduler):
"""
Linear scheduler.
Attributes
----------
max_epochs : int
Maximum number of epochs
warmup : int
Number of warmup epochs
initial_val : float
Initial value of the scheduler
final_val : float
Final value of the scheduler
"""
def step(self, epoch: float) -> None:
progress: float
if epoch >= self.max_epochs:
self._value = self.final_val
elif epoch <= self.warmup:
self._value = self.initial_val
else:
progress = (epoch - self.warmup) / (self.max_epochs - self.warmup)
self._value = progress * (self.final_val - self.initial_val) + self.initial_val
Custom Schedulers from BaseScheduler#
If your custom scheduler requires more complex functionality, then you can inherit from
BaseScheduler and implement the step, __call__, __getstate__, &
__setstate__ methods.
Example implementing a scheduler from BaseScheduler that updates the parameter every
\(N\) epochs.
The scheduler multiplies the initial value by a factor of \(\\gamma\) every \(N\) epochs for a total of
\(M\) updates.
class StepScheduler(BaseScheduler):
"""
Step scheduler.
Attributes
----------
max_steps : int
Maximum number of steps
step_size : int
Number of epochs between each step
gamma : float
Multiplicative factor of the step
initial_val : float
Initial value of the scheduler
"""
def __init__(
self,
max_steps: int,
step_size: int,
gamma: float,
initial_value: float = 1) -> None:
"""
Parameters
----------
max_steps : int
Maximum number of steps
step_size : int
Number of epochs between each step
gamma : float
Multiplicative factor of the step
initial_value : float
Initial value of the scheduler, default = 1
"""
super().__init__()
self.max_steps: int = max_steps
self.step_size: int = step_size
self.gamma: float = gamma
self.initial_val: float = initial_value
self._value: float = self.initial_val
def __call__(self) -> float:
return self._value
def __getstate__(self) -> dict[str, Any]:
# Get the scheduler parameters for saving
return {
'max_steps': self.max_steps,
'step_size': self.step_size,
'gamma': self.gamma,
'initial_val': self.initial_val,
}
def __setstate__(self, state: dict[str, Any]) -> None:
# Set the scheduler parameters for loading
self.__dict__.update(state)
self._value = self.initial_val
def step(self, epoch: int | float) -> None:
# Number of steps given the current epoch
steps: int = min(max(int((epoch - self.warmup) // self.step_size), 0), self.max_steps)
if not steps:
self._value = self.initial_val
return
self._value = self.initial_val * (self.gamma ** steps)
Built-In Schedulers#
All schedulers currently supported by NetLoader are listed below with a brief description of their functionality and
their initialisation arguments, excluding general initialisation arguments defined in
BaseScheduler.
For more details on each loss function, see API documentation.
BaseFuncScheduler : Abstract base class for simple function schedulers.
max_epochs:int– Maximum number of epochswarmup_epochs:int=0– Number of warmup epochsfinal_value:float=1– Maximum value of the schedulerinitial_value:float=0– Minimum value of the scheduler
BaseScheduler : Abstract base class for schedulers.
LinearScheduler : Linear scheduler.
max_epochs:int– Maximum number of epochswarmup_epochs:int=0– Number of warmup epochsfinal_value:float=1– Maximum value of the schedulerinitial_value:float=0– Minimum value of the scheduler
SigmoidScheduler : Sigmoid scheduler.
max_epochs:int– Maximum number of epochswarmup_epochs:int=0– Number of warmup epochsfinal_value:float=1– Maximum value of the schedulerinitial_value:float=0– Minimum value of the schedulercompression:float=0– Compression factor for the tanh function, higher values result in a steeper curve, default = 10
StepScheduler : Step scheduler.
warmup:int– Number of warmup epochsmax_steps:int– Maximum number of stepsstep_size:int– Number of epochs between each stepgamma:float– Multiplicative factor of the stepinitial_value:float=1– Initial value of the scheduler
TanhScheduler : Tanh scheduler.
max_epochs:int– Maximum number of epochswarmup_epochs:int=0– Number of warmup epochsfinal_value:float=1– Maximum value of the schedulerinitial_value:float=0– Minimum value of the schedulercompression:float=0– Compression factor for the sigmoid function, higher values result in a steeper curve, default = 5