Layers#

PyTorch Network Loader provides several built-in layers that can be used to construct neural networks. They are built upon BaseLayer and utilise Shapes for layer output shape tracking and input/output shape flexibility, alongside PyTorch safe weight saving.

See the configuration files under network_configs for examples on how to use layers in .json files. See models under netloader.models for examples on how to use layers in Python.

Layer Shape Compatibility#

Linear layers take inputs of \((N,\ldots,L)\) where \(N\) is the batch size and \(L\) is the length of the input. Recurrent layers require an input shape of \((N,C,L)\), where \(C\) is the number of channels/sequence length. Some layers, such as convolutional, require dimension \(C\), but can take 1D, 2D, & 3D inputs, so the inputs would have shapes \((N,C,L)\), \((N,C,H,W)\), or \((N,C,D,H,W)\), respectively, where \(D\) is the depth, \(H\) is the height, \(W\) is the width, and \(L\) is the length for 1D data.

The Reshape layer can be used to change the shape of the inputs for compatibility between the layers.

Extra Layer Features#

Features of `BaseLayer`#

All layers inherit from BaseLayer which has the signature:

Network.__init__(name, config, in_shape, out_shape, *, suppress_warning=False, root='', defaults=None)[source]

Parameters:

namestr: Name of the network configuration file
configstr | Config: Path to the network config directory or configuration dictionary
in_shapelist[int] | list[list[int]] | tuple[int, …]: Shape of the input tensor(s), excluding batch size
out_shapelist[int]: Shape of the output tensor, excluding batch size
suppress_warningbool, Optional: If output shape mismatch warning should be suppressed, default = False
rootstr, Optional: Root directory to prepend to config file path
defaultsdict[str, Any] | NoneType, Optional: Default values for the parameters for each type of layer

All layers can pass arguments to BaseLayer through their **kwargs argument. If an argument is passed to BaseLayer that is not recognised, it will log a warning and ignore the argument.

The BaseLayer class has the following attributes that are available to all layers:

class BaseLayer(*, idx=0, group=0, version='', description='', **kwargs)[source]

Bases: Module, ABC

Base layer for other layers to inherit.

Attributes:

groupint: Layer group, if 0 it will always be used, else it will only be used if its group matches the Networks
descriptionstr: Description of the layer

The group argument means that the layer will only be active if the network attribute group is equal to the layer’s group. This is most useful if the network has multiple heads that get changed during training. Skip layers take the output from a specified layer so should be used between groups so that all group paths have the same input. See layer_examples.json for an example.

The BaseLayer class has the following methods:

abstractmethod BaseLayer.forward(x, *_, **__)[source]

Forward pass for child layer ti implement.

Parameters:

xAny: Layer input

Returns:

Any: Layer output

BaseLayer.to(*args, **kwargs)[source]: See torch.nn.Module.to().

Along with the magic methods:

BaseLayer.__getstate__()[source]

Gets the state of the layer for saving.

Returns:

dict[str, Any]: State of the layer

You can also implement the extra_repr method to add extra information to the layer’s string representation. If inheriting from either netloader.layers.base.BaseLayer or netloader.layers.base.BaseSingleLayer, then the extra_repr method is not implemented, so the custom extra_repr should not call super, while for netloader.layers.base.BaseMultiLayer, the extra_repr method is implemented so super should be called.

Features of `BaseSingleLayer`#

Some layers utilise a simple sequential layer structure such as Linear & Conv, and so inherit from BaseSingleLayer which has the signature:

BaseSingleLayer.__init__(**kwargs)[source]

Parameters:

**kwargs: Leftover parameters to pass to base layer for checking

The BaseSingleLayer class has the following attributes:

class BaseSingleLayer(**kwargs)[source]

Bases: BaseLayer

Base layer for layers that only use the previous layer.

Attributes:

groupint: Layer group, if 0 it will always be used, else it will only be used if its group matches the Network’s
descriptionstr: Description of the layer
layersSequential: Layers to loop through in the forward pass

The BaseSingleLayer class has the following methods:

BaseSingleLayer.forward(x, *_, **__)[source]

Forward pass for a generic layer.

Parameters:

xTensorListLike: Input with tensors of shape (N,…) and type float, where N is the batch size

Returns:

TensorListLike: Output with tensors of shape (N,…) and type float

Features of `BaseMultiLayer`#

Some layers utilise the outputs from other layers such as Skip & Concatenate, and so inherit from BaseMultiLayer which has the signature:

BaseMultiLayer.__init__(net_check, layer, shapes, check_shapes, *, checkpoint=False, **kwargs)[source]

Parameters:

net_checkbool: If layer index should be relative to checkpoint layers
layerint | list[int]: Layer index(es) to concatenate the previous layer output with
shapesShapes: Shape of the outputs from each layer
check_shapesShapes: Shape of the outputs from each checkpoint
checkpointbool, Optional: If layer index should be relative to checkpoint layers, default = False
**kwargs: Leftover parameters to pass to base layer for checking

Creating Custom Layers#

To create a custom layer, first decide which base layer to inherit from:

BaseSingleLayer - Layers that can use a simple sequential forward pass.
BaseMultiLayer - Layers that require the outputs from other layers.
BaseLayer - Layers that require more complex functionality.

All layers __init__ method must have a shapes and a **kwargs parameter, and the **kwargs parameter must be passed to the base layer’s __init__ method. The shapes parameter is used to track the output shapes for each layer (excluding batch dimension); therefore, the output shape of the custom layer must be appended to the shapes parameter in the __init__ method (even if the shape remains unchanged). They must also implement the __getstate__ method calling the base layer’s __getstate__ method and returning a dictionary of the layer’s custom __init__ parameters.

Custom layers can also use the following __init__ parameters:

net_check : bool - Global flag to show if the network saves the outputs from each layer (False) or only from Checkpoint layers (True), default = False.
idx : int - The index of the layer in the network, if used it must be passed to the base layer’s __init__ method, default = 0.
root : str - The root directory for the project, used for relative file paths, default = ‘’.
version : str - The version of NetLoader, used for backwards compatibility, if used it must be passed to the base layer’s __init__ method, default = netloader.__version__.
net_out : list[int] - Output shape for the whole network.
check_shapes : Shapes - The output shape for each checkpoint layer.

The forward method must include the input x with type TensorListLike and the *args & **kwargs parameters in the forward method. Custom layers can use the following forward method parameters:

outputs : list[Tensor] - The outputs from each layer in the network if the network is not exclusively using checkpoint outputs.
checkpoints : list[Tensor] - The outputs from each checkpoint layer in the network.
net : Self - The network that the layer is in, used for accessing the network’s attributes.

Custom Layer From `BaseSingleLayer`#

If your custom layer can use Sequential for the forward pass, then you can inherit from BaseSingleLayer and append to the layers attribute in the __init__ method.

from typing import Any

from torch import nn
from netloader.utils import Shapes
from netloader.layers.base import BaseSingleLayer

class CustomLinear(BaseSingleLayer):
    def __init__(
            self,
            features: int,
            activation: str | None,
            shapes: Shapes,
            **kwargs: Any) -> None:
        super().__init__(**kwargs)
        self._features: int = features
        in_shape: list[int] = shapes[-1].copy()
        self.layers.add_module('Linear', nn.Linear(in_shape[-1], self._features))

        if activation:
            self.layers.add_module('Activation', getattr(nn, activation)())

        in_shape[-1] = features
        shapes.append(in_shape)

    def __getstate__(self) -> dict[str, Any]:
        return super().__getstate__() | {
            'features': self._features,
            'activation': self.layers.Activation.__class__.__name__
                if 'Activation' in self.layers._modules else None,
        }

Then to allow Network to recognise the custom layer, you can monkey patch the layer into netloader.layers:

from netloader import layers

from src.layers import CustomLinear

layers.CustomLinear = CustomLinear

Custom Layer From `BaseMultiLayer`#

If your custom layer requires the outputs from multiple layers, then you can inherit from BaseMultiLayer and implement the forward method.

from typing import Any

from torch import Tensor
from netloader.utils import Shapes
from netloader.layers.base import BaseMultiLayer

class CustomConcat(BaseMultiLayer):
    def __init__(
            self,
            net_check: bool,
            layer: int
            shapes: Shapes,
            check_shapes: Shapes,
            checkpoint: bool = False,
            dim: int = 0,
            **kwargs: Any) -> None:
        super().__init__(
            net_check,
            layer,
            shapes,
            check_shapes,
            checkpoint=checkpoint,
            **kwargs,
        )
        self._dim: int = dim
        out_shape: list[int] = shapes[-1].copy()
        out_shape[self._dim] = out_shape[self._dim] + self._target[self._dim]
        shapes.append(out_shape)

    def __getstate__(self) -> dict[str, Any]:
        return super().__getstate__() | {'dim': self._dim}

    def forward(
            self,
            x: Tensor,
            outputs: list[Tensor],
            checkpoints: list[Tensor],
            *_: Any,
            **_: Any) -> Tensor:
        dim: int

        if self._dim >= 0:
            dim = self._dim + 1
        else:
            dim = self._dim

        if self._checkpoint:
            return torch.cat((x, checkpoints[self._layer]), dim=dim)
        return torch.cat((x, outputs[self._layer]), dim=dim)

    def extra_repr(self) -> str:
        return f'{super().extra_repr()}, dim={self._dim}'

Custom Layer From `BaseLayer`#

If your custom layer requires more complex functionality, then you can inherit from BaseLayer and implement the forward method.

from typing import Any

from torch import Tensor
from netloader.utils import Shapes
from netloader.layers.base import BaseLayer

class CustomScale(BaseLayer):
    def __init__(
            self,
            scale: float,
            shapes: Shapes,
            **kwargs: Any) -> None:
        super().__init__(**kwargs)
        self._scale: Tensor = torch.nn.Parameter(
            torch.tensor(scale),
            requires_grad=True,
        ).to(self._device)
        shapes.append(shapes[-1].copy())

    def __getstate__(self) -> dict[str, Any]:
        return super().__getstate__() | {'scale': self._scale.detach().cpu().tolist()[0]}

    def forward(self, x: Tensor, *_: Any, **_: Any) -> Tensor:
        return x * self._scale

    def extra_repr(self) -> str:
        return f'param={self._scale.detach().cpu().tolist()[0]}'

    def to(self, *args: Any, **kwargs: Any) -> None:
        super().to(*args, **kwargs)
        self._scale = self._scale.to(*args, **kwargs)

Layer Types#

All layers currently supported by NetLoader are listed below with a brief description of their functionality and their initialisation arguments, excluding general initialisation arguments defined in BaseLayer. For more details on each layer, please refer to the API documentation.

Blocks#

Blocks are layers that consist of multiple sub-layers made with Python code. See Block Layers for more details.

ConvNeXtBlock : ConvNeXt block using BaseLayer.

drop_path : float = 0 – Probability of Drop Path dropout
layer_scale : float = 1e-06 – Initial scale factor for layer Scale
**kwargs : Additional keyword arguments passed to BaseLayer.

Composite#

Composites are layers that consist of multiple sub-layers made with .json files. See Composite Layers for more details.

Branch : Creates a branching layer that processes the input through multiple branches and combines the

branches : list[list[dict[str, Any]]] – List of layer configurations for each branch
checkpoint : bool = True – If layer index should be relative to checkpoint layers
dim : int = 0 – Dimension to concatenate along if method is ‘concat’
channels : int | NoneType = None – Number of output channels, won’t be used if shape is provided, if channels and shape aren’t provided, the input dimensions will be preserved
method : {concat, pack, sum} = concat – Method to combine the outputs from each branch, either concatenation, packing or summation
shape : list[int] | NoneType = None – Output shape of the block, will be used if provided; otherwise, channels will be used
defaults : dict[str, Any] | NoneType = None – Default values for the parameters for each type of layer
**kwargs : Additional keyword arguments passed to BaseLayer.

Composite : Creates a subnetwork from a configuration file.

name : str – Name of the subnetwork
checkpoint : bool = True – If layer index should be relative to checkpoint layers
channels : int | NoneType = None – Number of output channels, won’t be used if shape is provided, if channels and shape aren’t provided, the input dimensions will be preserved
config_dir : str = '' – Path to the directory with the network configuration file, won’t be used if config is provided
shape : list[int] | NoneType = None – Output shape of the block, will be used if provided; otherwise, channels will be used
defaults : dict[str, Any] | NoneType = None – Default values for the parameters for each type of layer
config : dict[str, Any] | Config | NoneType = None – Network configuration dictionary
**kwargs : Additional keyword arguments passed to BaseLayer.

Convolutional#

Convolutional layers are layers that perform convolution operations. See Convolutional Layers for more details.

Conv : Convolutional layer constructor.

filters : int | NoneType = None – Number of convolutional filters, will be used if provided, else factor will be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Number of convolutional filters equal to the output channels, or if layer is provided, the layer’s channels, multiplied by factor, won’t be used if filters is provided
groups : int = 1 – Number of input channel groups, each with its own convolutional filter(s), input and output channels must both be divisible by the number of groups
kernel : int | list[int] = 3 – Size of the kernel
stride : int | list[int] = 1 – Stride of the kernel
padding : int | {same} | list[int] = 0 – Input padding, can an int, list of ints or ‘same’ where ‘same’ preserves the input shape
dropout : float = 0 – Probability of dropout
activation : str | NoneType = ELU – Which activation function to use from PyTorch
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
padding_mode : {NoneType, replicate, zeros, reflect, circular} = None – Padding mode to use from PyTorch, if None zero padding is used if version is >3.9.4 else replication padding is used
**kwargs : Additional keyword arguments passed to BaseLayer.

ConvDepth : Constructs a depthwise convolutional layer.

filters : int | NoneType = None – Number of convolutional filters, will be used if provided, else factor will be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Number of convolutional filters equal to the output channels multiplied by factor, won’t be used if filters is provided
kernel : int | list[int] = 3 – Size of the kernel
stride : int | list[int] = 1 – Stride of the kernel
padding : int | {same} | list[int] = 0 – Input padding, can an int, list of ints or ‘same’ where ‘same’ preserves the input shape
dropout : float = 0 – Probability of dropout
activation : str | NoneType = ELU – Which activation function to use
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
padding_mode : {NoneType, zeros, reflect, replicate, circular} = None – Padding mode to use from PyTorch, if None zero padding is used if version is >3.9.4 else replication padding is used
**kwargs : Additional keyword arguments passed to BaseLayer.

ConvDepthDownscale : Constructs depth downscaler using convolution with kernel size of 1.

dropout : float = 0 – Probability of dropout
activation : str | NoneType = ELU – Which activation function to use
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
padding_mode : {NoneType, zeros, reflect, replicate, circular} = None – Padding mode to use from PyTorch, if None zero padding is used if version is >3.9.4 else replication padding is used
**kwargs : Additional keyword arguments passed to BaseLayer.

ConvDownscale : Constructs a strided convolutional layer for downscaling.

filters : int | NoneType = None – Number of convolutional filters, will be used if provided, else factor will be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Number of convolutional filters equal to the output channels multiplied by factor, won’t be used if filters is provided
scale : int = 2 – Stride and size of the kernel, which acts as the downscaling factor
dropout : float = 0 – Probability of dropout
activation : str | NoneType = ELU – Which activation function to use
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
**kwargs : Additional keyword arguments passed to BaseLayer.

ConvTranspose : Constructs a transpose convolutional layer with fractional stride for input upscaling.

filters : int | NoneType = None – Number of convolutional filters, will be used if provided, else factor will be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Number of convolutional filters equal to the output channels multiplied by factor, won’t be used if filters is provided
kernel : int | list[int] = 3 – Size of the kernel
stride : int | list[int] = 1 – Stride of the kernel
out_padding : int | list[int] = 0 – Padding applied to the output
dilation : int | list[int] = 1 – Spacing between kernel points
padding : int | {same} | list[int] = 0 – Inverse of convolutional padding which removes rows from each dimension in the output, default = 0
dropout : float = 0 – Probability of dropout
padding_mode : {zeros, reflect, replicate, circular} = zeros – Padding mode to use from PyTorch
activation : str | NoneType = ELU – Which activation function to use
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
**kwargs : Additional keyword arguments passed to BaseLayer.

ConvTransposeUpscale : Constructs an upscaler using a transposed convolutional layer.

filters : int | NoneType = None – Number of convolutional filters, will be used if provided, else factor will be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Number of convolutional filters equal to the output channels multiplied by factor, won’t be used if filters is provided
scale : int | list[int] = 2 – Stride and size of the kernel, which acts as the upscaling factor
out_padding : int | list[int] = 0 – Padding applied to the output
dropout : float = 0 – Probability of dropout
activation : str | NoneType = ELU – Which activation function to use
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
**kwargs : Additional keyword arguments passed to BaseLayer.

ConvUpscale : Constructs an upscaler using a convolutional layer and pixel shuffling.

filters : int | NoneType = None – Number of convolutional filters, will be used if provided, else factor will be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Number of convolutional filters equal to the output channels multiplied by factor, won’t be used if filters is provided
scale : int = 2 – Factor to upscale the input by
kernel : int | list[int] = 3 – Size of the kernel
dropout : float = 0 – Probability of dropout
activation : str | NoneType = ELU – Which activation function to use
norm : {NoneType, batch, layer} = None – If batch or layer normalisation should be used
padding_mode : {NoneType, zeros, reflect, replicate, circular} = None – Padding mode to use from PyTorch, if None zero padding is used if version is >3.9.4 else replicate padding is used
**kwargs : Additional keyword arguments passed to BaseLayer.

PixelShuffle : Used for upscaling by scale factor :math:`r` for an input :math:`(N,C\times r^n,D_1,…,D_n)` to

scale : int – Upscaling factor
**kwargs : Additional keyword arguments passed to BaseLayer.

Flows#

Flows are layers that perform normalising flow operations. See Normalising Flow Layers for more details.

SplineFlow : Neural spline flow layer constructor.

transforms : int – Number of transforms
hidden_features : list[int] – Number of features in each of the hidden layers
context : bool = False – If the output from the previous layer should be used to condition the normalising flow, default = False
features : int | NoneType = None – Dimensions of the probability distribution, if factor is provided, features will not be used
factor : float | NoneType = None – Output features is equal to the factor of the network’s output, will be used if provided, else features will be used
**kwargs : Additional keyword arguments passed to BaseLayer.

Linear#

Linear layers perform simple operations on the input. See Linear Layers for more details.

Activation : Activation layer constructor.

activation : str = ELU – Which activation function to use from PyTorch
activation_kwargs : dict[str, Any] | NoneType = None – Additional keyword arguments to pass to the activation function
**kwargs : Additional keyword arguments passed to BaseLayer.

Linear : Linear layer constructor.

features : int | NoneType = None – Number of output features for the layer, if factor is provided, features will not be used
layer : int | NoneType = None – If factor is not None, which layer for factor to be relative to, if None, network output will be used
factor : float | NoneType = None – Output features is equal to the factor of the network’s output, or if layer is provided, which layer to be relative to, will be used if provided, else features will be used
batch_norm : bool = False – If batch normalisation should be used
flatten_target : bool = False – If the target should be flattened so that features is equal to the product of the target multiplied by factor, if factor is provided
dropout : float = 0 – Probability of dropout
activation : str | NoneType = SELU – Which activation function to use from PyTorch
**kwargs : Additional keyword arguments passed to BaseLayer.

OrderedBottleneck : Information-ordered bottleneck to randomly change the size of the bottleneck in an autoencoder

min_size : int = 0 – Minimum gate size
**kwargs : Additional keyword arguments passed to BaseLayer.

Sample : Samples random values from a Gaussian distribution for a variational autoencoder,

idx : int – Layer number
**kwargs : Additional keyword arguments passed to BaseLayer.

Upsample : Constructs an upsampler.

idx : int – Layer number
shape : list[int] | NoneType = None – Shape of the output, will be used if provided, else scale will be used
scale : float | list[float] | tuple[float, EllipsisType] = 2 – Factor to upscale all or individual dimensions, first dimension is ignored, won’t be used if shape is provided
mode : {nearest, linear, bilinear, bicubic, trilinear} = nearest – What interpolation method to use for upsampling
**kwargs : Additional keyword arguments passed to BaseLayer.

Miscellaneous#

Miscellaneous layers are layers that do not fit into the other categories. See Miscellaneous Layers for more details.

BatchNorm : Batch Normalisation layer

**kwargs : Additional keyword arguments passed to BaseLayer.

Checkpoint : Constructs a layer to create a checkpoint for using the output from the previous layer in

**kwargs : Additional keyword arguments passed to BaseLayer.

DropPath : Constructs a drop path layer to drop samples in a batch.

prob : float – Probability of dropout
**kwargs : Additional keyword arguments passed to BaseLayer.

Dropout : Constructs a dropout layer to randomly zero out elements of the input tensor.

prob : float – Probability of dropout
**kwargs : Additional keyword arguments passed to BaseLayer.

Index : Constructs a layer to slice the last dimension from the output from the previous layer.

index : int | list[int | NoneType] | tuple[int | NoneType, EllipsisType] | slice | NoneType = None – Index or slice to apply to the last dimension of the input tensor(s) or if map_ is False, the DataList itself
map_ : bool = True – If slicing should be applied to each tensor in a DataList, or if the DataList should be sliced itself, only used if input is a DataList
**kwargs : Additional keyword arguments passed to BaseLayer.

LayerNorm : Constructs a layer normalisation layer that is the same as PyTorch’s LayerNorm, except

dims : int | NoneType = None – Number of dimensions to normalise starting with the first dimension, ignoring batch dimension, requires shapes argument, won’t be used if shape is provided
shape : list[int] | NoneType = None – Input shape or shape of the first dimension to normalise, will be used if provided, else dims will be used
**kwargs : Additional keyword arguments passed to BaseLayer.

Pad : Constructs a padding layer to pad the output from the previous layer.

pad : int | list[int] | list[tuple[int, int]] | tuple[int, EllipsisType] – Amount of padding to apply to each dimension of the input tensor, ignoring batch dimension and from right to left, if int is provided, same padding will be applied to left and right of all dimensions according to dims, if list of integers is provided, each integer will be applied to left and right of the corresponding dimension, if list of tuple of integers is provided, then asymmetric (left, right) padding will be applied, if tuple of integers is provided, each pair of integers will be applied to left and right of the corresponding dimension
dims : int = -1 – Number of dimensions to pad starting with the last dimension, ignoring batch dimension, if shapes is not provided, dims must be greater than 0, if negative, excludes the first abs(dims) dimensions, if 0 pads all dimensions, won’t be used if padding is a list of integers or list of tuple of integers
pad_kwargs : dict[str, Any] | NoneType = None – Additional keyword arguments to pass to the padding function
**kwargs : Additional keyword arguments passed to BaseLayer.

Reshape : Constructs a reshaping layer to change the data dimensions.

shape : list[int] – Desired shape of the output tensor, ignoring first dimension
factor : bool = False – If reshape should be relative to the network output shape, or if layer is provided, which layer to be relative to, requires tracking layer outputs
layer : int | NoneType = None – If factor is True, which layer for factor to be relative to, if None, network output will be used
**kwargs : Additional keyword arguments passed to BaseLayer.

Scale : Constructs a scaling layer that scales the output by a learnable tensor.

dims : int – Number of dimensions to have individual scales for
scale : float – Initial scale factor
first : bool = True – If dims should count from the first dimension after the batch dimension, or from the final dimension backwards
**kwargs : Additional keyword arguments passed to BaseLayer.

Multi Layers#

Layers that perform operations on multiple inputs or outputs. See Multi-Layers Layers for more details.

Concatenate : Constructs a concatenation layer to combine the outputs from two layers.

layer : int | list[int] – Layer index(es) to concatenate the previous layer output with
checkpoint : bool = False – If layer index should be relative to checkpoint layers or network layers, if checkpoints in net is True, layer will always be relative to checkpoints
dim : int = 0 – Dimension to concatenate to
**kwargs : Additional keyword arguments passed to BaseLayer.

Pack : Constructs a packing layer to combine the output from the previous layer with the output from

checkpoint : bool = False – If layer index should be relative to checkpoint layers
layer : int | list[int] | NoneType = None – Layer index to pack with the previous layer, if layer is None, then only the last layer will be packed
**kwargs : Additional keyword arguments passed to BaseLayer.

Shortcut : Constructs a shortcut layer to add the outputs from two layers.

layer : int | list[int] – Layer index(es) to add the previous layer output with
checkpoint : bool = False – If layer index should be relative to checkpoint layers or network layers, if checkpoints in net is True, layer will always be relative to checkpoints
**kwargs : Additional keyword arguments passed to BaseLayer.

Skip : Bypasses previous layers by retrieving the output from the defined layer.

layer : int – Layer index to get the output from
checkpoint : bool = False – If layer index should be relative to checkpoint layers or network layers, if checkpoints in net is True, layer will always be relative to checkpoints
**kwargs : Additional keyword arguments passed to BaseLayer.

Unpack : Enables a list of Tensors as input into the network, then selects which Tensor in the list to

index : int – Index of input Tensor list
checkpoint : bool = False – If layer index should be relative to checkpoint layers or network layers, if checkpoints in net is True, layer will always be relative to checkpoints
layer : int | NoneType = None – Layer index to get the output from if checkpoint is True
**kwargs : Additional keyword arguments passed to BaseLayer.

Pooling#

Pooling layers perform pooling operations. See Pooling Layers for more details.

AdaptivePool : Uses pooling to downscale the input to the desired shape.

shape : int | list[int] – Output shape of the layer
channels : bool = True – If the input includes a channels dimension
mode : {average, max} = average – Whether to use ‘max’ or ‘average’ pooling
**kwargs : Additional keyword arguments passed to BaseLayer.

Pool : Constructs a max or average pooling layer.

kernel : int | list[int] = 2 – Size of the kernel
stride : int | list[int] = 2 – Stride of the kernel
padding : int | {same} | list[int] = 0 – Input padding, can an int or ‘same’ where ‘same’ preserves the input shape
mode : {max, average} = max – Whether to use ‘max’ or ‘average’ pooling
pool_kwargs : dict[str, Any] | NoneType = None – Additional keyword arguments to pass to the pooling layer
**kwargs : Additional keyword arguments passed to BaseLayer.

PoolDownscale : Downscales the input using pooling.

scale : int – Stride and size of the kernel, which acts as the downscaling factor
mode : {max, average} = max – Whether to use ‘max’ or ‘average’ pooling
**kwargs : Additional keyword arguments passed to BaseLayer.

Recurrent#

Recurrent layers perform recurrent operations. See Recurrent Layers for more details.

Recurrent : Recurrent layer constructor for either RNN, GRU or LSTM.

idx : int – Layer number
batch_norm : bool = False – If batch normalisation should be used
layers : int = 2 – Number of stacked recurrent layers
filters : int = 1 – Number of output filters
dropout : float = 0 – Probability of dropout, requires layers > 1
mode : {gru, lstm, rnn} = gru – Type of recurrent layer
activation : str | NoneType = ELU – Which activation function to use, if mode is ‘rnn’, ReLU activation will always be used, default = ‘ELU’
bidirectional : {NoneType, sum, mean, concatenate} = None – If a bidirectional recurrence should be used and method for combining the two directions
**kwargs : Additional keyword arguments passed to BaseLayer.

Layers#

Layer Shape Compatibility#

Extra Layer Features#

Features of BaseLayer#

Features of BaseSingleLayer#

Features of BaseMultiLayer#

Creating Custom Layers#

Custom Layer From BaseSingleLayer#

Custom Layer From BaseMultiLayer#

Custom Layer From BaseLayer#

Layer Types#

Blocks#

Composite#

Convolutional#

Flows#

Linear#

Miscellaneous#

Multi Layers#

Pooling#

Recurrent#

Features of `BaseLayer`#

Features of `BaseSingleLayer`#

Features of `BaseMultiLayer`#

Custom Layer From `BaseSingleLayer`#

Custom Layer From `BaseMultiLayer`#

Custom Layer From `BaseLayer`#