Datasets & Data Formats#

PyTorch Network Loader provides the BaseDataset class for creating custom datasets in a format compatible with NetLoader architectures, along with additional useful classes and functions for data formatting.

Data Requirements for NetLoader Architectures#

NetLoader follows the standard PyTorch dataset conventions (see the PyTorch Datasets tutorial for more information), but with some specific requirements for compatibility with NetLoader architectures. The basic requirements for a custom dataset are:

__len__ method that returns the number of samples in the dataset.
__getitem__ method that returns a tuple of: ID, low-dimensional data (or None), high-dimensional data, and extra (or None) for a single sample.

High-dimensional data should always be set, if there is only one set of data (such as in unsupervised learning) then the low-dimensional data can be set to None. The ID is a unique identifier for each sample which can be used for tracking samples in the training and validation data loaders. The extra field can be used to store any additional information about the sample that can be used in the loss function of the architecture.

Custom Datasets from `BaseDataset`#

BaseDataset provides a convenient base class for creating custom datasets compatible with BaseArchitecture architectures.

The attributes of the BaseDataset object include:

class BaseDataset(*args, **kwargs)[source]

Bases: Dataset[tuple[int, DataListT, DataListT, Any]], Generic[DataListT]

Base dataset class for use with BaseNetwork.

Attributes:

extralist[Any] | ArrayLike | NoneType: Additional data for each sample in the dataset of length N with shape (N,…) and type Any
idxsndarray: Index for each sample in the dataset with shape (N) and type int
low_dimDataListT | NoneType: Low dimensional data for each sample in the dataset with shape (N,…)
high_dimDataListT | NoneType: High dimensional data for each sample in the dataset with shape (N,…), this is required

The minimum requirement to use BaseDataset is to inherit BaseDataset and define the attribute high_dim, which if an architecture only uses one set of data, it should use the high_dim attribute. However, depending on the architecture, the dataset may also need to define the low_dim and extra attributes. The idxs attribute is set to a list of indices for the samples in the dataset by default, but can be overridden if needed.

The BaseDataset class also has the following methods:

BaseDataset.get_extra(idx)[source]

Gets extra data for the sample of the given index

Parameters:

idxint: Sample index

Returns:

Any: Sample extra data

BaseDataset.get_high_dim(idx)[source]

Gets a high dimensional sample of the given index

Parameters:

idxint: Sample index

Returns:

DataListT: High dimensional sample

BaseDataset.get_low_dim(idx)[source]

Gets a low dimensional sample of the given index

Parameters:

idxint: Sample index

Returns:

DataListT: Low dimensional sample

BaseDataset.step(epoch)[source]

Step the dataset for each training iteration.

Parameters:

epochfloat: Current epoch number

Along with the magic methods:

BaseDataset.__getitem__(idx)[source]

Parameters:

idxint: Sample index

Returns:

tuple[int, DataListT, DataListT, Any]: Sample index, low dimensional data, high dimensional data, and extra data

BaseDataset.__len__()[source]

Returns the number of samples in the dataset

Returns:

int: Number of samples in the dataset

If pre-processing of the low-dimensional, high-dimensional, or extra data is needed for each time a sample is accessed (such as data augmentation), the methods get_low_dim, get_high_dim, and get_extra can be overridden to perform the necessary pre-processing, static pre-processing should be applied to the saved data, in the __init__ method, or after the dataset has been initialised and transformed with transforms from transforms (see Transforms for more information).

To create a custom dataset, inherit from BaseDataset and define the necessary attributes and methods based on the requirements of the architecture. For example, if using the Encoder architecture, the dataset should define the low_dim attribute as the labels and the high_dim attribute as the input data:

import pickle

import torch
from torchvision.transforms import v2
from netloader.data import BaseDataset

class CustomDataset(BaseDataset):
    def __init__(self, data_dir: str) -> None:
        super().__init__()

        with open(data_dir, 'rb') as file:
            self.low_dim, self.high_dim = pickle.load(file)  # e.g. labels & images

        # Data augmentations
        self.aug = v2.Compose([
            v2.RandomHorizontalFlip(),
            v2.RandomVerticalFlip(),
        ])

    def get_high_dim(self, idx: int) -> torch.Tensor:
        # Apply augmentations to the high-dimensional data
        return self.aug(self.high_dim[idx])

If any parameters in the dataset should be updated during training, such as the level of noise induced in the dataset, the method step can be implemented to update the necessary parameters at the end of each batch iteration during the training of the architecture:

from multiprocessing import Value
from multiprocessing.sharedctypes import Synchronized

class CustomDataset(BaseDataset):
    max_epochs: int = 100
    max_noise: float = 0.1

    def __init__(self, data_dir: str) -> None:
        super().__init__()
        # Shared value for the noise level so that all workers in the data loader can access and
        # update it
        self.noise: Synchronized[float] = Value('f', 0.0)

        with open(data_dir, 'rb') as file:
            self.low_dim, self.high_dim = pickle.load(file)

    def step(self, epoch: float) -> None:
        # Update the noise level at the end of each batch iteration
        self.noise.value = min(self.max_noise, self.max_noise * epoch / self.max_epochs)

    def get_high_dim(self, idx: int) -> torch.Tensor:
        # Add noise to the high-dimensional data
        return self.high_dim[idx] + torch.randn_like(high_dim_data) * self.noise_level.value

Data with Uncertainties#

If the dataset contains uncertainties, the data can be stored as a Data object, which contains the data and the corresponding uncertainties that can be naturally transformed using transforms (see Transforms for more information) and passed to the architectures; however, currently the architectures do not fully support the use of Data objects and will instead concatenate the data and uncertainties as a Tensor.

The Data class can be initialised with the following parameters:

Data.__init__(data, uncertainty=None)[source]

Parameters:

dataArrayCT: Data
uncertaintyArrayCT | NoneType, Optional: Data uncertainty

The attributes of the BaseDataset object include:

class Data(data, uncertainty=None)[source]

Bases: Generic[ArrayCT]

Stores data with uncertainties for uncertainty handling in BaseArchitecture.

Attributes:

dataArrayCT: Data
uncertaintyArrayCT | NoneType: Data uncertainty

See Data for the methods of Data class.

import torch
from netloader.data import Data
from netloader import transforms

values: torch.Tensor = torch.tensor([1.0, 2.0, 3.0])
uncertainties: torch.Tensor = torch.tensor([0.1, 0.2, 0.3])
data: Data = Data(values, uncertainty=uncertainties)
trans_data: Data = transforms.Log()(data)

Multiple Inputs/Outputs#

If the architecture has multiple inputs or outputs, the dataset can store the multi-data as a DataList object. DataList objects can store ndarray, Tensor, or Data objects where each array or Data object has the same number of samples (i.e. the same length along the first dimension). Some network layers from layers support DataList objects as inputs and/or outputs, such as Unpack (see Layers for more information).

The DataList class can be initialised with the following parameters:

DataList.__init__(data)[source]

Parameters:

datalist[DataT]: List of Data, Tensor, or ndarray objects

The DataList class also has the following magic methods:

DataList.__getitem__(idx)[source]

Gets a subset of each element in the DataList.

Parameters:

idx: int | slice: Index or slice to get

Returns:

DataList[DataT]: DataList with subset of each element in the DataList

DataList.__iter__()[source]

Iterates over each element in the DataList.

Returns:

Iterator[DataT]: Iterator over each element in the DataList

DataList.__len__()[source]

Returns the number of samples in each element.

Returns:

int: Number of samples in each element

Along with matching methods:

DataList.get(idx, list_=False)[source]

Gets a subset of the DataList or a subset of each element in the DataList.

Overloads:

get(self, idx: int, list_: Literal[True]) → DataT

get(self, idx: slice, list_: Literal[True]) → list[DataT]

get(self, idx: int | slice, list_: Literal[False]) → DataList[DataT]

get(self, idx: int, list_: bool) → DataT | DataList[DataT]

get(self, idx: slice, list_: bool) → list[DataT] | DataList[DataT]

Parameters:

idxint | slice: Index or slice to get
list_bool, Optional: If True, returns a subset of the DataList, else returns a subset of each element in the DataList, default = False

Returns:

DataT | list[DataT] | DataList[DataT]: Subset of the DataList or subset of each element in the DataList

DataList.iter(list_=True)[source]

Iterates over each element in the DataList.

Overloads:

iter(self, list_: Literal[True]) → Iterator[DataT]

iter(self, list_: Literal[False]) → Iterator[DataList[DataT]]

iter(self, list_: bool) → Iterator[DataT] | Iterator[DataList[DataT]]

Parameters:

list_bool, Optional: If True, iterates over the DataList, else iterates over each element in the DataList, default = True

Returns:

Iterator[DataT] | Iterator[DataList[DataT]]: Iterator over each element in the DataList

DataList.len(list_=True)[source]

Gets the length of the DataList or the length of each element in the DataList.

Parameters:

list_bool, Optional: If True, returns the length of the DataList, else returns the length of each element in the DataList, default = True

Returns:

int: Length of the DataList or length of each element in the DataList

The methods above have the list_ argument, which specifies if the method should act on the list containing the data, or the data itself. For example, DataList.len(True) will return the number of data arrays in the list, while DataList.len(False) will return the number of samples in each data array. For the magic methods, the list_ argument cannot be specified, so the default behaviour follows:

__getitem__: list_=False, returns a DataList with each data array indexed at the specified index.
__iter__: list_=True, iterates over the data arrays in the list.
__len__: list_=False, returns the number of samples in each data array.

See Data for the methods of DataList class.

import torch
import numpy as np
from netloader.data import Data, DataList

data1: np.ndarray = np.random.rand(4, 3, 10)
data2: torch.Tensor = torch.arange(12).view(4, 3)
data3: Data = Data(torch.ones((4, 3)), uncertainty=torch.zeros((4, 3)))
data_list: DataList = DataList([data1, data2, data3])

assert len(data_list) == 4
assert [tuple(data.shape) for data in data_list] == [(4, 3, 10), (4, 3), (4, 3)]
assert [tuple(data.shape) for data in data_list[0]] == [(3, 10), (3,), (3,)]

Datasets & Data Formats#

Data Requirements for NetLoader Architectures#

Custom Datasets from BaseDataset#

Data with Uncertainties#

Multiple Inputs/Outputs#

Custom Datasets from `BaseDataset`#