Data#

Classes#

class Data(data, uncertainty=None)[source]#

Bases: Generic[ArrayCT]

Stores data with uncertainties for uncertainty handling in BaseArchitecture.

Attributes:

dataArrayCT: Data
uncertaintyArrayCT | NoneType: Data uncertainty

Parameters:

dataArrayCT: Data
uncertaintyArrayCT | NoneType, Optional: Data uncertainty

static collate(data, *, data_field=True)[source]#

Collates a list of Data objects into a single Data object.

Overloads:

collate(data: list[Data[ArrayCT]], *, data_field: Literal[True]) → Data[ArrayCT]

collate(data: list[Data[ArrayCT]], *, data_field: Literal[False]) → ArrayCT

collate(data: list[Data[ArrayCT]], *, data_field: bool) → ArrayCT | Data[ArrayCT]

Parameters:

datalist[Data[ArrayCT]]: List of Data objects to collate
data_fieldbool, Optional: If True, returns a Data object, else returns the collated data as an ArrayLike, default = True

Returns:

ArrayCT | Data[ArrayCT]: Collated ArrayLike or Data object

__getitem__(idx)[source]#

Gets a subset of the data and uncertainty.

Parameters:

idxint | slice: Index or slice to get

Returns:

Data[ArrayCT]: Data with subset of data and uncertainty

__len__()[source]#

Returns the number of samples in the data.

Returns:

int: Number of samples in the data

apply(func, *args, types=(<class 'torch.Tensor'>, <class 'numpy.ndarray'>), **kwargs)[source]#

Applies a function or method to the data and uncertainty.

Parameters:

funcstr | ApplyFunc[ArrayCT]: Function, or method if string, to apply to the data and uncertainty
*args: Arguments to pass to the function or method
typestuple[type[ArrayLike], …]: Types to apply the function or method to, if data and uncertainty is not an instance of types, then data and uncertainty are returned unchanged, default = (Tensor, ndarray)
**kwargs: Keyword arguments to pass to the function or method

Returns:

Self: Self with function or method applied to the data and uncertainty

clone()[source]#

Clones the data and uncertainty.

Identical to Data.copy().

Returns:

Data[ArrayCT]: Cloned Data

concat(dim=0)[source]#

Concatenates data and uncertainty for passing into a network.

Parameters:

dimint, Optional: Dimension to concatenate along, default = 0

Returns:

ArrayCT: Data and uncertainty concatenated along the specified dimension if uncertainty is not None, else just data

copy()#

Clones the data and uncertainty.

Identical to Data.copy().

Returns:

Data[ArrayCT]: Cloned Data

cpu()[source]#

Moves data and uncertainty to CPU if they are Tensors.

Returns:

Self: Self with data and uncertainty on CPU

detach()[source]#

Detaches data and uncertainty from the computation graph if they are Tensors.

Returns:

Self: Self with data and uncertainty detached from the computation graph

numpy()[source]#

Converts data and uncertainty to numpy arrays if they are Tensors.

Returns:

Data[ndarray]: New Data object with data and uncertainty as numpy arrays

tensor()[source]#

Converts data and uncertainty to tensors if they are numpy arrays.

Returns:

Data[Tensor]: New Data object with data and uncertainty as tensors

to(*args, **kwargs)[source]#: See torch.nn.Module.to().

class DataList(data)[source]#

Bases: Generic[DataT]

A list that stores tensors, arrays, or Datas and provides batch operations.

Parameters:

datalist[DataT]: List of Data, Tensor, or ndarray objects

static collate(data, *, data_field=True)[source]#

Collates a list of DataList objects into a single DataList object.

Overloads:

collate(data: list[DataList[DataT]], *, data_field: Literal[True]) → DataList[DataT]

collate(data: list[DataList[DataT]], *, data_field: Literal[False]) → DataList[ArrayT]

collate(data: list[DataList[DataT]], *, data_field: bool) → DataList[ArrayT] | DataList[DataT]

Parameters:

datalist[DataList[DataT]]: List of DataList objects to collate
data_fieldbool, Optional: If True, collates Data elements into Data object, else collates into ArrayLike, default = True

Returns:

DataList[ArrayT] | DataList[DataT]: Collated DataList object

__add__(other)[source]#

Adds another DataList or data to the DataList.

Parameters:

otherDataT | DataList[DataT]: DataList or single element to add to the DataList

Returns:

DataList[DataT]: New DataList with the other DataList or data added to the end of the DataList

__getitem__(idx)[source]#

Gets a subset of each element in the DataList.

Parameters:

idx: int | slice: Index or slice to get

Returns:

DataList[DataT]: DataList with subset of each element in the DataList

__iter__()[source]#

Iterates over each element in the DataList.

Returns:

Iterator[DataT]: Iterator over each element in the DataList

__len__()[source]#

Returns the number of samples in each element.

Returns:

int: Number of samples in each element

append(data)[source]#

Appends an element to the DataList.

Parameters:

dataDataT: Element to append to the DataList

apply(func, *args, types=(<class 'netloader.data.Data'>, <class 'torch.Tensor'>, <class 'numpy.ndarray'>), **kwargs)[source]#

Applies a function or method to each element in the DataList.

Parameters:

funcstr | Sequence[str | ApplyFunc[ArrayT]] | ApplyFunc[DataT]: Function(s), or method(s) if string, to apply to each element in the DataList
*args: Arguments to pass to the function or method
typestuple[type[DataLike], …]: Types to apply the function or method to, if an element is not an instance of types, then that element is returned unchanged, default = (Data, Tensor, ndarray)
**kwargs: Keyword arguments to pass to the function or method

Returns:

Self: Self with function or method applied to each element in the DataList

clone()[source]#

Clones the DataList.

Identical to DataList.copy().

Returns:

DataList[DataT]: Cloned DataList

copy()#

Clones the DataList.

Identical to DataList.copy().

Returns:

DataList[DataT]: Cloned DataList

cpu()[source]#

Moves all tensors to CPU

Returns:

Self: Self with all tensors moved to CPU

detach()[source]#

Detaches all tensors from the computation graph

Returns:

Self: Self with all tensors detached from the computation graph

extend(data)[source]#

Extends the DataList by appending elements from the iterable.

Parameters:

dataSequence[DataT] | DataList[DataT]: Elements to extend the DataList with

get(idx, list_=False)[source]#

Gets a subset of the DataList or a subset of each element in the DataList.

Overloads:

get(self, idx: int, list_: Literal[True]) → DataT

get(self, idx: slice, list_: Literal[True]) → list[DataT]

get(self, idx: int | slice, list_: Literal[False]) → DataList[DataT]

get(self, idx: int, list_: bool) → DataT | DataList[DataT]

get(self, idx: slice, list_: bool) → list[DataT] | DataList[DataT]

Parameters:

idxint | slice: Index or slice to get
list_bool, Optional: If True, returns a subset of the DataList, else returns a subset of each element in the DataList, default = False

Returns:

DataT | list[DataT] | DataList[DataT]: Subset of the DataList or subset of each element in the DataList

get_data()[source]#

Gets the underlying data list.

Returns:

list[DataT]: Underlying data list

insert(idx, data)[source]#

Inserts an element into the DataList at the specified index.

Parameters:

idxint: Index to insert the element at
dataDataT: Element to insert into the DataList

iter(list_=True)[source]#

Iterates over each element in the DataList.

Overloads:

iter(self, list_: Literal[True]) → Iterator[DataT]

iter(self, list_: Literal[False]) → Iterator[DataList[DataT]]

iter(self, list_: bool) → Iterator[DataT] | Iterator[DataList[DataT]]

Parameters:

list_bool, Optional: If True, iterates over the DataList, else iterates over each element in the DataList, default = True

Returns:

Iterator[DataT] | Iterator[DataList[DataT]]: Iterator over each element in the DataList

len(list_=True)[source]#

Gets the length of the DataList or the length of each element in the DataList.

Parameters:

list_bool, Optional: If True, returns the length of the DataList, else returns the length of each element in the DataList, default = True

Returns:

int: Length of the DataList or length of each element in the DataList

numpy()[source]#

Converts all tensors to numpy arrays.

Returns:

DataList[ndarray | Data[ndarray]]: DataList with all tensors converted to numpy arrays

tensor()[source]#

Converts all numpy arrays to tensors.

Returns:

DataList[Tensor | Data[Tensor]]: DataList with all numpy arrays converted to tensors

to(*args, **kwargs)[source]#: See torch.nn.Module.to().

class BaseDataset(*args, **kwargs)[source]#

Bases: Dataset[tuple[int, DataListT, DataListT, Any]], Generic[DataListT]

Base dataset class for use with BaseNetwork.

Attributes:

extralist[Any] | ArrayLike | NoneType: Additional data for each sample in the dataset of length N with shape (N,…) and type Any
idxsndarray: Index for each sample in the dataset with shape (N) and type int
low_dimDataListT | NoneType: Low dimensional data for each sample in the dataset with shape (N,…)
high_dimDataListT | NoneType: High dimensional data for each sample in the dataset with shape (N,…), this is required

__getitem__(idx)[source]#

Parameters:

idxint: Sample index

Returns:

tuple[int, DataListT, DataListT, Any]: Sample index, low dimensional data, high dimensional data, and extra data

__len__()[source]#

Returns the number of samples in the dataset

Returns:

int: Number of samples in the dataset

get_extra(idx)[source]#

Gets extra data for the sample of the given index

Parameters:

idxint: Sample index

Returns:

Any: Sample extra data

get_high_dim(idx)[source]#

Gets a high dimensional sample of the given index

Parameters:

idxint: Sample index

Returns:

DataListT: High dimensional sample

get_low_dim(idx)[source]#

Gets a low dimensional sample of the given index

Parameters:

idxint: Sample index

Returns:

DataListT: Low dimensional sample

step(epoch)[source]#

Step the dataset for each training iteration.

Parameters:

epochfloat: Current epoch number

Functions#

data_collation(data, *, data_field=True)[source]#

Collates a list of ArrayLike, Data, or DataList objects into a single object.

Overloads:

data_collation(data: list[ArrayCT] | ArrayCT, *, data_field: bool) → ArrayCT

data_collation(data: DataList[DataT], *, data_field: bool) → DataList[DataT]

data_collation(data: list[DataList[DataT]], *, data_field: Literal[True]) → DataList[DataT]

data_collation(data: list[DataList[DataT]], *, data_field: Literal[False]) → DataList[ArrayT]

data_collation(data: list[Data[ArrayCT]], *, data_field: Literal[True]) → Data[ArrayCT]

data_collation(data: list[Data[ArrayCT]], *, data_field: Literal[False]) → ArrayCT

Parameters:

datalist[ArrayCT] | list[Data[ArrayCT]] | list[DataList[DataT]] | ArrayCT | DataList[DataT]: List of ArrayLike, Data, or DataList objects to collate, or if ArrayLike or DataList, will return as is, with shape (N,…)
data_fieldbool, Optional: If Datas should return Data else ArrayLike, default = True

Returns:

ArrayCT | Data[ArrayCT] | DataList[ArrayT] | DataList[DataT]: Collated Data or DataList object, or ArrayLike if data_field is False or input is ArrayLike, with shape (N,…)

loader_init(dataset, *, return_idxs=False, batch_size=64, ratios=None, idxs=None, **kwargs)[source]#

Initialises data loaders from a subset of the dataset with the given ratios.

Overloads:

loader_init(dataset: DatasetT, *, return_idxs: Literal[False], batch_size: int = ..., ratios: list[float] | tuple[float, ...] | None = ..., idxs: list[numpy.ndarray] | tuple[numpy.ndarray, ...] | numpy.ndarray | None = ..., **kwargs: Any) → tuple[torch.utils.data.dataloader.DataLoader[tuple[int, DataListT, DataListT, Any]], ...]

loader_init(dataset: DatasetT, *, return_idxs: Literal[True], batch_size: int = ..., ratios: list[float] | tuple[float, ...] | None = ..., idxs: list[numpy.ndarray] | tuple[numpy.ndarray, ...] | numpy.ndarray | None = ..., **kwargs: Any) → tuple[tuple[torch.utils.data.dataloader.DataLoader[tuple[int, DataListT, DataListT, Any]], ...], tuple[Sequence[int], ...]]

Parameters:

datasetDatasetT: Dataset to create data loaders from
return_idxsbool, Optional: If the indexes for each data loader should be returned, default = False
batch_sizeint, Optional: Batch size when sampling from the data loaders, default = 64
ratioslist[float] | tuple[float, …] | NoneType, Optional: Ratios of length M to split up the dataset into subsets, if idxs is provided, dataset will first be split up using idxs and ratios will be used on the remaining samples, default = (0.8,0.2)
idxslist[ndarray] | tuple[ndarray, …] | ndarray | NoneType, Optional: Dataset indexes for creating the subsets with shape (N,S), where N is the number of subsets and S is the number of samples in each subset
**kwargsAny: Optional keyword arguments to pass to DataLoader

Returns:

tuple[DataLoader[tuple[int, DataListT, DataListT, Any]], …] | tuple[tuple[DataLoader[tuple[int, DataListT, DataListT, Any]], …], tuple[Sequence[int], …]]: Data loaders for each subset of length N + M and optionally indexes used for each subset

Types#

InArrayT_contra#: TypeVar[bound = ndarray | Tensor | Data[ArrayCT]]

OutArrayT_co#: TypeVar[bound = ndarray | Tensor | Data[ArrayCT]]

Protocols#

class ApplyFunc(*args, **kwargs)[source]#

Bases: Protocol[InArrayT_contra, P, OutArrayT_co]

Protocol for functions that can be applied to Data objects in the apply method of Data and DataList.

__call__(data, /, *args, **kwargs)[source]#

Parameters:

dataDataT: Data to apply the function to
*args: Optional arguments to pass to the function
**kwargs: Optional keyword arguments to pass to the function

Returns:

DataT: Result of applying the function to the data