utils package

Submodules

utils.AUPRC module

Implements utils for getting AUPRC score given input and output pairs.

utils.AUPRC.AUPRC(pts)

Get average precision given a list of (true, predicted) pairs.

Parameters:: pts (List) – List of true, predicted pairs
Returns:: Average Precision Score.
Return type:: float

utils.AUPRC.ptsort(tu): Return first element of input.

utils.aux_models module

Implements a variety of modules for path-based dropout.

@author: juanma

class utils.aux_models.AlphaScalarMultiplication(size_alpha_x, size_alpha_y)

Bases: Module

Multiplies output element-wise by a scalar.

__init__(size_alpha_x, size_alpha_y)

Initialize AlphaScalarMultiplication module.

Parameters:

size_alpha_x (int) – Feature size for x input.
size_alpha_y (int) – Feature size for y input.

forward(x, y)

Apply Alpha Scalar Multiplication to both inputs, followed by a sigmoid.

Parameters:

x (torch.Tensor) – Layer input
y (torch.Tensor) – Layer input

Returns:

Layer output

Return type:

torch.Tensor

training: bool

class utils.aux_models.AlphaVectorMultiplication(size_alpha)

Bases: Module

Multiplies output element-wise by a vector.

__init__(size_alpha)

Initialize AlphaVectorMultiplication module.

Parameters:: size_alpha (int) – Size of alpha module.

forward(x)

Apply Alpha Vector Multiplication to input.

Parameters:: x (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.AuxiliaryHead(num_classes, filters=96)

Bases: Module

Implements Auxiliary Head Module.

__init__(num_classes, filters=96)

Instantiate AuxiliaryHead Module.

Parameters:

num_classes (int) – Number of classes in output
filters (int, optional) – Number of hidden channels. Defaults to 96.

forward(x)

Apply AuxiliaryHead to Layer Input.

Parameters:: x (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.Cell(operation_labels, configuration_indexes, connections, args)

Bases: Module

Implements a convnet classifier, using CellBlock instances and path-based dropout.

Generally unused.

__init__(operation_labels, configuration_indexes, connections, args)

Instantiate Cell Module.

Parameters:

operation_labels (list) – List of operation labels
configuration_indexes (list) – list of configuration indexes
connections (list) – list of connections
args (list) – list of args for Cell

forward(x1, x2)

Apply cell to layer input, and add residual connection.

Parameters:

x1 (torch.Tensor) – Input tensor 1
x2 (torch.Tensor) – Input tensor 2

Returns:

Output Tensor

Return type:

torch.Tensor

training: bool

class utils.aux_models.CellBlock(op1_type, op2_type, args)

Bases: Module

Implements a block of convolution cells, with path-based dropout.

__init__(op1_type, op2_type, args)

Instatiates CellBlock Module.

Parameters:

op1_type (int|str) – First convolution type
op2_type (int|str) – Second convolution type
args (obj) – Arguments for path-dropping and input/output channel number.

forward(x1, x2)

Apply block to layer input, and add residual connection.

Parameters:

x1 (torch.Tensor) – Input tensor 1
x2 (torch.Tensor) – Input tensor 2

Returns:

Output Tensor

Return type:

torch.Tensor

training: bool

class utils.aux_models.ChannelPadding(pad)

Bases: Module

Applies Channel Padding to input.

__init__(pad)

Initialize Tensor1DLateralPadding Module.

Parameters:: pad (int) – Padding amount

forward(inputs)

Apply Channel Padding to input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.ConvBranch(in_planes, out_planes, kernel_size, separable=False)

Bases: Module

Implements a convolution computational path for path-based dropout.

__init__(in_planes, out_planes, kernel_size, separable=False)

Initialize ConvBranch Module.

Parameters:

in_planes (int) – Input channel count
out_planes (int) – Output channel count
kernel_size (int) – Kernel size
separable (bool, optional) – Whether to use Separable convolutions or not. Defaults to False.

forward(x)

Apply ConvBranch to Layer Input.

Parameters:: x (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

utils.aux_models.CreateOp(conv_type, input_planes=64, output_planes=64)

Given a type of convolution, and the input/output channels, instatiate a convolution module.

Parameters:

conv_type (int|string) – Type of convolution.
input_planes (int, optional) – Input channel number. Defaults to 64.
output_planes (int, optional) – Output channel number. Defaults to 64.

Raises:

NotImplementedError – Convolution not implemented.

Returns:

Convolution instance

Return type:

nn.Module

class utils.aux_models.DropPath(keep_prob=0.9)

Bases: Module

Implements path-based dropout.

__init__(keep_prob=0.9)

Initialize DropPath module.

Parameters:: keep_prob (float, optional) – Probability to keep this path in the training pass. Defaults to 0.9.

forward(x, other_dropped=False)

Apply path-dropping to layer input.

Parameters:

x (torch.Tensor) – Layer Input
other_dropped (bool, optional) – Whether to always drop or not. Defaults to False.

Returns:

Tuple of the tensor ( zeros if dropped ), and a boolean of if the tensor was dropped.

Return type:

tuple(tensor, was_dropped)

training: bool

class utils.aux_models.FactorizedReduction(in_planes, out_planes, stride=2)

Bases: Module

Implements Factozied Reduction.

Reduce both spatial dimensions (width and height) by a factor of 2, and potentially to change the number of output filters https://github.com/melodyguan/enas/blob/master/src/cifar10/general_child.py#L129

__init__(in_planes, out_planes, stride=2)

Initialize FactorizedReduction Module.

Parameters:

in_planes (int) – Input channel count
out_planes (int) – Output channel count
stride (int, optional) – Stride of convolutions. Defaults to 2.

forward(x)

Apply factorized reduction to layer input.

Parameters:: x (torch.Tensor) – Layer Input
Returns:: Layer Output
Return type:: torch.Tensor

training: bool

class utils.aux_models.FixedCell(operation_labels, configuration_indexes, connections, args)

Bases: Module

Implements cell with fixed connections and no path-based dropout.

Generally unused, and probably buggy.

__init__(operation_labels, configuration_indexes, connections, args)

Instantiate Cell Module.

Parameters:

operation_labels (list) – List of operation labels
configuration_indexes (list) – list of configuration indexes
connections (list) – list of connections
args (list) – list of args for Cell

forward(x1, x2)

Apply cell to layer input, and add residual connection.

Parameters:

x1 (torch.Tensor) – Input tensor 1
x2 (torch.Tensor) – Input tensor 2

Returns:

Output Tensor

Return type:

torch.Tensor

training: bool

class utils.aux_models.GlobalPooling1D

Bases: Module

Implements 1D Global Average Pooling.

__init__(): Initialize GlobalPooling1D module.

forward(x)

Apply 1D Global Average Pooling to input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.GlobalPooling2D

Bases: Module

Implements 2D Global Average Pooling.

__init__(): Initialize GlobalPooling2D module.

forward(x)

Apply 2D Global Average Pooling to input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.Identity

Bases: Module

Implements an Identity Module.

forward(inputs)

Apply Identity to Layer Input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.IdentityModule

Bases: Module

Implements an Identity Module.

forward(inputs)

Apply Identity to Layer Input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.Maxout(d, m, k)

Bases: Module

Implements Maxout module.

__init__(d, m, k)

Initialize Maxout object.

Parameters:

d (int) – Input dimension.
m (int) – Number of features remeaining after Maxout.
k (int) – Pool Size

forward(inputs)

Apply Maxout to input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.PoolBranch(in_planes, out_planes, avg_or_max)

Bases: Module

Implements max pooling operations with 1x1 convolutions to fix output size.

__init__(in_planes, out_planes, avg_or_max)

Initialize PoolBranch module.

Parameters:

in_planes (int) – Input channel count
out_planes (int) – Output channel count
avg_or_max (str) – Whether to use average pooling (‘avg’) or max pooling ‘max’

Raises:

ValueError – Unknown Pooling Type.

forward(x)

Apply PoolBranch to Layer Input.

Parameters:: x (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.SeparableConv(in_planes, out_planes, kernel_size, bias=False)

Bases: Module

Implements Separable Convolutions.

__init__(in_planes, out_planes, kernel_size, bias=False)

Initialize SeparableConv Module.

Parameters:

in_planes (int) – Number of input channels.
out_planes (int) – Number of output channels.
kernel_size (int) – Size of kernel
bias (bool, optional) – Whether to add a bias to each convolution or not. Defaults to False.

forward(x)

Apply Separable Convolution to Layer Input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.SeparableConvOld(in_planes, out_planes, kernel_size, bias=False)

Bases: Module

(deprecated) Implements 1D Separable Convolutions.

__init__(in_planes, out_planes, kernel_size, bias=False)

Initialize SeparableConvOld Module.

Parameters:

in_planes (int) – Number of input channels.
out_planes (int) – Number of output channels.
kernel_size (int) – Size of kernel
bias (bool, optional) – (unused) Whether to add a bias to each convolution or not. Defaults to False.

forward(x)

Apply 1D Separable Convolution to Layer Input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.Tensor1DLateralPadding(pad)

Bases: Module

Applies 1DLateral Padding to input.

__init__(pad)

Initialize Tensor1DLateralPadding Module.

Parameters:: pad (int) – Padding amount

forward(inputs)

Apply Lateral Padding to input.

Parameters:: inputs (torch.Tensor) – Layer input
Returns:: Layer output
Return type:: torch.Tensor

training: bool

class utils.aux_models.WeightedCrossEntropyWithLogits(pos_weight)

Bases: Module

Implements cross entropy weighted by given weights.

__init__(pos_weight)

Initialize WeightedCrossEntropyWithLogits module.

Parameters:: pos_weight (np.array) – Weight for each position in batch.

forward(logits, targets)

Get WeightedCrossEntropy Loss.

Parameters:

logits (torch.Tensor) – Logit Tensor
targets (torch.Tensor) – Target labels

Returns:

Weighted cross entropy

Return type:

torch.Tensor

training: bool

utils.aux_models.random() → x in the interval [0, 1).

utils.evaluation_metric module

Implements various evaluation metrics for accuracies and MOSI/MOSEI.

utils.evaluation_metric.eval_mosei_senti(results, truths, exclude_zero=False)

Print out MOSEI metrics given results and ground truth.

Parameters:

results (list) – List of predicted results
truths (list) – List of ground truth
exclude_zero (bool, optional) – Whether to include zero or not. Defaults to False.

utils.evaluation_metric.eval_mosei_senti_return(results, truths, exclude_zero=False)

Evaluate MOSEI and return metric list.

Parameters:

results (np.array) – List of predicated values.
truths (np.array) – List of true values.
exclude_zero (bool, optional) – Whether to exclute zero. Defaults to False.

Returns:

Return statistics for MOSEI.

Return type:

tuple(mae, corr, mult_a7, f_score, accuracy)

utils.evaluation_metric.eval_mosi(results, truths, exclude_zero=False)

Evaluate MOSI results given predictions and ground truth.

Same as MOSEI evaluation.

Parameters:

results (list) – List of predicted results
truths (list) – List of ground truth
exclude_zero (bool, optional) – Whether to include zero or not. Defaults to False.

utils.evaluation_metric.multiclass_acc(preds, truths)

Compute the multiclass accuracy w.r.t. groundtruth.

Parameters:

preds – Float array representing the predictions, dimension (N,)
truths – Float/int array representing the groundtruth classes, dimension (N,)

Returns:

Classification accuracy

utils.evaluation_metric.weighted_accuracy(test_preds_emo, test_truth_emo)

Compute multiclass accuracy weighted by class occurence.

Parameters:

test_preds_emo (np.array) – List of predicted labels
test_truth_emo (np.array) – List of true labels.

Returns:

Weighted classification accuracy.

Return type:

float

utils.helper_modules module

Defines some helper nn.module instances.

class utils.helper_modules.Sequential2(a, b)

Bases: Module

Implements a simpler version of sequential that handles inputs with 2 arguments.

__init__(a, b)

Instatiate Sequential2 object.

Parameters:

a (nn.Module) – First module to sequence
b (nn.Module) – Second module

forward(x)

Apply Sequential2 modules to layer input.

Parameters:: x (torch.Tensor) – Layer Input
Returns:: Layer Output
Return type:: torch.Tensor

training: bool

utils.scheduler module

Implements learning rate schedulers used in training loops.

class utils.scheduler.FixedScheduler(lr)

Bases: object

Implements a fixed learning rate for each epoch.

__init__(lr)

Initialize a FixedScheduler Object.

Parameters:: lr (float) – Learning rate

step()

Update learning rate for scheduler.

Returns:: Updated learning rate
Return type:: float

update_optimizer(optimizer)

Update optimizer instance with current learning rate.

Parameters:: optimizer (nn.optim.Optimizer) – Optimizer instance to modify

class utils.scheduler.LRCosineAnnealingScheduler(eta_max, eta_min, Ti, Tmultiplier, num_batches_per_epoch)

Bases: object

Implements an LRCosineAnnealingScheduler for lr adjustment.

__init__(eta_max, eta_min, Ti, Tmultiplier, num_batches_per_epoch)

Initialize LRCosineAnnealingScheduler Object.

Parameters:

eta_max (float) – Max eta.
eta_min (float) – Minimum eta.
Ti (float) – Initial temperature
Tmultiplier (float) – Temperature multiplier
num_batches_per_epoch (int) – Number of batches per epoch.

step()

Apply scheduler one step, and adjust the learning rate accordingly.

Returns:: Adjusted learning rate
Return type:: float

update_optimizer(optimizer)

Apply current scheduler learning rate to optimizer.

Parameters:: optimizer (torch.optim.Optimizer) – Torch optimizer instance.

utils.search_tools module

Created on Tue Sep 11 18:42:17 2018 @author: juanma

utils.search_tools.compute_temperature(iteration, init, final, decay)

Compute temperature for a given round of the MFAS procedure.

Parameters:

iteration (int) – Iteration index
init (float) – Initial temperature
final (float) – Final temperature
decay (float) – Temperature decay rate

Returns:

Temperature for this round of MFAS.

Return type:

float

utils.search_tools.merge_unfolded_with_sampled(previous_top_k_configurations, unfolded_configurations, layer)

Given a list of top k configurations, and an unfolded single layer configuration, merge them together at the given layer index.

Parameters:

previous_top_k_configurations (list) – List of configurations of size (seq_len, 3)
unfolded_configurations (list) – Configuration for a single layer (,3)
layer (int) – Index of layer to add unfolded configuration to.

Raises:

ValueError – If there are no top k configurations, layer should be 0.

Returns:

Merged list of configurations.

Return type:

list

utils.search_tools.predict_accuracies_with_surrogate(configurations, surrogate, device)

Use surrogate to predict the effectiveness of different input configurations.

Parameters:

configurations (list[dict]) – List of configurations to search from.
surrogate (nn.Module) – Learned surrogate cost model for configuration space.
device (string) – Device to place computation on.

Returns:

Accuracy per configuration.

Return type:

list[float]

utils.search_tools.sample_k_configurations(configurations, accuracies_, k, temperature)

Sample k configurations from list of configurations and accuracies, based on past performance.

Parameters:

configurations (list) – List of configurations.
accuracies (list) – List of accuracies for each config.
k (int) – Number of configurations to sample on.
temperature (float) – Temperature for the sample distribution.

Returns:

List of sampled configurations.

Return type:

list

utils.search_tools.sample_k_configurations_directly(k, max_progression_levels, get_possible_layer_configurations_fun)

Sample k configurations given a set number of progression_levels.

Parameters:

k (int) – Number of configurations to sampl.e
max_progression_levels (int) – Maximum number of sample configurations.
get_possible_layer_configurations_fun (fn) – Function to get layer configurations given some index input.

Returns:

List of sampled configurations.

Return type:

list

utils.search_tools.sample_k_configurations_uniform(configurations, k)

Sample k configurations uniformly.

Parameters:

configurations (list) – List of configurations to sample from.
k (int) – Number of configurations to sample.

Returns:

List of sampled configurations.

Return type:

list

utils.search_tools.train_surrogate(surrogate, surrogate_dataloader, surrogate_optimizer, surrogate_criterion, ep, device)

Train surrogate model using surrogate dataloader.

Parameters:

surrogate (nn.Module) – Surrogate cost model instance.
surrogate_dataloader (torch.utils.data.DataLoader) – Data loader for surrogate instance, mapping configurations to accuracies
surrogate_optimizer (torch.optim.Optimizer) – Optimizer for surrogate parameters
surrogate_criterion (nn.Module) – Loss function for surrogate instance.
ep (int) – Number of epochs.
device (string) – Device to train on.

Returns:

Loss of surrogate with current training data.

Return type:

float

utils.search_tools.update_surrogate_dataloader(surrogate_dataloader, configurations, accuracies)

Update surrogate dataloader with new configurations.

Parameters:

surrogate_dataloader (SurrogateDataloader) – Data Loader for Surrogate Cost Model.
configurations (list) – List of new configurations to add.
accuracies (list[float]) – List of accuracies to train cost model with.

utils.surrogate module

Created on Thu Jul 26 12:10:35 2018 @author: juanma

class utils.surrogate.SimpleRecurrentSurrogate(num_hidden=100, number_input_feats=3, size_ebedding=100)

Bases: Module

Defines simple surrogate for MFAS using a single LSTM layer.

__init__(num_hidden=100, number_input_feats=3, size_ebedding=100)

Initialize SimpleRecurrentSurrogate Module.

Parameters:

num_hidden (int, optional) – Hidden dimension size of LSTM. Defaults to 100.
number_input_feats (int, optional) – Input dimension size. Defaults to 3.
size_ebedding (int, optional) – Hidden dimension size before LSTM portion. Defaults to 100.

eval_model(sequence_of_operations_np, device)

Apply SimpleRecurrentSurrogate to a list of operations ( a configuration ) to get accuracy predictions.

Parameters:

sequence_of_operations_np (np.array[int]) – List of operations for this configuration. Size len_seq x input_size.
device (torch.utils.data.device) – Device to train on.

Returns:

Array of predicted accuracies.

Return type:

np.array

forward(sequence_of_operations)

Apply SimpleRecurrentSurrogate to list of configurations, to get accuracy predictions.

Parameters:: sequence_of_operations (list) – List of configurations to predict accuracies.
Returns:: Predicted accuracies.
Return type:: nn.Tensor

training: bool

class utils.surrogate.SurrogateDataloader

Bases: object

Implements a data loader for the surrogate instance, predicting accuracies from configurations.

__init__(): Initialize SurrogateDataloader Instance.

add_datum(datum_conf, datum_acc)

Add data to surrogate data loader

Parameters:

datum_conf (list) – List of operations for a configuration.
datum_acc (list[float]) – Accuracies for this configuration.

get_data(to_torch=False)

Get data for training

Parameters:: to_torch (bool, optional) – Whether to turn output to torch tensors. Defaults to False.
Returns:: Data for surrogate instance to train on.
Return type:: list[np.array|torch.tensor]

get_k_best(k)

Get K best configurations, given all that has been sampled so far.

Parameters:: k (int) – Number of top configurations to get.
Returns:: Tuple of the list of configurations, their accuracies, and their position in the dataloader.
Return type:: tuple(configs, accuracies, index)

utils.surrogate.train_simple_surrogate(model, criterion, optimizer, data_tensors, num_epochs, device)

Train simple surrogate for MFAS procedure.

Parameters:

model (nn.Module) – Model to train on.
criterion (nn.Module) – Loss function to train on.
optimizer (nn.optim.Optimizer) – Optimizer to apply.
data_tensors (torch.Tensor) – Dataset to train on.
num_epochs (int) – Number of epochs to train this surrogate on.
device (torch.device) – Device to train on.

Returns:

Loss of this surrogate.

Return type:

float

utils package

Submodules

utils.AUPRC module

utils.aux_models module

utils.evaluation_metric module

utils.helper_modules module

utils.scheduler module

utils.search_tools module

utils.surrogate module

Module contents