fusions package

Subpackages

Submodules

fusions.MCTN module

Implements MCTN for Fusions.

class fusions.MCTN.Attention(hidden_size)

Bases: Module

Implements Attention Mechanism for MCTN.

__init__(hidden_size)

Initialize Attention Mechanism for MCTN.

Parameters:

hidden_size (int) – Hidden Size of Attention Layer.

forward(hidden, encoder_outputs)

Apply Attention to Input, with Hidden Layers.

Parameters:

  • hidden – Initial hidden state.

  • encoder_outputs – Outputs of Encoder object.

Returns:

Output of Attention layer

Return type:

output

training: bool
class fusions.MCTN.Decoder(hidden_size, output_size, n_layers=1, dropout=0.2)

Bases: Module

Apply a gated GRU to decode the input vector.

Paper/Code Sourced From: https://arxiv.org/pdf/1812.07809.pdf.

__init__(hidden_size, output_size, n_layers=1, dropout=0.2)

Initialize Decoder Mechanism for MCTN.

Parameters:

  • hidden_size (int) – Size of hidden layer.

  • output_size (int) – Size of output layer

  • n_layers (int, optional) – Number of layers in encoder. Defaults to 1.

  • dropout (float, optional) – Dropout percentage. Defaults to 0.2.

forward(input, last_hidden, encoder_outputs)

Apply Decoder Mechanism for MCTN.

Parameters:

  • input – Input to MCTN Mechanism.

  • last_hidden – Last hidden layer input.

  • encoder_outputs – Output of Encoder object.

Returns:

Output of this module.

Return type:

output

training: bool
class fusions.MCTN.Encoder(input_size, hidden_size, n_layers=1, dropout=0.2)

Bases: Module

Apply a gated GRU to encode the input vector.

Paper/Code Sourced From: https://arxiv.org/pdf/1812.07809.pdf.

__init__(input_size, hidden_size, n_layers=1, dropout=0.2)

Create Encoder.

Parameters:

  • input_size – Encoder input size

  • hidden_size – Hidden state size for internal GRU

  • n_layers – Number of layers in recurrent unit

  • dropout – Dropout Probability

forward(src, hidden=None)

Apply Encoder to input.

Parameters:

  • src – Encoder Input

  • hidden – Encoder Hidden State

training: bool
class fusions.MCTN.L2_MCTN(seq2seq_0, seq2seq_1, regression_encoder, head, p=0.2)

Bases: Module

Implements 2-Output MCTN.

__init__(seq2seq_0, seq2seq_1, regression_encoder, head, p=0.2)

Initialize L2_MCTN.

Parameters:

  • seq2seq_0 (nn.Module) – Seq2Seq Module converting input to target 1.

  • seq2seq_1 (nn.Module) – Seq2Seq Module converting input to target 2.

  • regression_encoder (nn.Module) – Encoder applied to joint embedding.

  • head (nn.Module) – Head module.

  • p (float, optional) – Dropout percentage. Defaults to 0.2.

forward(src, trg0=None, trg1=None)

Apply L2_MCTN to input.

Parameters:

  • src (torch.Tensor) – Input tensor.

  • trg0 (torch.Tensor, optional) – Target output for Seq2Seq Module 1 Teacher Forcing. Defaults to None.

  • trg1 (torch.Tensor, optional) – Target output for Seq2Seq Module 2 Teacher Forcing. Defaults to None.

Returns:

Output for L2_MCTN instance.

Return type:

torch.Tensor

training: bool
class fusions.MCTN.MCTN(seq2seq, regression_encoder, head, p=0.2)

Bases: Module

Implements MCTN.

__init__(seq2seq, regression_encoder, head, p=0.2)

Initialize MCTN object.

Parameters:

  • seq2seq (nn.Module) – Seq2Seq module for MCTN.

  • regression_encoder (nn.Module) – Encoder module for MCTN.

  • head (nn.Module) – Head for MCTN.

  • p (float, optional) – Dropout probability. Defaults to 0.2.

forward(src, trg=None)

Apply Cyclic Joint Embedding in MCTN.

Parameters:

  • src (torch.Tensor) – Input Tensor

  • trg (torch.Tensor, optional) – Output Tensor for Teacher-Forcing. Defaults to None.

Returns:

Output after applying MCTN.

Return type:

torch.Tensor

training: bool
class fusions.MCTN.Seq2Seq(encoder, decoder)

Bases: Module

Implements a Seq2Seq Layer for MCTN.

__init__(encoder, decoder)

Initialize Seq2Seq Module.

Parameters:

  • encoder (nn.Module) – Encoder for the Seq2Seq Layer.

  • decoder (nn.Module) – Decoder for the Seq2Seq Layer.

forward(src, trg, teacher_forcing_ratio=0.5)

Apply Seq2Seq Module to Input.

Parameters:

  • src (torch.Tensor) – Seq2Seq Input

  • trg (torch.Tensor) – Seq2Seq Output for Teacher Forcing.

  • teacher_forcing_ratio (float, optional) – Teacher Forcing Ratio. Set to 0 when evaluating. Defaults to 0.5.

Returns:

_description_

Return type:

_type_

training: bool

fusions.MVAE module

Implements MVAE.

class fusions.MVAE.ProductOfExperts(size)

Bases: Module

Return parameters for product of independent experts.

See https://arxiv.org/pdf/1410.7827.pdf for equations.

__init__(size)

Initialize Product of Experts Object.

Parameters:

size (int) – Size of Product of Experts Layer

forward(mus, logvars, eps=1e-08)

Apply Product of Experts Layer.

Parameters:

  • mus (torch.Tensor) – torch.Tensor of Mus

  • logvars (torch.Tensor) – torch.Tensor of Logvars

  • eps (float, optional) – Epsilon for log-exponent trick. Defaults to 1e-8.

Returns:

Output of PoE layer.

Return type:

torch.Tensor, torch.Tensor

training: bool
class fusions.MVAE.ProductOfExperts_Zipped(size)

Bases: Module

Return parameters for product of independent experts.

See https://arxiv.org/pdf/1410.7827.pdf for equations.

__init__(size)

Initialize Product of Experts Object.

Parameters:

size (int) – Size of Product of Experts Layer

forward(zipped, eps=1e-08)

Apply Product of Experts Layer.

Parameters:

  • mus (torch.Tensor) – torch.Tensor of Mus

  • logvars (torch.Tensor) – torch.Tensor of Logvars

  • eps (float, optional) – Epsilon for log-exponent trick. Defaults to 1e-8.

Returns:

Output of PoE layer.

Return type:

torch.Tensor, torch.Tensor

training: bool

fusions.common_fusions module

Implements common fusion patterns.

class fusions.common_fusions.Concat

Bases: Module

Concatenation of input data on dimension 1.

__init__()

Initialize Concat Module.

forward(modalities)

Forward Pass of Concat.

Parameters:

modalities – An iterable of modalities to combine

training: bool
class fusions.common_fusions.ConcatEarly

Bases: Module

Concatenation of input data on dimension 2.

__init__()

Initialize ConcatEarly Module.

forward(modalities)

Forward Pass of ConcatEarly.

Parameters:

modalities – An iterable of modalities to combine

training: bool
class fusions.common_fusions.ConcatWithLinear(input_dim, output_dim, concat_dim=1)

Bases: Module

Concatenates input and applies a linear layer.

__init__(input_dim, output_dim, concat_dim=1)

Initialize ConcatWithLinear Module.

Parameters:

  • input_dim – The input dimension for the linear layer

  • output_dim – The output dimension for the linear layer

Concat_dim:

The concatentation dimension for the modalities.

forward(modalities)

Forward Pass of Stack.

Parameters:

modalities – An iterable of modalities to combine

training: bool
class fusions.common_fusions.EarlyFusionTransformer(n_features)

Bases: Module

Implements a Transformer with Early Fusion.

__init__(n_features)

Initialize EarlyFusionTransformer Object.

Parameters:

n_features (int) – Number of features in input.

embed_dim = 9
forward(x)

Apply EarlyFusion with a Transformer Encoder to input.

Parameters:

x (torch.Tensor) – Input Tensor

Returns:

Layer Output

Return type:

torch.Tensor

training: bool
class fusions.common_fusions.LateFusionTransformer(embed_dim=9)

Bases: Module

Implements a Transformer with Late Fusion.

__init__(embed_dim=9)

Initialize LateFusionTransformer Layer.

Parameters:

embed_dim (int, optional) – Size of embedding layer. Defaults to 9.

forward(x)

Apply LateFusionTransformer Layer to input.

Parameters:

x (torch.Tensor) – Layer input

Returns:

Layer output

Return type:

torch.Tensor

training: bool
class fusions.common_fusions.LowRankTensorFusion(input_dims, output_dim, rank, flatten=True)

Bases: Module

Implementation of Low-Rank Tensor Fusion.

See https://github.com/Justin1904/Low-rank-Multimodal-Fusion for more information.

__init__(input_dims, output_dim, rank, flatten=True)

Initialize LowRankTensorFusion object.

Parameters:

  • input_dims – list or tuple of integers indicating input dimensions of the modalities

  • output_dim – output dimension

  • rank – a hyperparameter of LRTF. See link above for details

  • flatten – Boolean to dictate if output should be flattened or not. Default: True

forward(modalities)

Forward Pass of Low-Rank TensorFusion.

Parameters:

modalities – An iterable of modalities to combine.

training: bool
class fusions.common_fusions.MultiplicativeInteractions2Modal(input_dims, output_dim, output, flatten=False, clip=None, grad_clip=None, flip=False)

Bases: Module

Implements 2-way Modal Multiplicative Interactions.

__init__(input_dims, output_dim, output, flatten=False, clip=None, grad_clip=None, flip=False)
Parameters:

  • input_dims – list or tuple of 2 integers indicating input dimensions of the 2 modalities

  • output_dim – output dimension

  • output – type of MI, options from ‘matrix3D’,’matrix’,’vector’,’scalar’

  • flatten – whether we need to flatten the input modalities

  • clip – clip parameter values, None if no clip

  • grad_clip – clip grad values, None if no clip

  • flip – whether to swap the two input modalities in forward function or not

forward(modalities)

Forward Pass of MultiplicativeInteractions2Modal.

Parameters:

modalities – An iterable of modalities to combine.

training: bool
class fusions.common_fusions.MultiplicativeInteractions3Modal(input_dims, output_dim, task=None)

Bases: Module

Implements 3-Way Modal Multiplicative Interactions.

__init__(input_dims, output_dim, task=None)

Initialize MultiplicativeInteractions3Modal object.

Parameters:

  • input_dims – list or tuple of 3 integers indicating sizes of input

  • output_dim – size of outputs

  • task – Set to “affect” when working with social data.

forward(modalities)

Forward Pass of MultiplicativeInteractions3Modal.

Parameters:

modalities – An iterable of modalities to combine.

training: bool
class fusions.common_fusions.NLgate(thw_dim, c_dim, tf_dim, q_linear=None, k_linear=None, v_linear=None)

Bases: Module

Implements of Non-Local Gate-based Fusion.

See section F4 of https://arxiv.org/pdf/1905.12681.pdf for details

__init__(thw_dim, c_dim, tf_dim, q_linear=None, k_linear=None, v_linear=None)

q_linear, k_linear, v_linear are none if no linear layer applied before q,k,v.

Otherwise, a tuple of (indim,outdim) is required for each of these 3 arguments.

Parameters:

  • thw_dim – See paper

  • c_dim – See paper

  • tf_dim – See paper

  • q_linear – See paper

  • k_linear – See paper

  • v_linear – See paper

forward(x)

Apply Low-Rank TensorFusion to input.

Parameters:

x – An iterable of modalities to combine.

training: bool
class fusions.common_fusions.Stack

Bases: Module

Stacks modalities together on dimension 1.

__init__()

Initialize Stack Module.

forward(modalities)

Forward Pass of Stack.

Parameters:

modalities – An iterable of modalities to combine

training: bool
class fusions.common_fusions.TensorFusion

Bases: Module

Implementation of TensorFusion Networks.

See https://github.com/Justin1904/TensorFusionNetworks/blob/master/model.py for more and the original code.

__init__()

Instantiates TensorFusion Network Module.

forward(modalities)

Forward Pass of TensorFusion.

Parameters:

modalities – An iterable of modalities to combine.

training: bool

fusions.searchable module

Implements fusion network structure for MFAS.

See https://github.com/slyviacassell/_MFAS/tree/master/models for hyperparameter details.

class fusions.searchable.Searchable(layered_encoders, rep_size, classes, conf, sub_sizes, alphas=False)

Bases: Module

Implements MFAS’s Searchable fusion module.

__init__(layered_encoders, rep_size, classes, conf, sub_sizes, alphas=False)

Instantiate Searchable Module.

Parameters:

  • layered_encoders (List) – List of nn.Modules for each encoder

  • rep_size (int) – Representation size from unimodals

  • classes (int) – Number of classes

  • conf (Config) – Config instance that generates the layer in question.

  • sub_sizes (int) – Sub sizes

  • alphas (bool, optional) – Whether to generate alphas. Defaults to False.

alphasgen()

Generate alpha-layers if stated to do so.

central_params()

Define parameters for central module.

fcs()

Create fullyconnected layers given config.

forward(inputs)

Apply Searchable Module to Layer Inputs.

Parameters:

inputs (torch.Tensor) – List of input tensors

Returns:

Layer Output

Return type:

torch.Tensor

training: bool
fusions.searchable.get_possible_layer_configurations(max_labels)

Generate possible layer configurations.

Parameters:

max_labels (int) – Maximum number of labels

Returns:

List of Configuration Instances.

Return type:

list

fusions.searchable.train_sampled_models(sampled_configurations, searchable_type, dataloaders, use_weightsharing, device, unimodal_files, rep_size, classes, sub_sizes, batch_size, epochs, eta_max, eta_min, Ti, Tm, return_model=False, premodels=False, preaccuracies=False, train_only_central_params=True, state_dict={})

Train sampled configurations from MFAS.

Parameters:

  • sampled_configurations (List[config]) – List of configurations to train on.

  • searchable_type (rn) – Function to create full model from configurations.

  • dataloaders (List[torch.util.data.DataLoaders]) – List of dataloaders to train on.

  • use_weightsharing (bool) – Whether to use weightsharing or not.

  • device (torch.device) – Device to train on.

  • unimodal_files (List[path]) – List of unimodal encoder paths to train on.

  • rep_size (int) – Internal Representation Size

  • classes (int) – Number of classes

  • sub_sizes (int) – Sub sizes.

  • batch_size (int) – Batch size to train on.

  • epochs (int) – Number of epochs to train on.

  • eta_max (float) – Minimum eta of LRCosineAnnealingScheduler

  • eta_min (float) – Maximum eta of LRCosineAnnealingScheduler

  • Ti (float) – Ti for LRCosineAnnealingScheduler

  • Tm (float) – Tm for LRCosineAnnealingScheduler

  • return_model (bool, optional) – Whether to return the trained module as nn.Module. Defaults to False.

  • premodels (bool, optional) – Whether there are pre-trained unimodal models or not. Defaults to False.

  • preaccuracies (bool, optional) – (Unused). Defaults to False.

  • train_only_central_params (bool, optional) – Whether to train only central parameters or not. Defaults to True.

  • state_dict (_type_, optional) – (unused). Defaults to dict().

Returns:

List of model accuracies.

Return type:

List

fusions.searchable.train_track_acc(model, criteria, optimizer, scheduler, dataloaders, dataset_sizes, device=None, num_epochs=200, verbose=False, multitask=False)

Get best accuracy for model when training on a set of dataloaders.

Parameters:

  • model (nn.Module) – Model to train on.

  • criteria (nn.Module) – Loss function.

  • optimizer (nn.optim.Optimizer) – Optimizer instance

  • scheduler (nn.optim.Scheduler) – LRScheduler to use.

  • dataloaders (List) – List of dataloaders to train on.

  • dataset_sizes (List) – List of the sizes of the datasets

  • device (torch.device, optional) – Device to train on. Defaults to None.

  • num_epochs (int, optional) – Number of epochs to train on. Defaults to 200.

  • verbose (bool, optional) – (Unused) Defaults to False.

  • multitask (bool, optional) – Whether to train as a multitask setting. Defaults to False.

Returns:

Best accuracy when training.

Return type:

float

Module contents