psiz.keras
psiz.keras.constraints
Module of Keras constraints.
- class psiz.keras.constraints.Center(axis=0)
Constrains the weights to be zero-centered.
This constraint can be used to improve the numerical stability of an embedding.
- get_config()
Return configuration.
- class psiz.keras.constraints.GreaterEqualThan(min_value=0.0)
Constrains the weights to be greater/equal than a value.
- get_config()
Return configuration.
- class psiz.keras.constraints.GreaterThan(min_value=0.0)
Constrains the weights to be greater than a value.
- get_config()
Return configuration.
- class psiz.keras.constraints.LessEqualThan(max_value=0.0)
Constrains the weights to be greater/equal than a value.
- get_config()
Return configuration.
- class psiz.keras.constraints.LessThan(max_value=0.0)
Constrains the weights to be less than a value.
- get_config()
Return configuration.
psiz.keras.initializers
Module of Keras initializers.
psiz.keras.layers
Module of Keras layers.
- class psiz.keras.layers.ALCOVECell(*args, **kwargs)
An RNN-compatible cell implementing ALCOVE.
- build(input_shape)
Build.
- Parameters:
input_shape – Expects a dictionary that contains “categorize_stimulus_set” with shape = (batch_size, [1,]).
- call(inputs, states, training=None)
Call.
- Parameters:
inputs["stimulus_set"] – The indices of the stimuli. shape=(batch_size, 1)
inputs["objective_query_label"] – One-hot encoding of (objectively correct) query label. shape=(batch_size, n_output)
inputs["gate_weights"] (optional) – Tensor(s) containing gate weights. The actual key value(s) will depend on how the user initialized the layer.
states[0] – A tensor representing batch-specific attention weights that modify the Minkowski distance.
states[1] – A tensor representing batch-specific association weights that map RBF activity to class output activitiy.
- distance(inputs)
Compute distance
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- get_initial_state(inputs=None, batch_size=None, dtype=None)
Get initial state.
- Returns:
A list of tensors. The first Tensor represents “attention” weights. The second Tensor represents “association” weights. The attention weights include a sington dimension in anticipation of broadcasting operations in the call method.
- Return type:
initial_state
- humble_teacher_loss(y, y_pred)
Humble teacher loss as described in ALCOVE model.
- Parameters:
y – shape=(batch_size, units)
y_pred – shape=(batch_size, units)
- Returns:
Humble teacher loss.
- class psiz.keras.layers.BraidGate(*args, **kwargs)
A layer that routes inputs to subnetworks.
In a BraidGate the subnetwork outputs are re-combined at the end.
The subnetworks can take a list of inputs, but each subnetwork must output a single tensor. The final output shape must be the same for all subnetworks.
For more information see psiz.keras.layers.Gate
- call(inputs)
Call.
- Parameters:
inputs – Data Tensors. Can be an n-tuple or single-level dictionary containing Tensors. If n-tuple, the trailing Tensor is assumed to be a “gate weights” Tensor. If a dictionary, one of the fields is assumed to be gate_weights. The tuple format follows [data Tensor, [data Tensor, …], gate_weights Tensor]. The data Tensor(s) follows shape=(batch, m, [n, …]). The gate_weights Tensor follows shape=(batch, g)
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- class psiz.keras.layers.Combiner(*args, **kwargs)
A layer that combines multiple input channels based on provided mixing weights.
A Combiner layer is useful for a mixture of experts.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs –
A list of inputs. First input is assumed to be the mixing weights. mixing_weights: A Tensor of:
shape=(batch_size, n_split) or shape=(batch_size, sequence_length, n_split)
- Returns:
A combined signal.
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- class psiz.keras.layers.CosineSimilarity(*args, **kwargs)
Compute the cosine similarity between pairs of vectors.
Computes the cosine similarity: sum_i w_i u_i v_i / ((sum_i w_i u_i^2)(sum_i w_i v_i^2)), where w is a learnable vector of weights.
Follows the scipy implementation of weighted cosine similarity (scipy.spatial.distance.cosine), except outputs similarity, not distance.
The last axis of the input tensors is consumed in order to compute an inner product (see call method). It is assumed that both input tensors have the same rank, are broadcast-compatible, and have the same size for the last axis.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A list of two tensors denoting a the set of vectors to compute pairwise distances. Each tensor is assumed to have the same shape and be at least rank-2. Any additional tensors in the list are ignored. shape = (batch_size, [n, m, …] n_dim)
- Returns:
shape = (batch_size, [n, m, …])
- get_config()
Return layer configuration.
- class psiz.keras.layers.Drop(*args, **kwargs)
A wrapper layer that drops part of the inputs.
Assumues inputs provided to call is a list of Tensors.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – a n-tuple or list containing Tensors.
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- class psiz.keras.layers.EmbeddingGammaDiag(*args, **kwargs)
Gamma distribution embedding.
Each embedding point is characterized by a Gamma distribution.
- build(input_shape=None)
Build embeddings distribution.
- call(inputs)
Call.
- property embeddings
Return embeddings.
- get_config()
Return layer configuration.
- class psiz.keras.layers.EmbeddingLaplaceDiag(*args, **kwargs)
A distribution-based embedding.
Each embedding point is characterized by a Laplace distribution with a diagonal scale matrix.
- build(input_shape=None)
Build embeddings distribution.
- call(inputs)
Call.
- property embeddings
Return embeddings.
- property scale
Return embeddings.
- class psiz.keras.layers.EmbeddingLogNormalDiag(*args, **kwargs)
A distribution-based embedding.
Each embedding point is characterized by a Log-Normal distribution with a diagonal scale matrix.
- build(input_shape=None)
Build embeddings distribution.
- call(inputs)
Call.
- property embeddings
Return embeddings.
- property scale
Return embeddings.
- class psiz.keras.layers.EmbeddingLogitNormalDiag(*args, **kwargs)
A distribution-based embedding.
Each embedding point is characterized by LogitNormal distribution with a diagonal scale matrix.
- build(input_shape=None)
Build embeddings distribution.
- call(inputs)
Call.
- property embeddings
Return embeddings.
- property scale
Return embeddings.
- class psiz.keras.layers.EmbeddingNormalDiag(*args, **kwargs)
A distribution-based embedding.
Each embedding point is characterized by a Normal distribution with a diagonal scale matrix.
- build(input_shape=None)
Build embeddings distribution.
- call(inputs)
Call.
- property embeddings
Return embeddings.
- property scale
Return embeddings.
A class for wrapping a shared Embedding.
Build.
Call.
Getter method for embeddings.
Return distribution that creates copies of the source distribution for each stimulus and dimension.
The incoming distribution has, batch_shape=[] event_shape=[1, 1]
We require a distribution with, event_shape=[self.input_dim, self.output_dim].
Create layer from configuration.
This method is the reverse of get_config, capable of instantiating the same layer from the config dictionary.
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
A layer instance.
- Return type:
layer
Return configuration.
- class psiz.keras.layers.EmbeddingTruncatedNormalDiag(*args, **kwargs)
A distribution-based embedding.
Each embedding point is characterized by a Truncated Normal distribution with a diagonal scale matrix.
- build(input_shape=None)
Build embeddings distribution.
- call(inputs)
Call.
- property embeddings
Return embeddings.
- get_config()
Return layer configuration.
- property scale
Return embeddings.
- class psiz.keras.layers.EmbeddingVariational(*args, **kwargs)
Variational analog of Embedding layer.
- call(inputs, training=None)
Call.
- property embeddings
Getter method for (posterior) embeddings.
- property input_dim
Getter method for embeddings input_dim.
- property mask_zero
Getter method for embeddings mask_zero.
- property output_dim
Getter method for embeddings output_dim.
- class psiz.keras.layers.ExponentialSimilarity(*args, **kwargs)
Exponential family similarity function.
This exponential-family similarity function is parameterized as: s(x,y) = exp(-beta .* d(x,y).^tau) + gamma, where x and y are n-dimensional vectors. The exponential family function is obtained by integrating across various psychological theories [1,2,3,4].
By default beta=10. and is not trainable to prevent redundancy with trainable embeddings and to prevent short-circuiting any regularizers placed on the embeddings.
References
- [1] Jones, M., Love, B. C., & Maddox, W. T. (2006). Recency
effects as a window to generalization: Separating decisional and perceptual sequential effects in category learning. Journal of Experimental Psychology: Learning, Memory, & Cognition, 32 , 316-332.
- [2] Jones, M., Maddox, W. T., & Love, B. C. (2006). The role of
similarity in generalization. In Proceedings of the 28th annual meeting of the cognitive science society (pp. 405- 410).
- [3] Nosofsky, R. M. (1986). Attention, similarity, and the
identification-categorization relationship. Journal of Experimental Psychology: General, 115, 39-57.
- [4] Shepard, R. N. (1987). Toward a universal law of
generalization for psychological science. Science, 237, 1317-1323.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A tensor of distances.
- Returns:
A tensor of similarities.
- get_config()
Return layer configuration.
- class psiz.keras.layers.Gate(*args, **kwargs)
Abstract layer that routes inputs to subnetworks.
- build(input_shape)
Build.
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- class psiz.keras.layers.GateAdapter(*args, **kwargs)
A layer that adapts inputs for networks with Gates.
- input_keys
List of strings indicating required dictionary keys.
- gating_keys
See __init__ method.
- format_inputs_as_tuple
See __init__ method.
- build(input_shape)
Build.
- call(inputs, training=None, mask=None)
Call.
- Parameters:
inputs – A dictionary of Tensors.
training (optional) – see keras.layers.Layer
mask (optional) – see keras.layers.Layer
- get_config()
Get configuration.
- class psiz.keras.layers.GeneralizedInnerProduct(*args, **kwargs)
Compute the inner product between pairs of vectors.
Computes the inner product z_i^T W z_j, where z_i and z_j exist on R^n and W exists on R^n x R^n. By default, no constraints are placed on the matrix W.
The last axis of the input tensors is consumed in order to compute an inner product (see call method). It is assumed that both input tensors have the same rank, are broadcast-compatible, and have the same size for the last axis.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A list of two tensors denoting a the set of vectors to compute pairwise distances. Each tensor is assumed to have the same shape and be at least rank-2. Any additional tensors in the list are ignored. shape = (batch_size, [n, m, …] n_dim)
- Returns:
shape = (batch_size, [n, m, …])
- get_config()
Return layer configuration.
- class psiz.keras.layers.HeavyTailedSimilarity(*args, **kwargs)
Heavy-tailed family similarity function.
The heavy-tailed similarity function is parameterized as:
s(x,y) = (kappa + (d(x,y).^tau)).^(-alpha),
where x and y are n-dimensional vectors. The heavy-tailed family is a generalization of the Student-t family.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A tensor of distances.
- Returns:
A tensor of similarities.
- get_config()
Return layer configuration.
- class psiz.keras.layers.InnerProduct(*args, **kwargs)
Compute the inner product between pairs of vectors.
Computes the inner product z_i^T W z_j, where z_i and z_j exist on R^n and W exists on R^n x R^n. The constraints placed on the matrix “W” determine the type of the inner product operation. If W is constrained to be the identity matrix, the inner product is a dot product. If W is constrained to be symmetric positive definite, you get a generic inner product.
The layer is implemented using w_tril, the lower triangular Cholesky decomposition of W (W = w_tril @ w_tril^T). This is done because positive definiteness is easier to gaurentee using this parameterization. Internally, a mask is applied to w_tril to ensure the variable w_tril is a lower diagonal matrix.
The last axis of the input tensors is consumed in order to compute an inner product (see call method). It is assumed that both input tensors have the same rank, are broadcast-compatible, and have the same size for the last axis.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A list of two tensors denoting a the set of vectors to compute pairwise distances. Each tensor is assumed to have the same shape and be at least rank-2. Any additional tensors in the list are ignored. shape = (batch_size, [n, m, …] n_dim)
- Returns:
shape = (batch_size, [n, m, …])
- get_config()
Return layer configuration.
- property w
Return w attribute.
- property w_tril
Return w_tril attribute.
- class psiz.keras.layers.InverseSimilarity(*args, **kwargs)
Inverse-distance similarity function.
The inverse-distance similarity function is parameterized as: s(x,y) = 1 / (d(x,y)**tau + mu), where x and y are n-dimensional vectors.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A tensor of distances.
- Returns:
A tensor of similarities.
- get_config()
Return layer configuration.
- class psiz.keras.layers.Logistic(*args, **kwargs)
A layer for learning a parameterized logistic function.
Inputs are converted via the parameterized logistic function,
f(x) = upper / (1 + exp(-rate*(x - midpoint))))
with the following variable meanings: upper: The upper asymptote of the function’s range. midpoint: The midpoint of the function’s domain and point of maximum growth. rate: The growth rate of the logistic function.
- build(input_shape)
Build.
- call(inputs, training=None)
Return logistic function output.
- Parameters:
inputs – A tensor of inputs to the logistic function. shape=(batch_size, n, [m, …])
- Returns:
- The output of the parameterized logistic function.
shape=(batch_size, n, [m, …])
- Return type:
y
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- class psiz.keras.layers.Minkowski(*args, **kwargs)
Minkowski pairwise distance.
- A pairwise Minkowski distance layer that consumes the last axis of
the input tensors (see call method).
NOTE: It is assumed that both tensors have the same rank, are broadcast-compatible, and have the same size for the last axis.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A list of two tensors denoting a the set of vectors to compute pairwise distances. Each tensor is assumed to have the same shape and be at least rank-2. Any additional tensors in the list are ignored. shape = (batch_size, [n, m, …] n_dim)
- Returns:
shape = (batch_size, [n, m, …])
- get_config()
Return layer configuration.
- class psiz.keras.layers.MinkowskiStochastic(*args, **kwargs)
A stochastic Minkowski distance layer.
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A list of two tensors denoting a the set of vectors to compute pairwise distance. Each tensor is assumed to have the same shape and be at least rank-2. Any additional tensors in the list are ignored. shape = (batch_size, [n, m, …] n_dim)
- Returns:
shape=(batch_size, [n, m, …])
- get_config()
Return layer configuration.
- class psiz.keras.layers.MinkowskiVariational(*args, **kwargs)
Variational analog of Embedding layer.
- call(inputs)
Call.
- Parameters:
inputs – A list of two tensors denoting a the set of vectors to compute pairwise distance. Each tensor is assumed to have the same shape and be at least rank-2. Any additional tensors in the list are ignored. shape = (batch_size, [n, m, …] n_dim)
- Returns:
shape=(batch_size, [n, m, …])
- property rho
Getter method for (posterior) rho.
- property w
Getter method for (posterior) w.
- class psiz.keras.layers.Proximity(*args, **kwargs)
Abstract base class for pairwise proximity kernel layer.
- A pairwise proximity layer that consumes the last axis of the input
tensors (see call method).
NOTE: It is assumed that both tensors have the same rank, are broadcast-compatible, and have the same size for the last axis.
- build(input_shape)
Build.
- call(inputs)
Call.
- classmethod from_config(config)
Create from configuration.
- get_config()
Return layer configuration.
- class psiz.keras.layers.SoftRank(*args, **kwargs)
A (stateless) soft rank layer.
A (stateless) layer that outputs a soft rank of items based on incoming ‘strength’ associated with each option.
- call(inputs, training=None)
Return probability of a ranked selection sequence.
- Parameters:
inputs – A tensor indicating the strengths associated with each option. It is assumed that the last axis indicates the different options.
- Returns:
Probability of different behavioral outcomes.
- Return type:
outcome_prob
- class psiz.keras.layers.SoftRankBase(*args, **kwargs)
A base layer for ranking options.
A base layer that outputs a soft rank of items based on incoming ‘strength’ associated with each option.
Inputs are assumed to be the ‘strength’ in favor of an option. The probability of selecting a given option is proportional to its strength. The temperature parameter adjust the determinism of the ranking.
The number of options are inferred when the layer is built. Once, built, the soft rank layer can only be used for inputs with the specified number of options.
- build(input_shape)
Build.
- Parameters:
input_shape – Shape of strength tensor. shape=(batch_size, [m, n, …] n_option).
- classmethod from_config(config)
Creates an operation from its config.
This method is the reverse of get_config, capable of instantiating the same operation from the config dictionary.
Note: If you override this method, you might receive a serialized dtype config, which is a dict. You can deserialize it as follows:
```python if “dtype” in config and isinstance(config[“dtype”], dict):
policy = dtype_policies.deserialize(config[“dtype”])
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
An operation instance.
- get_config()
Return layer configuration.
- class psiz.keras.layers.SoftRankCell(*args, **kwargs)
A stateful soft rank layer.
A stateful layer that outputs a soft rank of items based on incoming ‘strength’ associated with each option. The outcome probabilities at the current timestep are biased by past outcome probabilities.
The mixing with past outcomes is determined by the learnable parameter inertia. When inertia is 0.0, there is no bias from past outcomes. As inertia approaches 1.0, the current outcome probabililites are increasingly dominated by the probabilies associated with past outcomes.
p_{t} = (1- inertia) * outcomes_{t} + inertia * p_{t-1}
- build(input_shape)
Build.
- call(inputs, states, training=None)
Return probability of a ranked selection sequence.
- Parameters:
inputs – A tensor indicating the strengths associated with each option. It is assumed that the last axis indicates the different options.
- Returns:
Probability of different behavioral outcomes.
- Return type:
outcome_prob
- get_initial_state(inputs=None, batch_size=None, dtype=None)
Get initial state.
- class psiz.keras.layers.StochasticEmbedding(*args, **kwargs)
Abstract base class for stochastic embeddings.
Intended to be a drop-in stochastic replacement for keras.layers.Embedding.
- call(inputs)
Call.
- get_config()
Return layer configuration.
- class psiz.keras.layers.StudentsTSimilarity(*args, **kwargs)
Student’s t-distribution similarity function.
The Student’s t-distribution similarity function is parameterized as: s(x,y) = (1 + (((d(x,y)^tau)/alpha))^(-(alpha + 1)/2), where x and y are n-dimensional vectors. The original Student-t kernel proposed by van der Maaten [1] uses a L2 distance (which is governed by the distance kernel), tau=2, and alpha=n_dim-1. By default, all variables are fit to the data.
References
- [1] van der Maaten, L., & Weinberger, K. (2012, Sept).
Stochastic triplet embedding. In Machine learning for signal processing (MLSP), 2012 IEEE international workshop on (p. 1-6). doi:10.1109/MLSP.2012.6349720
- build(input_shape)
Build.
- call(inputs)
Call.
- Parameters:
inputs – A tensor of distances.
- Returns:
A tensor of similarities.
- get_config()
Return layer configuration.
- class psiz.keras.layers.Variational(*args, **kwargs)
An abstract base class for variational layers.
This class can take advantage of a registered KL divergence between the posterior and prior is registered.
- kl_weight
The weighting of the kl term. Should be 1/n_train.
- kl_use_exact
Boolean indicating if a registered KL divergence should be used.
- kl_anneal
An annealing weight that can be accessed using a callback. Iniitalized to one so it has no effect if not used in a callback.
Notes
This layer is not registered as serializable because it is intended to be subclassed. Subclasses must implement call, which should sample from the posterior and call add_kl_loss.
- add_kl_loss(posterior_dist, prior_dist)
Add KL divergence loss.
- build(input_shape)
Build.
- classmethod from_config(config)
Create layer from configuration.
This method is the reverse of get_config, capable of instantiating the same layer from the config dictionary.
- Parameters:
config – A Python dictionary, typically the output of get_config.
- Returns:
A layer instance.
- Return type:
layer
- get_config()
Return configuration.
psiz.keras.models
- class psiz.keras.models.StochasticModel(*args, **kwargs)
An abstract Keras model that accomodates stochastic layers.
Incoming data is transformed by repeating all samples in the batch axis n_sample times for the forward pass. When n_sample is greater than 1, the computed losses and metrics are a better estimate of the expectation. As a side-effect, gradient updates tend to be smoother, reducing the risk of unstable training.
When making predictions, an average across samples is returned.
When calling the model in isolation via the call method, no modifications are made to the inputs.
- n_sample
See init method.
- See `keras.Model` for inherited methods.
- repeat_samples_in_batch_axis()
Transforms data structure by repeating all samples in the batch axis n_sample times.
- average_repeated_samples()
Transforms data structure by averaging over repeated samples.
- disentangle_repeated_samples()
Moves repeated samples to a new axis that has “repeated samples” semantics.
psiz.keras.regularizers
Module of Keras regularizers.
- class psiz.keras.regularizers.AttentionEntropy(rate=0.0)
Entropy-based regularization to encourage sparsity.
- get_config()
Return config.
- class psiz.keras.regularizers.Squeeze(rate=0.0)
Squeeze representation into a low number of dimensions.
Regularizer determines the “max usage” for each dimension by taking the maximum across stimuli. The regularizer places pressure on the representation to only use dimensions if necessary, “squeezing” out dimensions that are not essential for any of the stimuli.
- get_config()
Return config.