Weights initialization¶
-
class
model.weights.BaseWeights[source]¶ Bases:
objectBase class for weights initialization
References
-
get(size)[source]¶ Initialize the weigths matrix according to the specialization
- Parameters
size (tuple) – Weights matrix shape
- Returns
weights – Matrix of weights with the given shape
- Return type
array-like
-
property
name¶ Get the name of the weight initializer function
-
-
class
model.weights.GlorotNormal[source]¶ Bases:
model.weights.BaseWeightsGlorot normal initializer, also called Xavier normal initializer.
It draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / (inputs + outputs)) 4 where inputs is the number of input units in the weight matrix and outputs is the number of output units in the weight matrix.
References
- 4
Glorot & Bengio, AISTATS 2010. http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf
-
class
model.weights.GlorotUniform[source]¶ Bases:
model.weights.BaseWeightsGlorot uniform initializer, also called Xavier uniform initializer.
It draws samples from a uniform distribution within [-limit, limit] where limit is sqrt(6 / (inputs + outputs)) 3 where inputs is the number of input units in the weight matrix and outputs is the number of output units in the weight matrix.
References
- 3
Glorot & Bengio, AISTATS 2010. http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf
-
class
model.weights.HeNormal[source]¶ Bases:
model.weights.BaseWeightsHe normal initializer.
It draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / inputs) 6 where inputs is the number of input units in the weight matrix.
References
- 6
He et al., http://arxiv.org/abs/1502.01852
-
class
model.weights.HeUniform[source]¶ Bases:
model.weights.BaseWeightsHe uniform variance scaling initializer.
It draws samples from a uniform distribution within [-limit, limit] where limit is sqrt(6 / inputs) 5 where inputs is the number of input units in the weight matrix.
References
- 5
He et al., http://arxiv.org/abs/1502.01852
-
class
model.weights.LecunUniform[source]¶ Bases:
model.weights.BaseWeightsLeCun uniform initializer.
It draws samples from a uniform distribution within [-limit, limit] where limit is sqrt(3 / inputs) 2 where inputs is the number of input units in the weight matrix.
References
- 2
LeCun 98, Efficient Backprop, http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
-
class
model.weights.Normal(mu=0.0, std=1.0)[source]¶ Bases:
model.weights.BaseWeightsSample initial weights from the Gaussian distribution.
Initial weight parameters are sampled from N(mean, std).
- Parameters
mu (float (default=0.)) – Mean of initial parameters.
std (float (default=1.)) – Std of initial parameters.
-
class
model.weights.Ones[source]¶ Bases:
model.weights.BaseWeightsInitialize weights with one values
-
class
model.weights.Orthogonal(gain=1.0)[source]¶ Bases:
model.weights.BaseWeightsIntialize weights as Orthogonal matrix.
Orthogonal matrix initialization 7. For n-dimensional shapes where n > 2, the n-1 trailing axes are flattened. For convolutional layers, this corresponds to the fan-in, so this makes the initialization usable for both dense and convolutional layers.
- Parameters
gain (float or 'relu'.) – Scaling factor for the weights. Set this to
1.0for linear and sigmoid units, to ‘relu’ orsqrt(2)for rectified linear units, and tosqrt(2/(1+alpha**2))for leaky rectified linear units with leakinessalpha. Other transfer functions may need different factors.
References
- 7
Saxe, Andrew M., James L. McClelland, and Surya Ganguli. “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.” arXiv preprint arXiv:1312.6120 (2013).
-
class
model.weights.TruncatedNormal(mu=0.0, std=1.0)[source]¶ Bases:
model.weights.BaseWeightsGenerate draws from a truncated normal distribution via rejection sampling.
- Parameters
mean (float or array_like of floats) – The mean/center of the distribution
std (float or array_like of floats) – Standard deviation (spread or “width”) of the distribution.
out_shape (int or tuple of ints) – Output shape. If the given shape is, e.g.,
(m, n, k), thenm * n * ksamples are drawn.
Notes
The rejection sampling regimen draws samples from a normal distribution with mean mean and standard deviation std, and resamples any values more than two standard deviations from mean.
References
-
class
model.weights.Uniform(scale=0.05)[source]¶ Bases:
model.weights.BaseWeightsSample initial weights from the uniform distribution.
Parameters are sampled from U(a, b).
- Parameters
scale (float or tuple.) – When std is None then range determines a, b. If range is a float the weights are sampled from U(-range, range). If range is a tuple the weights are sampled from U(range[0], range[1]).
-
class
model.weights.Zeros[source]¶ Bases:
model.weights.BaseWeightsInitialize weights with zero values