There seem to be two ways of initializing embedding layers in Pytorch 1.0 using an uniform distribution.
For example you have an embedding layer:
self.in_embed = nn.Embedding(n_vocab, n_embed)
And you want to initialize its weights with an uniform distribution. The first way you can get this done is:
self.in_embed.weight.data.uniform_(-1, 1)
And another one would be:
nn.init.uniform_(self.in_embed.weight, -1.0, 1.0)
My question is: what is the difference between the first and second initialization form. Do both methods do the same thing?
Both are same
torch.manual_seed(3)
emb1 = nn.Embedding(5,5)
emb1.weight.data.uniform_(-1, 1)
torch.manual_seed(3)
emb2 = nn.Embedding(5,5)
nn.init.uniform_(emb2.weight, -1.0, 1.0)
assert torch.sum(torch.abs(emb1.weight.data - emb2.weight.data)).numpy() == 0
Every tensor has a uniform_ method which initializes it with the values from the uniform distribution. Also, the nn.init module has a method uniform_ which takes in a tensor and inits it with values from uniform distribution. Both are same expect first one is using the member function and the second is using a general utility function.
この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。
侵害の場合は、連絡してください[email protected]
コメントを追加