We can add layer normalization in Pytorch by doing: torch.nn.LayerNorm(shape). However, this is layer normalization with learnable parameters. I.e, it’s the following equation:
Does Pytorch have builtin layer normalization without learnable parameters?
>Solution :
You can use nn.LayerNorm, setting the elementwise flag to False. This way the layer won’t have learnt parameters. See source code for additional reference.