How to fix random seed in pytorch, while keeping the dropout randomness?

I am trying to approximate a Bayesian model by keeping the dropout probability during both training and inference (Monte Carlo dropout), in order to obtain the epistemic uncertainty of the model.

Is there a way to fix all source of randomness for reproducibility (random seed), but to maintain the randomness of the dropout?

# Set random seed for reproducibility
seed = 123
torch.manual_seed(seed)
random.seed(seed)
np.random.seed(seed)

# Training and Inference phase (with dropout)
dropout_mask = torch.bernoulli(torch.full_like(input, 1 - self.dropout))
skip = self.skip0(input * dropout_mask / (1 - self.dropout))

for i in range(self.layers):
    residual = x
    filter = self.filter_convs[i](x)
    filter = torch.tanh(filter)
    gate = self.gate_convs[i](x)
    gate = torch.sigmoid(gate)
    x = filter * gate

    dropout_mask = torch.bernoulli(torch.full_like(x, 1 - self.dropout))
    x = x * dropout_mask / (1 - self.dropout)

    s = x
    s = self.skip_convs[i](s)
    skip = s + skip
    if self.gcn_true:
        x = self.gconv1[i](x, adp) + self.gconv2[i](x, adp.transpose(1, 0))
    else:
        x = self.residual_convs[i](x)

    x = x + residual[:, :, :, -x.size(3):]
    if idx is None:
        x = self.norm[i](x, self.idx)
    else:
        x = self.norm[i](x, idx)

skip = self.skipE(x) + skip
x = F.relu(skip)
x = F.relu(self.end_conv_1(x))
x = self.end_conv_2(x)

return x

The above code produces same result everytime, which is not what I am trying to do.

>Solution :

Maybe consider setting and resetting the seed for the parts you want to be reproducible?

# Training and Inference phase (with dropout)
torch.manual_seed(torch.initial_seed())
dropout_mask = torch.bernoulli(torch.full_like(input, 1 - self.dropout))
torch.manual_seed(seed) 
skip = self.skip0(input * dropout_mask / (1 - self.dropout))

Might want to check out torch.cuda.manual_seed_all(seed) as well.

Also, there are some limits to reproducibility, at least if you don’t want a decrease in performance: https://pytorch.org/docs/stable/notes/randomness.html.