How do I fill NaN values with different random numbers on Python?

April 2, 2022

I want to replace the missing values from a column with people’s ages (which also contains numerical values, not only NaN values) but everything I’ve tried so far either doesn’t work how I want it to or it doesn’t work at all.

I wish to apply a random variable generator which follows a normal distribution using the mean and standard deviation obtained with that column.

I have tried the following:

Replacing with numpy, replaces NaN values but with the same number for all of them

df_travel['Age'] = df_travel['Age'].replace(np.nan, round(rd.normalvariate(age_mean, age_std),0))

Fillna with pandas, also replaces NaN values but with the same number for all of them

df_travel['Age'] = df_travel['Age'].fillna(round(rd.normalvariate(age_mean, age_std),0))

Applying a function on the dataframe with pandas, replaces NaN values but also changes all existing numerical values (I only wish to fill the NaN values)
```
df_travel['Age'] = df_travel['Age'].where(df_travel['Age'].isnull() == True).apply(lambda v: round(rd.normalvariate(age_mean, age_std),0))
```

Any ideas would be appreciated. Thanks in advance.

>Solution :

Series.fillna can accept a Series, so generate a random array of size len(df_travel):

rng = np.random.default_rng(0)
mu = df_travel['Age'].mean()
sd = df_travel['Age'].std()

filler = pd.Series(rng.normal(loc=mu, scale=sd, size=len(df_travel)))
df_travel['Age'] = df_travel['Age'].fillna(filler)