How to avoid bfill or ffill when calculating pct_change with NaNs

February 9, 2023

For a df like below, I use pct_change() to calculate the rolling percentage changes:

price = [np.NaN, 10, 13, np.NaN, np.NaN, 9]
df = pd. DataFrame(price, columns = ['price'])
df
Out[75]:
    price
0 NaN
1 10.0
2 13.0
3 NaN
4 NaN
5 9.0

But I get these unexpected results:

df.price.pct_change(periods = 1, fill_method='bfill')
Out[76]:
0 NaN
1 0.000000
2 0.300000
3 -0.307692
4 0.000000
5 0.000000
Name: price, dtype: float64

df.price.pct_change(periods = 1, fill_method='pad')
Out[77]:
0 NaN
1 NaN
2 0.300000
3 0.000000
4 0.000000
5 -0.307692
Name: price, dtype: float64

df.price.pct_change(periods = 1, fill_method='ffill')
Out[78]:
0 NaN
1 NaN
2 0.300000
3 0.000000
4 0.000000
5 -0.307692
Name: price, dtype: float64

I hope that while calculating with NaNs, the results will be NaNs instead of being filled forward or backward and then calculated.

May I ask how to achieve it? Thanks.

The expected result:

0 NaN
1 NaN
2 0.300000
3 NaN
4 NaN
5 NaN
Name: price, dtype: float64

Reference:

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pct_change.html

>Solution :

Maybe you can compute the pct manually with diff and shift:

period = 1
pct = df.price.diff().div(df.price.shift(period))
print(pct)

# Output
0    NaN
1    NaN
2    0.3
3    NaN
4    NaN
5    NaN
Name: price, dtype: float64

Update: you can pass fill_method=None

period = 1
pct = df.price.pct_change(periods=period, fill_method=None)
print(pct)

# Output
0    NaN
1    NaN
2    0.3
3    NaN
4    NaN
5    NaN
Name: price, dtype: float64