I have a dataframe with a few series that contain lists of floats that includes nan values. Eg.
s[0] = '[1.21, 1.21, nan, nan, 100]'
These strings I want to convert to lists using literal_eval. When I try I get the error ValueError: malformed node or string on line 1: because as per the docs, nan values cannot be converted as these values are not recognised.
What is the best way of converting the nan values within the string, to None and then converting back to np.nan values after applying literal_eval?
>Solution :
Solution like described in question – but get Nones instead NaNs:
s.str.replace('nan', 'None', regex=True).apply(ast.literal_eval)
If need np.nans use costom function:
def convert(x):
out = []
for y in x.strip('[]').split(', '):
try:
out.append(ast.literal_eval(y))
except:
out.append(np.nan)
return out
s.apply(convert)
Another idea is convert all values to floats:
f = lambda x: [float(y) for y in x.strip('[]').split(', ')]
s.apply(f)
pd.Series([[float(y) for y in x.strip('[]').split(', ')] for x in s],
index=s.index)