Background: I am very confused by my dataframe (df), which when I do some simple analyses is producing random rows for a specific value within my column named ‘ID’ (specifically, when ID == 42). As a result, I have started to do some troubleshooting.
When I try to list all the rows where ID = 42, I do:
data=df.loc[df['ID'] == 42]
And the rows look correct in this new variable called ‘data’. However, when I scroll manually through the original dataframe df (e.g., in the Variable Explorer on Spyder), I can see there are way more rows for ID=42 that are not being printed to ‘data’.
Then, to double check why the ‘ID’ values are showing this weird behavior, I did
print(df['ID'].unique())
And, weirdly, I get this:
[ 20. 31. 42. 42. 84. 142. 198. 248. 280. 288. 352. 378. 459. 498.]
— note that 42 is repeated!
My question is, how can there be two 42s when I use the .unique() function? I thought it was supposed to output all the unique values? If I could understand this better, I could start to understand the rest of the problems that ensue…
Am I missing something about how ‘unique’ works?
Ps. My files are big so I haven’t included them, but if I need to provide more (numerical) context please let me know.
Thanks!
>Solution :
Moving my comment to an answer, as it solved the problem:
print(df['ID'].astype(int).unique())