Remove duplicates with low score in list of dicts

February 18, 2022

I currently have the following list of dicts:

[
    {'id': '1', 'sim': 0.81},
    {'id': '1', 'sim': 0.72},
    {'id': '2', 'sim': 0.85},    
    {'id': '2', 'sim': 0.81},
    {'id': '2', 'sim': 0.72}
]

I’d like to remove the duplicates which have not the highest sim and get the following:

[
    {'id': '1', 'sim': 0.81},
    {'id': '2', 'sim': 0.85},
]

>Solution :

One way to go would be to use pandas :

import pandas as pd

d = [
    {'id': '1', 'sim': 0.81},
    {'id': '1', 'sim': 0.72},
    {'id': '2', 'sim': 0.85},
    {'id': '2', 'sim': 0.81},
    {'id': '2', 'sim': 0.72}
]

df = pd.DataFrame(d)

df = df.groupby(['id'], sort=False)['sim'].max()

Then you can keep using it as a Dataframe, or going back to nested dictionnaries depending on what you need.