I have a large table couple of millions of pairs of ints [[1,2],[45,101],[22,222] etc..].
What is the quickest way in Python to remove duplicates ?
Creating empty list and appending it "if not in" doesn’t work since it takes ages. Converting to Numpy and use "isin" I can’t seem to get it to work on pairs.
>Solution :
you can do the following
arr = [[1,2],[45,101],[22,222], [1,2]]
arr = set(tuple(i) for i in arr)
if you want to convert it back to list
arr = [list(i) for i in arr]