unique elements in a slice of maps

I’m unmarshaling some json files into []map[string]string{}
but very often the source is dirty with many repeated equal objects

[{"sa1":"8172"},{"sa3":"8175"},{"sa1":"8172"},{"sa3":"8175"},{"sa3":"8175"},{"sa3":"8175"},{"sa1":"8172"},{"sa3":"8175"},{"sa3":"8175"}]

resulting into

map[sa1:8172]
([]map[string]string) (len=9 cap=9) {
 (map[string]string) (len=1) {
  (string) (len=3) "sa1": (string) (len=4) "8172"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa1": (string) (len=4) "8172"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa1": (string) (len=4) "8172"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 }
}

How could i clean the map to contain only unique elements?

>Solution :

One option is to unmarshal the key value pairs directly into a comparable type, like a struct:

type Elem struct {
    k string
    v string
}

func (e *Elem) UnmarshalJSON(d []byte) error {
    m := map[string]string{}
    if err := json.Unmarshal(d, &m); err != nil {
        return err
    }

    for k, v := range m {
        e.k = k
        e.v = v
        return nil
    }
    return nil
}

Once you can compare the elements individually, you could also wrap that in a collection which filters the elements while unmarshaling. Whether to do this implicitly here, or after the fact is a matter of opinion. It may be a better separation of concerns to make filtering its own method, but I included it in UnmarshalJSON for brevity.

type Elems []Elem

func (e *Elems) UnmarshalJSON(d []byte) error {
    tmp := []Elem{}
    err := json.Unmarshal(d, &tmp)
    if err != nil {
        return err
    }

    seen := map[Elem]bool{}
    for _, elem := range tmp {
        if seen[elem] {
            continue
        }
        seen[elem] = true
        *e = append(*e, elem)
    }
    return nil
}

Then you can unmarshal into Elems:

    elems := Elems{}

    err := json.Unmarshal(js, &elems)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(elems)

Which will give you the two unique pairs: [{sa1 8172} {sa3 8175}]

https://go.dev/play/p/U0iqBAjvz-1