How to Remove Duplicates Strings from Slice in Go

Created Apr 25, 2022 at 10:11

Modified Apr 25, 2022 at 10:17

Using a Hash Map

We have defined a function where we are passing the slice values and using the map function we are checking the duplicates and removing them. The ones that are currently implemented are:

package main

import "fmt"

func removeDuplicates(s []string) []string {
  // Hash Map
  m := make(map[string]struct{})
  list := []string{}

  for _, v := range s {
    if _, ok := m[v]; !ok {
      m[v] = struct{}{}
      list = append(list, v)
    }
  }

  return list
}

func main() {
  s := []string{"a", "b", "a", "c"}

  n := removeDuplicates(s)
  fmt.Printf("%q\n", n)
}

["a" "b" "c"]

You can update the slice type, and it will filter out all duplicates data for all types of slices.

You can do in-place replacement guided with a map:

package main

import "fmt"

func removeDuplicates(s []string) []string {
  // Hash Map
  m := make(map[string]struct{})
  w := 0
  for _, v := range s {
    if _, ok := m[v]; !ok {
      m[v] = struct{}{}
      s[w] = v
      w++
    }
  }

  return s[:w]
}

func main() {
  s := []string{"a", "b", "a", "c"}

  n := removeDuplicates(s)
  fmt.Printf("%q\n", n)
}

["a" "b" "c"]

Using sort.Strings Function

You can use sort.Strings() function, it really simple. For example,

package main

import (
  "fmt"
  "sort"
)

func removeDuplicates(s []string) []string {
  if len(s) < 1 {
    return s
  }

  sort.Strings(s)
  w := 1
  for i := 1; i < len(s); i++ {
    if s[i-1] != s[i] {
      s[w] = s[i]
      w++
    }
  }

  return s[:w]
}

func main() {
  s := []string{"a", "b", "a", "c"}

  n := removeDuplicates(s)
  fmt.Printf("%q\n", n)
}