Map-Reduce-Filter pattern processing collection elements

title: Map-Reduce-Filter Pattern for Processing Collection Elements
date: 2022-07-04 10:37:00
toc: false
index_img: http://api.btstu.cn/sjbz/?lx=m_dongman&cid=9
category:

Go
tags:
Users
Traversal
Computation
Statistics
Strings
Functions

During the daily development process, it is common to handle collection types such as arrays, slices, and dictionaries by iterating through them. For example, extracting all age attribute values from a dictionary type user slice and then calculating their sum. The conventional approach is to loop through all the slices, extract the age field values from the user dictionary key-value pairs, and then perform the accumulation one by one.

For simple individual scenarios, this implementation is fine, but it is a typical procedural thinking and the code has almost no reusability: every time you deal with similar problems, you have to write the same code template, such as calculating other field values or modifying type conversion logic, you have to rewrite the implementation code.

In functional programming, we can make this functionality more elegant and reusable through the Map-Reduce technique.

Map-Reduce is not a whole, but needs to be implemented in two steps: Map and Reduce. The Map-Reduce model: first convert the dictionary type slice into a string type slice (Map, literally meaning one-to-one mapping), and then convert the converted slice elements to integers and accumulate them (Reduce, literally meaning reducing multiple collection elements through iterative processing to one).

Sometimes, in order to make the Map-Reduce code more robust (excluding invalid field values) or to perform statistical calculations only on specified ranges of data, we can introduce a Filter on top of Map-Reduce to filter the collection elements.

package main

import (
  "fmt"
  "strconv"
  "time"
)

func main() {
  var users = []map[string]string{
    {
      "name": "xmp",
      "age":  "11",
    },
    {
      "name": "wlm",
      "age":  "24",
    },
    {
      "name": "wjf",
      "age":  "30",
    },
    {
      "name": "lm",
      "age":  "50",
    },
  }
  fmt.Println("Total age:", ageSum(users))

  fmt.Println("Total age:", handlerMapReduce(users))

  fmt.Println("Total age (filtering above 40 and below 20):", handlerValidUsers(users))
}

func ageSum(users []map[string]string) int {
  startTime := time.Now()
  var result int
  for _, user := range users {
    age, _ := strconv.Atoi(user["age"])
    result += age
  }
  fmt.Println("\tageSum total time:", time.Since(startTime))
  return result
}

func handlerValidUsers(users []map[string]string) int {
  validUsers := filterMap(users, func(item map[string]string) bool {
    ageStr, ok := item["age"]
    if !ok {
      return false
    }
    age, err := strconv.Atoi(ageStr)
    if err != nil {
      return false
    }
    if age > 40 || age < 20 {
      return false
    }
    return true
  })
  return handlerMapReduce(validUsers)
}

func filterMap(items []map[string]string, f func(map[string]string) bool) []map[string]string {
  startTime := time.Now()
  var result []map[string]string
  for _, item := range items {
    if f(item) {
      result = append(result, item)
    }
  }
  fmt.Println("\tfilterMap total time:", time.Since(startTime))
  return result
}

func handlerMapReduce(users []map[string]string) int {
  startTime := time.Now()
  items := mapToString(users, func(user map[string]string) string {
    return user["age"]
  })
  result := fieldSum(items, func(str string) int {
    val, _ := strconv.Atoi(str)
    return val
  })
  fmt.Println("\thandlerMapReduce total time:", time.Since(startTime))
  return result
}

func mapToString(items []map[string]string,
  f func(map[string]string) string) []string {
  newSlice := make([]string, len(items))
  for _, item := range items {
    newSlice = append(newSlice, f(item))
  }
  return newSlice
}

func fieldSum(items []string, f func(string) int) int {
  var result int
  for _, item := range items {
    result += f(item)
  }
  return result
}

However, calling Map, Reduce, and Filter functions separately is not very elegant. We can nest them layer by layer through the decorator pattern, or use the pipeline pattern to make the calling logic more readable and elegant.