[Reprint] Those Go Language JSON Tips You Need to Know

title: 【转载】The Go Language JSON Tricks You Need to Know
date: 2021-08-09 16:33:33
comment: false
toc: true
category:

Golang
tags:
转载
Go
Json

This article is reprinted from: The Go Language JSON Tricks You Need to Know | Li Wenzhou's Blog

This article summarizes the issues and solutions I have encountered regarding the conversion between JSON data and structs in Go language during my projects.

Basic Serialization#

First, let's take a look at the basic usage of json.Marshal() (serialization) and json.Unmarshal (deserialization) in Go.

type Person struct {  
	Name   string  
	Age    int64  
	Weight float64  
}  

func main() {  
	p1 := Person{  
		Name:   "七米",  
		Age:    18,  
		Weight: 71.5,  
	}  
	// struct -> json string  
	b, err := json.Marshal(p1)  
	if err != nil {  
		fmt.Printf("json.Marshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
	// json string -> struct  
	var p2 Person  
	err = json.Unmarshal(b, &p2)  
	if err != nil {  
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("p2:%#v\n", p2)  
}

Output:

str:{"Name":"七米","Age":18,"Weight":71.5}  
p2:main.Person{Name:"七米", Age:18, Weight:71.5}

Introduction to Struct Tags#

Tag is the metadata of a struct, which can be read at runtime through reflection. The Tag is defined after the struct field and is wrapped in a pair of backticks, with the specific format as follows:

`key1:"value1" key2:"value2"`

Struct tags consist of one or more key-value pairs. Keys and values are separated by colons, and values are enclosed in double quotes. The same struct field can have multiple key-value pair tags, separated by spaces.

Using JSON Tags to Specify Field Names#

By default, serialization and deserialization use the struct field names. We can specify the field names generated during JSON serialization by adding tags to the struct fields.

// Use json tag to specify behavior during serialization and deserialization  
type Person struct {  
	Name   string `json:"name"` // Specify to use lowercase name during json serialization/deserialization  
	Age    int64  
	Weight float64  
}

Ignoring a Field#

If you want to ignore a certain field in the struct during JSON serialization/deserialization, you can add - in the tag as follows.

// Use json tag to specify behavior during json serialization and deserialization  
type Person struct {  
	Name   string `json:"name"` // Specify to use lowercase name during json serialization/deserialization  
	Age    int64  
	Weight float64 `json:"-"` // Specify to ignore this field during json serialization/deserialization  
}

Ignoring Empty Value Fields#

When fields in a struct have no value, json.Marshal() does not ignore these fields during serialization but outputs the zero value of the field type by default (for example, the zero value of int and float types is 0, the zero value of string type is "", and the zero value of object types is nil). If you want to ignore these fields with no value during serialization, you can add the omitempty tag to the corresponding fields.

For example:

type User struct {  
	Name  string   `json:"name"`  
	Email string   `json:"email"`  
	Hobby []string `json:"hobby"`  
}  

func omitemptyDemo() {  
	u1 := User{  
		Name: "七米",  
	}  
	// struct -> json string  
	b, err := json.Marshal(u1)  
	if err != nil {  
		fmt.Printf("json.Marshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
}

Output:

str:{"name":"七米","email":"","hobby":null}

If you want to remove empty value fields from the final serialization result, you can define the struct as follows:

// Add omitempty in the tag to ignore empty values  
// Note that here hobby,omitempty is combined as the json tag value, separated by a comma  
type User struct {  
	Name  string   `json:"name"`  
	Email string   `json:"email,omitempty"`  
	Hobby []string `json:"hobby,omitempty"`  
}

At this point, when you execute the above omitemptyDemo, the output will be as follows:

str:{"name":"七米"} // The serialization result does not include email and hobby fields

Ignoring Empty Value Fields in Nested Structs#

First, let's look at a few examples of nested structs:

type User struct {  
	Name  string   `json:"name"`  
	Email string   `json:"email,omitempty"`  
	Hobby []string `json:"hobby,omitempty"`  
	Profile  
}  

type Profile struct {  
	Website string `json:"site"`  
	Slogan  string `json:"slogan"`  
}  

func nestedStructDemo() {  
	u1 := User{  
		Name:  "七米",  
		Hobby: []string{"足球", "双色球"},  
	}  
	b, err := json.Marshal(u1)  
	if err != nil {  
		fmt.Printf("json.Marshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
}

When serializing the anonymous nested Profile, the resulting JSON string is single-layered:

str:{"name":"七米","hobby":["足球","双色球"],"site":"","slogan":""}

To make it a nested JSON string, you need to change it to named nesting or define field tags:

type User struct {  
	Name    string   `json:"name"`  
	Email   string   `json:"email,omitempty"`  
	Hobby   []string `json:"hobby,omitempty"`  
	Profile `json:"profile"`  
}  
// str:{"name":"七米","hobby":["足球","双色球"],"profile":{"site":"","slogan":""}}

If you want to ignore the field when the nested struct is empty, simply adding omitempty is not enough:

type User struct {  
	Name     string   `json:"name"`  
	Email    string   `json:"email,omitempty"`  
	Hobby    []string `json:"hobby,omitempty"`  
	Profile `json:"profile,omitempty"`  
}  
// str:{"name":"七米","hobby":["足球","双色球"],"profile":{"site":"","slogan":""}}

You also need to use a pointer to the nested struct:

type User struct {  
	Name     string   `json:"name"`  
	Email    string   `json:"email,omitempty"`  
	Hobby    []string `json:"hobby,omitempty"`  
	*Profile `json:"profile,omitempty"`  
}  
// str:{"name":"七米","hobby":["足球","双色球"]}

Ignoring Empty Value Fields Without Modifying the Original Struct#

We need to serialize User to JSON, but we do not want to serialize the password and do not want to modify the User struct. In this case, we can create another struct PublicUser that anonymously nests the original User, while specifying the Password field as an anonymous struct pointer type and adding the omitempty tag, as shown in the example code below:

type User struct {  
	Name     string `json:"name"`  
	Password string `json:"password"`  
}  

type PublicUser struct {  
	*User             // Anonymous nesting  
	Password *struct{} `json:"password,omitempty"`  
}  

func omitPasswordDemo() {  
	u1 := User{  
		Name:     "七米",  
		Password: "123456",  
	}  
	b, err := json.Marshal(PublicUser{User: &u1})  
	if err != nil {  
		fmt.Printf("json.Marshal u1 failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  // str:{"name":"七米"}  
}

Elegantly Handling String-Formatted Numbers#

Sometimes, the JSON data passed from the front end may use string-type numbers. In this case, you can add string in the struct tag to inform the JSON package to parse the corresponding field's data from the string:

type Card struct {  
	ID    int64   `json:"id,string"`    // Add string tag  
	Score float64 `json:"score,string"` // Add string tag  
}  

func intAndStringDemo() {  
	jsonStr1 := `{"id": "1234567","score": "88.50"}`  
	var c1 Card  
	if err := json.Unmarshal([]byte(jsonStr1), &c1); err != nil {  
		fmt.Printf("json.Unmarshal jsonStr1 failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("c1:%#v\n", c1) // c1:main.Card{ID:1234567, Score:88.5}  
}

Integers to Floating Points#

In the JSON protocol, there is no distinction between integers and floating points; they are collectively referred to as numbers. After deserialization, the numbers in the JSON string will all become float64 types in Go. The following code demonstrates this issue:

func jsonDemo() {  
	// map[string]interface{} -> json string  
	var m = make(map[string]interface{}, 1)  
	m["count"] = 1 // int  
	b, err := json.Marshal(m)  
	if err != nil {  
		fmt.Printf("marshal failed, err:%v\n", err)  
	}  
	fmt.Printf("str:%#v\n", string(b))  
	// json string -> map[string]interface{}  
	var m2 map[string]interface{}  
	err = json.Unmarshal(b, &m2)  
	if err != nil {  
		fmt.Printf("unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("value:%v\n", m2["count"]) // 1  
	fmt.Printf("type:%T\n", m2["count"])  // float64  
}

In this scenario, if you want to handle numbers more reasonably, you need to use a decoder for deserialization, as shown in the example code below:

func decoderDemo() {  
	// map[string]interface{} -> json string  
	var m = make(map[string]interface{}, 1)  
	m["count"] = 1 // int  
	b, err := json.Marshal(m)  
	if err != nil {  
		fmt.Printf("marshal failed, err:%v\n", err)  
	}  
	fmt.Printf("str:%#v\n", string(b))  
	// json string -> map[string]interface{}  
	var m2 map[string]interface{}  
	// Use decoder for deserialization, specifying to use number type  
	decoder := json.NewDecoder(bytes.NewReader(b))  
	decoder.UseNumber()  
	err = decoder.Decode(&m2)  
	if err != nil {  
		fmt.Printf("unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("value:%v\n", m2["count"]) // 1  
	fmt.Printf("type:%T\n", m2["count"])  // json.Number  
	// Convert m2["count"] to json.Number and call Int64() to get int64 type value  
	count, err := m2["count"].(json.Number).Int64()  
	if err != nil {  
		fmt.Printf("parse to int64 failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("type:%T\n", int(count)) // int  
}

The source code definition of json.Number is as follows:

// A Number represents a JSON number literal.  
type Number string  

// String returns the literal text of the number.  
func (n Number) String() string { return string(n) }  

// Float64 returns the number as a float64.  
func (n Number) Float64() (float64, error) {  
	return strconv.ParseFloat(string(n), 64)  
}  

// Int64 returns the number as an int64.  
func (n Number) Int64() (int64, error) {  
	return strconv.ParseInt(string(n), 10, 64)  
}

When handling number-type JSON fields, we need to first obtain the json.Number type and then call Float64() or Int64() based on the actual type of that field.

Custom Parsing of Time Fields#

The built-in JSON package in Go uses the time format defined in the RFC3339 standard, which imposes many restrictions on serializing time fields.

type Post struct {  
	CreateTime time.Time `json:"create_time"`  
}  

func timeFieldDemo() {  
	p1 := Post{CreateTime: time.Now()}  
	b, err := json.Marshal(p1)  
	if err != nil {  
		fmt.Printf("json.Marshal p1 failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
	jsonStr := `{"create_time":"2020-04-05 12:25:42"}`  
	var p2 Post  
	if err := json.Unmarshal([]byte(jsonStr), &p2); err != nil {  
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("p2:%#v\n", p2)  
}

The output of the above code is as follows:

str:{"create_time":"2020-04-05T12:28:06.799214+08:00"}  
json.Unmarshal failed, err:parsing time ""2020-04-05 12:25:42"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 12:25:42"" as "T"

This means that the built-in JSON package does not recognize commonly used string time formats, such as 2020-04-05 12:25:42.

However, we can implement custom time format parsing by implementing the json.Marshaler/json.Unmarshaler interfaces.

type CustomTime struct {  
	time.Time  
}  

const ctLayout = "2006-01-02 15:04:05"  

var nilTime = (time.Time{}).UnixNano()  

func (ct *CustomTime) UnmarshalJSON(b []byte) (err error) {  
	s := strings.Trim(string(b), "\"")  
	if s == "null" {  
		ct.Time = time.Time{}  
		return  
	}  
	ct.Time, err = time.Parse(ctLayout, s)  
	return  
}  

func (ct *CustomTime) MarshalJSON() ([]byte, error) {  
	if ct.Time.UnixNano() == nilTime {  
		return []byte("null"), nil  
	}  
	return []byte(fmt.Sprintf("\"%s\"", ct.Time.Format(ctLayout))), nil  
}  

func (ct *CustomTime) IsSet() bool {  
	return ct.UnixNano() != nilTime  
}  

type Post struct {  
	CreateTime CustomTime `json:"create_time"`  
}  

func timeFieldDemo() {  
	p1 := Post{CreateTime: CustomTime{time.Now()}}  
	b, err := json.Marshal(p1)  
	if err != nil {  
		fmt.Printf("json.Marshal p1 failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
	jsonStr := `{"create_time":"2020-04-05 12:25:42"}`  
	var p2 Post  
	if err := json.Unmarshal([]byte(jsonStr), &p2); err != nil {  
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("p2:%#v\n", p2)  
}

Custom MarshalJSON and UnmarshalJSON Methods#

The previous method of using custom types is somewhat verbose. Now let's look at a relatively convenient method.

First, you need to know that if you can implement the MarshalJSON()([]byte, error) and UnmarshalJSON(b []byte) error methods for a certain type, then this type will use your custom methods during serialization (MarshalJSON) and deserialization (UnmarshalJSON).

type Order struct {  
	ID          int       `json:"id"`  
	Title       string    `json:"title"`  
	CreatedTime time.Time `json:"created_time"`  
}  

const layout = "2006-01-02 15:04:05"  

// MarshalJSON implements a custom MarshalJSON method for the Order type  
func (o *Order) MarshalJSON() ([]byte, error) {  
	type TempOrder Order // Define a new type with the same fields as Order  
	return json.Marshal(struct {  
		CreatedTime string `json:"created_time"`  
		*TempOrder         // Avoid direct nesting of Order to prevent infinite loop  
	}{  
		CreatedTime: o.CreatedTime.Format(layout),  
		TempOrder:   (*TempOrder)(o),  
	})  
}  

// UnmarshalJSON implements a custom UnmarshalJSON method for the Order type  
func (o *Order) UnmarshalJSON(data []byte) error {  
	type TempOrder Order // Define a new type with the same fields as Order  
	ot := struct {  
		CreatedTime string `json:"created_time"`  
		*TempOrder         // Avoid direct nesting of Order to prevent infinite loop  
	}{  
		TempOrder: (*TempOrder)(o),  
	}  
	if err := json.Unmarshal(data, &ot); err != nil {  
		return err  
	}  
	var err error  
	o.CreatedTime, err = time.Parse(layout, ot.CreatedTime)  
	if err != nil {  
		return err  
	}  
	return nil  
}  

// Custom serialization method  
func customMethodDemo() {  
	o1 := Order{  
		ID:          123456,  
		Title:       "《七米的Go学习笔记》",  
		CreatedTime: time.Now(),  
	}  
	// Use the custom MarshalJSON method to achieve struct -> json string  
	b, err := json.Marshal(&o1)  
	if err != nil {  
		fmt.Printf("json.Marshal o1 failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
	// Use the custom UnmarshalJSON method to achieve json string -> struct  
	jsonStr := `{"created_time":"2020-04-05 10:18:20","id":123456,"title":"《七米的Go学习笔记》"}`  
	var o2 Order  
	if err := json.Unmarshal([]byte(jsonStr), &o2); err != nil {  
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("o2:%#v\n", o2)  
}

Output:

str:{"created_time":"2020-04-05 10:32:20","id":123456,"title":"《七米的Go学习笔记》"}  
o2:main.Order{ID:123456, Title:"《七米的Go学习笔记》", CreatedTime:time.Time{wall:0x0, ext:63721678700, loc:(*time.Location)(nil)}}

Adding Fields Using Anonymous Structs#

Using embedded structs can extend the fields of a struct, but sometimes we do not need to define a new struct separately; we can use anonymous structs to simplify operations:

type UserInfo struct {  
	ID   int    `json:"id"`  
	Name string `json:"name"`  
}  

func anonymousStructDemo() {  
	u1 := UserInfo{  
		ID:   123456,  
		Name: "七米",  
	}  
	// Use an anonymous struct to embed User and add an extra field Token  
	b, err := json.Marshal(struct {  
		*UserInfo  
		Token string `json:"token"`  
	}{  
		&u1,  
		"91je3a4s72d1da96h",  
	})  
	if err != nil {  
		fmt.Printf("json.Marsha failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
	// str:{"id":123456,"name":"七米","token":"91je3a4s72d1da96h"}  
}

Combining Multiple Structs Using Anonymous Structs#

Similarly, you can also use anonymous structs to combine multiple structs for serialization and deserialization:

type Comment struct {  
	Content string  
}  

type Image struct {  
	Title string `json:"title"`  
	URL   string `json:"url"`  
}  

func anonymousStructDemo2() {  
	c1 := Comment{  
		Content: "永远不要高估自己",  
	}  
	i1 := Image{  
		Title: "赞赏码",  
		URL:   "https://www.liwenzhou.com/images/zanshang_qr.jpg",  
	}  
	// struct -> json string  
	b, err := json.Marshal(struct {  
		*Comment  
		*Image  
	}{&c1, &i1})  
	if err != nil {  
		fmt.Printf("json.Marshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("str:%s\n", b)  
	// json string -> struct  
	jsonStr := `{"Content":"永远不要高估自己","title":"赞赏码","url":"https://www.liwenzhou.com/images/zanshang_qr.jpg"}`  
	var (  
		c2 Comment  
		i2 Image  
	)  
	if err := json.Unmarshal([]byte(jsonStr), &struct {  
		*Comment  
		*Image  
	}{&c2, &i2}); err != nil {  
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("c2:%#v i2:%#v\n", c2, i2)  
}

Output:

str:{"Content":"永远不要高估自己","title":"赞赏码","url":"https://www.liwenzhou.com/images/zanshang_qr.jpg"}  
c2:main.Comment{Content:"永远不要高估自己"} i2:main.Image{Title:"赞赏码", URL:"https://www.liwenzhou.com/images/zanshang_qr.jpg"}

Handling JSON with Uncertain Levels#

If the JSON string does not have a fixed format, making it difficult to define the corresponding struct, we can use json.RawMessage to save the raw byte data.

type sendMsg struct {  
	User string `json:"user"`  
	Msg  string `json:"msg"`  
}  

func rawMessageDemo() {  
	jsonStr := `{"sendMsg":{"user":"q1mi","msg":"永远不要高估自己"},"say":"Hello"}`  
	// Define a map with value type as json.RawMessage for more flexible handling later  
	var data map[string]json.RawMessage  
	if err := json.Unmarshal([]byte(jsonStr), &data); err != nil {  
		fmt.Printf("json.Unmarshal jsonStr failed, err:%v\n", err)  
		return  
	}  
	var msg sendMsg  
	if err := json.Unmarshal(data["sendMsg"], &msg); err != nil {  
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)  
		return  
	}  
	fmt.Printf("msg:%#v\n", msg)  
	// msg:main.sendMsg{User:"q1mi", Msg:"永远不要高估自己"}  
}

Reference Links:

https://stackoverflow.com/questions/25087960/json-unmarshal-time-that-isnt-in-rfc-3339-format

https://colobu.com/2017/06/21/json-tricks-in-Go/

https://stackoverflow.com/questions/11066946/partly-json-unmarshal-into-a-map-in-go

http://choly.ca/post/go-json-marshalling/