Advanced Protobuf and gRPC

title: Advanced protobuf and gRPC
date: 2021-09-14 15:05:00
toc: true
category:

Golang
gRPC
tags:
Golang
gRPC
Go
protobuf
proto
type
default value
command
package
sync
pitfalls
Map
enum
generic
message
nested
reference
object
property

Basic Types and Default Values of protobuf#

Refer to the table below:

Scalar message fields can have one of the following types – the table shows the types specified in the file and the corresponding types in the auto-generated classes: .proto.

.proto Type	Notes	C++ Type	Java/Kotlin Type	Python Type	Go Type	Ruby Type	C# Type	PHP Type	Dart Type
double		double	double	float	float64	Float	double	float	double
float		float	float	float	float32	Float	float	float	double
int32	Uses variable-length encoding. Encoding negative numbers is inefficient – if your field may have negative values, use sint32 instead.	int32	int	int	int32	Fixnum or Bignum (as required)	int	integer	int
int64	Uses variable-length encoding. Encoding negative numbers is inefficient – if your field may have negative values, use sint64 instead.	int64	long	int/long	int64	Bignum	long	integer/string	Int64
uint32	Uses variable-length encoding.	uint32	int	int/long	uint32	Fixnum or Bignum (as required)	uint	integer	int
uint64	Uses variable-length encoding.	uint64	long	int/long	uint64	Bignum	ulong	integer/string	Int64
sint32	Uses variable-length encoding. Signed integer value. They encode negative numbers more efficiently than regular int32s.	int32	int	int	int32	Fixnum or Bignum (as required)	int	integer	int
sint64	Uses variable-length encoding. Signed integer value. They encode negative numbers more efficiently than regular int64s.	int64	long	int/long	int64	Bignum	long	integer/string	Int64
fixed32	Always four bytes. More efficient than uint32 if values are usually greater than 2^28.	uint32	int	int/long	uint32	Fixnum or Bignum (as required)	uint	integer	int
fixed64	Always eight bytes. More efficient than uint64 if values are usually greater than 2^56.	uint64	long	int/long	uint64	Bignum	ulong	integer/string	Int64
sfixed32	Always four bytes.	int32	int	int	int32	Fixnum or Bignum (as required)	int	integer	int
sfixed64	Always eight bytes.	int64	long	int/long	int64	Bignum	long	integer/string	Int64
bool		bool	boolean	bool	bool	TrueClass/FalseClass	bool	boolean	bool
string	Strings must always contain UTF-8 encoded or 7-bit ASCII text, and the length cannot exceed 2^32.	string	String	str/unicode	string	String (UTF-8)	string	string	String
bytes	May contain arbitrary byte data in any order, but not exceeding 2^32.	string	ByteString	str	[]byte	String (ASCII-8BIT)	ByteString	string	List

You can find more information on "how various types are encoded when serializing messages" in the article Protocol Buffer Encoding.

The Role of option go_package#

When defining a .proto file, we need to declare which package this file belongs to, mainly for standardization and to avoid duplication. This concept also exists in other languages, such as the namespace concept in PHP and the package concept in Go.

Therefore, we define a package for each proto file based on the actual classification situation, generally keeping the package name consistent with the folder name where the proto file is located.

For example, if the file is in the proto folder, then the package we use is: proto;

The option name itself indicates options and configurations, and a common option is to configure go_package

option go_package = ".;proto";

Now, when the protoc command generates the Go package, if this line is not added, it will prompt an error:

➜  proto git:(master) ✗ protoc --go_out=:. hello.proto  
2020/05/21 15:59:40 WARNING: Missing 'go_package' option in "hello.proto", please specify:  
        option go_package = ".;proto";  
A future release of protoc-gen-go will require this be specified.  
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

So, what is the difference between go_package and the package proto; above? It can be a bit confusing.

Try changing it like this:

syntax = "proto3";  

package protoB;  

option go_package = ".;protoA";

Let's see what the generated Go package's package is. Open the generated Go file:

# vi hello.pb.go  

package protoA  

...

It turns out to be protoA, indicating that the Go package is influenced by option go_package. Therefore, when we do not apply this line, the system will use the package name of the proto file as a hint, prompting you to add the same go_package name.

Next, let's look at what =".;proto" means. Change it to:

option go_package = "./protoA";

After executing, it generates a protoA folder. Inside, the hello.pb.go file's package is also protoA.

So, does .; mean it's in the current directory? Okay then.

Now let's change it to an absolute path directory:

option go_package = "/";

So, to summarize:

package protoB; // This is used to set the package of the proto file itself  

option go_package = ".;protoA";  // This is used to set the package of the generated Go file. Generally, these two can be set to be the same.

Pitfalls When Syncing proto Files#

In short, it refers to the different sequence numbers of object properties in the proto files used by the client and server, leading to business logic confusion and being difficult to troubleshoot.

For example:

message StreamReqData{  
    string name = 1;  
    string url = 2;  
}

The logic of protobuf during serialization is roughly:

name = "biuaxia", -> 17biuaxia, where 1 is the sequence number of the property name, and 7 is the content length.

Similarly, url = "biuaxia.cn", -> 210biuaxia.cn.

If the sequence numbers of properties in the proto files of the server and client are different, it will cause difficult-to-troubleshoot issues.

The best solution to this is not to modify the proto files but to receive unified distribution for the proto files.

Nested References in proto Files#

For example, hello.proto references base.proto.

The hello.proto file is as follows:

syntax = "proto3";  

option go_package = "../proto";  

service Greeter {  
    rpc Ping(Empty) returns (Pong);  
}

The base.proto file is as follows:

syntax = "proto3";  

message Empty {  
}  

message Pong {  
    string id = 1;  
}

Directly generating or in the IDE, you can see that line 6 of hello.proto (i.e., Empty and Pong) will report an error, which can be resolved by using the import keyword.

Modify hello.proto by adding content on its third line:

syntax = "proto3";  

import "base.proto";  

option go_package = "../proto";  

service Greeter {  
    rpc Ping(Empty) returns (Pong);  
}

Note: When importing files, the subsequent content is the relative directory of the current file.

It can also be used to introduce proto files with package structures and use them, for example:

syntax = "proto3";  

import "base.proto";  
import "google/protobuf/empty.proto";  

option go_package = "../proto";  

service Greeter {  
    rpc Ping(google.protobuf.Empty) returns (Pong);  
}

Note: External files need to write the full package name, while local files do not.

To check what data is available in proto, you can view the following position in Idea, appending .proto to the red position.

Nested message#

You can nest directly, such as:

syntax = "proto3";  

option go_package = "../proto";  

message Hello {  
    message Result {  
        string code = 1;  
        string msg = 2;  
    }  

    string ret = 1;  
    Result msg = 2;  
    Result data = 3;  
}

Note: Importing nested files will cause the messages contained in the imported proto files not to be generated. The solution is to manually generate the proto files.

Using Enum Types#

Define directly, such as:

syntax = "proto3";  

option go_package = "../proto";  

enum Gender{  
    MALE = 1;  
    FEMALE = 2;  
}  

message Request {  
    Gender g = 1;  
}

Using Map#

Usage example:

syntax = "proto3";  

option go_package = "../proto";  

message Request {  
    map<string, string> mp = 1;  
}

Note: Be sure to specify the types of key and value just like generics.

Using Timestamp#

First, import the timestamp in the proto file

syntax = "proto3";  

import "google/protobuf/timestamp.proto";  

option go_package = "../proto";  

service Greeter {  
  rpc SayHello (HelloRequest) returns (HelloReply);  
}  

message HelloRequest {  
  string name = 1;  
  google.protobuf.Timestamp addTime = 2;  
}  

message HelloReply {  
  string message = 1;  
}

Then the client can directly call New(time.Time) under the directory "google.golang.org/protobuf/types/known/timestamppb" to generate the corresponding type. If you don't know how to find New(time.Time), it is recommended to use Goland, which will automatically provide hints.

package main  

import (  
    "biuaxia.cn/demo/grpc_test/proto"  
    "context"  
    "fmt"  
    "google.golang.org/grpc"  
    "google.golang.org/protobuf/types/known/timestamppb"  
    "time"  
)  

func main() {  
    conn, err := grpc.Dial("localhost:1234", grpc.WithInsecure())  
    if err != nil {  
        panic(err)  
    }  
    defer conn.Close()  

    c := proto.NewGreeterClient(conn)  
    reply, err := c.SayHello(context.Background(), &proto.HelloRequest{  
        Name:    "biuaxia",  
        AddTime: timestamppb.New(time.Now()),  
    })  
    if err != nil {  
        panic(err)  
    }  

    fmt.Println(reply.Message)  
}

The server does not need to modify:

package main  

import (  
    "biuaxia.cn/demo/grpc_test/proto"  
    "context"  
    "google.golang.org/grpc"  
    "net"  
)  

type Server struct {  
}  

func (s *Server) SayHello(ctx context.Context, request *proto.HelloRequest) (*proto.HelloReply, error) {  
    time := request.AddTime.AsTime().Format("2006-01-02 15:04:05.000")  
    return &proto.HelloReply{  
        Message: "hello " + request.Name + "_" + time,  
    }, nil  
}  

func main() {  
    s := grpc.NewServer()  
    proto.RegisterGreeterServer(s, &Server{})  

    listen, err := net.Listen("tcp", "localhost:1234")  
    if err != nil {  
        panic(err)  
    }  

    _ = s.Serve(listen)  
}

Metadata Mechanism#

It can be understood as the HTTP request header, used to carry token and other request information, and can be understood as separated from business information.

Note: The imported package is "google.golang.org/grpc/metadata".

Client usage example:

package main  

import (  
    "context"  
    "fmt"  
    "time"  

    "google.golang.org/grpc"  
    "google.golang.org/grpc/metadata"  
    "google.golang.org/protobuf/types/known/timestamppb"  

    "biuaxia.cn/demo/grpc_test/proto"  
)  

func main() {  
    conn, err := grpc.Dial("localhost:1234", grpc.WithInsecure())  
    if err != nil {  
        panic(err)  
    }  
    defer conn.Close()  

    c := proto.NewGreeterClient(conn)  

    // md := metadata.Pairs("timestamp", time.Now().Format("2006-01-02 15:04:05.000"))  
    md := metadata.New(map[string]string{  
        "name": "biuaxia",  
    })  
    ctx := metadata.NewOutgoingContext(context.Background(), md)  

    reply, err := c.SayHello(ctx, &proto.HelloRequest{  
        Name:    "biuaxia",  
        AddTime: timestamppb.New(time.Now()),  
    })  
    if err != nil {  
        panic(err)  
    }  

    fmt.Println(reply.Message)  
}

Server usage example:

package main  

import (  
    "context"  
    "fmt"  
    "net"  

    "google.golang.org/grpc"  
    "google.golang.org/grpc/metadata"  

    "biuaxia.cn/demo/grpc_test/proto"  
)  

type Server struct {  
}  

func (s *Server) SayHello(ctx context.Context, request *proto.HelloRequest) (*proto.HelloReply, error) {  
    md, ok := metadata.FromIncomingContext(ctx)  
    if !ok {  
        fmt.Println("get metadata failed")  
    }  
    fmt.Println("get metadata:", md)  
    time := request.AddTime.AsTime().Format("2006-01-02 15:04:05.000")  
    return &proto.HelloReply{  
        Message: "hello " + request.Name + "_" + time,  
    }, nil  
}  

func main() {  
    s := grpc.NewServer()  
    proto.RegisterGreeterServer(s, &Server{})  

    listen, err := net.Listen("tcp", "localhost:1234")  
    if err != nil {  
        panic(err)  
    }  

    _ = s.Serve(listen)  
}

Interceptors#

Server usage example:

package main  

import (  
    "context"  
    "fmt"  
    "net"  

    "google.golang.org/grpc"  

    "awesomeProject/grpc/interceptor/main/proto"  
)  

type Server struct{}  

func (s *Server) SayHello(ctx context.Context, request *proto.HelloRequest) (*proto.HelloReply,  
    error) {  
    return &proto.HelloReply{  
        Message: "hello " + request.Name,  
    }, nil  
}  

func interceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {  
    // Continue processing the request  
    fmt.Println("Received new request")  
    res, err := handler(ctx, req)  
    fmt.Println("Request processing completed")  
    return res, err  
}  

func main() {  
    var opts []grpc.ServerOption  
    opts = append(opts, grpc.UnaryInterceptor(interceptor))  

    g := grpc.NewServer(opts...)  
    proto.RegisterGreeterServer(g, &Server{})  
    lis, err := net.Listen("tcp", "0.0.0.0:50051")  
    if err != nil {  
        panic("failed to listen:" + err.Error())  
    }  
    err = g.Serve(lis)  
    if err != nil {  
        panic("failed to start grpc:" + err.Error())  
    }  
}

Client usage example:

package main  

import (  
    "context"  
    "fmt"  
    "time"  

    "google.golang.org/grpc"  

    "awesomeProject/grpc/interceptor/main/proto"  
)  

func interceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {  
    start := time.Now()  
    err := invoker(ctx, method, req, reply, cc, opts...)  
    fmt.Printf("method=%s req=%v rep=%v duration=%s error=%v\n", method, req, reply, time.Since(start), err)  
    return err  
}  

func main(){  
    //stream  
    var opts []grpc.DialOption  

    opts = append(opts, grpc.WithInsecure())  
    // Specify client interceptor  
    opts = append(opts, grpc.WithUnaryInterceptor(interceptor))  

    conn, err := grpc.Dial("localhost:50051", opts...)  
    if err != nil {  
        panic(err)  
    }  
    defer conn.Close()  

    c := proto.NewGreeterClient(conn)  
    r, err := c.SayHello(context.Background(), &proto.HelloRequest{Name:"bobby"})  
    if err != nil {  
        panic(err)  
    }  
    fmt.Println(r.Message)  
}

Verifying Request Information#

Server usage example:

package main  

import (  
    proto2 "awesomeProject/grpc/interceptor/samples/auth_verify/proto"  
    "context"  
    "fmt"  
    "google.golang.org/grpc/codes"  
    "google.golang.org/grpc/metadata"  
    "google.golang.org/grpc/status"  
    "net"  

    "google.golang.org/grpc"  
)  

type Server struct{}  

func (s *Server) SayHello(ctx context.Context, request *proto2.HelloRequest) (*proto2.HelloReply,  
    error) {  
    return &proto2.HelloReply{  
        Message: "hello " + request.Name,  
    }, nil  
}  

func interceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {  
    md, ok := metadata.FromIncomingContext(ctx)  
    if !ok {  
        fmt.Println("get metadata failed")  
        return resp, status.Error(codes.Unauthenticated, "No authorization information")  
    }  

    var (  
        appkey    string  
        appsecret string  
    )  

    if val, ok := md["appkey"]; ok {  
        appkey = val[0]  
    }  
    if val, ok := md["appsecret"]; ok {  
        appsecret = val[0]  
    }  

    if appkey != "vditor" || appsecret != "b1d0d1ad98acdd5d7d846d" {  
        return resp, status.Error(codes.Unauthenticated, "Authorization information is incorrect")  
    }  

    fmt.Println("get metadata:", md)  

    // Continue processing the request  
    fmt.Println("Received new request")  
    res, err := handler(ctx, req)  
    fmt.Println("Request processing completed")  
    return res, err  
}  

func main() {  
    var opts []grpc.ServerOption  
    opts = append(opts, grpc.UnaryInterceptor(interceptor))  

    g := grpc.NewServer(opts...)  
    proto2.RegisterGreeterServer(g, &Server{})  
    lis, err := net.Listen("tcp", "0.0.0.0:50051")  
    if err != nil {  
        panic("failed to listen:" + err.Error())  
    }  
    err = g.Serve(lis)  
    if err != nil {  
        panic("failed to start grpc:" + err.Error())  
    }  
}

Client usage example:

package main  

import (  
    proto2 "awesomeProject/grpc/interceptor/samples/auth_verify/proto"  
    "context"  
    "fmt"  
    "time"  

    "google.golang.org/grpc"  
    "google.golang.org/grpc/metadata"  
)  

func interceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {  
    start := time.Now()  

    md := metadata.New(map[string]string{  
        "appkey":    "vditor",  
        "appsecret": "b1d0d1ad98acdd5d7d846d",  
    })  
    ctx = metadata.NewOutgoingContext(context.Background(), md)  

    err := invoker(ctx, method, req, reply, cc, opts...)  
    fmt.Printf("method=%s req=%v rep=%v duration=%s error=%v\n", method, req, reply, time.Since(start), err)  
    return err  
}  

type customCredential struct {  
}  

func (c customCredential) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {  
    return map[string]string{  
        "appkey":    "vditor",  
        "appsecret": "b1d0d1ad98acdd5d7d846d",  
    }, nil  
}  

func (c customCredential) RequireTransportSecurity() bool {  
    return false  
}  

func main() {  
    var opts []grpc.DialOption  

    opts = append(opts, grpc.WithInsecure())  
    // Specify client interceptor  
    //opts = append(opts, grpc.WithUnaryInterceptor(interceptor))  
    opts = append(opts, grpc.WithPerRPCCredentials(customCredential{}))  

    conn, err := grpc.Dial("localhost:50051", opts...)  
    if err != nil {  
        panic(err)  
    }  
    defer conn.Close()  

    c := proto2.NewGreeterClient(conn)  
    r, err := c.SayHello(context.Background(), &proto2.HelloRequest{Name: "bobby"})  
    if err != nil {  
        panic(err)  
    }  
    fmt.Println(r.Message)  
}

The commented-out content in grpc/interceptor/samples/auth_verify/client/client.go:46 is the original calling method, while line 47 is the gRPC encapsulated method; the object needs to implement the GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) and RequireTransportSecurity() bool methods of the google.golang.org/grpc/credentials/credentials.go interface.