Protocol Buffers and Go: Getting started

In this tutorial, we will see how protocol buffers can be used in the context of GO Language.

What is Protocol Buffer

Protocol Buffers are data format which stores data in a structured format. Data in the protocol buffer format can be serialized and deserialized by multiple languages.

Sounds Confusing. You can think it on lines of JSON, XML but it has loads of advantages to offer. Still confusing? then don’t worry as we go along the tutorial we will understand why even a new data format is even needed.

Let’s see first with an example of the simplest protocol buffer file.

person.proto

syntax = "proto3";

message Person {
    string name = 1;
}

A couple of points to note about the above file.

• This file is just a blueprint of how our Protocol Buffer structure is. There is no data associated yet. This is different from JSON/XML , in which file also represents the actual data.
• In the above file there is a person message with a field name of type string.  “proto3” means that the message written is incompatible with Protocol Buffer version three.  
• From the above example, there is one difference you can notice from JSON i.e, it has type information. This type of information will be useful in the auto-generation of code in different languages. Let’s see an example of  auto-generation in Golang

Auto Generation of GO Code:

• We can generate a corresponding Golang code using the above person.proto file. But to do that we need to do some installations:

Installations:

• First install the C++ implementation of Protocol Buffers. Each platform has its own way of installation. See this link – https://github.com/protocolbuffers/protobuf/blob/master/src/README.md
• Install Golang
• Install protoc-gen-go – go get -u github.com/golang/protobuf/protoc-gen-go . This package will be used for auto generation of Go code

After the installation is done then cd to the directory which contains the person.proto file. Run this command:

protoc -I ./ --go_out=./ ./person.proto

It will generate a data access go file with name person.pb.go  in the same directory. 

// Code generated by protoc-gen-go. DO NOT EDIT.
// source: person.proto


package person


import (
    fmt "fmt"
    proto "github.com/golang/protobuf/proto"
    math "math"
)


// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf




// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
// A compilation error at this line likely means your copy of the
// proto package needs to be updated.
const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package


type Person struct {
    Name                 string   `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}




func (m *Person) Reset()         { *m = Person{} }
func (m *Person) String() string { return proto.CompactTextString(m) }
func (*Person) ProtoMessage()    {}
func (*Person) Descriptor() ([]byte, []int) {
    return fileDescriptor_4c9e10cf24b1156d, []int{0}
}


func (m *Person) XXX_Unmarshal(b []byte) error {
    return xxx_messageInfo_Person.Unmarshal(m, b)
}
func (m *Person) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
    return xxx_messageInfo_Person.Marshal(b, m, deterministic)
}
func (m *Person) XXX_Merge(src proto.Message) {
    xxx_messageInfo_Person.Merge(m, src)
}
func (m *Person) XXX_Size() int {
    return xxx_messageInfo_Person.Size(m)
}
func (m *Person) XXX_DiscardUnknown() {
    xxx_messageInfo_Person.DiscardUnknown(m)
}


var xxx_messageInfo_Person proto.InternalMessageInfo


func (m *Person) GetName() string {
    if m != nil {
        return m.Name
    }
    return ""
}


func init() {
    proto.RegisterType((*Person)(nil), "Person")
}


func init() { proto.RegisterFile("person.proto", fileDescriptor_4c9e10cf24b1156d) }


var fileDescriptor_4c9e10cf24b1156d = []byte{
    // 67 bytes of a gzipped FileDescriptorProto
    0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x29, 0x48, 0x2d, 0x2a,
    0xce, 0xcf, 0xd3, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x57, 0x92, 0xe1, 0x62, 0x0b, 0x00, 0xf3, 0x85,
    0x84, 0xb8, 0x58, 0xf2, 0x12, 0x73, 0x53, 0x25, 0x18, 0x15, 0x18, 0x35, 0x38, 0x83, 0xc0, 0xec,
    0x24, 0x36, 0xb0, 0x22, 0x63, 0x40, 0x00, 0x00, 0x00, 0xff, 0xff, 0xa4, 0xaf, 0x53, 0x72, 0x34,
    0x00, 0x00, 0x00,
}

Now the biggest question is what is this person.pb.go file which has been auto-generated by protoc using person.proto . The first couple of points to notice

• Person struct like below. See how type information of person.proto file is used to know what type of field Name is a string

type Person struct {
    Name                 string   `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}

•   Person struct also implements a couple of methods that make if of interface type proto.Message.

So basically this autogenerated file generates data accessors for Person struct and it provides methods that allow marshaling/unmarshalling of Person struct type to/from actual bytes. Now let’s write a main.go program to actually create concrete objects of Person struct. Here we will see a couple of advantages which Protocol Buffer has to offer. The below program also shows the read and write of Person struct to file.

main.go

package main


import (
    "fmt"
    "io/ioutil"
    "log"
    proto "github.com/golang/protobuf/proto"
)


func main() {
    person := &Person{Name: "XXX"}
    fmt.Printf("Person's name is %s\n", person.GetName())




    //Now lets write this person object to file
    out, err := proto.Marshal(person)
    if err != nil {
        log.Fatalf("Serialization error: %s", err.Error())
    }
    if err := ioutil.WriteFile("person.bin", out, 0644); err != nil {
        log.Fatalf("Write File Error: %s ", err.Error())
    }
    fmt.Println("Write Success")




    //Read from file
    in, err := ioutil.ReadFile("person.bin")
    if err != nil {
        log.Fatalf("Read File Error: %s ", err.Error())
    }
    person2 := &Person{}
    err2 := proto.Unmarshal(in, person2)
    if err2 != nil {
        log.Fatalf("DeSerialization error: %s", err.Error())
    }


    fmt.Println("Read Success")
    fmt.Printf("Person2's name is %s\n", person2.GetName())
}

To run this file first install protobuf/proto using “go get github.com/golang/protobuf/proto” and then run this file using the command “go run *.go”Output:

Person's name is XXX
Write Success
Read Success
Person2's name is XXX

Notice that in the above program we

• We write a concrete person struct to a file “person.bin”. It is a binary file and not human readable.
• Also we read from the file. It is able to successfully read and print “Person2’s name is XXX”

The astonishing thing about the “person.bin” file is that it is of only 5 bytes as compared if you create a JSON file with the same data that will be of size more than 15 bytes.  Also Marshal and UnMarshal from bytes to concrete objects and vice versa are also very fast as compared to unmarshal and marshaling of JSON files. 
Now we have provided the theory. Let’s write once again the advantages of using Protocol Buffers

1. More clear and less ambiguous than corresponding JSON and XML as there is also type information stored with them.
2. The data stored is relatively smaller in size of almost 2- 3 times smaller.
3. It is much faster. For example, serialization and deserialization is faster with protocol Buffers
4. Automated Code Generation – You write a protocol buffer file and you automatically get a corresponding GO file generated
5. A protocol buffer is used in GRPC which is next-generation replacement of REST protocol – Watch space for here, we will add an article on it soon.

Conclusion: Protocol Buffers have much to offer other than what we have discussed in the article. This provides a quick overview of what protocol buffers are and what are their advantages as compared to JSON/XML format.