A gentle intro to Protocol Buffers

This is part of a series of posts explaining cryptic tech terms in an introductory way.

This series is not intended to be a primary learning source. There may be follow-up posts with hands-on experiments or deeper technical content for some topics.

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. [src]

History

XML
One of the oldest data serialization standards driven from the SGML, the Standard Generalized Markup Language. Standardized 1996~1998, XML was the primary structured and semi-structured data serialization standard and the basis for SOAP protocol. It’s human-readable, structured and very verbose. [snippet src]

<?xml version="1.0" encoding="UTF-8"?>
<breakfast_menu>
<food>
    <name>Belgian Waffles</name>
    <price>$5.95</price>
    <description>
   Two of our famous Belgian Waffles with plenty of real maple syrup
   </description>
    <calories>650</calories>
</food>
<food>
    <name>Strawberry Belgian Waffles</name>
    <price>$7.95</price>
    <description>
    Light Belgian waffles covered with strawberries and whipped cream
    </description>
    <calories>900</calories>
</food>
<food>
    <name>Berry-Berry Belgian Waffles</name>
    <price>$8.95</price>
    <description>
    Belgian waffles covered with assorted fresh berries and whipped cream
    </description>
    <calories>900</calories>
</food>
<food>
    <name>French Toast</name>
    <price>$4.50</price>
    <description>
    Thick slices made from our homemade sourdough bread
    </description>
    <calories>600</calories>
</food>
<food>
    <name>Homestyle Breakfast</name>
    <price>$6.95</price>
    <description>
    Two eggs, bacon or sausage, toast, and our ever-popular hash browns
    </description>
    <calories>950</calories>
</food>
</breakfast_menu>

JSON
Short for JavaScript Object Notation, popularized in the early 2000s. It was a step forward: less verbose than XML, easier to support in browsers, faster to process, and it enforces consistent structure. This made it the dominant serialization standard for the modern web for many years.

// simple representation of a breakfast menu with only one item
[
  {
    "name": "Homestyle Breakfast",
    "price": "$6.95",
    "description": "Two eggs, bacon or sausage, toast, and our ever-opular hash browns",
    "calories": 950
  }
]

Protocol buffers (protobuf)

Google developed Protocol Buffers for use in their internal services. It is a binary encoding format that allows you to specify a schema for your data using a specification language, like so:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

The Protocol Buffers specification is implemented in various languages: Java, C, Go, etc. are all supported, and most modern languages have an implementation. Here is a Java example using the previous schema:

Person john = Person.newBuilder()
    .setId(1234)
    .setName("John Doe")
    .setEmail("jdoe@example.com")
    .build();
output = new FileOutputStream(args[0]);
john.writeTo(output);

Person john;
fstream input(argv[1],
    ios::in | ios::binary);
john.ParseFromIstream(&input);
id = john.id();
name = john.name();
email = john.email();

Using protocol buffers has several advantages over plain-text serialisation like JSON and XML:

• Dense binary encoding produces small output and reduces network overhead
• A declared schema makes parsing straightforward in most languages, with less boilerplate
• Fast processing
• Hard to decode without the schema, which provides some protection
• Backward compatibility as a structural side-effect

References

Google developers - Protocol Buffers