I began reading some interesting articles by Glenn Fiedler, who had a particular series about writing a custom UDP protocol.
At one point he expressed the concern for sending data over the network, stating that json is massive, and a binary serialization pattern that exposes a bidirectional workflow for both reading and writing works best.
I decided to give a try writing it in go, and at first I followed strict syntax rules, but since the binary documentation is very good I eventually went back to a much simpler project implementation with no attempt to handle variable sized types.
I also created a set of benchmarks to compare both the performance, memory management, and byte size between this system, msgp, and the gob package. _This is a proof-of-concept, so I am not attempting to compete._
While mine is not the fastest, it is by far the smallest, which proves that a very simple thin wrapper to abstract read and write behavior, and extra effort to write your own byte conversion logic, can be extremely beneficial in situations where every byte counts.
I also concluded that while the gain is significant, for a first-draft msgp is substantially easier to wield and might be worth the trade-off if you are dealing with a large number of structures and don’t have time to write binary serialization for each.
One major complexity with the go implementation that I couldn’t quite deal with was variable-sized data. Anytime an nondeterministic slice or string are written, extra effort must be made to keep tabs on the size when restoring it. While I had written a solution to do this, it was crappy and limited to strings only.
This project also helped me realize the significance between big and little endian encoding, as well as 32 vs 64 bit consistency. The fact that go abstracts endianness and optimizes little endian is awesome, but abstracted int types with variable size make for a hell of a mess when trying to connect a 32-bit client to a 64-bit server without having to add a myriad of casting for any case where you used a generic int (since data loss can occur when sending an int from a 64 bit service to a 32 bit service).
written on 2017-04-13