04 May 2015

The Past, Present, and Future of (Ruby) Protobuf

Past

Five years ago I was working at a struggling startup trying to make ends meet while building out an advertising platform centered around employment offers. I was hired to build a new analytics product for our clients and was fortunate to be given license to choose ruby as the language to build our new product with. We had an existing Java codebase that I needed to tie into and after some amount of research we decided that Protobuf was a good fit for cross-platform service communication.

I found the ruby-protobuf gem which had all we needed for the serialization usage. Unfortunately the RPC layer was unfinished and thus not an option. We certainly needed Protobuf for its RPC definition and so I wrote one. It was horrifying. I was newish to Ruby and definitely new to Protobuf and you can imagine how bad it was. All of my attempts to get patches merged to the existing library failed with nary a reply.

In that radio silence, I decided to fork the repo and released a separate gem called protobuf. Shortly afterwards I was laid off from that job and had to scramble to find something else, anything else. As fortune would have it, I stumbled on a new startup called MoneyDesktop (now MX) and was given a rad opportunity to rewrite an existing app from scratch.

Through the years at MX a whole slew of engineers helped me evolve the protobuf gem from a fledgling library that helped us out, to a library that has been adopted by lots of other great companies like Square and Lookout.

Present

In May of 2014 I left my friends at MX to continue growing at a different place under a different set of problems, and having left I now find it difficult to allocate time to working on the library. There’s been a fair amount of activity on the gem and I’ll admit to having a pretty high amount of maintainers guilt about not giving it the time I once had. It’s kind of dumb to feel guilty about not giving away my free time for free, but it’s pretty well documented that this isn’t just a me thing.

Case in point, I recently watched this great talk by Justin Searls about OSS maintenance problems. Justin really hit the nail on the head describing the lifecycle of OSS projects. It’s pretty clear I haven’t done a good job of keeping the community around this project healthy. The gem is after all, my* one hit wonder, so this is my attempt to change all of that.

After meeting with friends at MX recently, we developed a pretty solid gameplan to get things churning again.

Future

First of all, I’m moving the repo from my personal github account to a new github org called ruby-protobuf. Michael Ries at MX is spiking out some code that will eventually replace the serialization layer and public DSL of messages, enums, and service objects.

We want to modularize the gem into a series of libraries that you can include piecemeal based on individual usage (think RSpec). This will probably end up looking something like:

  • Core
    • protobuf-core
      • Extensible Compiler.
      • Message/Enum API (with minimal to no meta-programming).
      • Serialization.
  • RPC
    • protobuf-rpc
      • Depends on protobuf-core.
      • Service DSL with compiler for Service Descriptors.
      • Extensible Server and Client Implementations.
  • Transport (each depend on protobuf-rpc)
    • protobuf-rpc-socket
      • Vanilla socket client and server transport implementation.
    • protobuf-rpc-zmq
      • ZeroMQ client and server transport implementation.
    • protobuf-rpc-http (?)
    • protobuf-rpc-?
      • Message pack? DRB? AMQP? The list is endless of the number of transport mechanisms we could add.
  • Other
    • protobuf-rails
      • Rails generators.
    • protobuf-service-filters
    • protobuf-service-discovery

Longtime contributors to the project will recognize a familiar story here. The existing client and server implementation have undergone relatively few overhauls in the last 5 years and are in dire need of some simplification. The message/field APIs are even more crusty, having been structurally untouched since being forked from the original ruby-protobuf gem.

The unit and functional tests are extremely unreliable due to the difficult nature of testing the zmq transport implementation. It’s my belief that we will get a lot more mileage out of splitting the gem up than we’ve ever had. This is our chance to learn from the past and produce something spectacular.

The version target for the changes proposed here are >4.x, though that is definitely up for debate. It may be advantageous to start the versioning scheme over from scratch and let the protobuf gem as it is today die off. That is a conversation that I expect the community to make the decision on.

I’m also excited to say that I’ll be reaching out to some past contributors and asking if they’ll be a part of “ruby-protobuf core”. There have been some awesome contributions in the past from motivated devs and I want to foster more brains on the project.

Lastly, if it’s not clear from all of the above, I am ready to give up the reins of the project to the community that depends on it. I’ll probably still have my hand in a few parts of the organization at first, but this really is about distributing ownership so everyone can move faster without me as the bottleneck. I’m so grateful to have worked on something so useful for a significant portion of my career, and I look forward to a time where future projects can bring me back into regular usage and contribution.

If you’ve worked with me in the past on the gem, I want to say Thank You for your work, and your patience. Let’s make this thing even better.

*: Not mine, not really.