My journey to Go (stories in a PhD student's life)
Apr 4, 2016 · 9 minute readgolangscalastudent
Prologue: The Academic Mindset
Not that long ago, as I was working towards my doctorate, I spent a significant amount of time writing, benchmarking and optimizing distributed systems in various ways, shapes and forms.
These distributed systems were prototype implementations of whatever algorithm or protocol my team and I were working at the time. Our goal was simple: show that the algorithm is better (read: faster) than their closest competitors, figure out by how much, explain why, and publish a conference paper with our findings.
To do that, we had to make sure that the algorithm itself was the most important bottleneck in the system. Put in other words, all library code we used for things like serialization or RPC had to be as fast as possible. I thus ended up benchmarking various packages and libraries, with the sole purpose of choosing the fastest tools for the job.
Make no mistake: this was a purely academic endeavor. No serious engineer working on any kind of system that sees live traffic would care about performance above all else. But we were no engineers, and our code was never intended for live traffic. Stability, security and maintainability were normally tossed out the window. Deadlocks and crashes were at the order of the day. Unit tests were few and far between.
Anything was fair game, as long as we had “good numbers” and a paper at the end of the day.
Act 1: Java Beginnings
My academic career started in Java-land, on an inherited codebase. Many people would like working with Java: it is well supported, loads of documentation, runs extremely fast at steady state, and there are libraries for anything you would ever need.
final private static Map<Object, AtomicInteger> locks =
new ConcurrentHashMap<Object, AtomicInteger>();
Not me. Everything about Java kinda bothered me: Code is too repetitive; throws declarations are annoying; The JVM is too slow to start and takes up all my laptop’s memory when it finally does; Experiments need minutes of warm-up time; Tuning the garbage collector is a pain.
Throw in a codebase written in full academic spirit (remember maintainability?) that relies on some abandonware academic prototype for all communication purposes, and you can probably see where this is going.
For the sake of my own sanity, something had to be done. But what could possibly help in such a hopeless situation? Well, a full rewrite, of course!
Act 2: The Rewrite
Enter Scala. I don’t quite remember how I came to hear about Scala, but something clicked. The familiarity of the syntax, the intrigue of functional programming, and oh, the succinctness of type inference! What a breath of fresh air. Bring it on, I said.
First, it’s time to learn the language. Read some books, do some tutorials, all that. Fast forward several weeks (or was it months? hard to tell now…) and I think I have the basics down. I understand vals, vars, traits, case classes, pattern matching and implicits. I get options, futures, actors, higher order functions, filters, maps and folds. I can write simple programs and I can make them compile without writing a single line of XML for it. I didn’t fully grasp type parametrization, covariance, contravariance, nonvariance, upper bounds or lower bounds, but it’s taking me too long and I have to move on. I probably won’t need these anyway, and if I do, I’ll come back for them. Awesome, good to go!
It is now time to put on my architect’s hat and design how my system will look like. Oh, it will be so amazing. Components will be clearly separated. Heck, let’s make each component into an actor and let them communicate by message passing. What could go wrong?
I chose Akka to back all the actors, and would use its remoting features for all my inter process communication needs. Surely it’s a future-proof library – with all that hype around it there’s no way it would be abandoned anytime soon (it wasn’t).
One last stop before I get to coding: the API. I have the choice between designing my own, or borrowing from another library (let’s call it ScalaXYZ) that sort of does the same thing as what I need, but locally instead of distributed. I figure that my work could get more exposure if it was API-compatible with ScalaXYZ, so I take that route. But since this API was fairly complex, I decided to build my stuff as an add-on to ScalaXYZ for the sake of efficiency.
Fast forward past coding and, before long, it was alive! Long story short, the system worked as intended, and it reached parity almost on time. It was faster and more stable than it’s predecessor. I got a few papers out of it. It was even used by a few colleagues from my lab in their own papers.
Investing in learning Scala worked out for me, if only to keep me excited about my work. But, like anything else in this world, it was no panacea.
Act 3: Lessons Learned
KISS (keep it simple, stupid). That’s more or less the gist of it. And boy, did I get that wrong.
The single worst decision I made was using the ScalaXYZ API and implementing my system as a plug-in within it. You see, Scala is an insanely powerful language with a tremendously flexible syntax. ScalaXYZ took full advantage of it, and used it to present its users with the cleanest, sweetest API I had ever seen. But to get there though, it had to jump through some serious Scala hoops, making for a fairly complex architecture. And since I was extending this architecture to do something it was never meant to do, what I got out was a mess. I was fitting square pegs in round holes, and I couldn’t even tell.
(TODO: add image here)
Writing Scala code has this awesome feel to it. You feel like anything is possible and the world is your oyster. You just have to figure out how to do it, and it’s not easy. You have to think hard, staring at your problem square in its face. At some undefined later time you just get it, put on your cowboy hat, and write the most beautiful one-liner you have ever seen before. You feel proud of yourself, and then move on to the next problem.
For me the troubles started when I had to go back to older code for fixing bugs or adding new features. Turns out, code written this way gets really hard to understand after you a while, when you invariably forget the context you had when you wrote the code. You can sometimes spend minutes figuring out what a 2-3 line snippet of code really does. “What does it call? Where does it call it? What parameters does it use? Where are those parameters coming from? And WTF is the type of the result?” Rinse and repeat for the next snippet.
I’ll stop short of pointing fingers at anything but myself. I was clearly writing poor code, that would likely never make it past code review at any respectable engineering company. Deadlines and the throw-away nature of academic code definitely didn’t incentivize writing good quality, maintainable code. In retrospective, it feels that Scala itself contributed to some extent to my writing of poor code.
Finally, actors. Oh, I loved actors, and I still do! They’re such an amazing way to think about concurrency. In fact, I loved them so much, I grossly, hopelessly overused them. Not quantitatively, but qualitatively. Actors are not meant as a stand-in replacement for mutexes, especially in mostly sequential code. And yet, that’s exactly how I used them. Instead of wrapping shared state with a mutex, I wrapped it in an actor. The result? Spaghetti code, of course. To understand who calls what, I had to follow each message as it was bounced across actors, figure out what code handles it and where will it be thrown to next.
So my lessons are: simplicity trumps all, at both code and architectural level. If in doubt, optimize for simplicity. Pay more attention when using shiny new tools, as they can be deceiving to the inexperienced user.
Act 4: Reboot To Go
Sometime after the midpoint of my degree I had the opportunity to take on an internship at a technology company. Once there, I was presented with an interesting choice. On the one hand, I could use plain old and boring Java, which my team and I already knew. This was the safe choice. On the other hand, I could venture out into the unknown and learn this new programming language named Go. My team would support me, but lacked any relevant experience with the language. This was the exciting choice, despite having more risk involved.
For the next couple of days, I went through the Go tour, stormed through the documentation, read a lot of posts and articles, and watched the most interesting tech talks about Go that I could find. I was hooked, and finally decided there was no going back to Java for me. Go lived up to the challenge and the internship project was successfully completed.
As my internship ended and I returned to my research, I brought the new language back with me. My short experience with Go was enough to completely decimate my ability to tolerate the JVM, Java and to some extent even Scala with all its intricacies and unnecessarily complex syntax. Luckily enough, I was on the brink of starting a new project, so it only made sense I would implement it from scratch in Go.
This second blank slate meant I was free of any kind of technical debt and architectural cruft. I had my second chance to design a powerful but simple architecture, and implement it to be as fast as technically possible. I could make use of my newly obtained practical knowledge of what it means to write clean, maintainable, high-quality code in a corporate setting. By now I was also more aware of the power of simplicity, which I had observed first-hand in code reviews during my internship and indirectly within the design of the Go programming.
Did I manage to achieve these idealistic goals? Debatable. The architecture was objectively simpler than my previous attempt, but I still think standard engineering practices managed to evade me as the pressure to publish predictably mounted.
Either way, by the end of my degree, I had implemented and optimized several distributed coordination protocols in Go. I had benchmarked and profiled serialization formats, messaging libraries and compression algorithms. And finally, I had pushed more messages per second through individual nodes in a distributed system than I even thought was possible at that time. All in Go.
Epilogue
So yes, that concludes the story of my PhD (or journey into Go, depending on the point of view). I am convinced it isn’t as exciting or educative to you as it was for me. And it’s probably incoherent, but none of that matters. This post has a purpose, and that is to introduce you, the reader of my blog, to my background and perspective for the posts that will follow.
I’m planning to revive some of the experiments that I’ve done during my degree, and turn them into a series of blog posts. I will share any findings I make and try to draw some lessons.
Until next time.