SPECIAL: Between Go and Elixir

Dear readers,

Like I promised, I would send surprise issues of This Week in Elixir and Erlang from time to time. This one is special, because it is an article I just got off the press. It is a retrospection of my two years of working with Elixir and Go, and lays out my plans on bringing the two together in my future projects.

NOTE: The article is very opinionated, and may contain factological and other kinds of errors. I would appreciate it, if you give it a read and reply with your feedback and suggestions.

Cheers,

Preslav

The human mind is a strange beast. Rather than remaining focused on one thing throughout our lives, it constantly shape-shifts based on our surrounding environment. This is not only a survival but also an evolutionary mechanism. By oscillating between alternatives, the mind learns new concepts, and in doing so, elevates the human species (at least, in theory).

Two personalities are fighting for attention in every person’s mind - that of the pragmatic and the one of the emotional creative. Only when kept in balance do the two move us forward. Neglect one for too long, and your life gets miserable.

That’s a lesson I learned while trying to choose between Go and Elixir as the backbone of my business. For too long, I had criticized myself for being indecisive and unable to pick a side. Why go with two fundamentally different tech stacks if any one of them could solve the same kinds of problems with the same level of success? Reason wanted me to make a choice, and I am so glad I didn’t. Because the more I kept delving into both Elixir and Go, the more I found out how complementary the two can be to one another. Not just in my daily work but also in a more theoretical understanding of how software systems should be built.

In retrospect, what I initially saw as a potential weakness of my business, might be on its way to becoming one of its greatest strengths.

I discovered Elixir and Go at about the same time (2019). I had pivoted almost eight years of working as a Java developer, and part of me was looking for a new technical challenge. On my first encounter, I did not like either. Go felt far too simplistic to be an actual language. Elixir felt far too bizarre - a way of encouraging the Ruby community to try Erlang - a battle-tested language but shrouded with mysticism and urban legends. Little did I know, they would be instrumental in my decisions a couple of years later.

I did not choose Go over its syntax, neither was I initially fascinated with its concurrency mechanism. It was more of a pragmatic decision at the time. I needed a way to deliver and deploy small, blazing-fast binaries, and the language felt like it was made just for that. Coming from a Java project where we had virtually replaced the use of checked exceptions (“let it crash” had become our mantra), I was put off by checking errors after every second line of code. The lack of generics made me repeat myself more often than I liked. However, the overall package stood to the original promise of its creators - the tooling was easy to set up, comes with most of what you need, and was a no-brainer to get going.

That last bit was not the first impression I had of Elixir and the BEAM. Although I had quite some luck setting things up the first time (sadly, I still mess up my Elixir/Erlang updates most of the time), the sheer amount of stuff one got met with was overwhelming at first. I admit that the onboarding has improved a hundred times since then, thanks to the great work of the core team and that of all the people contributing code and learning materials. Plus, as a purist, I refused to try Phoenix before getting my hands dirty with some lower-level stuff first. I see many folks from the Rails community coming to Elixir mainly because of Phoenix. Only once hooked do they start uncovering more and more of the goodies (and dark corners) of the BEAM. I approached it the other way and was initially put off by the sheer amount of concepts one needs to keep in mind before working on the most basic tasks.

Thus, after my first unfruitful encounter with Elixir, I kept it in the back of my mind for about half a year. In the meantime, I had written enough Go to know what they meant by “there’s only one way of doing things.” Like others, I got through all phases of anger - from denial through trying to be creative to finally accepting the language for what it was.

The funny thing was, the more I got used to Go, the more I liked it. Constant error and type checks became less annoying. Quite the contrary, proper error checks saved my back quite a few times in production. Verbose as it was, the syntax made it easy to remember a few basic constructs and repeat them over and over again. Moreover, by having everything in front of your eyes most of the time, it is easier to decode your thoughts months after.

I have to clarify that easier does not mean easy - writing 500-line functions only to save a few clicks won’t make your code less spaghetti. However, it frees your mind from having to think of the proper abstraction all the time. Have an idea, sit down and test it - just like in the good old days of PHP’s heyday. Abstractions will form much later once you’ve made sure that what you’re working on actually makes sense.

Go is great for building commands. Commands, not only in the CLI meaning of the word, but in general - tools with limited scope that do one thing well. This includes small Web services (I will avoid the term “microservices” here) and all kinds of recurring and one-off tooling. Basically, anything that you can afford to lose, because it is easy to spawn it up again. Thanks to Kubernetes, Docker, or even good old systemd or nohup, one can afford to just “let it crash” and start on a clean slate mere seconds afterward.

That is not to say that Go is not suitable for long-running operations. I am maintaining a few Go apps that have undergone months without a restart. It is definitely possible. However, when I think about what DHH calls “majestic monoliths,” building a Go application beyond a particular scale starts requiring thorough checking and guarding every possible place that can crash the app. And it will still crash regardless.

That is what rekindled my interest in Elixir and the BEAM in late 2019. If any system could be prepared to handle crashes gracefully, that would be the BEAM. This time, I was prepared with a lot more patience and a couple of great books (like this one). Starting with a fresh Phoenix app, I quickly re-built PodRadio’s user-facing part in Elixir in only about a couple of weeks. A few months later, I dug into LiveView too, which resulted in AROUNDAWORLD - a fun little project with which I set out to beat the lockdown desperation of 2020 by giving people a compelling but straightforward challenge.

The more I worked with Elixir, the more I liked its syntax and the abstractions it adds on top of Erlang’s OTP (the core of what makes Erlang so famous) library. I am possibly one of the few people still set firm in Go-land who honestly find Erlang’s process/actor model easier to wrap their head around than orchestrating coroutines with channels. I instantly fell in love with the actor model, supervision trees, pipes, and pattern matching. I understand that neither of these concepts is unique to Elixir (or the BEAM), but for someone who has not dabbled deep into functional programming, this was the first stack where everything made sense to me.

With all the goodness around BEAM processes, it is easy to think of them as millions of nano-services running on a single machine, supervised and orchestrated by the application. Want more power? Connect another physical node, and suddenly your nano-services start spinning across the globe. In a world where the BEAM was the norm and so much the exception, that is—a world where the BEAM and not the OS was your OS.

Despite all the great efforts of contributors, it will probably remain challenging to think of BEAM applications as commands. Also, not many people really care about Erlang or as much as they do about having an ergonomic developer experience. True, nobody cares about Java’s byte code either - until something terrible happens, then everyone starts running in mad circles.

And this is precisely where Go and Kubernetes won the game for many. Neither requires the other. You can spin up a Kubernetes cluster running hundreds of different apps built with different technologies (including BEAM ones). You can also hook a 100-line Go binary to a cron job and let it make money for you by running virtually forever. While not in the same league, one may think of Kubernetes (also written in Go, BTW) as an alternative to the BEAM capable of orchestrating macro-scale processes (pods, services). In such an environment, Go opponents’ concerns that the “app may crash in any second” actually don’t matter much. In a way, it’s even better that way. Clean slate.

Of course, no one needs to go through the hoops of setting up or dealing with Kubernetes unless they really need to. For a large number of cases, running everything on a single machine can do miracles when done the right way. In my practice, I’ve seen setups costing 1000s of $/m that perform worse than a few Go binaries / a Phoenix app stuck on a single VM. In fact, most of my apps run on micro instances costing less $/m than some people pay for their morning coffee. Not because I cannot afford bigger ones, but because I don’t need to.

This naturally brings the question up, could Go and Elixir work together? Elixir providing the orchestration and user-facing plane, thanks to the resilience of the BEAM. Go, with its strength of building short-lived commands, providing speed of execution, and making it possible to extend the app beyond the boundaries of the BEAM.

Of course, it is, and that is what I have been doing for the past couple of years—using (or trying to) the best tool for the particular problem. Whether through ports, or simply calling them like regular Web services, an Elixir app can easily communicate with and coordinate dedicated Go binaries. And guess what, they can crash as much as they want; the BEAM will keep a poker face in front of the customer.

I am often getting asked why not mix Elixir and Rust instead, since they cooperate even better using Erlang NIFs. Without a doubt, Rust is a great language, and it has its place. I find it essential and have dedicated a small portion of my time trying to get along with its compiler. Yet, because Rust is so rigid about memory safety, I don’t find it easy to think in terms of solving business requirements. Some can say the same about Go, but thankfully, its creators have taken a set of choices early on that make the programmer and the compiler agree on things easier while maintaining a high level of safety.

Go has its deficiencies, but so does Elixir (or the BEAM for that matter). That largely applies to much of programming today. Or life, for that matter. We have to live with what we have.