Airflow at Scale, SQL+Jinja or Not, X is the next big thing; ThDPTh #64

What: Turns out I like crypto. But the part I’m sharing here reminded me sooo much about stuff going on in the data space right now, I had to share it…

My perspective: Ok, so now you gotta do some filling in. What are those strange words to you? And much more important: What are the stablecoins of data? ELT seems to be an oldie already. Observability and the modern data stack however seem to be on the front. As is the “data mesh”. But just as with stablecoins, the question is: For which one of these new fancy data words is the “why now?” moment, well, now?

Now it’s your turn, go read the piece and then conduct your own little analysis, what’s the X?

What: Eitan Chazbani from Databand shares his thoughts on running Airflow at scale, what problems it poses and how to solve them.

My perspective: I like how Eitan spells out the following thought: “you can run Airflow at scale, but its permissiveness makes it easy for you to mess it up”.

I always find it hard to believe that the alternatives dagster & prefect are supposed to scale better.

Eitan points out one good point at the beginning, Airflow is a pure orchestrator! Or at least it should be used as such. It’s not supposed to store data or execute itself.

I get that they address a bunch of issues and are more opinionated. But scaling is something that comes with the robustness of code because you want to be able to scale in a majority of environments. And there, airflow simply has a much larger contributor base, contributions, and life-years.

What: Furcy Pin recalls a legendary article written by Maxime Beauchemin and in particular a part about mountains of templated SQL, and why it’s a total antipattern.

“It’s pretty clear to me that combining SQL with Jinja templating doesn’t provide the proper foundation for these emerging constructs.”, Maxime

Furcy is going really deep in this post starting with the history of data frames. She also provides a POC on Github to make her point.

My perspective: Her points are sound, as are many other perspectives around the industry. On a metalevel we have two facts about SQL right now:

1. SQL as a language is massively popular due to its simplicity

2. SQL as a language is massively limited due to its simplicity.

Ouch, there you got the problem. Of course, we all know, the easy solution to tackling 2 while keeping 1 is to have a really good abstraction on top of SQL. Something that allows me to write:

But also

And have both executed in a similar fashion.

I don’t have a good answer, haven’t seen one out there yet. Sounds like a good business opportunity, one that will “grow the pie” for everyone but also one that needs a high level of systems thinking. I’m curious how this will play out in the future.

I recommend you read the article though because it is making a good point about the limitations of SQL and what we would really like to do with it.