Open Source Data Reliability, Joyful Data Devs, TechRadar 26; ThDPTh #65

What: Five properties that will make coding joyful, and to that extent really productive! The five properties are composable, Unix philosophy, predictable, idiomatic, and domain-based. This quote sums up the idea really well.

And …

My perspective: First of all, Dan put a lot of thought into this article. I like his well-structured approach to turning principles into properties while offering help for beginners and yet nuances for experts.

I also like the whole intent of the piece, he’s simply making the point that if you add any of the CUPID properties to your code (which will usually involve more reducting than adding), you and others will have an easier time working with it.

I think this makes CUPID in particular very attractive for data people, where strict rules and “principles” might not be the right place to start. Just take any one of the properties and add it to your codebase, your SQL, your ETL workload and you’ll likely be happier at the end of the day.

I for instance remember the day when I saw a data team that was building ETLs in a graphical user interface. The team built a third ETL process basically just ingesting REST API data. They realized, hey we can make this composable! Even within the graphical tool, they managed to provide a composable “REST API — template” which they used to then build on top off. Suddenly, adding a new ingestion process was “a that’s just a simple parameter added” and not “uh, that’s a whole new ETL job”. That’s composability at play — and it certainly seemed more fun.

I also remember when we changed from a “one tool to orchestrate everything” to a more “one tool, one purpose” strategy inside a data team I was part of. I even held a talk about exactly that idea called “Modern Cloud Data warehouses best practices”. Turns out, by decoupling and having one tool do one thing like only ingesting data, only displaying data, only transforming data, suddenly the whole work becomes super fast. Data quality increases, things were much easier to fix because usually, the error was just inside one of these components. That’s the Unix philosophy in action — and I can testify to that myself — it was much more fun!

I am sure you can also find good examples for the other three properties, but I’d encourage you to simply take it as is: a nice opportunity to increase the joy of working inside data and through that, the productivity of your team.

What: The Technology Radar is an opinionated content piece that curates technologies, platforms, languages, and techniques into four categories: Assess, Trial, Adopt, Hold.

My perspective: To put it into my words: This is always a must-read for everyone in technology, and it’s kind of like the guide to the future of software (& data). It’s published by ThoughtWorks and they very accurately shaped the software industry over the last 20 years.

I am already highlighting everything I find interesting on Twitter, but two things stand out. One is the data mesh! Zhamak finally got her book out and I am excited to read it. I already loved the first chapter about the inflection point (you might guess, I am also a fan of everything that Andy Grove wrote, including the “Inflection Point”). Oh yeah, and I also co-authored a book about that topic.

Second, ThoughtWorks is also noticing that stuff is going on with open-source. Especially open-source and data. There is more happening in general, more openness in general, and more commercial action in that space. It’s undirected for now, but it is happening and will be a good thing for everyone involved.

Now, go ahead and just dive right into it.