🐰 #37 The Data PM, Erik Bernhardsson on Podcast, Kolibris data mesh; ThDPTh #37 🐰

Data will power every piece of our existence in the near future. I collect “Data Points” to help understand & shape this future.

If you want to support this, please share it on Twitter, LinkedIn, or Facebook.

Data teams do a variety of things. When they build a new movie recommendation engine or a forecast that is used by consumers, they build products serving the end customer of the company.

But probably half of the data teams out there build products for internal users. Be it forecasts for the sales team, or the data warehouse with its BI tool on top.

These products are different from the usual B2C product. In essence, they are B2B products, serving their own company. The “First Round Review” makes a great case for why managing these products will fail if we apply the B2C playbook, and I feel this extends very well to products like a BI system or any other data product targeted at the own company.

The key to manage B2B products, and internal data products, in my opinion, is to adopt a hybrid approach to planning, one that involves both quick reactions to end-user feedback as well as long planning horizons.

For more details, I really suggest you read the piece.

I’ve recently enjoyed a few episodes of the Dbt Labs Analytic Engineering podcast. This episode I particularly liked because Bernhardsson really has some insightful ideas and lots of experience in the sector. It seems his experience in both data and software engineering produces good synergies.

He talks about how basically all data systems end up being monoliths without a good plan for productionizing & deploying them. It’s basically the world of software engineering 20 years ago, before microservices and CD. The fun part is that data developers simply have to use the same techniques software developers use, just translate them into their realm. It’s something I emphasized a bunch of times already in this newsletter and my blog posts. So listen to the episode if you want to hear Erik’s thoughts on this.

I really like the story of Kolibri games because it describes the stages of the data system evolution from the birth of the company! Not some pretty place in between. In 2016, the company was founded. In 2017, one marketing manager was writing some R scripts. In 2018, a team of three (!) built up an event hub, an SQL database, and finally Looker on top of it all. A year later with a team of five, they moved components over to databricks and snowflake.

After their acquisition, they started to roll out a more decentralized concept, a data mesh. They employed mostly full-stack teams inside business units:

They seem to enable this by using dbt on top of snowflake and an ELT pattern.

I really like the quote at the end, which is:

In my opinion, it’s not just sometimes. Build a data-driven culture, decentralize responsibility for data and the right tech stack will emerge…

Thanks for reading this far! I’d also love it if you shared this newsletter with people whom you think might be interested in it.