Real-Time dbt + trino, Cookies, Top50 Data; ThDPTh #66

What: a16z took the time to put together some data on the top50 data start-ups including valuation, funding, category, etc. It’s the first time I’m seeing such an analysis.

My perspective: A must-read.

What: Michiel De Smet shares a demo setup using trino, dbt, and a local data lake topped off with Hive & Iceberg. It provides real-time data leveraging dbt and trino. He’s using trino as a query engine and uses a simple lambda architecture where:

The architecture for sample purposes does not implement any kind of sampling mechanism on the real-time part as a lambda architecture sometimes does.

My perspective: I feel like both, neither data architectures nor the importance of real-time data gets enough attention these days. Real-time data will become the thing of the next decade. And for data to be effectively used, we need good architectures to enable them.

So I love that Michiel explains complicated architecture so plain and simple. Check it out and think about it. The obvious drawback of this solution is querying a live database on demand which will need either controlling or some kind of caching layer/copy of the production system to not crash the production system for analytical queries.

What: This is an article written by data scientist Alan Schelten, a colleague of mine about the practical troubles of doing learning-to-rank.

My perspective: I think this is the first time I share something written by a colleague in the newsletter. But I truly enjoyed reading through this piece. It fits very well into the newsletter. It’s practical and explains the troubles the everyday data scientist deals with. I particularly like both the optimization target which is chosen: “customer satisfaction”, as well as the following explanation of how to then choose a proxy to indirectly measure exactly that.

I think the summary is pretty simple: Machine learning & data science in real-life is not a Kaggle project. It is messy and almost nothing is about cool algorithms. But it also can be pretty amazingly effective, in the example, we’re talking about a possible 1% gain in revenue.