DataEng Digest - The very first issue!

One of the most popular article on data engineering written by Apache Airflow creator Maxime Beauchemin. This post clearly defines the role of a data engineer in a data-driven organization.

Follow-up article on “The Rise of the Data Engineer” which mainly focuses on the cons of being a data engineer.

In this post, we’ll explore how applying the functional programming paradigm to data engineering can bring a lot of clarity to the process. This post distills fragments of wisdom accumulated while working at Yahoo, Facebook, Airbnb and Lyft, with the perspective of well over a decade of data warehousing and data engineering experience.

Introduction to various data encoding formats including plain-text JSON/XML/CSV and binary formats like Avro, Protobuf etc.

Alex Petrov explains how B-Tree and LSM-Tree entered the world of modern data storages.

A collection of useful resources for novice on how to start diving into distributed systems realm.

The first part of the series of articles on distributed systems which mainly focuses on basic concepts and prepares distributed system engineer vocabulary.

This articles continues building up prerequisites for going into details of advanced consensus algorithms, starting with Shared Memory and building blocks for the distributed storage, then introduce Linearizability, figure out ways to implement it such as Two/Three Phase Commit and finally move to the Atomic Broadcast.

Max Beauchemin, author of Airflow & Superset, presents functional approach to building data pipelines. Video is based on his articles called “Functional Data Engineering — a modern paradigm for batch data processing”.

When models and data applications are pushed to production, they become brittle black boxes that can and will break. In this talk you’ll learn how to one-up your data science workflow with a little engineering!

Tom Augspurger explains how to scale your machine learning models using Dask.

What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data.