I always enjoy ThoughtWorks Technology radar, but the current one is particularly well suited for my recent thinking about data meshes & data platforms:
“Increasingly, organizations are adopting a platform team concept: set up a dedicated group that creates and supports internal platform capabilities — cloud native, continuous delivery, modern observability, AuthZ/N patterns, service mesh, and so on — then use those capabilities to accelerate application development, reduce operational complexity and improve time to market. This growing maturity is welcome and we first featured this technique in the Radar in 2017. But with increasing maturity, we’re also discovering antipatterns that organizations should avoid. For example, “one platform to rule them all” may not be optimal, “big platform up front” may take years to deliver value and “build it and they will come” might end up as a wasted effort. Instead, using a product-thinking approach can help you clarify what each of your internal platforms should provide, depending on its customers. Companies that put their platform teams behind a ticketing system like an old-school operations silo find the same disadvantages of misaligned prioritization: slow feedback and response, resource allocation contention and other well-known problems caused by the silo. We’ve also seen several new tools and integration patterns for teams and technologies emerge, allowing more effective partitioning of both.”
Now what ThoughtWorks is advocating is to follow the “product-thinking approach”. I think “product-thinking” can be misunderstood and is misunderstood a lot! There are at least two versions of “product-thinking” when it comes to platforms.
Version 1: I (product-)think we need a data platform. Let’s pick a team, task them with building one, and then see where things go.
- Step 1: Put the responsibility for X inside each single team (or unit, the knowledge system is an internal product aimed at individuals, there are other products aimed at hole business units)
- Step 2: Make them accountable. Wait and see what solutions emerge.
- Step 3: Support the solutions with products, if necessary.
This is the process a lot of product teams evolve, how Borg (later Kubernetes) emerged out of a need of one product team launching a new Google search engine. They built Borg to do that. After it was there, virtually all workloads switched over to Borg. But that’s the thing, the product or big parts of it, already were there.
Now here’s the key difference: It seems to me from the outside, that a lot of data mesh implementations and data platform implementations follow Version 1; There usually isn’t any part of the platform there. This will likely end in what ThoughtWorks describes as again “one platform to rule them all”.
So what can we do? It’s starting to start thin! And then even thinner, let a suitable solution emerge. Start with a simple wiki page, without any team at all to manage the platform. If that doesn’t catch on, chances are you don’t actually need a platform yet.
Oh, and of course I recommend every single edition of the Tech Radar.