The Data Mesh

I’ve previously spoken about the increasing diversity of data teams (and the associated challenges of properly serving every data practitioner in your organization). I recently discovered this 2019 paper How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh, and it became clear that this could be the answer to those challenges.

Dehghani paints a beautiful picture of what collaboration could look like for data teams of the next decade. The volution of the data ecosystem, in her opinion, will be towards “the Data Mesh”. As Dehghani defines it:

distributed data products oriented around domains and owned by independent cross-functional teams who have embedded data engineers and data product owners, using common data infrastructure as a platform to host, prep and serve their data assets.

This paper was very much an inspiration for Intuit’s Data Platform. Intuit’s VP Engineering, Mammad Zadeh, recently outlined this journey in Intuit’s Data Journey. Tristan Baker, a distinguished engineer and Chief Architect of Intuit’s Data platform, wrote a great follow-up to the piece as well.

Around the Ecosystem

• Congrats to the Census team on raising their Series A from Sequoia! They were an early voice in the Modern Data Stack conversation. Here’s their vision for “Data as a Product” which dovetails nicely with the Data Mesh.

• Next time you need more budget for a customer analytics project, make sure to show the leadership team this study from McKinsey: “How data analytics helps sales reps win more deals”

• How Microsoft Solved for Data Quality. They have an entire team focused on DQ called Datacop! The authors provde a great framework for Data Discovery and Monitoring:

• A short podcast with Ali Ghodsi, the CEO and Founder of Databricks, on “The 3 Phases of Startup Growth”.

On Data Communities

dbt’s community manager recently wrote a post on How to Build a Community. The dbt community has seen incredible growth over the past few years and a great hangout for data professionals.

On that topic, I recently joined the Locally Optimistic slack community, “a blog for current and aspiring data analytics leaders”

Locally Optimistic is an amazing venue for meeting data practitioners and bouncing ideas off them. It’s a very engaged community and I highly recommend joining. Ping me if you want an invite!