Introduction:
In today’s data-driven world, organizations are continually seeking innovative approaches to manage and harness the power of their data. Two emerging concepts that have garnered significant attention are Data Fabric and Data Mesh. Both aim to address the challenges of handling vast amounts of data, but they do so in distinct ways. In this blog, we’ll explore the key principles, benefits, and considerations of Data Fabric and Data Mesh to help you make informed decisions about your data architecture strategy.
What is Data Mesh
- Domain-oriented decentralized data ownership and architecture
- Data as a product
- Self-serve data infrastructure as a platform
- Federated computational governance
What is Data Fabric
- Augmented Data Catalog
- Persistence Layer
- Knowledge Graph
- Insights and Recommendations Engine
- Data Preparation and Data Delivery Layer
- Orchestration and Data Ops
- Automation AI/ML
- APIs
Data Fabric vs Data Mesh: What’s the difference?
- The difference between the two concepts lies in how users access data.
- Data fabric and data mesh provide architecture to access data across multiple technologies and platforms. “But a data fabric is technology-centric, while a data mesh focuses on organizational change.
- Data fabric is an all-in-one integrated architectural layer that connects data and analytical processes. It leverages existing metadata assets to support the design, deployment, and proper data utilization across all environments and platforms. Data fabric aims to accelerate inference from data through automated processes and provide real-time insights. It integrates data, analytics, and dashboarding into one and serves as a management solution, allowing frictionless access in a distributed environment. AI/ML is inbuilt.
- Data Mesh is a highly decentralized data architecture equipped to address challenges including lack of ownership of data, lack of quality data and scaling bottlenecks. The goal of data mesh is to treat data as a product, with each source having a data product owner who could be part of the cross-functional team of data engineers. Data mesh — introduced by Zhamak Dehghani of Thoughtworks in May 2019– overcomes the problems of traditional data lakes and data warehouses.
- Data Fabric fits under Data Mesh very well.
Approach:
Automation vs Human Inclusion
- Data Mesh approaches data from a people-and process-centric view and treats data as a product.
- Data fabric leverages human and machine capabilities to access data in place or support its consolidation where appropriate. It combines technologies that connect sources of data, types and locations with different methods for accessing the data.
- Gartner used the analogy of a self-driving car to explain the concept:
- Data fabric monitors the data pipelines as a passive observer and then suggests more productive alternatives. When both the data “driver” and the machine learning are comfortable with repeated scenarios, they complement each other by automating improvisational tasks while leaving the leadership free to focus on innovation.
Data storage: Centralized vs Decentralized.
Data access: APIs vs-controlled datasets
- In Data Mesh, data is made available via controlled datasets. First, the information is copied from the department data store to a shared location Unified data
- In Data Fabric, data is made available via objective-based APIs. The data is copied into specific datasets for specific use-cases, and the business unit that owns the data is in control.
- Data fabric continuously identifies, connects, and enriches real-time data from different applications to discover relationships between data points. It does so by building a graph storing interlinked data descriptions that algorithms can use for business analytics.
Data Fabric Vs. Data Mesh: Main Differences
| Data Fabric | Data Mesh | |
|---|---|---|
| Architecture | Data is centralized. Data made available through APIs. Aims to eliminate human effort with machine learning and AI. | Data is stored within each domain of a company. Data is copied into specific datasets for specific use-cases. Less emphasis on AI, since work is handled by domain experts. |
| Benefits | Self-service data consumption and collaboration. Automates governance, data protection, and security. Automates data integration and data engineering. | Agility and scalability with fast access and accurate data delivery. Platform connectivity and data security. Robust data governance and end-to-end compliance. |
| Use Cases | Business applications – challenges of data availability and reliance for business applications. Data discovery – what data is available and where. Machine learning – minimizes the data preparation phase when training ML models. | Financial sector – fast fraud threat analysis without copying data to a central database. Sales and marketing – targeted campaigns based on user profiles. Machine learning – create virtual data warehouses as a basis for training ML models. |
It is critical to note that Data Mesh and Data Fabric are not mutually exclusive concepts. Organizations can leverage both approaches across different use cases.
- Data Mesh is ideal for hybrid cloud networks.
- Data fabric enables single-point data access, address data quality and storage issues and handling of security threats.
- The difference between the two concepts lies in how users access data.
- Data Fabric and Data Mesh provide architecture to access data across multiple technologies and platforms, “But a data fabric is technology-centric, while a data mesh focuses on organizational change.
- Data Mesh is more about people and process than architecture, while a data fabric is an architectural approach that tackles the complexity of data and metadata in a smart way that works well together,”
Architecture
Conclusion on data Fabrics vs Data Mesh
- To summarize, both data fabric and data mesh provide powerful solutions to make your organization data-driven and even data-led. Data fabric allows everyone (within permission) easy access to data at the right time. Data mesh takes a decentralized approach by keeping separate domain-specific datasets.
- Choosing one over the other essentially boils down to the problem your organization is dealing with.
