In a fast-paced world, companies cannot rely on a single team to access data, analyze
and interpret it, and provide actionable insights. Democratisation of data has become essential as it allows users from all domains to access data and quickly make informed decisions without depending on one team of data experts.
This need for easily accessible data has led companies to transition from a monolithic
and centralised data architecture to a decentralised architecture. This decentralised
architecture is called data mesh. A data mesh is an architectural design that allows different departments and teams within a company to own and manage the data they produce.
There are various benefits of using a data mesh:
– It decentralises data management, eliminates data silos, and ensures a smooth flow of data between different systems and applications.
– Since the data is distributed, companies have more control over it.
– Scalability becomes easy as companies can add more nodes to manage large data volumes.
– Since there’s no single point of failure, companies don’t have to worry about unplanned outages or failures. The architecture is well-equipped to manage large traffic requests.
– The data quality and accuracy improve as data ownership gets distributed among different domains and business units within the company. Data ownership increases accountability among domain experts handling the data and encourages them to maintain its quality and integrity.
The need for data mesh has increased, with the world preparing to generate 180 zettabytes of data annually by 2025. It can help companies improve data connectivity, interoperability, and velocity and make data-driven decisions.
However, there are some limitations that companies must know before transitioning from monolithic architecture to data mesh architecture.
Challenges of Data Mesh Architecture
● Complexities: Every team within a company uses a different set of tools and processes to gather and store data. Receiving data sets from various sources and stitching them together to analyze can become complicated. Users may be left with information that’s incomplete or inaccurate.
● Data quality: A centralised data team typically manages data quality to ensure other teams can access accurate data. However, with data mesh architecture, that responsibility is handed over to multiple teams. This creates a problem because the quality is dependent on various teams. There’s no data standardization, as teams have different priorities and don’t use the same terminologies. The problem escalates further when one team tries to change something, compromising the data quality. The other teams would then have to rectify the problem to ensure everybody has access to the same data. Without clear data policies and standards, the teams will continue to rely on substandard and inconsistent data, compromising the accuracy and effectiveness of their decision-making.
● Organisational culture: Companies often face resistance to change because data
mesh architecture shifts the power of data ownership from a centralised data team to individual teams. There are also concerns about slow adoption, as many teams lack the skills to own, manage, and analyze data. The steep learning curve in managing and owning data creates a roadblock for adoption. The lack of understanding and skills could also lead to poor data interpretation and decision-making. Also, while data mesh might eliminate the problem of data silos, the silos between different teams can pose challenges for the company.
Without shared objectives, well-established standards and policies, and coherence between different teams, data mesh architecture cannot achieve the goals of data democratisation.
That’s why companies need data mesh governance.
What is data mesh governance?
Data mesh governance helps in striking a balance between establishing governance and
best practices centrally and providing autonomy to domain teams to manage and
execute data independently.
It improves data accessibility and decision-making without compromising on quality.
Here’s how it works:
Form a centralised data mesh governance group comprising data specialists, subject
matter experts, and security and legal experts. This central governance team –
– Creates guidelines, builds tools, and monitors the domains
– Ensure everything operates smoothly and safely
– Empowers all domain teams to access relevant data to make decisions
The domains that are a part of the mesh must strictly follow these guidelines and standards while accessing and using the data.
3 Reasons Why Data Mesh Governance Matters
1. Better governance: The level of collaboration and governance can help
companies maintain data consistency, quality, security, seamless collaboration,
and data-driven decision-making across all domains.
2. Data interoperability: It can enhance data interoperability and help different domain teams leverage various datasets available across the company to make decisions.
3. Adherence to compliance during changing times: More importantly, the data mesh governance team will have more control over policies and processes that will help safeguard data and adhere to the changing compliance laws.
7 Steps to Implement and Improve Data Mesh Governance
Here are a few things companies must do to implement and improve data mesh governance:
1. Identify the data sources and evaluate the existing infrastructure, team structure, and data governance practices.
2. Based on the evaluation, divide the teams into domain teams and align them with
the company’s functions and goals. These teams will act as the data custodians for that specific domain.
3. Conduct regular training sessions to ensure data accuracy and integrity.
4. Establish best practices like privacy policies, data cataloging practices, and access policies to improve governance.
5. Identify key stakeholders who would drive and enforce governance across the company and create a central governance team. This team will oversee IT, security, data management, and other compliance-related tasks for all domains.
6. Define Key Performance Indicators (KPIs) and monitor the data mesh governance based on those KPIs. Regular monitoring will help companies improve and iterate the governance model and quickly adapt to the changing business and compliance needs.
7. Finally, work with an expert to implement the data mesh architecture within the
company successfully.
Data mesh architecture drives companies toward the digital transformation journey.