The First step to a build and Data Governance & Quality program will be key to successful implementation and sustenance of a Big Data or Enterprise Data Warehouse initiative.
By Kishan Venkat Narasiah and Viros Sharma
While Big Data is a Jazzy Tech trend, it was nascent till last year for which CIOs were trying to figure out the business use cases. In year 2016, Big Data would be one of the key maturing trends in the IT industry and upcoming technology that can bring huge benefits to the business organizations.
Organization are in varied stages of their big data journey would want to derive the business value by either starting as Proof of Value or as an extension to their existing Enterprise Data Warehouse. Some of the organizations are planning to leap frog their Data Strategies straight to Big Data platforms. Data is the backbone that has many facets be it Integration, Replication, Archival, Security, Migration, Quality, Decision Workflows, Backup, Policies, Compliance, Insights that require Data Management using specialized mechanisms to manage multiple aspects such as Meta Data, Data Lineage, Governance, Change Management (Data change leading to Organizational change), Master data, Business Process Management, and Risk Management elements for any organization that wants to derive insights. Big Data is the most often heard term and has its relevance in just about every industry – BFSI, Manufacturing, CPG, Travel, Hospitality, Healthcare, and Education & Government. Any organization that can integrate multiple sources of data, cleanse and govern the data is able to reap insights combing all the different data sets in a timely manner, can benefit from Big Data.
Organizations in general are increasingly relying upon confidential data such as intellectual property, market intelligence, trade secrets and customers’ personal information. Maintaining the privacy and confidentiality of this data as well as meeting the requirements of a growing list of related compliance obligations are the top concerns for enterprises.
Challenges
The challenges of implementing and deriving the required insights will definitely be a continuous improvement process, especially in a Big Data environment due to Structural, Relationship and Frequency related complications. However, there are 2 key elements that need to be addressed for any Enterprise Data Warehouse or Big Data implementation to work efficiently – Data Quality & Governance. The key related challenges would be:
1. Exponential growth of data (Storage, Backup)
2. Integration of various data source (structured, unstructured & Semi-structured)
3. Dynamic changes to the data (Real-time, Near Real time, batch mode)
4. Increase visibility and access to relevant data(Data silos, Departmentalized BI)
5. Transparency & Governance to increase confidence and enable opportunities(Time to insight)
6. Quality of data to derive business insight is a huge problem(Data Reconciliation)
7. Varied levels of data completion, formats and completeness
8. Users access to data for which they are authorized
9. Definition/ Meaning of Data Attributes (Metadata)
10. Making meaning out of data and feeding the meaning back to Data Repositories (Machine Learning enablement)
Importance of Data Quality & Governance
However, given the business benefits and high level of interest in deploying Big Data solutions across the enterprise there are also some prerequisites and dependencies that should be well understood and identified before embarking on a global Big data rollout that consumes significant amount of effort.
Although most organizations would be already at be at various levels of the BI maturity curve and would not be starting the implementation entirely from scratch. They would have already developed Enterprise Data Warehouses, Data Marts and related Business Intelligence (BI) / Analytics solutions, most realize that big data analytics can start with augmenting their existing Data Warehouse system and over a period of time replacing their Data warehouse that they have used historically for data warehousing and BI.
With more users integrating data from various external data sources and using data discovery tools across the organization, improved quality of data and the respective governance is critical to ensure consistent, efficient and effective use of data while enabling users to make better business decisions. Self-service data and analytics are fast becoming the standard and, increasingly, business users are demanding direct access to data to gain their own insights.
Better business outcomes will ensue as data insights get more accurate. This will put the emphasis on value creation and the onus is on Information Management to organize the data better by identifying right sources and streams and easier aggregation and integration that any user can leverage in any application she wishes.
Better data quality leads to an enhanced customer experience and satisfaction, effective master data management provides greater customer insight, operational efficiency and new opportunities for marketing and revenue generation. Simplifying the data landscape and reducing manual processes lowers operational costs.
Developing the right technology framework is critical to your ability to automate, manage, and scale out the implementation program. The need to identify a robust data quality technology infrastructure combined with the right Governance model that can support the organizations processes, policies, standards, organization, and technologies that address the following key elements:
1. All enterprise data can be accessed, regardless of its source or structure
2. Data is available to users and applications—when, where, and how it is needed
3. Data is accurate and valid
4. The value, structure, and meaning of data are consistent and reconciled across systems, processes, and organizations
5. There is an audit trail on the data and internal controls have been appropriately implemented
6. Data can be accessed securely
Conclusion
The First step to a build and Data Governance & Quality program will be key to successful implementation and sustenance of a Big Data or Enterprise Data Warehouse initiative. In order to achieve enterprise data governance & quality, there is a critical need for proper planning along with executive sponsorship and a continuous improvement program that can help make the necessary adjustments in the company’s governance structure as it matures over a period of time. It is useful to start the program in a small way and gradually build it up across the organization. With this in mind, constructing a case for data governance & quality remains one of the most challenging things for an organization to successfully accomplish.
Narasiah is Consulting Lead – DWBI & Analytics, ITC Infotech, & Sharma is Vice President & Global Practice Head – DWBI & Analytics, ITC Infotech