Big Data technologies have transformed the way we store and process data and this has significantly impacted the storage companies.
By Neeraj Sabharwal
Big Data technologies have significantly impacted the technology roadmap of many solution and product companies. To being with, let’s talk about a few major developments. Software licenses or vendor lock in: Most of the big data technologies fall under the umbrella of open source or ASF, Apache Software Foundation and open source model challenging the traditional software companies.
For the sake of conversation, let’s talk about distributed computing systems like Hadoop or NoSql like Apache Cassandra. They are open source technologies getting picked by different vendors, where in they are making it enterprise ready and selling their support by providing enterprise ready software based on open source projects. Businesses, these days, want to focus on scalability, lower costs and effectiveness.
Big Data technologies have transformed the way we store and process data and this has significantly impacted the storage companies. Appliance companies are being affected by big data because of low cost solutions to process and store any amount of data, as focus has shifted towards horizontal scaling instead of vertical scaling. Businesses are focusing on data consolidation and walking away from creating more silos of data. The word Data Lake/Hub has emerged to store several petabytes being created by IOT devices, smart phone apps, social media etc. as putting all this data on traditional systems like expensive SAN is not possible because of hardware cost and performance of processing the data.
Security is one of the biggest concerns when it comes to storing, processing and ingesting data. We want to make sure that strong authentication, authorization, data encryption at rest and while data is moving, is robustly in place. Imagine discovering credit card or SSN or any PII out of Data Lake of several petabytes. The emergence of big data technology demands better and simpler methods to take care of the above mentioned security standards and this has changed the road map of various security product companies.
Networking companies have improved their products based on the high demand of secure transmission of large volume of data. As, we are talking about horizontal computing where we can scale up by adding more resources in a cluster, so better monitoring of bandwidth, network delay, availability and secure transmission is keeping networking companies on their toes to provide better and simpler ways to manage devices.
Now, it comes to analytics and data management, and if you are using smart phone app or social media, then you are a part of this as technology companies are working smart to create better analytics on all the data that is being generated by sensors, social media or anything that can track activities.
We have more new data types and this has created a new challenge for analytics companies to process all the data in near real time to provide faster, effective and quick analytics. The new data type like unstructured data demands better ways to transform the data and store in a better or compressed format to reduce the storage cost without impacting the processing performance. The demand of data analytics while data is in action or moving is growing.
To close this topic, let’s see how big data has impacted cloud. There is a shift happening as of now, CIO or CTO want to move everything to cloud because of various reasons like cost, no more worrying about buying hardware, ease of management and quicker time to market. Big Data has impacted the cloud companies because of large volumes of data and demand of faster analytics while data is still in action.
In the last few years, several new startups have emerged based on cloud solutions to address this demand and have come up with solutions to make it easier for technical and business users to handle big data challenges. There is a higher demand of hybrid cloud to copy or move data from on-premises to cloud and this has impacted traditional software companies to change their roadmaps to fulfil the demand of cloud computing. The demand of containerizing the development to reduce development cycle is changing the way we used to compute.
The author is director cloud and big data solutions, at Xavient Information Systems
