Over the last few months, working with businesses who chose to address their fast data needs, I have experienced firsthand the transition from batch systems to real-time systems. It is great to be fast, but to derive insight from piles of data is a very hard problem to solve. For businesses, how fast you act on a customer’s data to make a contextually relevant decision is as important as an insight derived from analyzing Big Data.
So as the title suggests – are the fast data and big data ecosystems competing with each other, as did David vs. Goliath, where for one to succeed the other must fail? I don’t believe so. Let me try to convince you, by the end of this post, that the interplay between these two distinct ecosystems is not only mutually beneficial but also necessary.
In the year 2010, Apache Hadoop (which had its roots in Nutch and a few whitepapers from Google) was barely four years old; only a handful of Web 2.0 companies had adopted it to index the whole web and make it searchable. In the year 2010, the original team that developed Hadoop was still part of Yahoo, and Cloudera, a young startup, was trying to compete with Amazon and still figuring out what its product should be. In the intervening years, I have seen Hadoop evolve into a much wider platform where businesses ranging from life sciences to financial services to manufacturing and many more sectors have modernized their infrastructures to realize its OpEx/CapEx benefits.
The global Big Data market is projected to reach $122B in revenue by 2020. By 2020, every person online will create roughly 1.7 megabytes of new data every second of every day, and that’s on top of the 44 zettabytes (or 44 trillion gigabytes) of data that will exist in the digital universe by that time. In a recent Computerworld article, the University of California Berkeley Professor Vincent Kelly predicted that more than a trillion devices would be in our world within the next 10 years. Frost & Sullivan forecasts global data traffic will cross 100 Zettabytes annually by 2025.
As the physical and digital worlds converge, it’s nothing short of the datafication of life. It will continue to grow at record rates and will overwhelm our ability to manage effectively this amount of data. With a trillion devices on the way – the volume, velocity, and variety of data are going to grow dramatically.
Yes, organizations are amassing larger and larger datasets from a growing number of sources. However, the ability to process this data has not kept pace. It’s a massive business opportunity but also a significant potential threat to businesses. In an increasingly data-intensive world, your business is only as fast and effective as your competitor’s ability to process its data to give real-time actionable insights. Here’s where the concept of “fast data” steps up to the plate.
Competitive edge – Fast Data gives your business wings