Embracing Big Data
Open source big data technologies like Hadoop have accomplished much to begin the transformation of analytics. We’re moving from expensive and specialist analytics teams towards an environment in which processes, workflows, and decision-making throughout an organisation can – in theory at least – become usefully data-driven. Established providers of analytics, BI and data warehouse technologies liberally sprinkle Hadoop, Spark and other cool project names throughout their products, delivering real advantages and real cost-savings, as well as grabbing some of the Hadoop glow for themselves. Startups, often closely associated with shepherding one of the newer open source projects, also compete for mindshare and custom.
And the opportunity is big. Hortonworks, for example, has described the global big data market as a $50 billion opportunity. But that pales into insignificance next to what Hortonworks (again) describes as a $1.7 trillion opportunity. Other companies and analysts have their own numbers, which do differ, but the step-change is clear and significant. Hadoop, and the vendors gravitating to that community, mostly address ‘data at rest’; data that has already been collected from some process or interaction or query. The bigger opportunity relates to ‘data in motion,’ and to the Internet of things that will be responsible for generating so much of this.
Adaptable organizations that embrace big data for a competitive advantage double-down on the idea of big data concepts both in content and format. Rapid experimentation is at the core of adaptability, and big data analytics is the best secret to rapid experimentation. At any point in time, your innovation funnel should be flooded with a variety of ideas that are rapidly moving their way to become product, service, or relationship offerings.