In
Gartner’s hype cycle, the term ‘big data’ was once a staple of the
yearly report. It moved swiftly into the peak of inflated expectations,
weathered its way through the trough of disillusionment, and is now
prevalent – somewhere between the slope of enlightenment and the plateau
of productivity.
Expectations
of the technology are high, a Gartner survey in September 2015 showed
more that 75% of companies are investing or planning to invest in big
data in the next two years, and 37% of those projects are being driven
from board level.
But
as a term, ‘big data’ still has no clear definition. For some, a
dataset over a terabyte is big data – for others, it might be a million
rows, and others still may have smaller datasets that is changing many
times a second.
In
the era of Google, Facebook, Amazon and web-scale data, no dataset
should be too difficult to analyse. It is all about having the right
tool for the job.
So
instead of defining big data as a number or a size, it is more
interesting and relevant to define it in terms of history, growth,
compute and value.
History:
29 million
That’s
the size, in number of records, of the world’s first big data project
in 1937. At that time, the administration in America were looking to
keep track of social security contributions from some 26 million
Americans, and 3 million employers and partners were sought.
IBM,
with its giant punch-card machines of the time, got the contract –
simultaneously setting the foundations for the Big Blue known now and
setting in motion the start of automatic record keeping and data
analysis on a massive scale.
Growth:
Zettabyte
For
the first time since its conception, global internet traffic will
surpass 1 zettabyte (1 billion terabytes) in 2016, according to a Cisco
research paper, having risen fivefold in the past five years.
A
separate study estimates that 90% of the world’s data was generated in
the past two years. Not only are we clicking, emailing, chatting and
taking photos or videos more than ever, companies have cottoned onto the
fact that data is valuable so are storing more and more data. Datasets
such as website access logs and click data are no longer being thrown
away – they are being archived and mined to generate valuable insights.
Compute:
24 months
That’s
the period that Gordon Moore, co-founder of Intel, observed was the
amount of time it took the number of transistors in a dense integrated
circuit to double. Similar observations have been found in other areas –
it has been estimated that the amount of data transmittable through an
optical fibre doubles every nine months and storage density doubles
roughly every 13 months.
Organisations
can process, transmit and store more data than ever, and all three are
exponentially increasing commodities. Companies are better placed than
ever before to deal with data.
Value:
$59 billion
That
is the estimated value of the big data market in 2015 and it is
expected to roughly double to $102 billion by 2019, according to IDC.
Big data is big bucks, a 21st century gold rush – and to extend the
metaphor, big data analytics is the modern-day equivalent of panning for
gold.
For
data-first companies, monetisation comes in the form of advertising –
and big data analytics helps them to show an appropriate advert as well
as analyse the results. For other companies it is often about increasing
sales (the ‘I see you bought this, what about this?’ offers),
automating decisions (big data gives the proof that an option is the
correct one to take) and decreasing costs (driving efficiencies in the
supply chain).
It’s clear big data is growing – it’s here to stay and, right up to board level, companies have woken up to its potential.
Companies
see that with big data analytics they can find insights that enable
them to outperform their competitors and reduce costs. With the right
tools, these insights are easier to obtain than ever.
Blogger Comment