Making sense of a tsunami of big data

Making sense of a tsunami of big data

A motorist passes a car damaged by a tsunami, in Tanjung Lesung, Indonesia, Monday, Dec. 24, 2018. (AP)
A motorist passes a car damaged by a tsunami, in Tanjung Lesung, Indonesia, Monday, Dec. 24, 2018. (AP)
Short Url

Managing a tsunami of personal information presents firms with new challenges and opens a Pandora’s box on whether harvesting big data is a legitimate exercise, or one that pushes society into Big Brother territory.
But what is big data? The quick answer is that it is a term that makes a fuss out of something businesses have always had to manage — vast quantities of information. There is however more to it than that, because now everybody generates data 24/7, and it is digital, rather than being stored on paper in filing cabinets. The data available to those who want to pursue it is astonishing with IBM estimating that each one of us generates 2.5 gigabits of data every day. This is the footprint left by tweets, emails, online transactions, swiping rewards cards at tills to collect loyalty points, and logging in to our networked devices.
It does not stop there. On top of that comes the data that businesses themselves generate — emails, internal newsletters, invoices and purchase orders. Then comes the digital information that pours out of governments and official bodies — economic indicators, share price movements, fresh stock market indices, an endless stream.
Is there a benefit for those who want to harvest this data? How does a business manage this tsunami of information, never mind make sense of it? Computerised data used to be tidily placed onto structured databases that businesses owned themselves and tapped into at intervals. Now, data is often unstructured and comes from a myriad of sources. The old tools do not work anymore — new platforms and new algorithms have to be created and applied to allow efficient storage, access and analysis.
There is also the sheer volume. Because storage is still relatively inexpensive — though this could change — far more data is stored than ever before. This explosion of data demands a new way of thinking about it. There’s financial data, there’s social media, which complicates matters, but is an important element, especially for the Z generation (of 10 to 25-year-olds). Social media is at the heart of what businesses have to pay attention to. That is not just Twitter or Facebook, but such platforms as LinkedIn, YouTube, Audioboo and blogs. Sentiment about a firm can be spotted, tracked, aggregated and analyzed across all these platforms. And self-appointed influencers can have a sudden impact on the way a business is perceived.
Then to complicate matters, there is also velocity, the speed at which data flows. Businesses need to be nimble not only in collecting and analyzing data, but also in responding to it. As well as keeping tabs on what is happening now, the smart company can use big data to manage its future business. However useful an analytic tool it can be, big data throws up issues of management and compliance for firms of all sizes. Previously, businesses kept a tight rein on data created inside the company — what came in and what went out. But the move toward bring-your-own-device policies, where staff fire up their own laptops and mobile phones at work, mean it is much harder for an IT department to track data, as well as possible leaks to competitors.
There are concerns for firms that allow users to pick up their corporate emails on personal mobiles. Who has access to that data, how is it being shared? If a user is working on his own laptop that connects to the corporate network, what if that laptop contains copyright-violating material?
The cloud — which refers to data and software stored on network servers and accessed via the Internet, rather than on individual devices — can help with many of the data issues that firms face, from storage and security to analysis.
But is this data harvesting ultimately worthwhile? Just having lots of data and being able to access it securely is not enough. Users have to be able to analyze data constructively. A company might, for example, find it wants to track all references to its products whether these are stored as images, sounds or words, and is likely to want to cross-reference and compare them in counter-intuitive ways.
But when enormous quantities of data have been hoovered up without reference to how best to present them for use in analysis, and stored in varying formats on different pieces of hardware, this is no small task and needs a new breed of big data analysts. So, in theory, this is a golden age for businesses — data is easy to collect, storage is cheap and the IT industry is increasingly able to offer smart deciphering services. But what are the downsides?
Data fatigue comes to mind as one can wind up with a huge amount of data and be overwhelmed by it, making it impractical to use. So perhaps the biggest challenge of big data is how to make sense of it by using good old-fashioned human common sense and intuition, a rare trait for some. In the meantime, just keeping a handle on big data is going to keep company chief information officers busy for a while to come.

Dr. Mohamed Ramady is a former senior banker and professor of finance and economics at King Fahd University of Petroleum and Minerals in Dhahran.

Disclaimer: Views expressed by writers in this section are their own and do not necessarily reflect Arab News' point of view