Big data: The obstacles we’re seeing and how to overcome them
- 21 October, 2016 15:12
Picture: Ron Mader, Flickr
The digital universe is doubling every two years, according to EMC. That means, by 2020, about 1.7 MB of new information will be created every second, for every human being on the planet. In business, this previously unimaginable proliferation of data can be overwhelming and seemingly impossible to navigate. But it also poses the question - how can we ignore something that promises to solve every conceivable business problem?
Seductive, yes. Impossible, no.
We are rapidly approaching a digital inflection point, and big data analytics is no longer just a “good idea.” Turning mountains of data into insight, efficiently and cost effectively, perhaps in real time even, is a business imperative that is critical to staying competitive. Or for some, staying in business.
What follows are the obstacles we’re seeing, and how to overcome them.
Mostly, the conversations we’re hearing on big data today tend to focus on its technology challenges. The biggest hurdle that enterprise customers face is that they don’t know which technology to act on, or to bet on, for their big data strategy. This is further muddled now that all of the players - Microsoft, IBM, Oracle, and so on – all have offerings on the table.
Big data in itself, however, is fundamentally not a technology issue, and the first step in the right direction is in recognising that point. The arguably more difficult element of solving the big data puzzle is actually cultural. Big data’s pervasiveness – the way it can disrupt people and processes across every part of an organisation – means any big data initiative requires careful and significant thought.
The change that’s more essential to tapping into big data’s potential is one of perspective.
Big data offers a new way to look at the world through a completely different lens. For this reason, big data initiatives require a comprehensive, holistic approach to the ‘problem’, with a balanced focus on the pillars of people, processes and technology.
This includes considerations around:
- How people work, think and communicate,
- Their existing expertise and skills,
- The culture of the organisation and its receptiveness to change,
- Current processes and the adjustments required, however big or small,
- The technology needs and necessities of the company.
The more data of different forms that is introduced into the organisation, the more pressing the need for a sophisticated and scalable approach to managing that big data environment.
When implementing a big data initiative, there’s often a temptation to expedite projects by skipping elements and taking shortcuts. But like most things, a push for immediate results is likely to end in disappointment and unmet expectations.
[Related: What big data can reveal about your staff]
‘Going rogue’, as I call it, is rarely an effective approach to big data and may even undermine the effort entirely. Rogue projects tend to be focused on isolated technology deployment, lack collaboration, cost too much, and end up nearly impossible to sustain.
The fact big data projects are by nature, highly iterative, does not eliminate the need for a plan. Rather, the plan needs to be flexible and adaptable to consistent, dynamic change. Importantly, establishing open channels of communication and interaction across relevant stakeholders is critical to making sure that everyone remains aligned and collaborative on the plan and its staging.
According to Paul Gibbons, author of The Science of Successful Organisational Change, “The human side of analytics is the biggest challenge to implementing big data.” Why? Because most enterprise cultures are hierarchical, and this corporate style may present as an obstacle to achieving the continuous improvement and innovation that’s necessary to survive in today’s dynamic digital world.
Instead, organisations should cultivate a learning culture that values problem identification, knowledge transfer, and reflective post-audits across the full team.
The existing franchise
Big Data projects also need to be an integral part of your existing business and IT infrastructure. So, it’s critical to involve internal process experts from the beginning. Inspecting and adapting processes and data flow with the right people in the organisation at regular intervals is key to growing and continuously improving big data initiatives by reducing cycle times, modelling change, increasing analytics speed and getting the right information to decision makers faster.
Focusing on small, discreet projects is a good place to start. Identifying and prioritising those that solve concrete problems helps to establish realistic expectations and demonstrate value for those that may be resistant to change.
No matter what big data technologies are used, they require the same commitment as your other information systems. Big data is not an island, and big data projects have a significant impact on existing IT infrastructure and operations.
To conclude, a successful Big Data adoption program should consider the pillars of people, place and technology, and how each is impacted or influenced by the other. To get to the stage where your business is benefiting from clear, data-driven insights, requires an iterative process, incorporating learnings along the way. When it comes to the enabling technology, it’s vital that big data platforms integrate with existing enterprise applications in order to optimise the cost of underlying infrastructure, and ensure that your foray into big data is a fruitful one.
Gavin Selkirk is VP APAC for BMC Software