​Big data: The obstacles we’re seeing and how to overcome them

Consider the pillars of people, place and technology, and how each is impacted or influenced by the other

Picture: Ron Mader, Flickr

Picture: Ron Mader, Flickr

The digital universe is doubling every two years, according to EMC. That means, by 2020, about 1.7 MB of new information will be created every second, for every human being on the planet. In business, this previously unimaginable proliferation of data can be overwhelming and seemingly impossible to navigate. But it also poses the question - how can we ignore something that promises to solve every conceivable business problem?

Seductive, yes. Impossible, no.

We are rapidly approaching a digital inflection point, and big data analytics is no longer just a “good idea.” Turning mountains of data into insight, efficiently and cost effectively, perhaps in real time even, is a business imperative that is critical to staying competitive. Or for some, staying in business.

What follows are the obstacles we’re seeing, and how to overcome them.

The talk

Mostly, the conversations we’re hearing on big data today tend to focus on its technology challenges. The biggest hurdle that enterprise customers face is that they don’t know which technology to act on, or to bet on, for their big data strategy. This is further muddled now that all of the players - Microsoft, IBM, Oracle, and so on – all have offerings on the table.

The truth

Big data in itself, however, is fundamentally not a technology issue, and the first step in the right direction is in recognising that point. The arguably more difficult element of solving the big data puzzle is actually cultural. Big data’s pervasiveness – the way it can disrupt people and processes across every part of an organisation – means any big data initiative requires careful and significant thought.

The way

The change that’s more essential to tapping into big data’s potential is one of perspective.

Big data offers a new way to look at the world through a completely different lens. For this reason, big data initiatives require a comprehensive, holistic approach to the ‘problem’, with a balanced focus on the pillars of people, processes and technology.

This includes considerations around:

  • How people work, think and communicate,
  • Their existing expertise and skills,
  • The culture of the organisation and its receptiveness to change,
  • Current processes and the adjustments required, however big or small,
  • The technology needs and necessities of the company.

The method

The more data of different forms that is introduced into the organisation, the more pressing the need for a sophisticated and scalable approach to managing that big data environment.

When implementing a big data initiative, there’s often a temptation to expedite projects by skipping elements and taking shortcuts. But like most things, a push for immediate results is likely to end in disappointment and unmet expectations.

[Related: What big data can reveal about your staff]

‘Going rogue’, as I call it, is rarely an effective approach to big data and may even undermine the effort entirely. Rogue projects tend to be focused on isolated technology deployment, lack collaboration, cost too much, and end up nearly impossible to sustain.

The fact big data projects are by nature, highly iterative, does not eliminate the need for a plan. Rather, the plan needs to be flexible and adaptable to consistent, dynamic change. Importantly, establishing open channels of communication and interaction across relevant stakeholders is critical to making sure that everyone remains aligned and collaborative on the plan and its staging.

According to Paul Gibbons, author of The Science of Successful Organisational Change, “The human side of analytics is the biggest challenge to implementing big data.” Why? Because most enterprise cultures are hierarchical, and this corporate style may present as an obstacle to achieving the continuous improvement and innovation that’s necessary to survive in today’s dynamic digital world.

Instead, organisations should cultivate a learning culture that values problem identification, knowledge transfer, and reflective post-audits across the full team.

The existing franchise

Big Data projects also need to be an integral part of your existing business and IT infrastructure. So, it’s critical to involve internal process experts from the beginning. Inspecting and adapting processes and data flow with the right people in the organisation at regular intervals is key to growing and continuously improving big data initiatives by reducing cycle times, modelling change, increasing analytics speed and getting the right information to decision makers faster.

Focusing on small, discreet projects is a good place to start. Identifying and prioritising those that solve concrete problems helps to establish realistic expectations and demonstrate value for those that may be resistant to change.

[Related: Is big data and artificial intelligence a (r)evolution in outsourcing?]

No matter what big data technologies are used, they require the same commitment as your other information systems. Big data is not an island, and big data projects have a significant impact on existing IT infrastructure and operations.

To conclude, a successful Big Data adoption program should consider the pillars of people, place and technology, and how each is impacted or influenced by the other. To get to the stage where your business is benefiting from clear, data-driven insights, requires an iterative process, incorporating learnings along the way. When it comes to the enabling technology, it’s vital that big data platforms integrate with existing enterprise applications in order to optimise the cost of underlying infrastructure, and ensure that your foray into big data is a fruitful one.

Gavin Selkirk is VP APAC for BMC Software

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags business

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Gavin Selkirk

PC World
Show Comments

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Tom Pope

Dynabook Portégé X30L-G

Ultimately this laptop has achieved everything I would hope for in a laptop for work, while fitting that into a form factor and weight that is remarkable.

Tom Sellers


This smart laptop was enjoyable to use and great to work on – creating content was super simple.

Lolita Wang


It really doesn’t get more “gaming laptop” than this.

Jack Jeffries


As the Maserati or BMW of laptops, it would fit perfectly in the hands of a professional needing firepower under the hood, sophistication and class on the surface, and gaming prowess (sports mode if you will) in between.

Taylor Carr


The MSI PS63 is an amazing laptop and I would definitely consider buying one in the future.

Christopher Low

Brother RJ-4230B

This small mobile printer is exactly what I need for invoicing and other jobs such as sending fellow tradesman details or step-by-step instructions that I can easily print off from my phone or the Web.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?