Meet Apache Spot, a new open source project for cybersecurity

The effort taps big data analytics and machine learning for advanced threat detection

Hard on the heels of the discovery of the largest known data breach in history, Cloudera and Intel on Wednesday announced that they've donated a new open source project to the Apache Software Foundation with a focus on using big data analytics and machine learning for cybersecurity.

Originally created by Intel and launched as the Open Network Insight (ONI) project in February, the effort is now called Apache Spot and has been accepted into the ASF Incubator.

"The idea is, let's create a common data model that any application developer can take advantage of to bring new analytic capabilities to bear on cybersecurity problems," Mike Olson, Cloudera co-founder and chief strategy officer, told an audience at the Strata+Hadoop World show in New York. "This is a big deal, and could have a huge impact around the world."

Based on Cloudera's big data platform, Spot taps Apache Hadoop for infinite log management and data storage scale along with Apache Spark for machine learning and near real-time anomaly detection. The software can analyze billions of events in order to detect unknown and insider threats and provide new network visibility.

Essentially, it uses machine learning as a filter to separate bad traffic from benign and to characterize network traffic behavior. It also uses a process including context enrichment, noise filtering, whitelisting and heuristics to produce a shortlist of most likely security threats.

By providing common open data models for network, endpoint, and user, meanwhile, Spot makes it easier to integrate cross-application data for better enterprise visibility and new analytic functionality. Those open data models also make it easier for organizations to share analytics as new threats are discovered.

Other contributors to the project so far include eBay, Webroot, Jask, Cybraics, Cloudwick, and Endgame.

“The open source community is the perfect environment for Apache Spot to take a collective, peer-driven approach to fighting cybercrime,” said Ron Kasabian, vice president and general manager for Intel's Analytics and Artificial Intelligence Solutions Group. “The combined expertise of contributors will help further Apache Spot’s open data model vision and provide the grounds for collaboration on the world’s toughest and constantly evolving challenges in cybersecurity analytics.”

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Katherine Noyes

IDG News Service
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?