Aiming for SEC's big data project, Sungard and Google bet on the cloud

Sungard is using Google cloud services to prototype a system that stores and formats all U.S. financial trading data

At the Google Next roadshow in New York, held on June 12, the company extolled the virtues of cloud computing.

At the Google Next roadshow in New York, held on June 12, the company extolled the virtues of cloud computing.

To bid for a massive U.S. government contract, Sungard has teamed up with Google to build a prototype cloud system that could store six years' worth of U.S. stock and financial trading data and let regulators and stock traders scrutinize this mountain of information.

The work is being done to compete for a U.S. Securities and Exchange Commission contract, called the Consolidate Auditing Trail (CAT). The SEC's goal is to build a system that provides more transparency into financial markets, a response, in part, to the computer-driven 2010 "flash crash" that briefly cratered U.S. stock prices.

"The CAT is a huge undertaking," said Neil Palmer, Sungard's chief technology officer for its consulting services practice. "It is the biggest big data problem in the financial industry today."

Palmer described the prototype Friday at the Google Next user conference in New York. Sungard, a provider of financial software and services, is one of six finalists for the work, and has partnered with Google for technology infrastructure.

The flexibility of cloud computing provides the ability for Sungard to pursue such an ambitious job, Palmer told a group of reporters after the keynote.

With building a system in-house, "there are just too many unknowns," he said, referring to the intense hardware and operational demands that would come with building an on-premises system to execute this work.

The system will cost anywhere from $350 million to $1 billion to build, the SEC has estimated.

Once operational, CAT will generate a tremendous amount of data, Palmer said. The system must record every quote and every trade from every financial company participating in the public U.S. markets. The companies must submit their data on a daily basis, and the system must keep this data for six years.

Each day, the system will ingest about 50 terabytes of data, comprised of about 100 billion events. The six year window of when records will be actively kept will amount to about 30 petabytes of data, Sungard estimated.

All this data must be validated, indexed, and posted within four hours.

Tools must also be available to query all this data. "There is no point storing that much data and not being able to generate any actionable information from it," Palmer said.

In addition to the SEC, cloud accessible financial data could also be of immense value to the financial firms themselves, Palmer noted. A comprehensive centralized copy of all financial trading information would reduce the needs for such firms to store that data in house. They could test algorithms on the history market data, to see how will they can predict upcoming changes.

Sungard assembled the infrastructure for the prototype using a variety of Google Cloud Platform components.

The prototype uses Google Cloud Storage to hold the data, and Google BigTable to structure the information. A Google Dataflow service can validate the data. Google's Big Query can be used to publish the data and provide a way for users to analyze it, either directly or through third-part software such as Microsoft Excel or various business intelligence (BI) packages.

In the first round of tests, the prototype was able to process 10 billion events an hour, or about three gigabytes of data process a second.

Best of all, the prototype was built in six weeks.

"There is no way we could have done that, even with similar technologies, if we had to stand up our own infrastructure," Palmer said.

The team still has work to do, Palmer said. The system must be able to ingest data at four times the speed of the current prototype.

If Sungard wins the contract, or if another team with a cloud-based approach wins, it will represent a significant step towards the acceptance of the cloud computing model in the U.S. financial industry, which to date has been reluctant to embrace the approach.

Sungard's testimony was thematically in tune with the overall theme of the keynote, which was how cloud computing can give companies resources to forge into new markets that they otherwise wouldn't have had access to.

Carl Schachter, Google vice president of the cloud platform, said that companies like Uber and AirBnB have used cloud computing to disrupt traditional markets. "Markets that have been previously thought established are now re-inventable," he said.

The other finalists for the CAT contract are Epam systems; Thesys; the Financial Industry Regulatory Authority (FINRA); a team comprised of AxiomSL and Computer Sciences Corporation; and a consortium of companies including Hewlett Packard and Booz Allen.

The SEC has not offered a date when it expects to award the contract.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the PC World newsletter!

Error: Please check your email address.

Tags Googlesungardcloud computinginternetInfrastructure services

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

Crucial Ballistix Elite 32GB Kit (4 x 8GB) DDR4-3000 UDIMM

Learn more >

Gadgets & Things

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Family Friendly

Lexar® JumpDrive® S57 USB 3.0 flash drive 

Learn more >

Stocking Stuffer

Plox Star Wars Death Star Levitating Bluetooth Speaker

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

GGG Evaluation Team

Kathy Cassidy

STYLISTIC Q702

First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni

STYLISTIC Q572

For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Steph Mundell

LIFEBOOK UH574

The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.

Andrew Mitsi

STYLISTIC Q702

The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.

Simon Harriott

STYLISTIC Q702

My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?