IBM develops new clustered analytics processing platform

The GPFS-SNC distributed computing architecture supports Posix

IBM today announced that it has created a new distributed computing architecture with a General Parallel File System technology that is twice as fast as existing clustered file systems and that provides management and advanced data-replication techniques.

Calling it the General Parallel File System-Shared Nothing Cluster (GPFS-SNC), IBM said the new architecture is designed to provide higher availability through advanced clustering technologies.

Prasenjit Sarkar, a master inventor in storage analytics and resiliency for IBM's research branch, said the system scales linearly, so that a file system with 40 nodes would have 12GB/sec. throughput, and a system with 400 nodes could achieve 120GB/sec. throughput.

"It's very cost-effective bandwidth. You get 1MB/sec. per dollar," Sarkar said. "If you try to replicate that with a [storage area network], it gets very costly."

The new architecture is aimed at enabling applications that support high-performance analytics, data warehousing applications and cloud computing, he said.

Sarkar described the GPFS's "shared nothing" cluster technology as each node or standard x86 server having access to its own metadata, cache, the data storage and management tools, while also having access to every other node in the cluster at the same time through Gigabit Ethernet ports.

"What we have done, in contrast to the Google file system, which has a single domain node, is we've distributed every aspect of the file system -- the metadata, the allocation, the lock management, the token management," he said. "Even if you take out a rack of servers [from the cluster], we'll still be able to continue to work."

By "sharing nothing," Sarkar said, new levels of availability, performance and scaling can be achieved with the clustered file system. Each node in the GPFS-SNC architecture is also self-sufficient. Tasks are divided up between these independent computers, and no one has to wait on another, Sarkar said.

The GPFS-SNC code also supports Posix, which enables a wide range of traditional applications to run on top of the file system, allowing both reads and writes to be performed.

"You can open a file, you can read a file, then you can append to the file and overwrite any section. With Google's Hadoop distributed file system, you cannot append to a file, you can't overwrite any sections, so you're very limited in what you can do," Sarkar said.

GPFS-SNC also supports the whole range of enterprise data storage features, such as snapshots, backup, archiving, information life-cycle management, data caching, WAN data replication, and management policies. The architecture has a single global domain namespace, allowing virtual machines to be moved between hypervisor nodes.

"So for example in our cluster, you can run Hadoop as well as a clustered DB2 or Oracle databases," Sarkar said. "This allows us to have a general-purpose file system that [can be used by] a wide range of users."

IBM would not say when the GPFS-SNC file system would make it out of the labs and into the marketplace, but Sarkar said that once it it's available, it will be targeted at three use cases: data warehousing, Hadoop MapReduce applications and cloud computing.

"The cloud may not be intuitive of a parallel architecture, but we have [many] virtual machines on each hypervisor node, and we have a lot of hypervisor nodes in parallel. Each virtual machine is accessing its own storage independently of every other virtual machine. So in effect you're getting a lot of parallel access to storage," Sarkar said.

IBM's current GPFS technology offering is the core technology for the company's high-performance computing systems, Information Archive, Scale-Out Network-Attached Storage (SONAS), and Smart Business Compute Cloud.

The GPFS-SNC technology's ability to run real-time Hadoop applications on a cluster won IBM a first-place award at the Supercomputing 2010 conference in New Orleans this week.

Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian or subscribe to Lucas's RSS feed. His e-mail address is lmearian@computerworld.com.

Join the PC World newsletter!

Error: Please check your email address.

Tags applicationsIBMsoftwarebusiness intelligence

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Lucas Mearian

Computerworld (US)
Show Comments

Essentials

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?