Online dating site eHarmony.com continues to amass scores of data to fuel its online relationship service for 15 million registered users. The company 18 months ago deployed clustered storage systems to handle the photos, images, logs and other content types it has added to the online profiles it stores for its users. In an interview this week at the Computerworld Storage Networking World conference in San Diego, Mark Douglas, vice-president of technology at eHarmony, talked about the company's use of storage clusters, how he chooses storage vendors, why the company is phasing out tape backup, and about new projects planned to keep online dating rolling along.
What makes up eHarmony's IT storage environment?
We use two key vendors. For storage arrays, we use 3PAR Data. We have Oracle database servers fiber-attached directly to those arrays for all TP, database access and data warehousing. For everything else and for storage access, we have clustered ONStor NAS gateways connected to 3PAR gateways. They provide access to everything on our farm and our two data centers. Everything is physically on 3PAR. Everything except the database is accessed via the OnStor NAS arrays -- photos, profiles, anything being stored.
Why did you choose ONStor for storage clustering?
Because we could use a gateway approach and we didn't have to put in another underlying storage technology. More important is reliability. We've been running them about 1.5 years and we've had zero seconds of downtime. Reason two is because they're clustered. Even for planned downtime, we can do software upgrades where we take down the nodes and cluster and upgrade it. Also, the performance is very good. At any given point in time, there's a significant number of people on our Web site. Given what we do, they're all looking at photos and reading profiles at the same time. The ONStor gateways are getting heavy usage but we haven't had to dedicate people to manage them.
How important is storage clustering for your uptime and redundancy?
Clustering is everything to keeping a site like this, or any 24x7 Web sites, [going]. It's primarily a job of removing every single possible point of failure. Even our monitoring systems have to be redundant. If the thing that is monitoring things fails, it all could fail. It involves a lot of diligence and you have to be extremely thorough. You've uncovered every point of failure and have it clustered to prevent that. In every web company, something is always broken but the user never sees it because they have redundant parts running.
How have you been able to balance the need for upgrades versus keeping costs in line at a company that's seen amazing growth since you joined in 2004?
In April of 2005 we had one terabyte of storage. In April 2007, only 24 months later, we have 100TB. Keeping up with that growth is a constant challenge. What we don't want to do is keep changing vendors and changing technologies. It's important to make sure everything you're buying can scale horizontally, meaning it can all be clustered. We have continued growth but we've haven't had to add people because we're leveraging these technologies. We have 100TB of storage and we have zero people managing that -- the storage arrays are basically self-managing.
Couldn't you have accomplished the same thing using a storage area network?
We don't run a SAN, so we don't have to worry about a whole bunch of storage management tasks that most organizations face. If you're running an EMC or Network Appliance array, there's a whole storage layout that has to be determined. You probably have 100 drives and [have to ask], 'where am I going to lay out my data?' 3PAR does that automatically. When we bought these technologies two years ago, we didn't have anyone with 'storage' in their title - we still don't. We just have a database team.
What led you to adopt a storage cluster configuration?
Before [clustering, our environment] was all [Hewlett-Packard] MSA1000 direct-attached storage arrays connected to the database servers. We outgrew them when we couldn't physically put any more drives on the box. That drove us to look for more enterprise class technology. The prime motivator [for clustering] was that we wanted leverage. We didn't want to end up with a whole big 'storage effort,' and didn't want to introduce more points of failure. It's not cheap but it's cost-effective. One thing I have observed in other companies is they wind up with storage all over the organization. We've been able to centralize it. All our storage comes form our shared arrays and gateways.
Why didn't you turn to the largest storage vendors 18 months ago to address your storage concerns?
We evaluated those vendors, in particular EMC. There was just too much cost and effort involved in managing those technologies. On top of that, they're not the highest performance and highest reliability. Even on an EMC array, software upgrades generally [require] planned downtime. It just didn't add up. The only reason to justify buying from established vendors is basically their names. That's just not enough for us. We didn't have a vested interest in buying EMC because we didn't own [their products]. We didn't ignore the business risk. We just looked at the purely technical [needs] and so far it has paid off for us."
Why are you abandoning tape in favor of online backup?
We're starting to use de-duplication gateways and eliminating tape all the way around. Instead of using three months of tape for backup, [for the same cost] I can put the same amount of data online for a year. Tape is not considered reliable. By improving reliability, you're enabling faster access and now you can keep backups for a much longer time and [cut] costs.
What type of new storage projects is eHarmony currently working on?
In terms of features and functionality, there are always different things we're trying on the site. We [are developing] a product to help married couples share their relationship. Both parties take a questionnaire and each can tell how the other person feels. Also, we have 15 million registered users. That generates a huge amount of data that is useful for everyone in the company to [use to] improve our business.
What advice do you have for companies looking to better control their storage environments?
Do more with less. The biggest point of failure is mistakes -- literally human error. When you can get leverage from a technology and you don't have to touch it as much, you don't have to break it as much. My advice for anyone in the storage arena is you really should be constantly looking for new and emerging technologies that are going to bring reliability, cost effectiveness, and performance that you need.