Behind the scenes of Internet2

New network operations manager excited about Dynamic Circuit Network, multicast and IPv6
  • Bob Brown (Network World)
  • 31 January, 2008 12:16

You might think the network you oversee is big, but consider Chris Robb's new job: network operations manager for Internet2, which in October announced completion of a new research and education network boasting initial capacity of 100Gbps nationwide. Robb takes on his new position as an assigned staff member from the Global Research Network Operations Center (GRNOC) at Indiana University. Network World Editor Bob Brown interviewed Robb by e-mail to get an idea of what lies ahead for him and Internet2.

You'll be responsible for day-to-day operations of the nationwide Internet2 Network infrastructure. That sounds daunting. Give me a sense of what sorts of network gear you are actually responsible for and what you expect to be doing day to day.

The new Internet2 Network is really an evolution from Internet2's former Abilene IP network which was officially decommissioned in Fall 2007 when we completed the buildout of a completely different nationwide infrastructure in partnership with Level 3 Communications. The new network is what we call a 'hybrid.' Originally envisioned by the research and education advanced networking community, hybrid networks combine the best of IP and optical networking and allow users to dynamically set up circuits as their applications demand.

On the IP network, members connect via distinct optical wavelengths, deterministic SONET-based 'lightpaths,' or via the common-bearer IP service. We continue to use the Juniper T640 platform to provide IP services. They've provided rock solid performance with expansion room to help us support other IP-based services like our new Commodity Peering Service.

The optical network, which is built on the Infinera Digital Transport Chassis, is unique in its management model. Level 3 has responsibility for the day-to-day maintenance of the optical equipment (replacing cards, monitoring Layer 1 network health), but Internet2 has direct control over provisioning circuits across the network via our Network Operations Center. This allows the NOC to focus on our core mission of controlling and maintaining the network.

The Dynamic Circuit Network is still in its infancy. The overall idea is that a user can request an end-to-end circuit between two points on the network, and plumb a path that's unique to their project.

Because this is a new technical platform as well as a brand new way of operating a network, it has required some new thinking by our NOC in terms of monitoring and coordination. It's no longer the case that you can always just troubleshoot just your piece of the network cloud. Because each circuit can cross multiple network provider domains, much more metadata and coordination is required in resolving network issues from end to end.

Today, setting up large inter-domain circuits can be very labor intensive with coordination across multiple providers, across countries. One of our immediate goals is to implement an automated system that creates all the monitoring hooks that are needed for network operators to troubleshoot in minutes or less. It's a challenging task, but certainly one that has a lot of eyes on it right now.

Can you elaborate on the Dynamic Circuit Network and IP network?

The Dynamic Circuit Network is a circuit-based network cloud that is directly driven by our users. The closest analogy is the phone system. We provide the transport and the tools to make a "phone call" (i.e. create the dedicated circuit connection) without the day-to-day assistance of a network engineer. In the future, through the use of a new Web-based interface, users will be able to reserve their bandwidth for a set period of time, transfer their data using their protocol of choice, and then "hang-up" the circuit for the next researcher to use. It's a Ciena CoreDirector-based network, so the underlying transport is SONET. Our demarcation to domestic users is vanilla Ethernet, so it meshes well with most current campus networks.

Page Break

Simply plumb a VLAN to the edge of your network, where you meet the Dynamic Circuit Network, and initiate the circuit.

We're rolling this out in 2008 at no additional cost to our users for the next year, so we can seek input and feedback on the process. Because of the collaborative nature of the service, we can't do this in a vacuum, so we're actively seeking out collaborators and researchers that want to participate.

The IP network is really an augmentation of what was available three years ago. We've added a Commodity Peering Service as a separable feature. It's an opt-in mechanism whereby we peer with content and ISPs and our customers receive those routes over their high-speed circuits into the Internet2 network cloud. It's obviously a cost-savings mechanism for our members and enhances network performance for users, but we also see it as mechanism to help push some advanced services toward the commercial world. Most of our members have a well-built infrastructure that supports IPv4/IPv6 multicast and IPv6 unicast. We have a large user base that's of interest to content providers. We want to partner with commercial networks and collaborate on bringing those services into the light. If a large content provider were to start operating a multicast version of a video delivery technology that would be a big win for everyone. It saves bandwidth and uses the network in a much more efficient manner.

You'll also be shaping the priorities for the NOC. What will be say, your top three priorities out of the chute?

The first priority is to maintain the network in a manner that's consistent with our members' needs and expectations, which means supporting top quality service, reliability and performance at all times.

In addition, I plan to start focusing more resources on the operational issues that will be raised when you start handing off some control of the network to your users. The Dynamic Circuit Network in the future will allow users to auto-provision circuits from a Web-based interface, which in turn means we need to be ready to accommodate unanticipated changes to the network.

Also, since the dynamic optical paths will cross networks beyond the Internet2 Network, there are a lot of questions to be answered, like: Which network provider has ownership of troubleshooting a failed circuit? How do those failures become detected? And how do you start troubleshooting?

The beauty of running a research network is that we are pioneering these techniques with our collaborators worldwide and we believe these methodologies will help inform commercial Internet providers in the future as they adopt similar technologies and architectures.

Lastly, I hope to immediately begin re-evaluating the way we communicate information about the network to our membership base. When we started operating the network in 1998, none of the recent dynamic Web technologies or RSS feeds existed or were widely supported. Now that these technologies are more mature, we want to take a look at how we can more effectively publish information and how to make information easier, more targeted and more useful to our members.

What sort of technologies will you get to "play" with that you wouldn't on a more mainstream corporate or university network?

Multicast and IPv6 come to mind when comparing the research and education space to the corporate environment, as these are both technologies that are absolutely necessary for the scalability of a network. As IPv4 address space becomes depleted, and as users start utilizing more bandwidth-intensive applications from increasingly smaller and numerous devices, research and education network operators are in a good position to push deployment deeper into their campuses. The recent explosion in smart phone sales is a perfect example of the changing environment that research and education operators need to face at scales beyond what you would normally see on a corporate campus environment.

The Internet2 Dynamic Circuit Network we believe will be the next big disruptive technology both from a user-experience standpoint, and an operations standpoint. Creating dynamic network overlays is a relatively new area for the research and education community, but one that we need to support and continue to develop so researchers have the networking resources they require to keep innovating.

Page Break

How would you describe your end users/customers?

Varied. There are a lot of students that put a fairly consistent demand on the network. But there's a smaller subset of researchers that really challenge us to grow the network and keep adding services. Most of them are fairly network-savvy and understand the capabilities and limitations of a high-speed national data network. We seek to collaborate with these researchers not only on meeting their current demands, but on trying to explore and shape what will be needed in the future.

What are some of the cooler applications you've seen Internet2 used for of late?

The Dynamic Circuit Network is obviously very exciting because of the new directions it can take us. I'm very pleased to be involved in a project that can do so much for research. It's humbling to collaborate with researchers working on projects that will provide real benefit for future generations.

The Internet2 Network is expected to play an essential role for physicists to investigate the properties of dark matter and the origins of the universe when the Large Hadron Collider, a new particle accelerator currently under development at CERN in Switzerland, begins operating next year. The project will be massive - expected to produce roughly 15 million gigabytes of data annually for analysis by scientists around the globe. Over 70 Internet2 university members expect to participate, each with a need to download or transmit, about 2 terabytes of data over a four-hour window, every two weeks. By leveraging the "on demand" capabilities of the network, researchers should be able to provision the bandwidth needed for their allotted time.

We are seeing other interesting scientific applications by researchers in fields like radio astronomy who are experimenting with dynamic circuit networks to support breakthroughs. Specifically, there is a technique in radio astronomy called Very Long Baseline Interferometry [VLBI], which is considered one of the most powerful methodologies for high-resolution imaging of distant radio sources in the universe. VLBI has traditionally been done by physically shipping tapes or disk packs from the telescopes to a central correlator for processing, which is for obvious reasons, a labor- and time-intensive task. Internet2 members along with our global network partners are helping to make electronic transmission of VLBI data, or eVLBI, possible - this approach allows scientists to dynamically link telescopes around the world via dedicated optical circuits to the correlator.

In doing so, scientists can have immediate access to correlation results, even while experiments are in progress, which allows them to analyze the data immediately to make adjustments or changes in their approach to maximize their results. In doing so, eVLBI essentially creates a virtual radio telescope with a diameter nearly the size of the Earth. Astronomers are excited about the significant impact this type of technology will have on their work.

How much of your job involves network security and what is Internet2's approach to keeping the bad guys at bay?

Security is a 24/7 job. We partner with the REN-ISAC (Research and Education Network - Information Sharing and Analysis Center) to do distributed denial-of-service monitoring of netflow data. We provide both REN-ISAC and our members with mechanisms to proactively mitigate attacks without involving us, but provide support as needed. Campus networks are a prime target for botnet solicitation and other such exploits. The security of campus networks is best handled by the campuses themselves, but we do provide support where we can from a backbone perspective. Being an academic network, we don't want to make any presumptions about how open or closed campus networks want to be. Our job is to secure the network infrastructure itself.

Anything else come to mind?

Another exciting new aspect of the Internet2 Network is our ability to run multiple networks on the same infrastructure through the use of distinct lambdas or virtual circuits through the Dynamic Circuit Network. In building the network in this manner, we are able to provide dedicated facilities to network researchers who are pushing the envelope on the development of brand new protocols and architectures for networking --- this could include a researcher at a university, or a corporate start-up that needs a short-term nationwide wave footprint to test a new concept in networking that a commercial carrier might not want to sell them. Providing testbed facilities on a separate wavelength allows researchers to essentially break the network without harming the production traffic. We look forward to working with network researchers on exciting new network research projects such as the National Science Foundation's GENI project.