Usenix: Dartmouth expanding diff, grep Unix tools

Dartmouth researchers are working on variants of diff and grep that can parse more complex data structures

With some funding from Google and the U.S. Energy Department, a pair of computer scientists at Dartmouth University are updating the venerable grep and diff Unix command line utilities to handle more complex types of data.

Such updates are needed because "we now tend to have more model-based configuration languages that have meaningful constructs spanning more than one line," said Gabriel Weaver, a Dartmouth graduate student who, along with Dartmouth computer science professor Sean Smith, is creating the variants of grep and diff. Weaver presented the new utilities at a poster session at the Usenix Large Installation System Administration (LISA) conference, being held this week in Boston.

The new programs will allow administrators to extract meaningful data from configuration files, log files and other sources of operational data, the researchers maintain.

Grep and diff are command line-based text analysis tools available in all Linux and Unix distributions. Both are designed to parse documents on a line-by-line basis. Grep offers the ability to search through multiple text files and folders for a specific chunk of text or regular expression. Diff compares two documents and highlights the differences between them.

As with most Unix utilities, the output from either of these programs can be linked, or piped, to other utilities, so they can be incorporated into scripts that automate routine system administration tasks.

The new programs, called Context-Free Grep and Hierarchical Diff, will provide the ability to parse blocks of data rather than single lines. For each new type of data structure, a vendor would provide a pattern library identifying the basic structure of the data, which the software would then use to "extract the constructs of interest from the document," Weaver said.

Such utilities could provide administrators the ability to work with more complex forms of data now being generated by network equipment and infrastructure software. For instance, Cisco's IOS (Internetwork Operating System), which is the company's operating system for its routers and switches, will provide operational data in block-like data structures.

With this data, a tool such as diff "can be too low-level," Weaver said. "Diff doesn't really pay attention to the structure of the language you are trying to tell differences between." He has seen cases where dif reports that 10 changes have been made to a file, when in fact only two changes have been made, and the remaining data has simply been shifted around.

Grep has issues with data blocks as well. "With regular expressions, you don't really have the ability to extract things that are nested arbitrarily deep," Weaver said.

Context-Free Grep is still in the design stage, but should be completed within the next few months. A prototype of Hierarchical Diff has been completed, though the researchers have not posted the code yet.

Google's interest in this technology springs from the company's efforts in cloud computing, where it must automate operations across a wide range of networking gear, Weaver said. The DOE foresees that this sort of software could play a vital role in smart grids, in which millions of energy consuming end-devices would have connectivity of some sort. The software would help "make sense of all the log files and the configurations of the power control networks," Weaver said.

In addition to system administration duties, the utilities could also be used in with non-technical languages as well. They could be used to parse legal documents, for instance, Weaver suggested.

A number of Usenix attendees praised the idea for its potential usefulness. "You wonder why it hasn't been done before," one said. Another commented that such tools could also be really handy for code repositories such as Git.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is

Join the PC World newsletter!

Error: Please check your email address.

Tags unixopen sourceU.S. Energy DepartmentLinuxUtilitiessoftwarenon-Windowsoperating systemssystem managementDartmouth UniversityGoogle

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

D-Link PowerLine AV2 2000 Gigabit Network Kit

Learn more >

D-Link TAIPAN AC3200 Ultra Wi-Fi Modem Router (DSL-4320L)

Learn more >

ASUS ROG Swift PG279Q – Reign beyond virtual world

Learn more >

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Crucial® BX200 SATA 2.5” 7mm (with 9.5mm adapter) Internal Solid State Drive

Learn more >

Gadgets & Things


Learn more >

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Lexar Professional 2000x SDHC™/SDXC™ UHS-II cards

Learn more >

Family Friendly

Lexar Professional 2000x SDHC™/SDXC™ UHS-II cards

Learn more >

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

ASUS VivoPC VM62 - Incredibly Powerful, Unbelievably Small

Learn more >

Stocking Stuffer

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Lexar Professional 2000x SDHC™/SDXC™ UHS-II cards

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Best Deals on PC World

Latest News Articles


GGG Evaluation Team

Kathy Cassidy


First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni


For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Steph Mundell


The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.

Andrew Mitsi


The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.

Simon Harriott


My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.


Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?