Eye tracking within Windows 10's Fall Creators Update is an assistive technology with potential

Microsoft’s Eye Control for the Windows 10 Fall Creators Update offers up a noble goal: controlling your PC with your eyes, using a feature called Eye Control in the Windows 10 Fall Creators Update. And it works, mostly.

Eye tracking has quietly emerged as an alternative to the mouse, led by companies like Tobii. The concept is simple enough: Sensors within an eye tracker bar you mount on your monitor “watch” your gaze, guessing pretty accurately what you’re looking at. Though Tobii sees eye tracking as a tool for productivity and gaming, Microsoft currently sees it as a purely assistive feature, rather than a productivity tool.

We’ve reviewed two generations of eye-tracking hardware from Tobii—Eye Control was specifically designed for its latest eye tracker, the Tobii 4C—which Tobii positioned first as a productivity assistant, then as a gaming peripheral. However, Microsoft designed Eye Control as a virtual replacement for both a traditional mouse and keyboard, specifically for those with ALS (Amyotrophic Lateral Sclerosis) or other degenerative diseases.

tobii 4c eye tracker — To enable eye tracking currently, you need to own a Tobii 4C, which mounts beneath a desktop monitor. In real life, the LED light output is dialed down considerably.

Eye Control setup is a snap

Besides the traditional mouse, keyboard, and PC, Windows 10’s Fall Creators Update doesn’t require any specialized hardware. Eye Control does. You’ll need a Tobii Eye Control 4C to enable it. Eventually, Eye Control will support the Tobii Dynavox PCEye Mini, PCEyePlus, EyeMobile Plus, and I-series, according to Microsoft. (Tobii hardware is also being built into some gaming notebooks like the MSI GT72S G Tobii-805, eliminating the need for a mounting strip.) In addition to tracking your eyes, an eye tracker also serves as a way to enable Windows Hello on PCs that don’t already support it, like desktops.

Essentially, the 4C works the same way as the earlier Tobii EyeX: You mount the long, wand-shaped eye tracker on your monitor, plug it into an available USB port for power, and complete the short setup process.

tobii eye tracker options — Tobii’s eye tracker can quickly move your cursor around your screen, or alternatively can be used to switch between apps or virtual desktops.

It’s worth noting that Tobii has its own ideas about how eye tracking could be used within Windows. Tobii’s own native software can be configured so that your mouse “teleports” to the area to the screen that you’re gazing at, triggered by tapping a certain key or simply twitching your mouse. Tobii’s software can optionally be configured so that the cursor also “clicks” the portion of the screen that it lands on. In general, though, Tobii sees eye tracking as a supplement to your mouse, not as a replacement.

Windows, though, goes a step further. Once the eye tracker is installed, you can go into Settings > Ease of Access > Other Options and enable the beta eye-tracking feature. (Microsoft states that it doesn’t save or share eye-tracking data, incidentally.) Windows doesn’t require any training or configuration—toggle it on, and eye tracking immediately begins.

eye control setting eye control mouse Microsoft Windows 10 Fall Creators Update — Eye tracking is buried in Microsoft’s ease-of-use assistive settings.

Once Eye Control is turned on, a small box appears at the top of your screen, with four icons. The first, a pair of up and down arrows, simply relocates this navigational box to the top or bottom of your screen. The others—a keyboard, a dialog box, and some sort of a mouse/directional pointer—represent the Eye Control functions: a traditional keyboard, “shape writing,” and the mouse functionality.

eye tracking overlay Microsoft Windows 10 Fall Creators Update — Enable eye tracking within Windows, and you’ll see this overlay.

Early problems, now resolved

When I first tried out Eye Control, the app failed, and failed hard. Tobii’s own “Gaze Trace” tool places a small bubble to represent your gaze as it moves about the screen. If you let your gaze linger on the “mouse” icon on the Windows overlay, that function will launch. Windows, too, uses a circular target as a mouse “cursor” of sorts.

Originally, when compared against Microsoft’s own Eye Control “mouse,” I could measure the discrepancy between the two in inches. As it happened, a bug didn’t account for my multi-monitor setup and the differences in resolution. That, plus a driver update, resolved my issues.

tobii vs windows 2 larger — The differences in eye tracking between Windows and Tobii’s own software prompted me to check in with Microsoft. Fortunately, this does *not* represent the current

(As our earlier testing shows, the size, shape, and color of your eye matters significantly; Tobii’s eye tracker simply didn’t interpret narrow eyes as accurately as wide-eyed people, and glasses affected accuracy, too. As a blue-eyed white male, however, my eyes were among the most accurately tracked of the ten different users we picked.)

As your gaze drifts across the screen in mouse mode, letting it linger over a spot will bring up the “target” overlay for the mouse, below. Here, you can fine-tune your spot, then “click” using one of the buttons. Once you’ve “clicked,” you’ll need to bring your gaze up to the mouse icon again to once again enter mouse mode.

eye control mouse Microsoft Windows 10 Fall Creators Update — If your gaze lingers on any one spot for a second or two, the Windows’ assistive eye-tracking mouse function will pop up this overlay, which you can “fine tune” by clicking the arrow keys, then the mouse. Once an action has been taken, you’ll need to click the mouse or keyboard icon to begin again.

This is how Eye Control’s keyboard works

Text entry is performed via the keyboard icon, again within the overlay. Within the keyboard, “hovering” your gaze over a letter enters it on the screen, while a spacebar, Function, Control and a backspace key help you navigate about the screen. It’s a laborious process.

eye tracker eye control Microsoft Windows 10 Fall Creators Update — Microsoft’s assistive, eye-tracking keyboard.

What Microsoft has referred to as “shape writing” attempts to use prediction to facilitate the process, especially in a scenario where the user is using eye tracking to communicate in real time. Like a Windows phone keyboard, shape writing allows users to let their gaze drift over the letters of a word, only requiring them to linger or “dwell” on the first and last letters. Microsoft’s predictive technology then tries to guess the word. Here, Microsoft’s assistive technology is used to enter words into a text field that can be “played” or read aloud. Users also have the option to pick from a few preconfigured words (such as “hello,” or “please”) at the top of the screen.

eye control text to speech eye control mouse Microsoft Windows 10 Fall Creators Update — Microsoft’s “shape writing” keyboard uses predictive technologies and text to speech to facilitate communication.

What’s the future of eye tracking within Windows?

In promoting Microsoft CEO Satya Nadella’s new book, Hit Refresh, Nadella and his wife revealed that they have two special-needs children, one with cerebral palsy. Over the past few years, Microsoft has been instrumental in adding and promoting new modalities for interacting with PCs: touch, pen, voice, and, now, gaze. Though Microsoft designs its products for billions of consumers worldwide, Nadella’s public disclosures paint Microsoft’s admirable work in assistive technologies in a new light.

I’m a fan of these additional forms of input: I ask Cortana to remind me of things I need to do, and I doodle notes or drawings with my Surface stylus when I need to. Eye tracking could be a real boon to individuals with any level of ability. It’s unclear, though, what the role of eye tracking may be within Windows: assistive technology or productivity aid? Time will tell, apparently.