When Consumer Reports stopped recommending the Microsoft Surface Laptop and latest Microsoft Surface Book, it did so based not on those specific products’ reliability. Rather, the company used reliability surveys of earlier Surface devices (including Surface 3, Surface Pro 3, and the early Surface Book), according to a report the publication released Thursday.
Microsoft apparently privately confirmed this as well, via an internal memo obtained by Thurrott.com this week, which shows that while return rates of products like the Surface Pro 3 and Surface Book were initially atrocious, they improved substantially over time. Later Surfaces, like the Surface Hub, also never really suffered from the same issues.
In addition to removing its “recommended” rating from four Microsoft Surface devices, Consumer Reports stated it wouldn’t recommend any Surface products at all. Devices receiving the no-confidence vote include the Microsoft Surface Laptop, with 128GB and 256GB of storage, and the Microsoft Surface Book, with 128GB and 512GB of storage. Consumer Reports also predicted that 25 percent of all Surface laptops and tablets will generate problems within the first two years of ownership.
Though it named four specific products, a Consumer Reports spokesman said it was penalizing the Surface brand as a whole. “We can’t recommend any of [Microsoft’s] laptops or tablets until we get data that shows better reliability,” a Consumer Reports spokesman said in an email.
Microsoft, for its part, said that the company respectfully disagreed with Consumer Reports’ findings, and that its real-world return and support rates are on par with competing devices and “significantly lower than 25 percent,” Microsoft Devices corporate vice president Panos Panay wrote.
Dan Masaoka Consumer Reports pulled its recommendation of the Surface Laptop, even though the notebook wasn’t covered in the survey.
Why this matters: Consumer Reports extrapolated reliability reports from products like the Surface 3, Surface Pro 3, and early Surface Book—all of which suffered from some hardware flaws—to unrelated products like the Surface Laptop, which wasn’t even in production during the time of the survey.
The question is whether, in an analogous situation, a Chevrolet Malibu sedan that had suffered repeated manufacturing defects would cause the publication to stop recommending the Chevrolet Silverado pickup truck. While a specific warning about the Surface Book might be defensible, claiming that a quarter of Book and Laptop buyers will experience problems seems like a leap of logic.
Can past Surface performance predict the future?
Consumer Reports, which bases its reliability reports on surveys that it sends to its subscribers, compiled data on 90,741 tablets and laptops that its members bought from 2014 until the beginning of 2017, it said. That data included devices from a variety of manufacturers, including Microsoft Surface devices.
The reliability predictions represent extrapolations made from data reported on older devices, such as the Surface 3, Surface Pro 3, and other devices sold during that time, the CR report said. Actual problems that consumers experienced included freezing or unexpected shutdowns, and touchscreens that weren’t responsive enough, the publication said. Using that past data, Consumer Reports then generated a reliability rating for the brand as a whole.
Martyn Williams The Microsoft Surface Pro 3 tablet suffered some early hardware problems.
“Essentially, our analysis isn’t of the Surface laptops or any specific model line or family,” a Consumer Reports spokesman explained in an email. “Rather we conduct annual laptops and tablets brand reliability surveys. We are not making claims about a specific laptop or tablet model released at a specific point in time. We are making reliability predictions at the brand level, based on the devices purchased between 2014 and the first quarter of 2017.”
The spokesman acknowledged the problem with using past data on a new product that wasn’t even shipping when the survey took place. “It’s not 100 percent perfect,” the spokesman said of the publication’s methodology.
Still, the spokesman added, it represents the best data the company has on the brand as a whole. “Our analysis revealed that Microsoft as a brand has a comparatively worse reliability rate than other brands, so we can’t recommend Microsoft tablets and computers – no matter the model family,” the Consumer Reports spokesman said. “This wouldn’t necessarily change if Microsoft came out with a brand new tablet or computer next week.”
Consumer Reports’ findings surprised many at Microsoft and elsewhere, which had seen steadily increasing reliability metrics from generation to generation. Surface Pro 3 owners had struggled through battery issues and overheating, while the Surface Pro 4 and Surface Book suffered screen flickering and their own power problems. But by 2017, the Surface Book had become fairly reliable.
Given that the Surface Laptop began shipping in June, after the survey concluded, many wondered how Consumer Reports could reach its conclusions. “I own and have used every version of Surface products for extensive periods of times, not just reviews,” Patrick Moorhead, principal analyst at Moor Insights, said in an email. Moorhead’s experiences, while anecdotal, are far more positive. “Since 2012 I have only had an issue with two systems: software issues with Windows RT systems in 2012 and in 2016 with the first batch of Skylake-based systems. After six months, the Skylake driver issues disappeared.”
Moorhead’s largely trouble-free experience extends to Microsoft’s latest models. “I have both the Surface Pro (5) and Surface laptop and have not experienced an issue with stock (non-beta) software.”
Blair Hanley Frank Panos Panay, Microsoft’s Corporate Vice President of Surface Computing, demonstrates the new Surface Book at the company’s press event in New York City on October 26, 2016.
Consumer Reports has been here before
Consumer Reports is no stranger to controversy. The publication made headlines earlier this year when it said it uncovered issues with the Apple MacBook Pro’s battery life. Its testing used a developer setting that included a undiscovered bug, however, which affected its results. Those tests prompted Consumer Reports to withhold a buying recommendation, though CR later reinstated the recommendation after the bug was resolved.
Microsoft has its own side of the story. The company maintains an internal testing facility and generates its own estimates of how its Surface devices compare to its competition—which is basically Apple. Its internal defect rate was “not even close” to 25 percent, according to Microsoft employees. Surface Computing vice president Panos Panay wrote that the “incidents per unit” were less than 1 percent, or presumably less than 1 out of 100 devices.
Consumer Reports noted that it ranked the new Surface Pro very highly in recent Consumer Reports tests, including display quality, battery life, speed, and ergonomics. (PCWorld’s Surface Pro review nicked the tablet for lack of innovation, while our Surface Book review was more positive). The CR spokesman also acknowledged in an email that models within a brand may vary, and that changes in design and manufacture might affect future reliability. By then, though, the damage had been done.
Updated at 10:48 AM on August 14 with additional Microsoft return data, as provided to Thurrott.com.