Google makes it easier for news sites to opt-out

Publishers can now use robots.txt to opt-out of both Google search and news

In what will be seen as a concession to media baron Rupert Murdoch, Google has made it easier for news sites -such as those Murdoch controls - to opt-out of Google News.

Where they used to have to fill out an online form to opt-out of Google's news aggregation site, publishers will soon have a means to opt-out or set other options automatically, using a small file placed on their sites.

Murdoch has previously threatened to take News Corp content, including the Australian and the Times of London, off Google when at some point in the future they become paid sites. His Wall Street Journal and Barron's are already largely subscription-based.

Murdoch told a US Federal Trade Commission hearing that "there is no such thing as free news" and reiterated his statement that News Corp sites would move to a paid model.

As for the aggregators, "these people are not investing in journalism," Murdoch said. "They're feeding off the hard-earned efforts and investments of others."

"To be impolite, it's theft," he added.

His remarks targeting Google prompted Huffington Post founder Arianna Huffington to respond that "aggregation is part of the Web's DNA" and that old media needs to "get real."

Murdoch has also reportedly been in talks with Microsoft that would result is News Corp content being removed from Google and enhanced on Microsoft's Bing, which would pay News Corp a fee in return for exclusivity. Recent reports, however, say the talks have been overplayed in the media.

Google, which also attended the FTC meeting, made its announcement in a blog post outlining extensions to the Robots Exclusion Protocol, already used to prevent Google and other search engines from indexing Web sites. The extensions will give publishers control over how their sites are treated by Google News.

"Now, with the news-specific crawler, if a publisher wants to opt out of Google News, they don't even have to contact us - they can put instructions just for user-agent Googlebot-News in the same robots.txt file they have today," wrote Google's Josh Cohen in the post.

Robots.txt is a small file that developers can place in the root directory of their Web sites that contain the Robots Exclusion Protocol commands.

"In addition, once this change is fully in place, it will allow publishers to do more than just allow/disallow access to Google News. They'll also be able to apply the full range of REP directives just to Google News. Want to block images from Google News, but not from Web Search? Go ahead. Want to include snippets in Google News, but not in Web Search? Feel free. All this will soon be possible with the same standard protocol that is REP," Cohen added.

It's not likely most users will notice any difference as a result of the change, unless a large number of publishers decide to abandon Google News and the estimated 1 billion clicks-a-month it generates for participating publishers (including PC World)

"Most people put their content on the web because they want it to be found, so very few choose to exclude their material from Google. But we respect publishers' wishes. If publishers don't want their websites to appear in web search results or in Google News, we want to give them easy ways to remove it. We're excited about this change and will start rolling it out today," Cohen said in concluding his post announcing the change.

Join the PC World newsletter!

Error: Please check your email address.

Tags Google NewsGooglenews corpmedia

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

David Coursey

PC World (US online)
Show Comments

Essentials

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?