I always look forward to reading Microsoft-funded research, because over the years it's evolved into a kind of game. The results – of course – are always amazingly good for Microsoft, but hidden away in there, like a secret at the heart of a complex puzzle, there's something that we're not supposed to notice that undermines the final result.
So I was delighted to come across this post on the Internet Explorer UK blog:
Fantastic news just in at Internet Explorer HQ! We are pleased to announce that, for the third quarter running in 2010, Internet Explorer has continued to be the number one browser to protect users against malware attacks. Today, NSS Labs has released its Q3 2010 web browser study that shows our investment in blocking malware is helping protect millions of people.
According to the latest NSS Labs report, Internet Explorer is the industry leader in protecting users against malware and specifically, IE9 is able to block or warn against 99% of socially-engineered malware. But what exactly does this mean? Well put simply, Microsoft blocks almost five times more malware threats than Firefox, nine times more malware threats than Safari and 33 times more malware threats than Chrome. Internet Explorer has blocked over 1.2 billion malware and fishing attacks, further cementing it as the leading browser in the fight against malware attacks. In addition, Internet Explorer 8 is the only browser to achieve an improved score compared to earlier reports.
Well, of course, my curiosity is piqued by this kind of result, so I clicked on the link in that post, which took me here, which gave me another link which took me here, which finally led me to this link [.pdf], which produced the report in question (couldn't they just give the link directly?)
The document explains what exactly is being tested:
Modern web browsers offer an added layer of protection against these threats by leveraging in-the-cloud, reputation-based mechanisms to warn users. This report examines the ability of six different web browsers to protect users from socially-engineered malware. Each of the web browsers has added security technologies to combat web-based threats. However, not all of them have taken the same approach, nor claim to stop the same breadth of attacks.
Browser protection contains two main functional components. The foundation is an "in-the-cloud" reputation-based system which scours the Internet for malicious websites and categorizes content accordingly; either by adding it to a black or white list, or assigning a score (depending on the vendor's approach). This categorization may be performed manually, automatically, or using both methods. The second functional component resides within the web browser and requests reputation
information from the in-the-cloud systems about specific URLs and then enforces warning and blocking functions.
So, in other words, what this study really tests is two things: how good that "in-the-cloud" reputation-based system is, and how well a browser uses that system. This means that the relatively poor results of non-Microsoft browsers claimed in the study are down to a combination of both of those factors. This is not an aspect of security that I'm particularly familiar with, so I leave it to others to analyse in greater depth. It would also be interesting to hear directly from the Mozilla and Google teams concerned about the low scores of both Firefox and Chrome in combination with Google's Safe Browsing feed.
But it seems to me that there is a more important point here that this focus rather ignores. Maybe Internet Explorer's system for spotting malware through reputation-based systems is indeed better than its rivals, but let's consider a minute what exactly that bad stuff is. Here's what the report says:
For clarity, the following definition is used for a socially-engineered malware URL: a web page link that directly leads to a download that delivers a malicious payload whose content type would lead to execution, or more generally a website known to host malware links. These downloads appear to be safe, like those for a screen saver application, video codec upgrade, etc., and are designed to fool the user into taking action. Security professionals also refer to these threats as "consensual" or "dangerous" downloads.
This makes it clear that we are talking about code that is downloaded and then executed. According to the report, all the tests were carried out on a Windows 7 system. So in other words, we are talking about Windows malware. The undoubtedly thorough tests in the present report simply underline the huge scale of the Windows malware problem, and hints at the considerable costs it imposes on users, businesses and the economy as a result. What emerges from this test, then, is that Internet Explorer is better at solving problems of Microsoft's own making than third parties without direct access to the Windows code and its flaws.
Frankly, I would expect no less: it is Microsoft's responsibility to sort out these weaknesses in its own software, and if it produced a browser that exacerbated the problem it would be doubly culpable. But for a really fair test, what we would need to see would be Firefox running on a GNU/Linux system, Safari running on a Mac box and Chrome on ChromeOS, and then to compare those systems with Microsoft's own combo of Internet Explorer and Windows. I'm pretty sure that Internet Explorer would not emerge as such a star in these circumstances.
But failing that kind of comparison, what the report's test shows is quite simple: that irrespective of which browser you use, you really shouldn't be running Windows at all if you want to minimise your exposure to malware.