How to completely misuse a report

28 July 2005

One of the more unseemly sides of the Windows versus Linux war is the use of research to bolster the position of each side. We get releases describing how report X shows up the weaknesses of Windows and report Y that shows how expensive Linux to run even though its core code is ostensibly free to use. Some of the claims even match up to what the original report said. Not the latest claim from Red Hat.

Red Hat claimed yesterday the SANS Institute had published a report that said only two of the top 20 defects listed by the researchers affected its operating system. Because of that, the company claimed: "Linux network security [is] higher than other platforms". I had to check with Red Hat which report the company was using to back up its claims, because I couldn't find anything out of SANS that came close to the claim made in the press release. Even after finding out, making the connection wasn't much easier.

The release apparently referred to the Q2 update of SANS' Top 20 Internet security vulnerabilities. This is where Red Hat's claims begin to fall down. It turns out that, according to the SANS criteria for the Top 20, Linux bugs could not account for more than 50 per cent of vulnerabilities in the report in the first place. This is not because the SANS Institute is fall of Microsoft-hating zealots but because the Top 20 was never meant to be used as a way of counting up who has the most bad bugs in their code. It is just meant to provide advice to sysadmins who want to know which holes they should plug first.

As SANS points out: "This SANS Top-20 2004 is actually two Top Ten lists: the ten most commonly exploited vulnerable services in Windows and the ten most commonly exploited elements in UNIX and Linux environments. Although there are thousands of security incidents each year affecting these operating systems, the overwhelming majority of successful attacks target one or more of these twenty vulnerable services."

Now, you can argue that Windows gets special treatment because it has more bugs than bin full of three-week-old raw meat. But this is not the report to use to make that point. You will just look either a bit stupid or just be treated as trying to make out everybody else is stupid. To a large degree, the Top 20 does not name and shame unpatched flaws but the things that tend to exhibit problems and which tend to get attacked. The Q1 and the Q2 updates issued this year did cite specific faults, but they don't add up to 20 and they also have the Windows and Other Platforms split. The Q2 document contained six named flaws in the former category and eight in the latter. Some of those affect Linux. So, you might get to the "two bugs that have been patched" claim from Red Hat.

The company might as well have claimed Linux is more secure "because my dad said so". I look forward to the next installment when someone uses the prophecies of Nostradamus to show that Windows has fewer bugs per thousand lines of code than Linux.