ESET Threat Blog

The anti-malware industry isn't a suitable environment for the thin-skinned. We get used to receiving "more kicks than ha'pence" (see http://www.virusbtn.com/spambulletin/archive/2006/11/vb200611-OK)..

In particular, I've grown accustomed to the fact that many people expect all the following from an AV product:

• Absolute Protection
• Absolute Convenience
• Absolutely no  False Positives
• Absolutely no charge

False positives (FPs) are not a minor issue: my experience is that many people (especially in corporate environments) are more infuriated by an FP than by a false negative (i.e. where security software fails to detect real malware).

So it was a pleasant surprise to come across a blog from a source not usually associated with fulsome praise of the AV industry that was actually rather positive, in a backhanded sort of way. The author states "…while I rant quite a bit about the AV industry, you have to give this one to them: the number of false positives is really low. For example, in the AV-Comparatives test 20 false positives is considered many, even though the collection is over 1 500 000 samples (so the acceptable FP rate is below 0.0015%!)." [Sadly, this calculation doesn't altogether make sense, either to us or to AV-Comparatives. In fact, it seems to be based on comparing the number of "acceptable" false positives to the AV-Comparatives malware test set. It isn't based on the AV-Comparatives clean set: AV-C don't publish that information, as they consider FP ratios in percentages to be potentially misleading.]

The blog isn't actually about us, I should make clear: it's about the importance of false positives in intrusion detection, an area with just enough methodology and terminology shared with anti-malware to be confusing – for instance, both technologies talk about FPs and signatures, but often mean something slightly different by them. This led me to another blog that makes a useful distinction between "real" false positives – misdetection of innocent objects as being malicious – and "noise" such as "inappropriate traffic that is legitimately detected that we just don't care about." Actually, because modern anti-malware has a much wider remit than traditional virus detection, that is a concern we share with IDS and IPS vendors – of course, there's an overlap – most AV products now include some IPS (Intrusion Protection System) functionality. But it also has an analogue in more traditional anti-malware detection, which tends to cover "noise" programs such as adware and "possibly unwanted" code as well as out-and-out malware. 

Then there's the miscellaneous detritus that may be left behind when malware is removed: it may be harmless in itself, but what happens when two security programs are used on the same system and one is more cautious about what it removes than the other? Which is "correct" is a subjective judgement rather than a right/wrong issue.

To take another common case, no-one wants to see those misidentifications of innocent system files as malicious that affect even the most cautious products from time to time, but what about files that are detected as malicious because they're packed with a run-time packer that's often (but not necessarily always) used by malware authors? Well, that's not an issue I'm going to be able to give a short and simple answer to here. But I will say this (and often have before): given the complexity of the malware problem, not to mention the sheer volume of samples, the wonder is that mainstream anti-malware generates so few FPs. And it's a real pleasure to find someone agreeing on that point who can't be accused of undue bias in favour of this industry.

By the way, these blogs reminded me of an old but still relevant paper on "The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection" by Stefan Axelsson. If you're interested in the false positive conundrum and  the words "Bayes Theorem" don't strike terror to your heart, you might find it well worth reading.

David Harley BA CISSP FBCS CITP
Director of Malware Intelligence

ESET Threatblog (TinyURL with preview enabled): http://preview.tinyurl.com/esetblog
ESET Threatblog notifications on Twitter: http://twitter.com/esetresearch
ESET White Papers Page: http://www.eset.com/download/whitepapers.php

Securing Our eCity community initiative: http://www.securingourecity.org/

It’s that time of year when everyone wants a top ten: the top ten most stupid remarks made by celebrities, the ten worst-dressed French poodles, the ten most embarrassing political speeches, and so on. Our research team came up with a few rather more serious ideas, most of which are considered at some length in our about-to-be-published Annual Global Threat report and November Threatsense report, but we thought it might be nice to post some of the information in one or two of those top ten lists here for those who may find the length of the full reports a little daunting, as well as a taster for those who don’t. Rather than simply reproduce those lists, we’ll consider individual items at more length over the next few days.

Perhaps one of the more useful ideas that was tossed around was a top ten of things that people can do to protect themselves against malicious activity. This is the item that we pretty much all agreed should be top of the list. 

Disable Autorun in Windows: this facility is consistently exploited by the class of malware ESET detects as INF/Autorun, among other threats. We’ve been considering this issue in detail for quite a while, now: for instance, in Randy Abrams’ blog here. That class of malware has been consistently at or near the top of our monthly worldwide top ten reported threats as long as I’ve been tracking them. Don’t assume, though, that that single precaution will save you from every example of that type of threat. Most malware uses more than one technique to infect targeted systems.

More later.

David Harley BA CISSP FBCS CITP
Director of Malware Intelligence

AV-Comparatives, one of the major anti-malware testing organizations, has just announced its retrospective test for November. Retrospective or "frozen" testing involves testing the ability of one or more products to detect threats proactively, using techniques such as advanced heuristics rather than signature detection.

The test used new and unique samples received between 4th and 31st August, while the 16 products testing were "frozen": that is, they didn’t receive updates and signatures after the 4th. This means, of course, that they were only able to detect these samples using proactive or generic techniques, since they had no access to signatures generated after the samples were discovered, so it’s a fair test of the detection capabilities of a product with good passive heuristics. It was an on-demand test, using static analysis. For advanced heuristics using some form of behaviour analysis, dynamic testing might be a fairer test: however, it would be extraordinarily time- and resource-intensive to run a full dynamic test with such a large sample set. (Which is probably why fully dynamic testing is still so rare.) However, a really interesting feature of this test is that the results take into account not only proactive detection, but false positives.

As it happens, I’ve been doing quite a few presentations on testing in the past few weeks, and one of the points I’ve been making that usually generates lively discussion concerns the continual trade-off between maximizing "true positives" and minimizing false positives (FPs). (Of course, that’s also a theme that features strongly in the AMTSO-approved document on the principles of testing.) When people talk about FPs, they usually think of embarrassing and very public incidents like Windows system files being incorrectly identified as malware. Of course, that happens: as the anti-malware industry tries to focus more and more on proactive detection, it’s amazing that it doesn’t happen more often. Far more common these days is the identification (or, arguably, misidentification) of corrupted or truncated samples as malware even when such sample can’t actually execute or do damage. While you can certainly argue that a program meant to be malicious should be detected even if it isn’t able to deliver, there are as many views and ways of addressing that issue as there are vendors. Then, there’s the issue of what constitutes a "suspicious" file. Some products come quite close to equating "executable" with "suspicious". Many come very close to diagnosing any file processed with a runtime packer as suspicious, even though there are still legitimate reasons for using runtime compression, for instance. There are justifiable reasons for taking this approach. It’s certainly fair to be cautious about scenarios like this:

• It’s packed (compressed, maybe multiply, with a runtime packer)
• It’s packed with a known “black” packer (suggests malice)
• It’s packed with a custom variant packer (suggests obfuscation and malice)
• It’s packed, but was already a small executable (suggests obfuscation and malice)

 However, this isn’t malware-specific detection. It’s more like the defenses evolved in the heyday of mass-mailers, i.e. blocking attachments like .EXE, .SCR, .COM and so on. In other words, rather than blacklisting a single malicious object, or a family of malware, a whole class of executable objects is being blocked. That’s fine if the customer understands what is happening, but it’s not a malware-specific detection method, and, potentially,  it institutionalizes false positives. Furthermore, it’s not a universal practice: maybe it should be, but that’s a whole different discussion. So tests that assume that suspicious==malicious are making an inappropriate value judgement, and to see a test that takes these issues into account is very encouraging.

And yes, we seem to have done pretty well, thank you for asking. We were one of the front runners in detecting test samples, and did even better in terms of minimizing false positives, and that combination has put us in the very gratifying position of having the only product certified as Advanced+.

But let’s be clear about this: we’re not talking about absolute values here, but a continuum. Some companies have a much higher tolerance for false positives than others. And there are many other factors that you might need to consider when you evaluate a product. Certainly, you should read the full report, rather than take my word for anything. (Go to the AV-Comparatives site and check report number 20 on the comparatives page.) Still, it’s always nice to find a major testing group prioritizing similar criteria to those that we value.

David Harley CISSP FBCS CITP
Director of Malware Intelligence

…and it’s still hybrid. Or multi-layered, if you prefer. What anti-malware companies (and malware authors, if it comes to that) are constantly doing is revisiting concepts that have worked before so that they fit the current environment better: there’s nothing wrong with an evolutionary approach, but changing the terminology doesn’t make it revolutionary. So what Larry Seltzer is describing in a recent eWeek article isn’t exactly groundbreaking technology, it’s what all anti-malware companies originating in traditional AV are doing, to a greater or lesser extent. "File reputation" is pretty much what we used to call integrity checking, and is close to a limited application of whitelisting that’s been in common use since the 1990s. The main difference is that whereas earlier incarnations of anti-virus tended to bundle an integrity checker as a separate application along with a known-virus (signature) scanner, it’s now common to whitelisting or a near equivalent into the main application. And in those days, no-one described their own server networks as a cloud.

What’s more interesting is Larry’s critique of various "classic methods" of malware scanning.

• Clearly, we aren’t about to argue that it’s feasible to have a signature for "every new variant": that’s a model we moved away from many years ago.
• I would argue, though, that "true heuristics" is an odd term to use for what we describe as "passive heuristics", where an object is scanned for malicious characteristics statically – that is, looking at the code without executing it. (We have a fairly comprehensive, vendor-agnostic review of heuristic analysis on our white papers page, by the way.) The term heuristics has a far wider range of applications than that, even within the anti-malware industry, which uses it in quite a specialized sense. Clearly, there is still a place for this analogue to static analysis, as the term is used in forensics, but it isn’t nearly as effective as it used to be, because the bad guys use a variety of obfuscatory techniques to hide malicious code from signature and basic heuristic detection. There’s nothing "untrue" about heuristic analysis when it analyses code when it’s actually running (an analogue to dynamic analysis): the point Larry seems to have missed is that when products like ours run that code, it’s in a protected environment, so (assuming a sound implementation of the product) the code being run shouldn’t present a risk to the system. (By the way, this is the second time in two days I’ve talked about static versus dynamic…)

What interests me most, however, is his yearning for "a simple solution like absolute whitelisting." It does seem that we’re always looking for the 100% solution that will render current anti-malware solutions unnecessary. The way that firewalls, IDS, IPS, reputation services, NAC and a dozen other panaceas du jour were once seen as The Answer. But the fact is that whitelisting itself is hybrid (by which I mean that you can’t whitelist an application without using other technologies to confirm that it’s what AMTSO like to call "innocent". And it works best as one layer of a defensive strategy, at any rate in the version of the internet in which we currently find ourselves.

David Harley CISSP FBCS CITP
Director of Malware Intelligence