However, John Leyden of The Register contacted us both when he was writing an article on the controversy following Kaspersky Lab's dramatic demonstration of the way in which false positives can cascade from one vendor to another. This is a major issue, because it can and does introduce a serious bias into comparative detection testing and analysis. After responding to John's questions, we continued the discussion subsequently by email and found that we (along with most of the AV industry) were in agreement on all major points, and decided that it was more important to clarify those points, than to continue debating the detail of the demonstration.

The fact that the demonstration used Virus Total as a channel for cascading the "artificial" false positives to other vendors should not be seen as in any way detrimental to Virus Total. Hispasec have never endorsed the use of the service as a substitute for comparative testing or for sample validation, either of which are very likely to generate misleading results.

Multiple scanners are not in themselves the problem, whether they're hosted on public sites, specialist resources, or used by testers or anti-malware companies in-house. As tools for comparative analysis or precursors to more detailed analysis, they have a great deal of value. However, that value depends on the user's knowledge and understanding of how to make the most appropriate use of them.

Mainstream testers and security vendors have extensive understanding of these issues: however, many tests do not take them sufficiently into account. The Kaspersky Lab experiment did at least bring the issue to the attention of some of the press and publishers who most need to be aware of it, and who would probably have taken far less notice of a less controversial presentation.

As supporters of AMTSO, the Anti-Malware Testing Standards Organization, we are in emphatic agreement that away from static testing and toward dynamic testing is a positive direction. We hope that more reviewers now appreciate that dynamic testing with small but properly validated sample sets offers more realistic assessment of detection capability with less risk of unintended bias. If more people realized this, it would allow vendors to spend more time on real threats and less on making sure they detect samples that shouldn't be included in a test set.

David Harley, ESET Research Fellow & Director of Malware Intelligence
Magnus Kalkuhl, Senior Virus Analyst, Kaspersky Lab

Kaspersky did something rather interesting, though a little suspect. They created 20 perfectly innocent executable files, then created fake detections for ten of them. Then they uploaded (actually, the blog says re-uploaded) the files to Virus Total. They claim that after ten days "up to" 14 other vendors also had detection for the files for which Kaspersky had a fake detection.

(No, ESET wasn't one of them! We don't have a marketing axe to grind here.)

In fact, several vendors only detected one or two of those files, not the whole set. Furthermore, in at least one case all the files were detected as "suspicious" rather than malicious, and only by that company's online scanner: in that context, it's possibly defensible, certainly as a short-term flag.

So is there really a problem here? When Virus Total sends a sample of what may be a missed detection to the company whose scanner missed it, they probably don't expect the company simply to add detection without validating the sample: otherwise, a false positive could cascade through the industry in a very short time. [Update: there's an interesting related blog from Hispasec, who provide the Virus Total service, at http://translate.google.com/translate?u=http%3A%2F%2Fwww.hispasec.com%2Funaaldia%2F4013&sl=es&tl=en&hl=&ie=UTF-8 - I'm afraid it's an automatic translation, but you get the idea... (Thanks, Pedro, for calling my attention to it.)]

But as Larry suggests, there are other factors at play there, rather than a simple mindset of "if Kaspersky detects it, it must be malicious". With all due respect to the company, no-one in this business has escaped problems with false positives, as John Leyden pointed out today in an article in the Register.

In the Kaspersky blog, Magnus suggests that this is really a problem with testing. Well, he has a point. Where tests are carried out with huge sample sets and minimal time and financial resources, it's inevitable that some testers will rely on "source reputation and multi-scanning" rather than manual validation. In fact, Ján Vrabec recently cited problems with a test where a much smaller test set was "validated" according to whether four or more scanners detected it as malicious.

However, AV companies (and the major test organizations) don't, in general, rely on these approaches. Obviously, a significant proportion of analysis for a virus lab has to be manual, though where tens of thousands of samples are received daily, there has to be as much automation as possible. So it may be that there are many factors at work here, such as:

• Timing issues – a sample may be wrongly detected at one point because of "reputation sourcing" and detection removed after manual analysis. A Virus Total analysis report is a snapshot of one moment in time: it shouldn't be seen as a blot on a company's permanent record.
• In fact, the same problems frequently pointed out in other contexts with the inappropriate use of Virus Total also apply here. VT uses a battery of command line scanners, and different products will behave differently in different contexts. Taken into account along with the timing issue mentioned above, the numbers cited here don't seem to mean much.
• As Larry suggests, it's not unknown for an aggressive heuristic to generate a false positive. And as a spokesman for another product suggested, products with a wide range of functionality may react differently according to what backend, gateway or desktop settings are enabled.

In fact, the problem Kaspersky has flagged may have longer-lasting effects than the company has acknowledged. Larry Seltzer's article seems to say that subsequently, a "hello world" program generated with the same compiler and compiler settings was also flagged by at least two programs. Could this be a compiler-specific heuristic implemented as a result of Kaspersky's artificial false positives? If so, Kaspersky also bear some of the blame.

The sad thing is, that genuine problems may take second place here to the easy story of "who copies from whom." By going straight to the press and presenting journalists with the innocent executables, encouraging them to use Virus Total (inappropriately, in our view) to check their conclusions, Kaspersky has made inevitable what they flagged as a risk. Reproducible experimentation is a worthy target, as long as the experimental methodology is sound, but does that really apply here?

But the real problem here is that Magnus is perpetuating a fallacy that already worries many people in the industry. He suggests that the problem here is with static testing, and that the move to dynamic testing is going to fix things. However, that's neither the problem nor the cure in this case. The problem is non-validation, and the cure is validation. Right now, lots of tests claim to be dynamic, because that's what AMTSO advocates. Quite rightly, in principle: good dynamic testing has the potential to be a more accurate test of a product's efficacy than static testing. But good static testing remains a better approach than bad dynamic testing, which few testers are doing well at the moment. A significant part of that problem is, unfortunately, still validation.

The Research Team
ESET LLC

In particular, I've grown accustomed to the fact that many people expect all the following from an AV product:

• Absolute Protection
• Absolute Convenience
• Absolutely no  False Positives
• Absolutely no charge

False positives (FPs) are not a minor issue: my experience is that many people (especially in corporate environments) are more infuriated by an FP than by a false negative (i.e. where security software fails to detect real malware).

So it was a pleasant surprise to come across a blog from a source not usually associated with fulsome praise of the AV industry that was actually rather positive, in a backhanded sort of way. The author states "…while I rant quite a bit about the AV industry, you have to give this one to them: the number of false positives is really low. For example, in the AV-Comparatives test 20 false positives is considered many, even though the collection is over 1 500 000 samples (so the acceptable FP rate is below 0.0015%!)." [Sadly, this calculation doesn't altogether make sense, either to us or to AV-Comparatives. In fact, it seems to be based on comparing the number of "acceptable" false positives to the AV-Comparatives malware test set. It isn't based on the AV-Comparatives clean set: AV-C don't publish that information, as they consider FP ratios in percentages to be potentially misleading.]

The blog isn't actually about us, I should make clear: it's about the importance of false positives in intrusion detection, an area with just enough methodology and terminology shared with anti-malware to be confusing – for instance, both technologies talk about FPs and signatures, but often mean something slightly different by them. This led me to another blog that makes a useful distinction between "real" false positives – misdetection of innocent objects as being malicious – and "noise" such as "inappropriate traffic that is legitimately detected that we just don't care about." Actually, because modern anti-malware has a much wider remit than traditional virus detection, that is a concern we share with IDS and IPS vendors – of course, there's an overlap – most AV products now include some IPS (Intrusion Protection System) functionality. But it also has an analogue in more traditional anti-malware detection, which tends to cover "noise" programs such as adware and "possibly unwanted" code as well as out-and-out malware. 

Then there's the miscellaneous detritus that may be left behind when malware is removed: it may be harmless in itself, but what happens when two security programs are used on the same system and one is more cautious about what it removes than the other? Which is "correct" is a subjective judgement rather than a right/wrong issue.

To take another common case, no-one wants to see those misidentifications of innocent system files as malicious that affect even the most cautious products from time to time, but what about files that are detected as malicious because they're packed with a run-time packer that's often (but not necessarily always) used by malware authors? Well, that's not an issue I'm going to be able to give a short and simple answer to here. But I will say this (and often have before): given the complexity of the malware problem, not to mention the sheer volume of samples, the wonder is that mainstream anti-malware generates so few FPs. And it's a real pleasure to find someone agreeing on that point who can't be accused of undue bias in favour of this industry.

By the way, these blogs reminded me of an old but still relevant paper on "The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection" by Stefan Axelsson. If you're interested in the false positive conundrum and  the words "Bayes Theorem" don't strike terror to your heart, you might find it well worth reading.

David Harley BA CISSP FBCS CITP
Director of Malware Intelligence

ESET Threatblog (TinyURL with preview enabled): http://preview.tinyurl.com/esetblog
ESET Threatblog notifications on Twitter: http://twitter.com/esetresearch
ESET White Papers Page: http://www.eset.com/download/whitepapers.php

Securing Our eCity community initiative: http://www.securingourecity.org/

The Win32/Induc.A virus does not infect like most viruses do. Delphi is a programming language. Induc infected the Delphi IDE so that when the programmers compile their programs the programs are already infected.

As far as we are able to determine at this time, this virus went undetected since April 2009. Most of the samples of infected files we have seen are other trojans, mainly those that steal bank information. So, we detected the Trojan, but didn’t know that it was also infected.

For the average user the virus is essentially harmless. The problem is that some software development companies use Delphi, got infected, and when we added detection for Win32/Induc.A their programs were detected. Some of these companies accused ESET of having false positives when their programs were actually infected!

In reviewing our internal malware collections our researchers have found over 4,000 infected samples. Our Threatsense.Net network has identified over 30,000 unique infected samples in the first 24 hours after we added detection.

For a write up about this virus you can visit http://www.eset.eu/encyclopaedia/win32-induc-a-virus?lng=en

Ironically, some other malicious software that was previously undetected by antivirus vendors will now be detected because it is infected with Induc.A!

It’s pretty rare now to be able to talk about a widespread virus that probably won’t cause you any harm.

Randy Abrams
Director of Technical Education

In fact, the idea of maintenance viruses goes back at least as far as Dr. Fred Cohen, who pretty much "wrote the book" on early viruses, almost literally. In fact, looking at his book "A Short Course on Computer Viruses" (in which he specifically discusses "benevolent viruses") again, I was struck by how far malware and anti-malware technologies today still reflect his ideas.

However, the idea of the benevolent maintenance worm, which last year resurfaced in a paper in which Microsoft researchers had an interest, has tended to meet with a chilly reception from the mainstream researcher community.

An event today really drove home the point about how bad a good worm could really be.

 We had an unfortunate false positive event this morning at ESET. A synchronization error allowed a new heuristic module to be released along with a new signature update. Each worked great on its own, but together they created false positives on some Windows operating system files. Both components went through QC, but not together. We’ve got that process fixed now.

The problem was spotted almost immediately, and within ten minutes the faulty update was removed from the download site, thereby preventing 95% of our users from encountering a problem. As for the other 5%, many of those people never knew there was a problem because the next update replaced the quarantined files on all versions of our products except for version 2.7. We have also issued a new tool to help anyone who needs assistance, and that also works on all versions, including 2.7. Our people in tech support can provide the tool upon request, and, of course, tech support is free for all ESET customers.

So, what does it really mean, when we say that 95% of our users were unaffected? Unfortunately, It means that there were still a couple of million users who did encounter the problem. A couple of million users in 10 minutes, and this was done without the help of a worm. The Slammer worm infected 90% of all vulnerable connected computers in 10 minutes. No matter how good the intent, there will be problems with software from time to time. With a faulty signature we can turn off the source almost instantly and it goes no farther. With a worm, well, you know what breaks loose. Don’t just think Slammer, think Code Red, Blaster, and many, many mass mailers (some of which are still very much current).

 No matter how good the intent, self-replicating code is not a good idea for a distribution mechanism. There is always a better alternative for distribution.

So, you may wonder, how do we get our updates out to a couple of million users in ten minutes or less without the benefit of a tame worm? Ironically, by a mechanism analogous to that used to update zombie PCs in a botnet. But we’d like to think that’s a case of the malware community taking ideas from us, rather than vice versa.

Randy Abrams & David Harley
ESET Research Team

The European publisher Heise, apparently recently reinvented as The H, has pointed out that both GData and Bitdefender were inaccurately flagging winlogon.exe as Trojan.Generic.1423603. In case you were wondering, this doesn’t mean the whole anti-malware industry has gone mad: GData’s product uses two engines, one of which is  Bitdefender’s.

In fact, it doesn’t mean that Bitdefender have gone crazy, either. False positives – diagnosing an infection where none actually exists – is one of those regrettable issues from which no vendor (or AV customer) is safe. Sometimes it’s a major inconvenience for customers, and that’s why Heise have been characteristically abrasive on the issue: "This latest case gives rise to the assumption that false alarms caused by anti-virus software will continue to unnerve users. " But the point is well taken: FPs can be alarming, confusing, inconvenient and downright expensive for customers (especially where innocent but essential files are quarantined or deleted), which is why we go to considerable lengths to minimize the risk (but even ESET can’t eliminate it altogether, and it really is an issue we work very hard on!)

In fact, high profile misidentifications are surprisingly rare, given the complexity and sophistication of the technology. There are probably quite a few instances of low-profile FP, affecting applications less commonly used, that are hardly noticed. Despite the anger that antivirus and antispam FPs sometimes inspire, much of the security industry is founded on the idea that for some organizations and individuals, a degree of inconvenience caused by sensitive filters (in the broadest sense of the word filter) is acceptable.

For example, it’s common for mail filters to block whole swathes of executable filetypes. It’s unlikely that a filename like something.lnk would ever be attached to a legitimate email file attachment (the suffix denotes a shortcut rather than a "real" executable file), so blocking that suffix hardly ever results in misidentification. However, there are a great many innocent files that have the .EXE suffix, yet we often accept the filtering of .EXE attachments because we consider the risk from malicious attachments worth the risk of not receiving a legitimate .EXE through the mail, or the inconvenience of finding an alternative mode of transportation. In fact, some organizations have been doing this for at least ten years (and similar considerations apply in the world of spam management, as I discussed more recently in a Virus Bulletin article).

The fact is that the industry has had to move on from the early days of exact identification and one sighature to one threat. Nowadays, products use combinations of approaches that are nearer to what we used to call generic detection like change detection: whitelisting, filtering by filetype, change detection, heuristics, behaviour analysis, and so on. (Of course, products vary widely in terms of which techniques they use, how they are implemented, and so on.)

Our labs are now seeing over 100,000 unique samples a day. Other vendors may use different ways of counting, but they all face the glut problem, which is why there is so much emphasis in the industry on automation, distributed processing, and dynamic analysis. The miracle is, in the face of these difficulties, that significant false positives are so rarely encountered.

But there you go: problems attract far more attention than a product doing pretty much what it was designed to do, especially an antimalware product…

David Harley BA CISSP FBCS CITP
Director of Malware Intelligence

What else causes data loss? Sometimes improperly configured antivirus products themselves cause data loss. A couple of years ago one of the largest antivirus vendors in the world had a false positive problem that deleted many Microsoft office files. False positives happen to all of us from time to time, but files should be quarantined so that if it is a false positive the data can be recovered. The product in question was revised to make quarantine the default setting after so many of their customers who did not perform proper backups lost a lot of data.

Fingers are high up on the list. Did you ever accidentally delete something and not have a back up?

Hard drive failures can also cause massive data loss. A recent event that our own Aryeh Goretsky brought to my attention is what made me decide to write this blog entry. Hard drive failures are typically fairly rare, but there is a biggie out there now.

It seems that Seagate has released droves of 1 terabyte hard drives that have a problem causing them to die. Having good backups may be the only way to recover your data without spending a few thousand dollars in such a situation.  

There are about 18 pages of comments on a Seagate forum (as of this writing). The drives may be working fine for 3 months and then die instantly.

Below are a few links about the issue. The last link is the Seagate forum I mentioned.

http://www.techreport.com/discussions.x/16232

http://www.theinquirer.net/inquirer/news/374/1050374/seagate-barracudas-7200-11-failing 

http://www.dslreports.com/forum/r21737309-AVOID-seagate-ST31000340ASSD15-drives 

http://forums.seagate.com/stx/board/message?board.id=ata_drives&thread.id=3668&view=by_date_ascending&page=1

It is unfortunate that Seagate refuses to comment on the situation. Late last year it was discovered that Seagate had shipped about 1,800 brand new drives with malware on them.

If your data is important enough to put on a hard drive, be sure you back it up. There are many threats to your data, and viruses are the least of those threats.

Randy Abrams
Director of Technical Education

You can find information in our knowledgebase here about how to forward malware samples or false positive samples to our labs: it doesn’t specifically mention malicious URLs, but you can send those to the same address.

It’s not that we’re not willing to pass on information to the labs ourselves (and I’ve already forwarded the link), but it’s much more effective for you to send the information direct. As a case in point, it turns out that by the time I saw this comment and forwarded the link, our products had already been updated to detect this particular binary.

Thanks!

David Harley CISSP FBCS CITP
Director of Malware Intelligence