Since January, ESET has been in discussions with NSS Labs to address industry concerns regarding their recent “Advanced Endpoint Protection 1.0” test. During these discussions, it was agreed that deadlines would be honored by both parties, communication would be improved, specified content would be provided and all outstanding issues would be resolved. We are not the only vendor to raise these concerns.
We now feel it incumbent upon us to publicly communicate what we believe are violations of testing standards. We remain committed to providing open and transparent communication to our industry partners and to the public. Below, we have included a series of timelines to convey what was mutually agreed upon between ESET and NSS Labs, and their commitments to us, which many, as you will see, remain unfinished.
Timeline of Deadlines Missed
False Positives
Issue: Time after time, NSS Labs did not deliver on their self-imposed but announced deadlines. For example, we have recorded the times for which NSS Labs indicated we would receive additional information about the False Positives (FP’s) we flagged during the test. As you can see, deadlines were ignored completely.
· Friday 27 Jan: Remaining: False Positives – ETA Saturday
· Saturday 28 Jan: Remaining: False Positives – ETA Saturday
· Sunday 29 Jan:Remaining: False Positives – ETA Monday
· Monday 30 Jan: Remaining: False Positives – ETA Monday
· Tuesday 31 Jan: Remaining: False Positives – ETA Tomorrow
· Saturday 2 Feb: Finally, we received the details of the False Positives. Sadly these were only hashes, without meta-data and the majority of the samples could not be found. So of course ESET asked for the samples from this set.
· Monday, 4 Feb: NSS responds that “we do not provide samples for this category based on the nature of false positive testing”. Seriously? They can claim vendors have FP’s on legitimate files but they won’t supply the files to prove it?
· Saturday 11 Feb: Although there are several issues outstanding, NSS Labs - under pressure from an RSA Release, sent the “final” spreadsheet with the results. Lots of new columns were added with new important information. One of them was the column with the filenames of the samples from the FP Test set. To ESET’s surprise, all of these files were documents, spreadsheets, pdf’s and rtf’s. Where the entire AEP test is based on binaries, the FP test was done only against data-files. This is extremely unbalanced. Furthermore, products in the test that are basing their judgement on white- and blacklisting will never produce the same FP on data files as the ones in the test-set. They don’t even “scan” as those files change contents with normal usage. Opening a Word Document and closing it, even *without* changing a character, will change the document written to the hard disk, the “last accessed” and “accessed by” fields get updated. Any white- and blacklisting product, and there were several in the test, would have issues with these files if they were included in black- and whitelisting.
· Thursday 16 Feb: At a meeting during RSA, the issues around the FP test-set were discussed. NSS’ CEO agreed that for transparency, of course the test-set would have to be shared. It was also agreed that NSS Labs would follow up with a blog clarifying that the FP test-set was of a certain type only, which is insufficiently measuring the FP impact and biasing the results. Since nothing really happened during the following week, ESET called for a teleconference with NSS Labs for Monday 27 February. Regardless of the fact that NSS agreed to the call, on the actual day and just 40 minutes prior, NSS cancelled the call and moved it to the next day.
· Thursday 28 Feb: We reiterated the issue of FP’s to the NSS Labs CEO and received their promise to share the samples, advising that ESET still hadn’t received the samples from the set. Again NSS promised to send the samples, and during the call we also agreed to a deadline: Monday 6 March.
· Monday 6 Mar: The deadline again expired on *all* of the outstanding issues, including the FP test-set. Again we contacted the NSS CEO.
· Tuesday 7 Mar: *Finally* we received the NSS FP sample set, all other outstanding issues again were “postponed”.
Test Samples
Issue: NSS Labs informed us that they utilized a different number of samples and different sample sets for each vendor they tested, which is quite unusual in the world of AV testing and they didn’t mention this in their testing methodology. When we spoke to them about this, they conveyed that their intent was to disrupt the collusion of some anti-malware vendors during the testing process. We mutually agreed that NSS Labs would communicate this information publicly and reflect in their AEP test methodology. However, as of March 13, they have not made any public statement about this topic.
· Thursday 16 Feb: Remaining: Agreed: Public Communication about the different samples and collusion will be done by NSS Labs
· Tuesday 28 Feb: Remaining: Public Communication – ETA Monday 6 March
· Monday 6 Mar: https://www.nsslabs.com/blog/ features a new blog on RSA/AEP but not touching the agreed points. So Public Communication – <ETA who knows>
Config files
Issue: During our discussions, it was established that defining something as a threat all depends on the context. Several misses were lacking php/ini files, which could identify the difference between harmless and malicious events. NSS Labs was to deliver the configuration files to ESET for these occurrences, however they have failed to do so.
· Thursday 16 Feb: Configuration files will be provided
· Tuesday 28 Feb: Remaining: Configuration files not provided, ETA 6 March
· Monday 6 Mar: Remaining: Configuration files not provided, ETA 14 March
Corrupted Samples
Issue: NSS Labs agreed to verify the observation we made about three corrupted samples in the test and adjust accordingly after verification. This issue still remains outstanding and unresolved.
Outstanding Issues
Issues: NSS Labs is to provide additional information on outstanding issues so ESET can understand the nature of the miss to improve ESET products. We had four outstanding issues. We provided them to NSS Labs and conveyed that our own solutions have recognized three of these since 2015. NSS has not provided any evidence that we do not detect them. The other miss (http) is questionable as to whether it can be detected at all, especially without the php-file that holds the location of a redirect. At the time of verification, it is redirected to a non-existing page.
Conclusion
We did not set out to have a bad experience with NSS Labs. We are not doing this to improve our score in their test but to improve transparency of the test and testing methodology. We’re communicating openly, because it’s the right thing to do. It represents what ESET stands for, and we will always stand up for what is right.
We welcome discussions, open dialogue and we even support having a healthy debate. However, when an organization interferes with our ability to learn from a test, which allows us to better protect our customers, we feel it is incumbent upon us to tell the full story. We remain committed to providing the best endpoint protection for all of our current and future customers and we have a duty communicate openly and honestly with the public. We are here to serve our customers and protect their digital worlds.
-ESET