Malwarebytes and testing

[Ping]

Just a short question. Why is MalwareBytes so badly in this test(?):
PDF-File from MRG Effitas 360 Assessment & Certification Q3 2016
https://www.mrg-effitas.com/wp-content/uploads/2016/11/MRG-Effitas-360-Assessment-Q3-2016.pdf

Is this test at all a test? Or even nonsense? Or is the selection of the samples just bullshit?

[David H. Lipman]

Who knows.  They do not provide a list of the actual files they tested the applications with and since MBAM is not an anti virus and has a limited file targeting list, it could be they used non-targeted files.

MBAM is not an anti virus application and does not replace an an anti virus application.  MBAM is an adjunct, complimentary, anti malware application.
 
In its role as a adjunct, complimentary, anti malware application it has limitations in aspects that the anti virus application performs in its role.
 
MBAM does not target script files. That means MBAM will not target; JS, JSE, PY, .HTML, HTA, VBS, VBE, .CLASS, SWF, SQL, BAT, CMD, PDF, PHP, WSF, etc.
It also does not target document files such as; PDF, DOC, DOCx, DOCm, XLS, XLSx, PPT, PPS, ODF, RTF, etc.
It also does not target media files;  MP3, WMV, JPG, GIF, etc.

Until MBAM, v1.75, MBAM could not access files in archives but with v1.75 came that ability so it can unarchive a Java Jar (which is a PKZip file) but it won't target the .CLASS files within. Same goes with CHM files (which is a PKZip file) but it doesn't target the HTML files within. MBAM v1.75 specifically will deal with; ZIP, RAR, 7z, CAB and MSI for archives. And self-Extracting; ZIP, 7z, RAR and NSIS executables (aka; SFX files).

MBAM specifically targets binaries that start with the first two characters being; MZ
They can be; EXE, CPL, SYS, DLL, SCR and OCX. Any of these files types can be renamed to be anything such as;  TXT, JPG, CMD and BAT and they will still be targeted just as long as the binary starts with 'MZ'.
 

MBAM targets mainly non-viral malware.  The exception being a virus dropper ( a malware file that drops a virus and starts a virus infection but is not infected with the virus ) and worms ( such as Internet worms and AutoRun worms ).
 
MBAM is incapable of removing malicious code that has been prepended, appended or cavity injected into a legitimate file.  That means if a file infecting virus infects a legitimate file MBAM will be unable to remove the malicious code.  An anti virus application should be able to remove malicious code from an infected file and hopefully bring it back to its preinfected state.  Which may or may not return the file to its original, non infected, checksum value.
 
A file infecting virus will prepend, append or cavity inject malicious code into a legitimate file.  Once infected, that infected file can further the infection by infecting other legitimate files.
 
On the other hand there are trojans that will prepend, append or cavity inject malicious code into a legitimate file.   However that file can not infect other files.  The infection stops with that targeted file.  These files are either deemed to be "trojanized" or "patched".  Since MBAM can not remove the added malicious code, at best MBAM will try to replace the trojanized file with a legitimate, unaltered, file.
 
Where a traditional anti virus application is weak, MBAM is strong.  Today's malware is much more complex than 10 years ago.  When we saw the Melissa virus ( I-Worm via SMTP  ), Lovsan/Blaster worm (  I-Worm via RPC/RPCSS @ TCP port 135 ) etc, they were distributed for the effect, damage and bragging rights.  Today's malware is more sophisticated in that it is "all about the money".  Malicious actors use malware to profit from.  Either by stealing, distribution affiliation revenue, data exfiltration, personal identification impersonation, etc.  To effect that the malicious actors don't want the victim to know that their system was compromised or they are so blatant about it by generating advertisements,  Yesterday's malware was simple and less obtrusive.  Today's malware is very intrusive and makes numerous modifications to the Operating System.  Those numerous modifications to the Operating System is where the traditional anti virus application does poorly and where MBAM specializes.
 
MBAM is not a historical anti malware solution.  That means it will not target old malware.  It's intent is to target 0-Day malware.  Malware that is infecting computers Today with malware found in-the-wild, Today.  That means that something like the BugBear which infected years ago will not be targeted by MBAM.  Malwarebytes will actually cull their signature database for malware that is no longer seen in-the-wild Today.   This is why Malwarebytes requests samples that are submitted for detection consideration be no older than 3 months old.

 

 

[exile360]

As far as I can tell they only used the Free (i.e. on-demand scan/remediation only) version of Malwarebytes Anti-Malware, not Premium, which likely would have been capable of preventing/blocking many of the threats they tested against, especially with our malicious website blocking feature included in Premium.  Additionally, given the fact that many (i.e. most at this point, as far as I can tell based on my own investigations) modern malware threats/attacks are initiated by exploits, if they had also included Malwarebytes Anti-Exploit (even just the free version, not necessarily the paid version) it likely would have prevented a huge number of the threats they were testing against since they were simulating actual real world infection/attack vectors (i.e. visiting malicious sites rather than just executing samples downloaded into a folder or archive on the desktop).

If you take a look at what's coming in Malwarebytes 3.0, it has all the tools necessary to deal with virtually all of the threats and threat types they mention in their test, including ransomware thanks to our new Anti-Ransomware technology/module which will be integrated in that release (along with Anti-Exploit; which also will be integrated into version 3.0).  I'm anxious to see how we fare in tests like this once these labs get ahold of Malwarebytes 3.0 to pit against the competition.  I'm confident that its protection will stand up quite well against virtually all of today's real world threats because it features many new technologies which do not rely on signatures/database updates and instead focus on behaviors and known attack vectors to stop an attack in its tracks, often before it even gets the chance to try dropping an MZ/PE payload onto the system.

Edited November 26, 2016 by exile360

[Ping]

David, thankyou for the explanation. It also illuminates this "test".

[Ping]

Seems to be a Java Script. Should MalwareBytes 3 AntiRansom not yet report an alert? I know MBAM itself does not recognize JS. Nevertheless ... most of the Ransomware is hidden in JS, PDF, DOCx, XLSx. Will this change in the future?

 

[David H. Lipman]

24 minutes ago, Ping said:

Seems to be a Java Script. Should MalwareBytes 3 AntiRansom not yet report an alert? I know MBAM itself does not recognize JS. Nevertheless ... most of the Ransomware is hidden in JS, PDF, DOCx, XLSx. Will this change in the future?

 

No that is not true.  Crypto trojans are not hidden in Scripted Malware or MS Office Documents and I have not seen PDF files associated with them either.

Scripted Malware ( HTA, WSF, JS/JSE, VBS/VBE ) are scripted downloader trojans.  They are coded to download and execute a crypto trojan,  They are dependent upon Internet access at the time the script is executed.

MS Office documents are used but the Crypto trojans are not embedded.  The malicious actors take advantage of the MS Office Macro language which is a variant of VBS called VBA.  In effect they too are are scripted downloader trojans.  They too are dependent upon Internet access at the time the Document is opened and the Macro is allowed to execute.

I have not seen a PDF being used in a crypto trojan infection ploy.  It is possible but if it is employed it is very rare and the crypto trojan would not be embedded within the PDF.

[AdvancedSetup]

This too is not a valid test of real world detection or protection as these infections are not triggered by Explorer executing them as the user is doing here. This is quite similar to the topic here

https://forums.malwarebytes.org/topic/191104-the-pc-security-channel-tpsc-malwarebytes-premium-v3-beta-review/

 

[Ping]

@David, of course, I'm aware that * .js or * .doc up to * .bat are on user level Trojan Downloader. I'm sorry.

@AdvancedSetup
Thank you for your reference. You are certainly right with your statement. The link I have unfortunately no access authorization: You are not authorized to view this content. Maybe you've a screenshot for post here.

[John L. Galt]

Hi, Ping,

The same video in that URL is also mentioned in this public post: