Jump to content

BSODs (BAD_POOL_HEADER) w/ Malicious Website Protection


Recommended Posts

On February 27 I bought the Pro (as it was called then) version of Malwarebytes.  Shortly thereafter, I started getting BAD_POOL_HEADER BSOD reboots, averaging between 1 and 2 per day, although sometimes none in a day.  I didn't suspect Malwarbytes because NVIDIA also updated my graphics driver around the same time, and most BSODs I had before then were due to video drivers.

 

Time went on, and NVIDIA updated the graphics driver two more times, with no mention of BSOD fixes in the release notes, and the BSOD crashes continued as always.  So last Friday I finally ran WhoCrashed, and - surprise! - it blamed all the BAD_POOL_HEADER crashes on mwac.sys (mwac+0x53D1).

 

Searching the web, I found several similar reports of mwac.sys causing the same kind of crash.  Guessing that "Malwarebytes Web Access Control" (mwac) implements Malwarebytes "Malicious Website Protection" (MWP), last Friday I disabled MWP.  I didn't have another crash then for 5 1/2 days, by far a record.

 

Last night I thought it was time to fix this for real ^_^   So I did a "clean remove" of Malwarebytes, rebooted, and did a fresh install of the latest version (2.0.2.1012), leaving MWP enabled.

 

Alas, after a bit more than a full day, I got the familiar BAD_POOL_HEADER BSOD again a little while ago.  Various log files are attached.

 

Note:  I cannot reproduce this at will.  I'm always on the web when it happens, but that's nothing special - I'm always on the web, period <_<   For now, I've disabled MWP again.

 


Link to post
Share on other sites

Hello and :welcome:

Let's try this first....

Next lets add exclusions in your Antivirus....

Please exclude the following files from your Antivirus Software (not sure what version of you are using):

Note: If using a software firewall besides the built in Windows Firewall you'll need to exclude them from it as well

For 64 bit versions of Windows Vista or Windows 7 & 8:

  • C:\Program Files (x86)\Malwarebytes Anti-Malware\mbam.exe
  • C:\Program Files (x86)\Malwarebytes Anti-Malware\mbamdor.exe
  • C:\Program Files (x86)\Malwarebytes Anti-Malware\mbampt.exe
  • C:\Program Files (x86)\Malwarebytes Anti-Malware\mbamservice.exe
  • C:\Program Files (x86)\Malwarebytes Anti-Malware \mbamscheduler.exe
  • Note: If using a software firewall besides the built in Windows Firewall you'll need to exclude MBAM.EXE and MBAMSERVICE.EXE from it as well

    Note: Once that's done, please make sure that if either of those programs has any sort of web filter, that you add the following as a trusted site:

    data-cdn.mbamupdates.com
    The FAQ contains examples of setting file exclusions for some known AV products

    Also as a side note... I noticed you have some software from IObit..

    The company behind this product was found to be stealing Malwarebytes database.

    Personally I would not trust installing any software from a company that resorts to stealing someone's technology to sell their product.

    Please see the following links and make up your own mind if you want to keep this on your system. If needed we can help you remove it.

    Please post back and let us know how it went.

    Thank You,

    Firefox

Link to post
Share on other sites

Hi, Firefox  :)

 

About "Let's try this first....", I did all those before my first post.  If I had not, you would not have been able to comment on the content of my log files  ;) .

 

About AV:

  • It's F-Secure, but obscured because my ISP (Charter Communications) slapped their branding on it.
  • I added the 5 programs you listed to F-Secure's exclusion list.
  • But this is a 32-bit system, so there's no " (x86)" in the file paths here.
  • It does not have a web filter, so I didn't add "data-cdn.mbamupdates.com" to anything.  The only web filtering I do, apart from Malwarbytes, is via editing the HOSTS file (and you can see three obscure sites I added to that in one of the log files).

About IOBit:  nasty!  I didn't know that - thanks.  I'm only using their free disk defragmenter, and will look for another.

So, in all, the only relevant actionable item I took from this was adding 5 Malwarebytes programs to F-Secure's exclusion list.  I enabled Malicious Website Protection again, and will post results later (probably within 2 days if I get another BSOD - which seems most likely).

Thanks!

Link to post
Share on other sites

This time it only took a few hours to get the next BSOD after re-enabling Malicious Website Protection  :( .  Of course I've disabled MWP again.  This is what WhoCrashed has to say about it - not much, but the same as all the other crashes:
 

On Fri 5/30/2014 9:21:28 PM GMT your computer crashedcrash dump file: C:\Windows\Minidump\Mini053014-01.dmpThis was probably caused by the following module: mwac.sys (mwac+0x53D1)Bugcheck code: 0x19 (0x20, 0xFFFFFFFF89129458, 0xFFFFFFFF89129470, 0x8030008)Error: BAD_POOL_HEADERfile path: C:\Windows\system32\drivers\mwac.sysproduct: Malwarebytes Web Access Controlcompany: Malwarebytes Corporationdescription: Malwarebytes Web Access ControlBug check description: This indicates that a pool header is corrupt.This appears to be a typical software driver bug and is not likely to be caused by a hardware problem. This might be a case of memory corruption. More often memory corruption happens because of software errors in buggy drivers, not because of faulty RAM modules.A third party driver was identified as the probable root cause of this system error. It is suggested you look for an update for the following driver: mwac.sys (Malwarebytes Web Access Control, Malwarebytes Corporation).Google query: Malwarebytes Corporation BAD_POOL_HEADER

I'll attach the crash file - seems to me what this really needs is someone with access to the source code to think about what it's doing at this point.  Note that I changed the filename extension from ".dmp" to ".txt" - the forum software wouldn't let me attach a .dmp file.Mini053014-01.txt

Link to post
Share on other sites

I only recommended the starting point to make sure we had a clean install, as you had not mentioned that. It has worked for many users. Thanks for confirming that you did do that. As for the x86 path, your right it will not exist if you are using a 32bit OS. I did not catch that, nor did I notice that, but looks like you figure out where the files were, good job.

As for the logs, and the BSOD, we now have to wait for someone on staff/development team to review them. Thanks

Link to post
Share on other sites

Firefox, no problem here  :), and I hope you realize I didn't mean to chide you by anything I wrote, I just wanted to be very clear about how what I've done does, and doesn't, exactly match what you asked for.

For more info, here are what look to be very similar reports:

http://answers.microsoft.com/en-us/windows/forum/windows_7-system/bsod-badpoolheader/48206536-70bc-477b-8740-668f853a9e31

 

http://www.pchelpforum.com/threads/8-bsods-in-past-10-days.167457/

 

If you plug my .dmp file into a kernel debugger, the stack trace is very similar - but not identical - to the stack traces in the two messages above.

I'm afraid this may be hard to fix: by the time corruption is detected, the code that caused the corruption may be long gone  :(.

 

So I'll emphasize what seem to me to be the best 4 clues we have at this point:
 

  1. BSODs are frequent (1 or 2 per day, but averaging closer to 1) with Malicious Website Protection enabled.
  2. BSODs never occur with Malicious Website Protection disabled.
  3. The failure is always the same:  a BAD_POOL_HEADER BSOD, with very similar stack traces in the .dmp files.
  4. The stack traces always involve kernel routines mucking with IP packets, which is consistent with that the crashes always occur with MalwareBytes functionality (i.e, MWP) concerned with web access.
Link to post
Share on other sites

  • Staff

Hello anachromat,

 

Thank you for your cooperation with us.

 

I will alert the development team and work with them to find a solution to this.  Until then, could you also give us the full MEMORY.DMP file that should be generated for the last BSOD your system had?

  • C:\Windows\MEMORY.DMP

Also, thank you Firefox for your assistance! :D

Link to post
Share on other sites

Jekko, happy to give the file, but how?  The raw MEMORY.DMP is over 250 million bytes, and is still over 83 million bytes in .zip format:
 

C:\Users\Tim\Uploads>dir mem*...05/30/2014  04:22 PM       265,674,426 MEMORY.DMP06/02/2014  09:19 PM        83,604,298 MEMORY.zip

 

Link to post
Share on other sites

anachromat here is how you can get the files to them...

Upload File(s) to WeTransfer:

  • Visit WeTransfer.com
  • Click on I Agree

    4ENbg3P.png

  • Click on the icon on the lower left indicated in the below image

    qKOjzXD.png

  • Select the Link option

    Cyzhcx1.png

  • Click on +Add Files

    CvZMyrC.png

  • Browse to the location of the file and double-click on it or click once on it and select Open

    S5Ty834.png

  • Click on Transfer

    8eYfZGi.png

  • Once the transfer completes, click on Copy link

    fkb0tkR.png

  • Once you receive the Copied! message as indicated below, paste the link into your next reply

    ndpEstA.png

Link to post
Share on other sites

  • Staff

Anachromat,

 

Thank you very much for the full memory dump.  I have alerted our development team and we will do our best to research and alleviate the issue.  We will post back here or in a PM when we have fixed the issue.  You have been a great help!  :lol:

Link to post
Share on other sites

Hi, Jon.  FYI,

 

  • With Malicious Website Protection disabled (but Malware Protection still enabled), the system has continued to run without any problems.
  • I realize that corruption problems can be very hard to debug, and there may simply not be any way to determine the true cause from static dump files.  So if the developers would like, I'm happy to do just about anything to help - for example, re-enable MWP in a special debug build of the product, and/or run the Windows verifier.exe on some set of drivers.  Just ask  :).
Link to post
Share on other sites

Possible clue:  I ran the Windows verifier.exe and asked it to check (only) mwac.sys, using its "standard" checks.  Also enabled the Malwarebytes setting to "Enable Malicious Website Protection when Malwarebytes Anti-Malware starts".

After making those changes, it was impossible to boot the computer normally:  got a BSOD when attempting to reboot, during some (unknown to me) phase of startup.  The dump file blamed mwac.sys.  Tried it twice, same result.

So I booted into Safe mode instead, and undid the changes (told verifier.exe to stop checking anything, and told Malwarebytes not to enable MWP on startup).  Now everything is fine again.

Best use of verifier.exe requires having two machines, with one running a debugger connected to the failing machine.  Alas, I'm not set up for that.

But if what I tried also fails on your machines, it's an easy and seemingly reliable way to provoke an "immediate" BSOD related to mwac.sys.  But also unknown whether that kind of BSOD is related to the kind of BSOD this bug report is about.

Link to post
Share on other sites

More fun with verifier.exe  ^_^

 

The boot-time deaths were IRQL_NOT_LESS_OR_EQUAL BSODs.  I figured they were irrelevant to my real problem, since (1) my "real" BSODs were all BAD_POOLHEADERs, and (2) I never got a BSOD while booting up (it always took hours to over a day before a BAD_POOL_HEADER BSOD).

So my guess is that while mwac.sys may be doing something dicey at startup to trigger IRQL_NOT_LESS_OR_EQUAL, and that would be a bug, it's not a relevant bug.

So I tried again, this time disabling verifier.exe's "Force IRQL checking" default (in the "standard" flag set).

Much better!  Now the machine booted fine.  But it died very soon after the first time I opened a browser.  Here's what WhoCrashed said about it:
 

On Sat 6/7/2014 4:17:37 AM GMT your computer crashedcrash dump file: C:\Windows\memory.dmpThis was probably caused by the following module: mwac.sys (mwac+0x5A9C)Bugcheck code: 0xC4 (0x0, 0x2, 0x200, 0x0)Error: DRIVER_VERIFIER_DETECTED_VIOLATIONfile path: C:\Windows\system32\drivers\mwac.sysproduct: Malwarebytes Web Access Controlcompany: Malwarebytes Corporationdescription: Malwarebytes Web Access ControlBug check description: This is the general bug check code for fatal errors found by Driver Verifier.The driver requested a zero-byte pool allocation. This appears to be a typical software driver bug and is not likely to be caused by a hardware problem.A third party driver was identified as the probable root cause of this system error. It is suggested you look for an update for the following driver: mwac.sys (Malwarebytes Web Access Control, Malwarebytes Corporation).Google query: Malwarebytes Corporation DRIVER_VERIFIER_DETECTED_VIOLATION

Now verifier.exe was forcing mwac.sys to satisfy allocation requests from a special tagged pool, and verifier.exe checks operations on that pool for sanity.  If it's true that mwac.sys "requested a zero-byte pool allocation", that's a software error:

 

http://msdn.microsoft.com/en-us/library/windows/hardware/ff544520%28v=vs.85%29.aspx

http://msdn.microsoft.com/en-us/library/windows/hardware/ff544501%28v=vs.85%29.aspx

Note  Do not set NumberOfBytes = 0. Avoid zero-length allocations because they waste pool header space and, in many cases, indicate a potential validation issue in the calling code. For this reason, Driver Verifier flags such allocations as possible errors.

 

 

 

Alas, it's not a "smoking gun" error - it's not itself a source of corruption, but is considered to be such a Bad Idea that verifier.exe won't forgive it to go on to something more interesting.

Link to post
Share on other sites

It's very surprising to me that mwac.sys falls over left and right without even trying under the Windows Driver Verifier - that's a basic tool I assume all Windows driver developers routinely test against.  I hold Malwarebytes in high regard, so would be pretty amazed if they didn't test against it.

So is this something unique to my machine?  It's 32-bit Windows Vista Ultimate.  mwac.sys is 51,928 bytes, and its hex MD5 checksum is 799613ba73d25641402aa81b6403eff8.

For contrast, I tried installing the trial Malwarebytes on an unrelated 64-bit Windows 7 Professional box.  Entirely different experience:  on that box, running mwac.sys under verifier.exe (with "standard" flags) has shown no problems at all so far.  BUT, it's obviously a different file than on the 32-bit Vista box:  on the Win7 box, mwac.sys is 63,704 bytes.

So is this unique to me?  Unique to 32-bit boxes?  Unique to Vista?  Puzzling  :(.

Note:  while I'm keen to know, please don't anyone else try running verifier.exe unless you're very comfortable with command lines and possibly needing to restore your whole system from a backup (it's possible - although unlikely - to get into a state where you can't even boot into Safe mode).

Link to post
Share on other sites

Mate,

 

I am just an amateur, but I would suggest that you update your network drivers.

Looking at the stack mwac was doing something then Windows tried to "Construct IP Header For Transport" and then free some memory.

 # ChildEBP RetAddr  00 c3516844 82ef9184 nt!KeBugCheckEx+0x1e01 c35168bc 89a9f2d2 nt!ExFreePoolWithTag+0x17f02 c3516a4c 89af1835 tcpip!IppInspectBuildHeaders+0x56703 c3516a9c b16f03d1 fwpkclnt!FwpsConstructIpHeaderForTransportPacket0+0x161WARNING: Stack unwind information not available. Following frames may be wrong.04 c3516af0 b16f076e mwac+0x53d105 c3516b0c b16f06d7 mwac+0x576e06 c3516b38 b16f0e34 mwac+0x56d707 c3516b54 b16ee833 mwac+0x5e3408 c3516b6c b16edea4 mwac+0x383309 c3516bc0 b16edad2 mwac+0x2ea40a c3516be0 b16ee6f0 mwac+0x2ad20b c3516bfc b16f338a mwac+0x36f00c c3516c1c b16f2293 mwac+0x838a0d c3516c2c 82e5097a mwac+0x72930e c3516c44 83052e25 nt!IofCallDriver+0x630f c3516c64 830535ca nt!IopSynchronousServiceTail+0x1d910 c3516d00 83054694 nt!IopXxxControlFile+0x6b711 c3516d34 82e56c96 nt!NtDeviceIoControlFile+0x2a12 c3516d34 77da5d14 nt!KiSystemServicePostCall13 13faf798 00000000 0x77da5d14

 

 

If your going to run Driver Verifier, then perhaps Select:

---- Special Pool
---- Pool Tracking

 

In regards to the  IRQL_NOT_LESS_OR_EQUAL that could be another problem. Driver Verifier makes sure that Drivers are behaving correctly. So if a driver is poorly written then it will trip Driver Verifier.

 

Hope this helps.

Link to post
Share on other sites

@aparsons, yup, I agree the IRQL death is almost certainly unrelated.  Unfortunately, under Driver Verifier (with "Force IRQL checking" enabled), it happens every time during bootup on my Vista box.  That's pretty bizarre.  I'd suspect my mwac.sys is corrupt, but Googling on its MD5 checksum shows that mwac.sys with the same checksum is well-known.

My network drivers are up to date.  If you look up, I posted links to two other reports of BAD_POOL_HEADER mwac.sys deaths with very similar stack traces.  There's also another report with a similar stack trace currently active on this forum.

I already ran Driver Verifier with 
"Force IRQL checking" disabled (and, yes, "Special Pool" and "Pool Tracking" were left enabled then as part of the standard flags).  That died with the DRIVER_VERIFIER_DETECTED_VIOLATION error reported a few messages up; and a complaint that mwac.sys "requested a zero-byte pool allocation".  I can't get beyond that on the Vista box.

None of these Driver Verifier deaths occurred on the 64-bit Win7 box, though.

The most likely cause of a bad pool header is a "bad store".  In C terms, say a pointer-to-character variable p, pointing to N allocated bytes, followed by a store to (say) p[N] or (far less common) p[-1].  That's out of bounds, but normally the hardware won't complain about it.  And it's a common-as-mud kind of error in C code.  Alas, by the time the bad store is detected (in this case by a kernel routine releasing allocated memory and noticing that it's internal bookkeeping memory has values that make no sense), the code that did the bad store may be long gone.  In theory, it may have happened days ago  ^_^.

Driver Verifier's Special Pool option plays tricks with alignment and page-level hardware protection in such a way that there's a good chance of catching a bad store at the instant it happens (instead of an arbitrarily long time later, when the memory-freeing routine notices it).  But with pool checking enabled, on my Vista box Driver Verifier causes system death long before any of that (due to the mwac.sys zero-byte pool allocation explained earlier).

So I don't think there's anything more I can do on my own with this.  The multiple similar stack traces from multiple machines don't "prove" mwac.sys is doing a wild store, but they're strongly suggestive.  Driver Verifier could tell us a lot more, but mwac.sys dies on my Vista box then dies far too early to get to anything probably related.

Link to post
Share on other sites

Mate,

 

Just to cover bases there is a Intel Driver update utility HERE. You didn't state whether or not you had checked the Intel web site or not. So I apologize if you already had.

 

The only crash dump file that I had access to was in your post number 4. It didn't have Special Pool or Pool tracking enabled. (See Below)

Verify Flags Level 0x00000000  STANDARD FLAGS:    [X] (0x00000000) Automatic Checks    [ ] (0x00000001) Special pool    [ ] (0x00000002) Force IRQL checking    [ ] (0x00000008) Pool tracking    [ ] (0x00000010) I/O verification    [ ] (0x00000020) Deadlock detection    [ ] (0x00000080) DMA checking    [ ] (0x00000100) Security checks    [ ] (0x00000800) Miscellaneous checks  ADDITIONAL FLAGS:    [ ] (0x00000004) Randomized low resources simulation    [ ] (0x00000040) Enhanced I/O checking    [ ] (0x00000200) Force pending I/O requests    [ ] (0x00000400) IRP logging    [X] Indicates flag is enabled  

If Special Pool was enabled we *might* have found the culprit, because verifier additionally checks the padding that is used when an allocation is rounded up to an even page boundary, that allocation is then surrounded by Invalid pages.

Do you have a crash dump with just Special Pool enabled?

If does work it would be worth tagging all non-microsoft drivers.

I can't see what, if any, drivers were tagged because it is a mini kernel dump.

 

To be fair testing on another 64bit box is not a real help, unless it was a 64 bit version of your Gateway FX530.

 

I don't have access to your IRQL crashes or even the one where Special Pool was enabled, unless you want to make them available, so I can't comment on those.

 

Another possible problem is that you appear to have multiple AV / Malware products installed. (F-Secure, Spybot - Search & Destroy 2, McAfee and of course Malwarebytes.)

 

As a suggestion, you could uninstall them and just use Microsoft Windows Defender and Malwarebytes Pro just to see if the problem persists. Sometime some AV / Malware products get over zealous.

 

*if* the Dev's find that it is Driver related, you might not be able to get a fix because your hardware is now 7 years old and some vendors may have marked your hardware end-of-life.

Link to post
Share on other sites

:@aparsons, I have never posted any dump file here obtained when Driver Verifier was running.  The topic of the bug report is BAD_POOL_HEADER BSODs running Malwarebytes normally with Malicious Website Protection enabled, so the only dump files I've posted here (the mini-dump, and a full MEMORY.DMP in a private message to Jekko) are concerned with only that.

I started trying Driver Verifier on my own, hoping to get more clues.  That has not been successful  <_<.  What I would like to know about that is whether other people also find the same version of mwac.sys dying left and right under Driver Verifier.  The only reason I tried it on a 64-bit box too is that's the only other machine I had easy access to, and was so surprised by the bad behavior on the Vista box.  If someone from Malwarebytes says "wow!  that's sure unexpected! let's dig into that!", then I'll open a different topic about that and attach all the Drivier Verifier dump files they can download  :lol: But before then, seems to me it's just another distraction (I can generate stack traces from dump files and read them, and these really aren't interesting without access to mwac.sys's source code).

Network drivers:  the Intel app identifies the relevant component, but then punts with "your operating system is not supported by this utility".  So it's a manual search.  Best I can tell, they're up to date.  But I have no reason to suspect network drivers anyway.  Yes, code mucking with the IP stack shows up in the stack trace from my mini-dump, but that's not surprising:  Malicious Website Protection is all about intercepting web access, and this box has been running for years without a hint of any network problems.

AV/Malware products:  F-Secure is my AV, and Malwarebytes is my anti-malware app,  Spybot is inactive:  I don't use its proxy or its TeaTimer, and no Spybot processes are running.  About once a month I use it to run a manual scan.  McAfee is really "McAfee Security Scan Plus", some freebie I have because I forgot to uncheck a box when installing a Java update - LOL  :).   It runs a very brief (about 2 minutes) system scan once a week, but is otherwise inactive (and no McAfee process are running either). Ya, I'll uninstall it, but I'd bet a great deal it's irrelevant to the problem at hand.

As to the age of the machine, it is getting old.  Too bad, really - it's been well cared for and has been extremely stable.  In fact, I've had more problems with Malicious Website Protection over the past few months than from all other causes combined over the life of the machine.

I'm just surprised nobody has suggested I do a RAM test yet  ^_^ (LOL - but I already did).

 

Take care  :)

Link to post
Share on other sites

Oops, forgot to address this part:
 

 

...

 

If Special Pool was enabled we *might* have found the culprit, because verifier additionally checks the padding that is used when an allocation is rounded up to an even page boundary, that allocation is then surrounded by Invalid pages.

 

In case this wasn't clear, I can't get anywhere on my Vista box with Special Pool enabled, because it quickly crashes complaining that mwac.sys is requesting a 0-byte pool allocation.  While I didn't spell this out before, that happens even with the tiny flag set "/flags 9" (only Special Pool and Pool Tracking enabled).  This can't be the fault of any other software on the box:  Driver Verifier is complaining about an allocation request made directly by mwac.sys.

Link to post
Share on other sites

..., that happens even with the tiny flag set "/flags 9" (only Special Pool and Pool Tracking enabled).  This can't be the fault of any other software on the box:  Driver Verifier is complaining about an allocation request made directly by mwac.sys.

 

@aparsons, just to flesh one of these out, "/flags 1" (enable only Special Pool) is enough to kill mwac.sys quickly on my Vista box.  This is what I did:

- Enabled the Malwarebytes Malicious Website Protection (I normally have it off now, because I don't like random BSODs  ;)).

- Ran this from an admin prompt:

verifier /volatile /flags 1 /adddriver mwac.sys

- Opened Firefox and logged into Facebook.

BOOM!  That's all it usually takes.  I'll attach the mini-dump for your amusement, but noting that it almost certainly has nothing to do with the real topic here (note that I changed the .dmp extension to .txt, because the forum software doesn't allow attaching .dmp files - just rename it).

 

Mini060914-01.txt

Link to post
Share on other sites

mwac.sys BSOD (0xC9  DRIVER_VERIFIER_IOMANAGER_VIOLATION) under verifier.exe checks on my Windows 7 x64 laptop. BSODs quite early in the boot sequence, no memory.dmp file is able to be successfully written sadly. Doesn't seem to matter what flags I try. Was just using standard flags at first then tried w/o Special Pool and Force IRQL Checking, makes no difference for me.

Link to post
Share on other sites

Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
Back to top
×
×
  • Create New...

Important Information

This site uses cookies - We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.