Improvements to our Updating Process

[RubbeR DuckY]

It's been a rough week here at Malwarebytes, and I'm sure for many of you as well. We've spent the entire week focused on supporting the users affected by Monday's false positive, as well as implementing systems to prevent this type of problem from ever happening again. If you have not yet received help, please route everything to our support team so we can reach out to you -- the forums aren't an ideal place to track responses, and once you're in our helpdesk system we can help you more quickly.

With that said, I'd like to take a closer look at what we've done to prevent false positives in the future.

1. We've installed a false positive shim server. This server will have virtual machines running a wide range of different configurations and operating system versions, to mirror the range of setups our customers run. Before an update gets pushed out, it will be tested on this server, on every configuration. If a false positive is detected, it will prevent our research team from uploading a database update.

2. We've modified the tools that compress and encrypt our definition updates. The false positives on Monday were not traditional, they were caused by a corrupted file that our encryption tool did not flag. We've made immediate changes to the tool and are testing it with a roll-out date to the entire research team by the end of the week.

3. We've started hiring for our support team. While I am proud of how our support team handled the situation, they were, and still are, very overwhelmed. We realize that Malwarebytes needs to scale proportionally as a team and the support team needs more members. We're going to reach out to our community and hire additional forum members as well.

4. Phone support has been on our plate for quite some time. We've been exploring several different options and approaches. This incident has opened our eyes to how important this really is and we're taking all the steps necessary to make it happen.

We remain fully committed to providing the top quality products you expect from Malwarebytes and to earning and keeping your trust.

Marcin

[DarkSnakeKobra]

This sounds like a very good plan. The first one is the most important that would have prevented this issue. I'm glad this was taken into consideration and implemented to prevent future issues.

[Firefox]

Kudos for owning up to the mistakes and learning from them..... Thanks for putting measures in place so quickly and being open and informing the public of the changes made so we are aware what they were, puts my mind at ease knowing that something was really done and implemented.

Its great to know the you will be increasing your support team, and looking forward to PHONE support!!!

[catscomputer]

That sounds excellent Marcin! Very belt and braces. I like the sound of the shim server thing. I can't imagine how false positives on system files could bypass such a set up. This has completely put my mind at ease.

I hope things are starting to settle down at Malwarebytes now so you can all be back to 'business as usual' (perhaps after a well deserved rest!!), and put these nightmare last few days behind you. I noticed updates have still been coming through thick and fast in spite of the overwhelming demands for support and the work involved in setting things up to prevent similar occurrences.

Thanks again for your honesty and transparency throughout, and for taking the action that you have as a result. You are an awesome company and your product has just got even better with these changes. MBAM PRO will be one of the very first things going on my next lap top!

[mountaintree16]

This sounds excellent! Thank you for the update! This has completely put my mind at ease, and I feel comfortable leaving the setting on now if I want to.

Kudos to Malwarebytes, an amazing company run by amazing people

[neant]

Thank you for the update, Marcin! Really appreciate the information you gave to us.

[sunandroses]

Just been reading about 'Mondays false positives'' & thought what the heck is that? I didn't notice anything different about MB.I just do the updates often & that's that; just hope MB does it's job. I may not know malware if I do have it ! Not good ! I haven't noticed any difference in the updating process..? Actually, I'm on the forum today cause I was wondering why MB needs such frequent updating. It's always out of date.???

[shadowwar]

Malware morphs almost hourly anymore that is why we need frequest updates to stay on top of them. On the client end you will not notice any difference as far as updates go. It all happens backend. If you didnt update during the minutes the db was out then you wouldnt have been affected.

[callthatgirl]

As a follower, fan and client of Malwarebytes for many years now, I will support them through this incident. I had approximately 32 users with crashed systems on that Monday. I think was due to the fact they had their Malwarebytes installed at that time the update was released. A preventative procedure would be to have a pop up for the clients (when self installing or technician installs because techs forget to schedule the scans/updates as well ) to remind them to schedule updates and scans for reboot update. Moving forward, I am choosing them on reboot so if this happens again, I can call/email all my Mbam clients and prevent it until the update is fixed. Currently, it's set for default on just scans I believe.

[shadowwar]

callthatgirl. Thanks. This was already resolved after the immediate and short time the other db was out. Its been fixed for a while. FP Server is installed and online for us too now.

[gerardwil]

Thanks Marcin for keeping us informed and measurements taken to avoid future incidents.

[cgtracydms]

Marcin,

How about a status update as to how the new changes are working and progress toward phone support?

Thanks!