That box sits behind hardware and software firewalls and hosts a CCTV system, Media Server and VirtualBox Windows 8.1 Workstation instances.
It also has Norton Internet Security Suite and (until recently) MalwareBytes, so it requires a deliberate effort (or crass stupidity on my part) to experience unwanted access to that machine.
Its been running just great ... for months, apart from an instance a few months back when an automatic update of MalwareBytes forced me to turn off the MalwareBytes 'Web Protection' module because it started causing random blocking of the IP CCTV cameras. That lasted several months til a recent update seemed to fix that. So for probably a couple of months now all has been good.
Seven days ago I came down in the morning and started my VBox VM workstation and barely got past the Windows login screen before the things started to go into a progressively frozen state.
First the VM UI froze, then the host network failed, then several monitoring UI's I have running on the host froze, then the host explorer froze.
I could do nothing but power off.
Powering back on, everything seemed to be starting ok but part way through, everything again started to gradually freeze ... requiring another POR.
Long, long, long, long, long, story short (days of effort and God knows how many POR's, image restores and tinkering with software), which involved;
- swapping out the host system SSD drive
- restoring prior backups of both the host and VM partitions
- thank god for EASEUS TodoⓇ and a bootable USB recovery image !!
- substituting all 4 sticks of the 32GB of RAM
- re-seating everything I could get my hands on
- trawling through VBox, Windows, Malware and NISS logs, all of which had not a single clue as to what was going wrong
- exhaustive memory tests
- disk SMART and access checking
- monitored system voltages and temperatures
- scanned for corrupt files, malware and rootkits
I also initially thought it was related to when one particular VBox VM was loaded, because I could load other VMs and the host would not lock up
... but then ...
I gradually discovered that it was more sinister than that. I could actually crash the host without even loading VBox!
Then I noticed it was sporadically related to whenever applications attempted to access disk resources ... any disk, any partition, ... well almost ... but NOT the CCTV storage partition ... why not ??
None of the logs showed any clues AT ALL!
There had been no NISS, Windows or VirtualBox updates, besides which, the restores to earlier images should have eliminated that possibility.
What I missed was that there must have been a change (update perhaps) to MalwareBytes.
Remember that earlier in the year I had to disable the MWB Web Protection component to ensure CCTV camera feed stability?
Well I read a very old forum article that mentioned someone had to disable the Ransomware component to stop their Win7 x64 machine from constantly freezing.
"Let me try shutting down MWB" ... thinks I.
So I quit MWB and disabled the service.
BINGO!
Problem 100% resolved! (and has been for over a week now)
It also explained why accessing the CCTV disk storage didn't crash the system ... that partition is excluded from MWB interaction.
I'm still unclear why I could run some of my VM's ... maybe because some of them don't access the Host disk storage apart from loading their VM images.
I have yet to post this to MalwareBytes for their investigation, as I very much want to have the added assurance of MalwareBytes doing its specialisation in concert with the other protections in place.
If you have questions or your own helpful comments, please do post a comment to this article.