According to the University of Texas, 94% of companies that suffer catastrophic data loss do not survive.
Further, 93% of companies that lose access to their data for 10 days or more will go out of business within 12 months, according to the National Archives & Records Administration in Washington.
This is a true story about a company that avoided that fate.
Anatomy of the Ransomware Attack
It’s early morning and the IT team is looking forward to the upcoming holiday weekend. It’s been a good week. A quiet week. Exactly how the team likes it. No problems or distractions.
A call to the help desk on the East Coast is the first sign of potential trouble. Something isn’t right. The caller can’t open their files. That call is quickly followed up by another as users in another region begin to report issues with remote access systems. They can’t log into core servers. Other systems appear to be completely unavailable.
Instantly, systems administrators across the IT team swing into action. Almost immediately they notice that their virtual environment is utilizing 100% of available capacity for no apparent reason. A feeling of dread begins to sink in. Most of these systems are inaccessible as well.
The team continues to move quickly, troubleshooting with their technology vendors to identify what might be causing this activity. And then, in an instant, it all became clear.
A user created a support ticket reporting that multiple PDF files that were fine yesterday, will no longer open. Curiously, they didn’t seem to be PDFs any more. The file extension for every file was changed to .CURATOR.
There would be no 4-day holiday weekend. No celebrations. No extra time off. In an instant, they knew that this was a fight for their life. This wasn’t some faceless, nameless ransomware attack that they had only read about — this was happening to their company.
At this moment, they are in the cross hairs of a sophisticated team of cyber-criminals willing to do anything to extort the most money from them, even if it means destroying every bit of data.
Curator has a relatively low level of distribution compared to other ransomware variants, but it causes a high level of damage. Its signature is the .CURATOR file extension, and the inclusion of a text file in every infected directory that contains instructions for contacting the attackers.
To date, there are no known tools available to repair infected files other than the decrypter provided by attackers.
The IT team watches as the attack begins to spread quickly. Although, it doesn’t seem to follow any discernible pattern. They know that although their firm has multiple offices that share a single global file system, each office has its own file partition to make it easier to identify key files.
Inside headquarters, senior leadership frantically huddled with their IT team to discuss their options.
In the heat of the moment, the spread of the attack looked to be random. The firm’s Information Technology Manager noticed that “There was no easily definable sequence to the attack. On some of our systems, it looked like files were being attacked in alphabetical order, but others had no apparent pattern. It was as if someone had obtained a list of directories and processed it in whatever order they received it. More dangerously, the attack went wide before it went deep. They were trying to attack every folder on our network.”
Obviously, the cyber criminals they were up against were experts at causing maximum destruction.
James Cannon, a seasoned Support Engineer, answered the call. Like the others on the Panzura cloud security team, he had seen more than a few ransomware attacks — and he knew exactly what to do.
Within minutes, Cannon helped the IT team disable write access to the filer that was spreading the attack. He did this by disabling their Windows CIFS/SMB license. This automatically disabled communication with the affected filer and forced all locations into read-only mode, preventing further contamination to the file network.
To save time later in the clean-up process, the Panzura team disabled user snapshots, to stop the system from creating automatic restore points of encrypted data that would be deleted anyway.
Less than 20 minutes later, the ransomware attack was contained to local systems, preventing any further files from being encrypted and giving the firm’s IT team breathing room to identify the source and magnitude of the attack.
Evaluating and Assessing the Damage
After containing the ransomware, the firm’s IT team began a thorough assessment of all systems — taking stock of their situation. Processor spikes continued to make it difficult to access their virtual environment, but with Panzura’s continued help, they identified the server causing most of the activity. They disconnected it and rebooted the entire virtual environment.
It worked. That reboot created enough bandwidth to be able to log in.
Working systematically through each virtual server, the IT team disconnected and assessed the damage of each node on their environment. This took the remainder of the day.
Once complete, they knew the full extent of the comprehensive attack they had faced. Thousands of their files had been encrypted, and hundreds of employees across the country were affected. Projects were at a standstill.
It became painfully obvious that recovery was going to require a staged approach. They would need to prioritize access to production files, as that would allow employees to get back to work, and projects to resume.
Cleaning and Planning for Recovery
Over the following week, the firm’s team focused on identifying all the servers affected by encryption, evaluating their previous and current states to track the potential root cause, and establishing steps to safely bring systems online.
During this discovery phase, they worked with a cyber-security remediation firm to set up new endpoint protection software and provide monitoring and mitigation services against the attack.
All the while this remediation was going on, users could access local Panzura file system data, and continue working – remaining productive.
As one subset of the firm’s team worked on system reviews, another group worked with Panzura’s global services experts to get critical access to clean, current project data. Panzura’s cloud security experts rapidly tested and deployed a custom script that immediately detected the creation of any new .CURATOR files, and prohibited them from being written to any filer.
To ensure that no encrypted files would slip through the cracks, the global services team implemented a counter to track attempts to write this file type. This enabled the system admins to monitor for new suspicious activity as systems were being brought online.
Within hours, the number of .CURATOR files being generated slowed. The IT team could see that they were getting the attack under control. But they weren’t done yet. They now had to clean up the mess this wide-spread attack had made for them.
Using the Data Services layer embedded in the Panzura global file system, the Panzura team provided the firm with a complete list of all encrypted files affected, and their location and creation dates. This analysis showed that every file partition, for every office, had been affected by the attack, and that data recovery would require methodical, careful planning.
Moving purposefully, the Panzura support team got to work helping the firm restore files to their unencrypted state, using a triage system to ensure that the most critical folders were restored first.
To minimize any data loss, the IT team utilized Panzura analytics for each file to pinpoint exactly when files and folders were encrypted. Using that information, they were able to restore the last good version of each file before it was encrypted by the attack.
With complete file system snapshots run every 60 minutes and user snapshots every 60 seconds, the IT team could assure the firm’s leadership that they were protected.
Restoring the data would be a different process — unlike restoring from a traditional backup solution. Since the Panzura solution catalogs infinite changes to every file and stores that data in the metadata, the IT team simply needed to roll back each file to the best version of the file. No need to transport data around the globe. No need for expensive egress fees. Just a tweak to the metadata.As a result, the Panzura snapshot restoration would take only a fraction of the time otherwise required from an offsite backup, and with much greater reliability.
Despite the success they were having with their users’ data, it was obvious to the IT team that they still had weeks of work ahead of them. They would now need to assess and recover the data and systems stored outside of Panzura.
And the outlook was grim.
To put it candidly, the IT Manager summed it up succinctly:
Slowing the Attack, Early Detection and Faster, Better Recovery
The inherent architecture of Panzura provides a tremendous advantage over traditional ways to store and protect data.
Early detection, identification of the key sources of the attack, and a swift mitigation response were critical to the success of the recovery of user data. No ransom was paid. No extenuated downtime was felt.
Because Panzura caches the most used files locally, a ransomware attack is limited to the files that are local. That’s because the malware has no way to know whether the file it’s currently encrypting is in the cache or not – it simply crawls directories.
When the malware encounters files that aren’t in cache, it retrieves them from the cloud store. This takes time. And it instantly creates red flags as bandwidth is monopolized.
As well, the increased movement of data out of and back into the cloud is also easy to identify and alert on, providing an early warning and saving valuable time trying to diagnose the problem. Using Panzura Data Services, many such encryption attempts are stopped within minutes — limiting the scope of damage and required recovery time.
When asked how different the situation would have been without Panzura, the IT Manager said “We would have had tape backup. To be honest, we would still be swapping out tapes and recovering data now, if that were the case. Time to recovery is probably the biggest benefit of using Panzura, as well as the amount of effort that it has taken us to recover.”
A Deeper Dive
Ransomware attacks have a single purpose – to encrypt files in such a way that business decision makers believe they can only be unlocked with the attacker’s help. Typically, ransomware attacks focus on encrypting or destroying backup data and snapshots as well as primary data — with a clear goal of removing any ability for a company to restore useful data on their own and evade the ransom demand.
Panzura does not allow that to happen.
Panzura’s global cloud file system CloudFS allows enterprises to store data in any public or private cloud, using object storage for scalability and durability. Users work on familiar files in a familiar directory or folder, but underneath that effort, Panzura turns any creation or changes to a file into object blocks that can be stored in any object store.
Using Panzura, any data in the cloud object store is immutable and cannot be overwritten. As users edit files within the file system, changes they make are synced to the cloud as new data objects.
Panzura never overwrites existing data. The metadata for each file is updated with every edit, recording which object blocks are needed to form “a file” at any given time. Stored data is further protected by read-only snapshots, taken at configurable intervals.
As a result, the ransomware attack the firm experienced was not encrypting the data that Panzura had secured in their cloud storage. Instead, it was creating data that was being written to cloud storage as new objects, leaving pre-existing data untouched.
For organizations running on legacy file systems, ransomware presents a serious problem. By storing data that needs to be editable, legacy file systems are inherently vulnerable.
When attacked, they do exactly what they are designed to do, and allow files to be changed. That means recovering “clean” files is exceptionally difficult and time consuming. As well, backup processes tend to run on a scheduled basis, and often just once daily due to the resources they consume. Restoring from a backup after a disaster almost always involves some data loss, and often a considerable amount.
Popular approaches to data resilience do little to stop that data loss.
Traditionally, IT makes a copy of user data, storing that data separately from the primary data, often at another company location. For additional resilience, additional copies are stored offsite. Again, this approach to disaster recovery results in data loss – especially if the files have to be restored from tape. The significant amount of time between the backup being run and the actual time of the attack creates a gap in operational integrity.
So What Works?
This particular firm’s investment in Panzura goes far beyond the need to allow their users to collaborate in real time on the same set of data across multiple locations — a solution unique to Panzura in the world of unstructured data applications.
Smart technology that immediately moves data to wherever it needs to be, means little when faced with the type of attack witnessed here by sophisticated cyber-criminals.
Core to the resilience displayed in this quick recovery is Panzura’s data immutability that withstands high-velocity attempts to encrypt data with malware.
Coupled with the resiliency of cloud storage itself, this particular firm has data durability of at least 13 9’s — along with ransomware protection that many organizations wish they had.
That data durability made their critical data essentially impervious to an aggressive attack, minimizing disruption and maximizing their speed of recovery.
Most such stories don’t end so well.
But it’s possible with an intelligent hybrid cloud global file system like Panzura that prioritizes data protection, and maintains a pristine data set that can be swiftly restored – minimizing downtime and data loss, and preserving data integrity.