According to the University of Texas, 94% of companies that suffer catastrophic data loss do not survive. Further, 93% of companies that lose access to their data for 10 days or more will go out of business within 12 months, according to the National Archives & Records Administration in Washington.
This is a true story about a company that avoided that fate.
Anatomy of the Attack
It’s early morning and the IT team is looking forward to the upcoming holiday weekend. It’s been a good week. A quiet week. Exactly how the team likes it. No problems or distractions.
A call to the help desk on the East Coast is the first sign of potential trouble. Something isn’t right. The caller can’t open their files. That call is quickly followed up by another as users in another region begin to report issues with remote access systems. They can’t log into core servers. Other systems appear to be completely unavailable.
Instantly, systems administrators across the IT team swing into action. Almost immediately, they notice that their virtual environment is at maximum capacity for no apparent reason. A feeling of dread begins to sink in. Most of these systems are inaccessible as well.
The team continues to move quickly, troubleshooting with their technology vendors to identify what might be causing this activity. And then, in an instant, it all became clear.
A user created a support ticket reporting that multiple PDF files that were fine yesterday, will no longer open. Curiously, they didn’t seem to be PDFs any more. The file extension for every file was changed to .CURATOR.
There would be no 4-day holiday weekend. No celebrations. No extra time off. In an instant, they knew that this was a fight for their life. This wasn’t some faceless, nameless ransomware attack that they had only read about — this was happening to their company.
At this moment, they are in the cross hairs of a sophisticated team of cyber-criminals willing to do anything to extort the most money from them, even if it means destroying every bit of data.
Curator has a relatively low level of distribution compared to other ransomware variants, but it causes a high level of damage. Its signature is the .CURATOR file extension, and the inclusion of a text file in every infected directory that contains instructions for contacting the attackers.
To date, there are no known tools available to repair infected files other than the decrypter provided by attackers.
It Was Sobering
For a minute, the team looked at each other. They understood that they were fighting for their data. Their company. And their jobs. Still huddled in a crowded conference room, the IT team made a call to Panzura’s global services team. They needed urgent help. And they needed it immediately. This was a race against the clock.
Thankfully, a seasoned Support Engineer answered the call. Like the others on the Panzura cloud security team, he had seen more than a few ransomware attacks — and knew exactly what to do.
Within minutes, Panzura helped the customer’s IT team disable write access to the node perpetuating the attack by cutting off its Windows CIFS/SMB license. This automatically disabled communication with the affected filer and forced all locations into read-only mode, preventing further contamination to the file network.
To save time later in the clean-up process, the Panzura team disabled user snapshots, to stop the system from creating automatic restore points of encrypted data that would be deleted anyway.
Less than 20 minutes later, the ransomware attack was contained to local systems, preventing any further files from being encrypted and giving the firm’s IT team breathing room to identify the source and magnitude of the attack.
During a ransomware attack, every moment counts. Machine processes like these can encrypt files many times faster than a human can respond...Panzura gives you the ability to stop the attack’s effect on your stored files without having to identify the cause first.
At the same time, any encrypted files that have reached your cloud storage have been written as new data, leaving existing files completely untouched. That means all your data is there to be restored, and you can focus your efforts on stopping the attack...
Evaluating and Assessing the Damage
After containing the ransomware, the firm’s IT team began a thorough assessment of all systems — taking stock of their situation. Processor spikes continued to make it difficult to access their virtual environment, but with Panzura’s continued help, they identified the server causing most of the activity. They disconnected it and rebooted the entire virtual environment.
It worked. That reboot created enough bandwidth to be able to log in.
Working systematically through each virtual server, the IT team disconnected and assessed the damage of each node on their environment. This took the remainder of the day.
Once complete, they knew the full extent of the comprehensive attack they had faced. Thousands of their files had been encrypted, and hundreds of employees across the country were affected. Projects were at a standstill.
It became painfully obvious that recovery was going to require a staged approach. They would need to prioritize access to production files, as that would allow employees to get back to work, and projects to resume.
Cleaning and Planning for Recovery
Over the following week, the firm’s team focused on identifying all the servers affected by encryption, evaluating their previous and current states to track the potential root cause, and establishing steps to safely bring systems online.
During this discovery phase, they worked with a cyber-security remediation firm to set up new endpoint protection software and provide monitoring and mitigation services against the attack.
All the while this remediation was going on, users could access data on individual Panzura deployments and stay productive in the face of a ransomware attack.
As one subset of the firm’s team worked on system reviews, another group worked with Panzura’s global services experts to get critical access to clean, current project data. Panzura’s cloud security experts rapidly tested and deployed a custom script that immediately detected the creation of any new .CURATOR files, and prohibited them from being written to any filer.
To ensure that no encrypted files would slip through the cracks, the global services team implemented a counter to track attempts to write this file type. This enabled the system admins to monitor for new suspicious activity as systems were being brought online.
Within hours, the number of .CURATOR files being generated slowed. The IT team could see that they were getting the attack under control. But they weren’t done yet. They now had to clean up the mess this wide-spread attack had made for them.
Identifying Affected Files
Using the Data Services layer embedded in the Panzura global file system, the Panzura team provided the firm with a complete list of all encrypted files affected, and their location and creation dates. This analysis showed that every file partition, for every office, had been affected by the attack, and that data recovery would require methodical, careful planning.
Moving purposefully, the Panzura support team got to work helping the firm restore files to their unencrypted state, using a triage system to ensure that the most critical folders were restored first.
To minimize any data loss, the IT team utilized Panzura analytics for each file to pinpoint exactly when files and folders were encrypted. Using that information, they were able to restore the last good version of each file before it was encrypted by the attack.
With complete file system snapshots run every 60 minutes and user snapshots every 60 seconds, the IT team could assure the firm’s leadership that they were protected.
Restoring the data would be a different process — unlike restoring from a traditional backup solution. Since the Panzura solution catalogs infinite changes to every file and stores that data in the metadata, the IT team simply needed to roll back each file to the best version of the file. No need to transport data around the globe. No need for expensive egress fees. Just a tweak to the metadata. As a result, the Panzura snapshot restoration would take only a fraction of the time otherwise required from an offsite backup, and with much greater reliability.
Despite the success they were having with their users’ data, it was obvious to the IT team that they still had weeks of work ahead of them. They would now need to assess and recover the data and systems stored outside of Panzura.
And the outlook was grim.
To put it candidly, the IT Manager summed it up succinctly:
While we were able to rid ourselves of any encrypted files on the Panzura system early on, we were still going through this process with the rest of our other systems seven weeks later.
Slowing the Attack, Early Detection and Faster, Better Recovery
The inherent architecture of Panzura provides a tremendous advantage over traditional ways to store and protect data.
Early detection, identification of the key sources of the attack, and a swift mitigation response were critical to the success of the recovery of user data. No ransom was paid. No extenuated downtime was felt.
Because Panzura caches the most used files locally, a ransomware attack is limited to the files that are local. That’s because the malware has no way to know whether the file it’s currently encrypting is in the cache or not – it simply crawls directories.
When the malware encounters files that aren’t in cache, it retrieves them from the cloud store. This takes time. And it instantly creates red flags as bandwidth is monopolized.
As well, the increased movement of data out of and back into the cloud is also easy to identify and alert on, providing an early warning and saving valuable time trying to diagnose the problem. Using Panzura Data Services, many such encryption attempts are stopped within minutes — limiting the scope of damage and required recovery time.
When asked how different the situation would have been without Panzura, the IT Manager said:
We would have had tape backup and [would be stuck] swapping out tapes and recovering data now. Time to recovery is probably the biggest benefit of using Panzura.
No critical production data was lost in this attack, nor were production files unavailable for any significant period of time. Practically speaking, that meant the firm could continue to meet all deadlines and ensure that their clients were unaffected by the disruption.
A Deeper Dive
Ransomware attacks have a single purpose – to encrypt files in such a way that business decision makers believe they can only be unlocked with the attacker’s help. Typically, ransomware attacks focus on encrypting or destroying backup data and snapshots as well as primary data — with a clear goal of removing any ability for a company to restore useful data on their own and evade the ransom demand.
Panzura does not allow that to happen.
Panzura’s CloudFS global file system allows enterprises to store data in any public or private cloud, using object storage for scalability and durability. Users work on familiar files in a familiar directory or folder, but underneath that effort, Panzura turns any creation or changes to a file into object blocks that can be stored in any object store.
Using Panzura, any data in the cloud object store is immutable and cannot be overwritten. As users edit files within the file system, changes they make are synced to the cloud as new data objects.
The metadata for each file is updated with every edit, recording which object blocks are needed to form “a file” at any given time. Stored data is further protected by read-only snapshots, taken at configurable intervals.
As a result, the ransomware attack the firm experienced was not encrypting the data that Panzura had secured in their cloud storage. Instead, it was creating data that was being written to cloud storage as new objects, leaving pre-existing data untouched.
For organizations running on legacy file systems, ransomware presents a serious problem. By storing data that needs to be editable, legacy file systems are inherently vulnerable.
When attacked, they do exactly what they are designed to do, and allow files to be changed. That means recovering “clean” files is exceptionally difficult and time consuming. As well, backup processes tend to run on a scheduled basis, and often just once daily due to the resources they consume. Restoring from a backup after a disaster almost always involves some data loss, and often a considerable amount.
Popular approaches to data resilience do little to stop that data loss.
Traditionally, IT makes a copy of user data, storing that data separately from the primary data, often at another company location. For additional resilience, additional copies are stored offsite. Again, this approach to disaster recovery results in data loss – especially if the files have to be restored from tape. The significant amount of time between the backup being run and the actual time of the attack creates a gap in operational integrity.
So What Works?
This particular firm’s investment in Panzura goes far beyond the need to allow their users to collaborate in real time on the same set of data across multiple locations — a solution unique to Panzura in the world of unstructured data applications.
Smart technology that immediately moves data to wherever it needs to be, means little when faced with the type of attack witnessed here by sophisticated cyber-criminals.
Core to the resilience displayed in this quick recovery is Panzura’s data immutability that withstands high-velocity attempts to encrypt data with malware.
Coupled with the resiliency of cloud storage itself, this particular firm has data durability of at least 13 9’s — along with ransomware protection that many organizations wish they had.
That data durability made their critical data essentially impervious to an aggressive attack, minimizing disruption and maximizing their speed of recovery.
Most such stories don’t end so well.
But it’s possible with an intelligent data management system like Panzura, with a global file system that prioritizes data protection, and maintains a pristine data set that can be swiftly restored – minimizing downtime and data loss, and preserving data integrity.