96
A Kaseya Breach Victim Speaks
View this email in your browser
The weekly newsletter of the Business of Tech, giving you new insights into the world of IT service delivery.
Looking for stories from the podcast stories? Check out the pod itself on Apple Podcasts, Spotify, or daily in your inbox. Stories are available to everyone for five days,and Patreon supporters forever.
Was this forwarded to you? Join the list!
A Kaseya Breach Victim Speaks
I’ve said it before, and I’ll say it again: no one thinks a cybersecurity break will happen to them until it does. And I’m not the only one sounding the alarms. We’ve heard from experts in the field time and time again that a breach isn’t a matter of if – it’s a matter of when.
So, what actually happens when you’re the MSP caught in the middle of a breach? That happened to Robert Cioffi, a recent guest on a bonus episode of the Business of Tech. When the infamous Kaseya breach went down in 2021, it was up to him as the co-founder and CTO of Progressive Computing to juggle the needs of clients, lawyers, and the folks at Kaseya.
He walked me through his experience on the ground, and his story is something all of us should take seriously. Here’s a rundown (and a cautionary tale) of what happened from the MSP POV:
A Refresh on Kaseya
If you don’t remember the full story of the ransomware attack on software provider Kaseya, here’s what you need to know:
On July 2nd, 2021, ransomware attackers leveraged a vulnerability in Kaseya’s VSA software, targeting multiple MSPs and their customers.
They infected all of the victims via an automatic software update that delivered the REvil ransomware package.
The package then encrypted all the systems that it touched across the networks impacting roughly 50 MSPs and between 800 and 1500 of their customers before Kaseya shut down their cloud and issued a shutdown directive for all on-premise systems.
Kaseya’s internal team worked with security experts to determine what happened, alerting law enforcement and several government agencies. They teamed up with the FBI and CISA during the process.
Backers demanded a ransom of $70 million to offer blanket decryption for all victims, and $5 million each to owners like you.
Kaseya did not pay the ransom, and in late July, they released a universal key that decrypted all systems.
Two foreign nationals were arrested on October 8th, and one faces charges in Texas.
But that’s just a broad look at the story. What about folks on the ground?
Cioffi’s First Reaction
I asked Cioffi to walk us through his own version of events. He described the MSP leader’s worst nightmare: by the end of the breach, the hackers pushed the script out to 2,500 endpoints across 80 of his customers in 200 physical locations.
The day of the breach started out quite nicely for Cioffi – weather-wise, he described it as one of the best ten days of the year. The news broke, of course, when he was on lunch (which was weird, because he never usually breaks around noon). While in the office kitchen, he noticed Jay, his ops director, was pacing around while taking a phone call. Cioffi knew something was wrong just by the way Jay was walking. In fact, something was so off with Jay’s body language that Cioffi actually thought somebody had died.
When Jay finally told Cioffi that their customers were undergoing a ransomware attack, he froze:
“It’s almost too impossible for your brain to kind of wrap around. It’s almost like saying, hey, go count the amount of grains of sand there are on a beach. Like, where do you even start, right? Like, how do you even tackle a problem like that?”
As Jay listed off the 10 of their top 15 accounts going down, he could almost picture the revenue and decades-long relationships collapsing like dominoes. He knew immediately that it was an RMM attack – how else could unconnected clients all be victims – but all he could do was stare at their fading desktop icons (a telltale sign something is seriously wrong) and start making phone calls.
Cioffi wants listeners to know that, in the moment, it’s not enough to have a loose incident response plan. They had small things in place that semi-prepared them for this, but not to the extent that they needed to:
“Even if you have all of those things completely documented and buttoned up, there was this wave of psychological emotion that grips you that pulls you to a place, a very, very dark place. Having those processes will increase your survivability, It certainly doesn’t prepare you for the handling of that emotional state.”
Cioffi’s Timeline
So, how long does it actually take to make the first moves during a crisis like this? While shaking off the initial shock, Cioffi and his team spent the first couple of hours calling experts in their network and industry friends with relevant knowledge.
However, they delayed one call for a bit too long: their cyber liability insurance team. When they eventually realized they needed to make that call, they were glad they did – it put them in contact with break counsel (essentially lawyers) who understood the legalities of the next steps. They helped formulate a communication plan for customers so they weren’t passing along random or inaccurate information.
Cioffi noted that he did find himself pushing back on the lawyers’ advice out of a desire to be more transparent and authentic with customers, but he still urges listeners to listen to the council. Specific to customer communication, Cioffi still shared this:
“I wish that we had more planning done, at least a one-page sheet of flowcharts, something that would have given us some map to say, you know, here you are.”
Because they learned the lesson the hard way that even if you have nothing new to say, you should still let customers know that you plan to communicate with them soon. They also had to send out a lot of messages reminding customers to leave systems alone and hold off on attempting solo recovery.
As for communication with Kaseya, Cioffi confirmed that even in the thick of the crisis, he thinks their team did a good job. Within a few days, Cioffi’s people reached about 95% recovery, and Kaseya released the key just a couple weeks later.
A Look at Costs
We see a lot of data about the average cost of breaches, but what does that actually look like for the MSP? In Cioffi’s case, the biggest financial hurdle was customers who withheld payment or slowed payments down (they had to chase money for a while). It made payroll harder, MRR took a hit, and they had to wait for roughly $30,000 of backpay from a client who terminated their agreement.
A Note on Community
Cioffi credits Progressive’s financial survival to the many IT community members who stepped up to support his team. Beyond the many experts who took his calls that first day, he had peers from the IT Nation Evolve community jump into action. MSP owners literally showed up at Cioffi’s doorstep with entire teams – from leaders to engineers, volunteers were integrating themselves into Progressive’s recovery efforts for weeks. A friend of a friend even flew in from Iowa to help for fifteen whole days. By the end, about 27 MSPs and subcontractors volunteered their time and talent.
After seeing the community step up to help, Cioffi launched www.MSP911.org – a resource page for MSPs like himself who need help thinking clearly, finding support, and executing next steps with confidence.
To wrap up our convo, Cioffi hammered home this point:
“There is no shame in victimhood. And there is strength in vulnerability. So if you are ever a victim, you must open your mouth and say something. Don’t feel that you are exposing yourself as being weak. No, you are actually showing your strength by saying, I’m a victim and I need some help, right?”
I hope Cioffi’s story gets your gears turning on what you would do in his situation. If you want to get involved in his mission, head to www.MSP911.org to register as an MSP first responder. Or, worst case scenario, you know where to go if the worst happens.
That’s it for this week! As always, I’m available to connect if you’d like to share a story, question, or whatever else is on your mind.
More from MSP Radio
Missed Things?
How about our latest videos to catch you up?
The Daily Podcast available as videos
The Need for an AI Operating System: A Conversation with Alexander De Ritter
The AI Dilemma: A Conversation with Juliette Powell and Art Kleiner
Customer Experience and the Importance of Engagement with Gerwai Todd
The Future of Distribution: Predictions and Perspectives from Industry Leaders
Exploring Autonomous Project Monitoring and Management with Mike Psenka
Want the Daily News?
All the stories from the daily Business of Tech Podcast are available in the daily digest, and stories are available to everyone for the first five days, and Patreon supporters forever. Catch the audio of the show anytime on Apple Podcasts, Spotify, YouTube, or wherever you find podcasts. Links at businessof.tech
Copyright © 2024 MSP Radio, All rights reserved.
Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.