As Baltimore rebuilds from 2019 ransomware attack, is $10 million for a cure better than prevention?

What does $10 million, committed for curing a city’s ransomware affliction, actually buy you?

AJ Nash, VP of intelligence at cybersecurity company ZeroFOX, says it depends on how you want to handle the situation. Firms can negotiate the ransom. For Baltimore, trying to recover from the devastating 2019 ransomware attack, that ransom was $76,000 worth of Bitcoin.

Beyond these cyber companies, intelligence organizations can help a victim better understand the cyberattack and the organization claiming responsibility. Contractors can come in and attempt to decrypt data, as well as restore backups and networks.

These are all specialties and there are very few organizations, if any, that can provide all the different services that are necessary during an attack. Perhaps that helps explain why the city of Baltimore ultimately spent $10 million purely on IT recovery.

Documents that Technical.ly obtained from Baltimore’s city government, via a FOIA request, paint a picture of how this money was spent — and who it was spent on — if not exactly how much each entity received.

For instance, on May 24, 2019, the city bought a month’s worth of IT support from Trigyn Technologies. From SecuLore Solutions of Odenton, Maryland, the city purchased cyber forensics and monitoring assistance, as well as security software. IT solutions company Quest sold the city consultation with archive managers. Microsoft sold it consultation and implementation of active directory remediation and hardening, which included the billable hours of a solutions architect, senior consultant, senior project manager and account delivery executive. FireEye, doing business through Mandiant, was contracted by the city as a security consultant and engineer. California-based Dyntek was hired to rebuild Microsoft products. A consultant from Glen Burnie, Maryland-based Skyline Technology Solutions was contracted to act as the city’s Chief Information Security Officer and, after Todd Carter was promoted, acted as director and deputy director of infrastructure.

Regardless of how all this contracted work shook out, COO Terry Bazemore Jr. of cybersecurity firm Ey3 Technologies noted that cyber companies are also, in a sense, being paid so clients can have a certain sense of coverage.

“As a cybersecurity company, you come in as either the hero or the scapegoat,” he said. “I’m coming in to help you but let’s say, while we’re in the process of helping this company, something goes wrong on their end, or something happens later. Now they can say, ‘Hey, well, we had this company come in to do this work.’ What you’re paying for is also coverage for yourself as an organization.”

The cost of all these contractors may explain why cities pay $20 million in cybersecurity insurance. Otherwise, the full expense comes out of the taxpayers’ pockets. Instead of paying a $569,000 premium to pay the contractors or the ransom, a city pays millions.

“It’s a challenge. What’s the right amount of money to spend? What’s the right amount of prevention to have,” Nash said. “The challenge in cybersecurity is: Prevention is hard, sometimes, to document what the value was. The value comes after something bad happens.”

Another option, and why it’s not often used

Cities could theoretically hire internal cybersecurity professionals. This raises another problem, however: Cities cannot always offer salaries that compete with what the private sector offers. Baltimore’s former CIO Frank Johnson in 2019 had the city government’s highest salary at $250,000 a year, according to The Baltimore Sun. As the highest-paid employee, that’s still less than the median salary for a CIO throughout the US. It’s no secret that you make can more in the private sector than in the public one.

The city IT budget skyrocketed after the 2019 ransomware attack without actually leading to an increase in positions — a detail that city councilmember Eric Costello noted during an FY2021 city budget meeting at which the Baltimore City Information & Technology Department (BCIT) proposed its plans.

[youtube https://www.youtube.com/watch?v=w9Z73_Fn9do?start=489]

That BCIT proposal was actually trying to stabilize the budget over the next few years. That’s estimated to be around $5 million in FY22, which is higher than before the ransomware attack but about half the budget now.

Along with that budget, the city’s philosophy IT security has changed under new Chief Information Security Officer Kevin Kearney. As he told Technical.ly, security must be an integral and early part of IT planning.

“Security is less effective and often costs more to try and bolt on later,” he said. “The city has been investing in IT improvements to support the mayor’s pillars and security is being integrated into all of our IT projects.”

Nash is similarly proactive about cybersecurity and estimates a cost of about  $3 or $4 million to establish a decent cyber intelligence organization. To keep it running still costs a couple of million per year.

“If you spend a lot of money year over year and bad things don’t happen, there’s a tendency to try to cut that budget,” Nash said. “‘Maybe we’re overspending here. We don’t really know what we’re defending.’ Until something bad happens. Then you fire a couple people. Spend a whole bunch of money on people and new technologies. We see that happen a lot of times [in both the]private sector and public sector.”

Kearney also confirmed that the city has an incident response plan, and all city agencies are required to update their continuity-of-operation plans this year.

“I joined the city after the ransomware attack, but in talking to leaders and employees throughout the city, the most common feedback I’ve heard is that communication during the incident could have been better,” Kearney recalled. “Employees and leadership sometimes felt like they didn’t have the information they needed. When we updated our incident response plan, that was an area that received a lot of attention.”

A longterm shift, and a persistent question

The change in mentality about cybersecurity is great and to be expected after an attack that shut down city services for a month. But memories fade as time goes on, and the budget always reflects a city’s current priorities.

“[Victimized organizations] still have a mentality that security is important to get it fixed but not important to pay for,” Bazemore said. “They still oftentimes think $50 an hour should cover that. If you understand the caliber of people that have to come in and clean this up, triage and understand the advanced nature of the individuals that hit you. We’re going to have to bring in some really advanced people.”

Nash comes from an NSA and US Air Force background. Bazemore built his cybersecurity toolkit with Lockheed Martin a little over a decade ago and has worked with Maryland cybersecurity firms ever since. Both of them have specialties under the IT umbrella that demand top dollars, and the market reflects their importance: Of the 1,500 senior tech managers that Robert Half Technology recently surveyed, 30% say that security, privacy and compliance professionals are among the most challenging to find.

“In demand” and “hard to find” means “expensive to hire,” sure, but it’s also expensive to hire consultants after the fact. The question for cities and businesses big and small remains: Is an ounce of prevention worth a pound of the cure?

Donte Kirby is a 2020-2022 corps member for Report for America, an initiative of The Groundtruth Project that pairs young journalists with local newsrooms. This position is supported by the Robert W. Deutsch Foundation.