Despite word of 'radical malware attack,' it took hours to shut down Suffolk's computer network 

The email sent at 11:18 a.m. on Sept. 8 from a top computer manager at the Suffolk County Clerk’s Office to the Bellone administration’s technology commissioner was as blunt as it was chilling.

“We are currently experiencing a radical malware attack and we shut down all outside access to the systems until such time as we are safe,” said the email, which was obtained by Newsday.

Yet, more than four hours had elapsed before the rest of the county’s computer networks, encompassing nearly 600 servers from Hauppauge to Riverhead, were severed from access to the outside world, starting the clock on the county's broader response to one of the most devastating ransomware attacks faced by a U.S. municipality of any size in the history of such cyberattacks.

A series of emails obtained by Newsday from the day of the attack and the day prior show that awareness of the attack had been slowly dawning on technology staff and officials in the 24 hours preceding the shutdown. Among those was the actual ransomware message, first circulated at 10:53 a.m. on Sept. 8, 25 minutes before the clerk's office shut down.

Whether the four-hour lag in shutting down all county computer networks caused a sizable loss of data is open for debate. One tech expert called it "significant," but said that considerably more data could have been taken in the days and weeks before the Black Cat/ALPHV message was first noted in a 10:53 email. Suffolk Comptroller John M. Kennedy Jr. said it likely made the difference between the clerk's unscathed backup data and the impacts that continue to ripple through Bellone administration operations. 

The emails obtained by Newsday provide a limited look inside the attack at the time it was happening, chiefly involving correspondence to and from the clerk's office. Newsday has requested information from County Executive Steve Bellone's administration, which has been hobbled by a loss of employees' past emails.

In an email Friday, Suffolk County spokeswoman Marykate Guilfoyle said: “Following the September cyberattack, the county hired an industry-leading forensic investigation team to determine what happened and when it occurred. Until that examination is complete, any assertions or determinations are purely speculative, obviously self-serving and within the context of this criminal matter, wholly inappropriate.”

Separately, the county said it took "aggressive measures on Sept. 8 to help contain the cyber intrusion" and noted that in comparison to the clerk's network, the broader county networking environment "involves a significant amount of infrastructure with multiple departments responsible for the administration of those environments." The county "took immediate steps to coordinate with all and shut down the entire network within a four-hour time frame."

As Suffolk approaches its third month of crippling impacts, new details about how the attack occurred, how the county responded, and how it could impact hundreds of thousands of county residents and employees are slowly coming to light.

Last week, Suffolk announced that up to 26,000 current and former county employees’ Social Security numbers and other personal data may have been accessed in the attack. Two weeks earlier, it announced that the records of some 470,000 moving violations may also have been accessed — records dating to 2013. County officials won’t say if they expect to find further impacts.

An expensive attack

But even before the time and cost tallies are in, it’s clear Suffolk’s could be one of the most expensive cyberattacks among municipal governments in U.S. history.

Suffolk thus far has spent more than $10 million to fix the county computer systems hobbled by the attack, according to information sent to Newsday and an estimate by Kennedy. That figure would amount to more than twice the national average cost of data breaches in both public and private entities.

Suffolk said forensic assessment and restoration services cost approximately $5.2 million, but acknowledged that doesn't include the cost of other system and security upgrades it is making, which it said were needed anyway.

An annual report produced by IBM Security based on data compiled by the Ponemon Institute, a Michigan-based information security research institute, found that the average cost of a data breach was $4.35 million.

Suffolk’s cost likely will reach $15 million, according to Kennedy. That would appear to make it the most expensive cyberattack on a municipality in the country, according to a Newsday review of ransomware attacks on municipalities nationwide.

A March 2018 ransomware attack on Atlanta knocked out more than a third of the city’s essential computer programs and cost $9.5 million to restore by June of that year, according to news reports.

The city of Tulsa, Oklahoma, was hit with a ransomware attack in 2021. The city spent $2 million to repair its systems, which were offline for eight months, according to news reports.

In New York, hackers attacked Albany’s computer system on March 30, 2019. Unlike Suffolk, the attack ended within five hours and left critical systems intact. But some data, such as building permits, was lost and took months to restore. Newsday has reported that the city paid $300,000 to fix the damage.

Suffolk’s recovery from the ransomware attack is taking an unusually long time, according to cybersecurity experts interviewed by Newsday.

“It is surprising,” said Justin Cappos, associate professor of computer science and engineering at the Tandon School of Engineering at New York University. “I would not have expected it to take months to do this.”

He said the reason for that was that governments tend to have antiquated computer systems that are tailored to specific tasks, making it more complicated to repair.

Michael Nizich, adjunct and associate professor of computer science at New York Institute of Technology in Old Westbury, agreed. He said the problem with older computer systems is twofold: They rely on code that isn't in widespread use anymore, and the people who have the expertise in them are either retired or gone.

“When those systems were possibly affected, how do we get those back on if we don’t even have the people who know how to operate them?” he said.

Additionally, different departments use different systems, meaning that “You have all kinds of different vulnerabilities that can get exploited.”

A cache of emails shown to Newsday from the earliest moments of the attack indicate that the 4 p.m. shutdown of Suffolk’s computer systems was only the end point of two days of fielding alerts of intrusions and other signals.

Two minutes before sending the 11:18 a.m. email informing Scott Mastellon, commissioner of the county’s department of information technology, of the malware attack, the clerk’s office noted it also had shut down “all backups,” blocking malware actors from infiltrating, locking up or stealing terabytes of real estate records — a move that has allowed the clerk’s office to return to on-site operations within weeks. (The overall county lockdown continues to restrict clerk services from online access.)

One email at 10:53 a.m. from the clerk’s office to a supervisor in Bellone's information technology department also included a copy of the entire ransomware note, complete with instructions on how to respond.

“What happened?" the cyberattackers say by way of introduction. 

“Important files on your network was (sic) ENCRYPTED and now they have “st1t73b” extension,” the note goes on, explaining how the malware had taken over the files. “In order to recover your files you need to follow instructions below.”

The day before

But even before such an overt signal of infiltration was known, techs in the clerk's office had been sounding the alarm. Indeed, the ransomware message amounted to confirmation of what county techs had been puzzling over for more than 24 hours. 

One message at 2:14 p.m. on Sept. 7 from the clerk's office to Brian Bartholomew, the county’s information security coordinator, an outside contractor, spoke of "malicious" detections. 

“Brian … we need to deal with this ASAP … 3rd cortex today … with the last two being malicious,” the clerk’s office IT manager wrote to Bartholomew the afternoon before the countywide shutdown. Cortex messages alert computer techs of malware intrusions and steps to block them. 

Concerns about the attack also led the clerk’s office to alert the Suffolk County District Attorney’s Office on Sept. 7, according to emails. That same day, Laura de Oliveira, a deputy bureau chief in the district attorney's office, responded by referring the matter back to Bellone’s security coordinator.

“We think the best course is for Brian Bartholomew to continue his forensic investigation and then report back to us when he either has more information or a need for a legal process,” de Oliveira wrote.

Newsday previously has reported that the Bellone administration had been alerted to a possible ransomware event after a tip from an FBI agent in June. Suffolk District Attorney Ray Tierney in October told Newsday his office had been “assured at that time by the Coordinator in charge of County IT that their checks revealed no malicious activity."

“Unfortunately,” he added, “that assessment was incorrect. We are working with the FBI and our law enforcement partners to apprehend and prosecute the offenders.”

Tierney’s office on Friday acknowledged receiving a call from the clerk’s office on Sept. 7 notifying it of a “potential intrusion.” It said the information was passed up the chain of command at the district attorney’s office and help was offered help in investigating the intrusion, but noted the district attorney’s office “has no capability to monitor county IT systems as that is an IT and not a law enforcement function.”

By the morning of Sept. 8, at 8:28 a.m., the clerk’s office had grown so concerned about the intrusions that an email was sent to 10 technology staff and officials, including Deputy IT Commissioner Ari McKenzie, informing them the clerk had received an alert from its security software. The clerk's office requested their help, including “specific methods and/or tools to remove” a known piece of malware that has “already been propagated" on its systems. 

Whether the early alerts, and the message from the clerk’s office indicating it had shut down access at 11:18 a.m., should have prompted Suffolk’s parent domain to do the same across the broader network is a point of debate.

Kennedy, who has been sparring with the Bellone administration as he attempts to get a level of autonomy over his financial management computer systems, said the county should have pulled the plug when the clerk did.

“I’m a computer idiot, but my understanding is that by disconnecting their systems, they [the clerk's office] did not sustain the same amount of damage,” Kennedy said. “The county side didn’t go dark until after 4 p.m. My sense is that the executive staff in IT are without the skills necessary to protect us.”

Nizich said that while the county’s delay in shutting down was "significant," it's also likely that much of the data had likely been stolen or compromised. That's because, he suspects, the thieves had been in the system long before Suffolk discovered the attack.

The four terabytes of data reportedly taken by the hackers would have taken much more than six hours to download, Nizich suspects. “I think the data was stolen earlier than that,” he said.