What Suffolk County has to do to recover from crippling cyberattack

The costly, painstaking process of rebuilding Suffolk County’s computer networks in the wake of a ransomware attack may be complicated by uncertainties about how the attack occurred, how much data was lost and whether hackers can re-exploit vulnerabilities, experts say.

One month after the BlackCat/ALPHV intrusion on Suffolk networks was discovered and the county was forced to stop it with measures as blunt as physically pulling network cables from their sockets, only parts of the vast system of police, court, health department and real estate systems are back online, some in limited form. Email and phone systems were widely affected, and a source with knowledge of the situation said there are questions about whether years of email records can be restored. 

“You have to make a determination on how to wall off your network,” said Mike Balboni of the Manhattan consulting firm Redland Strategies, which has been a computer security contractor to the county. He declined to discuss specifics of the attack.

Suffolk's main vendor for firewalls, PaloAlto Networks, also declined to discuss what happened. 

"For this story, we're not going to be able to assist with your questions, but I appreciate you reaching out," Kelly Kane, PaloAlto senior manager for threat communications, said in an email.

The Sept. 8 attack infiltrated departments across the sprawling county system, from the Department of Health to the county clerk, affecting the ability of the police to write tickets and the government to make payments to vendors and local governments and provide certain real estate records searches. The hackers are seeking an unspecified "small reward" for identifying vulnerabilities in the system, which the county is not paying.

Industry protocol suggests that a massive forensic investigation is underway even as the county works to methodically restore critical parts of the network. The investigation is a crucial part of helping the county understand not only how the intruders breached the network, but also to map the depth of the intrusion and hunt for lurking threats, according to experts from academic institutions, government and the computer industry. There are likely hundreds of "compromised" locations on the county's network, they said.

Attack may have lasted 4 hours

Now, experts and a public official said, the county likely has begun the process of installing scores of new computer servers, the core of its network, while examining massive backup files to make sure they haven't been corrupted. 

The fact that many services are not yet back online or have been moved to new domains is indicative of the depth of the intrusion, experts said, suggesting that there may not yet be a level of certainty that all known “back doors” for intrusion, and corrupted files capable of relaunching an attack, have been identified and removed.

It's still unclear how much time elapsed between the discovery of the ransomware and when computer staff started pulling network connections. One person familiar with the attack suggested it was around four hours.

Worse, said one expert, the intruders may have been lurking on the county's system far longer.

“The breach could have been going on for days, maybe weeks. They could have been pulling data for a long time," said Michael Nizich, director of the Entrepreneurship and Technology Innovation Center and computer science faculty member at New York Institute of Technology.

Suffolk said it has begun a "rolling restoration" of its computer operations, starting with a portion of the 911 emergency dispatch system, followed by its property title search capabilities, but officials offered no timeline for when all systems will be back online.

“The fact that it’s taking longer to get back up and running is indicative of another problem,” Nizich said. “They might do all that work and the same thing might happen,” a scenario the county is working to avoid.

“They could be vulnerable to a new group that knows the same vulnerabilities,” he added.

Ransomware attacks cost organizations $4.54 million on average, not including the ransom itself, between March 2021 and March 2022, according to a 2022 annual IBM study examining data breaches among 550 large organizations in the public and private sectors. It took entities on average 326 days to identify and contain a ransomware threat, according to the report.

Critical data may be for sale

Worse for Suffolk is the notion that the ransomware group, or likely several groups, may be offering critical data taken during the attack for sale to other bad actors, threatening waves of cascading intrusions and impacts. A copy of the ransomware letter sent to Suffolk indicates the group changed the names of certain Suffolk data files and encrypted them, meaning Suffolk could no longer access them unless it paid the demanded ransom and received a special “key” from the attackers to unlock them. The group already has selectively published some county records, and experts said much worse could be in store. 

"Another layer we’ve seen, particularly with ransomware, is that attackers don't just stop at system level with encryption and extortion, but in almost every case, they exfiltrate data off the network," said TJ Sayers, cyberthreat intelligence manager at the Center for Internet Security, a multistate information sharing and analysis nonprofit. "They want to sell it to make extra cash and they try to double extort you. … Now you're under the threat that all property and patient information, legal criminal records even, will get leaked to the public." 

BlackCat/ALPHV has gained notoriety by not only locking up data but destroying it as well, putting a high reliance on Suffolk’s backups to rebuild.

“There is a heavy fog about the attacks of the Black Cat group, and the reason for this is the profile of the victims,” Ido Cohen, founder of DarkFeed, which tracks ransomware intrusions, said in an email. “In the last month, this group [reported] that they managed to encrypt and steal information of very sensitive companies related to the military and government sectors.”

The implications aren’t lost on firms that closely monitor county government. On Wednesday, after Newsday sent a series of questions to Fitch Ratings concerning implications of the attack, the Wall Street firm published a statement noting that while Suffolk believes costs tied to the attack could be absorbed in its current operating budget, “More significant costs could weaken the county’s recently improved liquidity and reserve positions.”

“The breach of confidential information could also trigger legal and political risks regarding data privacy and confidentiality,” Fitch senior director Shannon McCue wrote.

Suffolk spokeswoman Marykate Guilfoyle said the county "believes it is well positioned to absorb any costs associated with this cyber intrusion. Lost revenues remain minimal, and as part of the restoration process, the county will be implementing several new security measures." 

New York State is said to be absorbing part of the costs, including new work stations for title searches that recently went online, according to a source with knowledge of the situation. Gov. Kathy Hochul's office didn't respond to questions about the help. 

Guilfoyle said with the county's continuity plan, "Additional services continue to be brought back online. The county’s team of cybersecurity experts is continuing their assessment to test and verify all systems."

Experts say the county appears to have taken the approach of replacing vital computer servers — potentially hundreds of them — and slowly relaunching thoroughly vetted platforms and data as soon as it can be assured it’s safe.

Costs can add up

The process is exhaustive and time-consuming and “just so expensive,” Nizich said. “It’s going to cost us a fortune.” Teams of tech consultants alone rack up costs quickly, often working overtime, and overnight. Pay can top $500 an hour, he said.

“The safe bet is to replace the entire system,” said Victor Congionti, chief executive of Proven Data, a Manhattan-based computer security firm that helps companies recover files and in some cases negotiate and pay ransom. Restoring a network as vast as Suffolk’s is a “pretty daunting task,” he said.

For affected systems, it’s “essentially starting from scratch. Do they have backups that are older so they at least have some data? They’d have to reimport all that data” after careful scrubbing to make sure it isn’t compromised.

“That’s why the majority of companies opt to pay the ransom, just to get back up and running as soon as possible,” Congionti said, “especially if that data is not replaceable.”

For that reason, questions about how well protected the Suffolk systems were at the time of the attack are certain to be raised anew.

For example, the Suffolk County Budget Review Office's 2022 report reviewing the proposed capital budget for 2023 noted that $1,277,579 out of a total of $2,707,000 that had been appropriated for the IT Department — to acquire equipment and system upgrades for a disaster recovery program — remained unspent. The report did not offer an explanation why.

Also, earlier this year, the county clerk’s office was repeatedly rebuffed in its attempts to upgrade its computer system, according to minutes of the county legislature’s Ways & Means Committee, which tabled the request several times.

Legis. Bridget Fleming (D-Noyack) was among those who advocated for approval.

“We can’t be vigilant enough about bad actors who want to jeopardize the work of government, and I felt the clerk’s request to upgrade her system was critically important,” said Fleming, a candidate for the House of Representatives. “I have been asking questions about security around computer systems for some time.”

Fleming, a former federal prosecutor, said the county is now “doing the careful job of migrating people back to a clean system to ensure they won’t be affected,” while her staff is being asked to “add back in profiles, one by one, into a system that’s new so it’s cleaned and being scanned regularly.”

“It’s important to exercise caution as we get things moving again,” she said.

With Sandra Peddie