Suffolk cyberattack anniversary: Many fixes made, but not everything is back
Richard Donoghue (center), a former U.S. Deputy Attorney General, has been acting as special counsel to a Suffolk legislative committee investigating the Sept. 8, 2022, cyberattack on the county. Credit: Howard Schnapp
The year since Suffolk County sustained one of the nation’s longest and most costly ransomware attacks has brought an intense focus on a sprawling computer network that lacked security leadership and enough staff to proactively monitor threats, according to months of testimony and Newsday investigations.
Suffolk County on Friday marked the anniversary of an attack that locked down a cross-section of services — from civil-service testing and traffic and parking violations functions to police dispatch to functions of the comptroller and clerk's offices — that also might have exposed personal data of some 500,000 people. Officials said the impacts continue to resonate, even months after the county declared itself “back online.”
Suffolk’s main website was restored to service in February, and most of the services that the county offers through its computer system of more than 1,000 servers and 10,000 desktops largely have been restored. A chief information security officer was appointed in March, years after the post was recommended, and new technology, such as multifactor authentication to shore up defenses, is now in place. The county also has moved to centralize computer security after years of operating in "silos" by departments, which made uniform security policies difficult to enforce.
But not everything is back, and last month Suffolk issued the 12th emergency declaration tied to the Sept. 8, 2022, attack. The county declined to say whether it declared another one this month. Interviews with elected and appointed officials indicated some services remain out, including in the clerk and comptroller's offices. County legislators last month complained they still don’t have access to their preattack emails.
“What’s very unusual about this is the amount of time it took the county to become operational again,” said Richard Donoghue, a former U.S. Deputy Attorney General who is special counsel to a Suffolk legislative committee investigating the attack. In an interview on Wednesday, he took note of several “established” findings from his probe to paint a wider picture of the issues that may have opened the door for the intrusion, and prolonged its restoration.
Newsday requested to speak to Suffolk's new chief information security officer, Kenneth Brancik, to discuss developments since the attack, but the request was denied. County Executive Steve Bellone's administration did not respond to specific questions for this article.
The hackers initially demanded a $2.5 million ransom, but reduced the amount to $650,000, Bellone previously said. Suffolk, which did not have cyber insurance, did not pay any ransom. Bellone in December said the county at that point had spent $3.4 million on the restoration and $2 million on an investigation, but Newsday has reported the figure was closer to $17 million, including hardware and software the county said it needed to purchase anyway.
Donoghue indicated one finding about the attack is clear: There’s plenty of blame to go around.
“There’s no way that you can point to one person or even one department and say they were the cause of this,” said Donoghue, in a reference to the Bellone administration pinning much of the blame on a director in the clerk's office. “There are too many factors that went into it," many of them tied to the administration's Department of Information Technology.
Donoghue pointed to testimony last month before the legislative committee that showed a handful of high-level employees in the IT department, and security staff led by an outside contractor, were so overwhelmed by red-flag alerts in the weeks and months leading up to the attack that they diverted them to a spam-like folder to minimize “message fatigue.” The alerts, from a defense system known as Cortex, inform tech workers about malware or intrusion attempts to help thwart bigger intrusions.
“I think you can look at the [missed] Cortex alerts as probably the last chance to avoid what happened on Sept. 8,” Donoghue said. He noted that the Cortex endpoint protection system was “not fully deployed across the county,” leaving an untold number of systems exposed, while a critical patch for the prime vulnerability behind the attack was “not fully remediated across the county.”
The findings thus far present a contrast to an assessment by Bellone, who in December largely laid the blame on one employee in the clerk’s office, IT director Peter Schlussler. Bellone alleged the director “knew of the vulnerability” exploited by the ransomware, but “failed to protect the county clerk’s IT infrastructure from this threat.” Schlussler was suspended with pay.
Bellone also said there was “no way” for the IT department to “become directly involved to validate, troubleshoot or resolve any of the alerts from the clerk’s system” because clerk staff “insisted that county IT security should have no visibility or access to the clerk’s systems.”
Schlussler and former County Clerk Judy Pascale, in public testimony, have rebutted Bellone’s claims. Schlussler, who also prepared a 157-page report for the legislative committee rebutting Bellone's claims, was the first IT employee to alert officials of the attack, and to shut down the clerk’s systems, hours before the broader county did. He has blamed the failure of IT staff to follow up on the alerts and the lack of a chief information security officer for the county as being among the root causes for the attack.
Donoghue suggests the problem was broader than one employee.
“This notion that the Department of Information Technology had no insight at all into the clerk’s office is I think pretty clearly established at this point not to be accurate,” Donoghue said. “They had insight through a variety of means, probably most importantly through the Cortex deployments,” as other witnesses have testified. The alerts “gave them critical insight into what was going on in the clerk’s office.”
Legis. Anthony Piccirillo (R-Holtsville), who chairs the cyberattack committee, agrees there's plenty of blame to go around. “It’s clear there were systemic problems throughout the county, not just the clerk’s office," he said. What’s also clear is that “we’re understaffed, undertrained and we have to take cybersecurity more seriously.”
Newsday in May reported on an independent analysis done by Dell Technologies that found more than 600 instances of malware on computers a month after the attack, some that might have been there for years.
Donoghue pointed to the county’s lack of a disaster recovery plan for the technology department as a factor that largely hindered a quick recovery.
“The county simply didn’t have one, which effectively is technology malpractice,” he said of the written plans, which are to be kept close by to provide a step-by-step plan for recovery. Suffolk spokeswoman Marykate Guilfoyle didn't respond to Newsday's inquiry about whether the county since has created such a plan.
Michael Nizich, director of the Entrepreneurship and Technology Innovation Center and computer science faculty member at New York Institute of Technology, said the lack of such a written plan likely explains the long recovery time.
"The preparation that's supposed to be in place in a textbook world doesn't seem to have been in place at all" at Suffolk, said Nizich, who noted corporate recovery times from such attacks often are measured in hours, not weeks or months. "In a perfect environment, they should have gone back to Sept. 7 [to back up systems] and been back up in a day or so," or a week at most, he said.
Meanwhile, some services remain offline, but are expected to return.
Last week, the clerk’s office brought online a “kiosk” of services that allow residents to view select property deeds and mortgage information, although a similar service that allows title search experts to do more sophisticated searches online won’t be available until year’s end, said Christopher Como, senior official examiner of title. Professional title searchers still can visit the clerk’s office in person.
Another system, of homeowner alerts for property record searches, is slated to come back online soon.
Suffolk Comptroller John Kennedy said a once-heavily used self-service tool his office offered to vendors to check the status of payments remains offline, and he’s still waiting on whether he can shift his financial management software to the cloud.
Kennedy said he was able to complete on Aug. 21 the certified financial statement for the county that is normally finished by June 30 — seven weeks late, he said, because of ransomware-related impacts.
He said his office is still unable to provide a full picture of the total cost of the attack in response to a Newsday Freedom of Information Law request, as staffers continue to review purchase orders and contracts issued without competitive bidding under the emergency status. Kennedy said he recently began a review of purchase orders tied to computer services during the remediation. “It’s involving something which under normal circumstances we don’t engage in,” he said of the outside audit.
Some needed fixes are in place, such as two-factor authentication and a new chief information security officer. Donoghue noted but he was uncertain whether the county was able to obtain the cyber insurance it lacked before the attack.
“The real benefit of cyber insurance, in addition to having your back when things go bad, is the fact that it compels you to have good cyber-hygiene,” Donoghue said. “And the lack of cyber insurance is part of what allowed the environment to develop the way that it did."
Donoghue expects to finish interviewing witnesses in the investigation within the next month or two and issue a report. “It will give people a much fuller picture of all the issues and problems and challenges that contributed to this so that we’re in a position to make sure it doesn’t happen again,” he said.
“Undoubtedly the county will be attacked again,” Donoghue added. “The question is, what’s your posture going to be at that point to make sure you limit the damage and recover quickly?”
The year since Suffolk County sustained one of the nation’s longest and most costly ransomware attacks has brought an intense focus on a sprawling computer network that lacked security leadership and enough staff to proactively monitor threats, according to months of testimony and Newsday investigations.
Suffolk County on Friday marked the anniversary of an attack that locked down a cross-section of services — from civil-service testing and traffic and parking violations functions to police dispatch to functions of the comptroller and clerk's offices — that also might have exposed personal data of some 500,000 people. Officials said the impacts continue to resonate, even months after the county declared itself “back online.”
Suffolk’s main website was restored to service in February, and most of the services that the county offers through its computer system of more than 1,000 servers and 10,000 desktops largely have been restored. A chief information security officer was appointed in March, years after the post was recommended, and new technology, such as multifactor authentication to shore up defenses, is now in place. The county also has moved to centralize computer security after years of operating in "silos" by departments, which made uniform security policies difficult to enforce.
But not everything is back, and last month Suffolk issued the 12th emergency declaration tied to the Sept. 8, 2022, attack. The county declined to say whether it declared another one this month. Interviews with elected and appointed officials indicated some services remain out, including in the clerk and comptroller's offices. County legislators last month complained they still don’t have access to their preattack emails.
WHAT TO KNOW
- It's been a year since Suffolk County sustained one of the nation’s longest and most costly ransomware attacks, and impacts continue to resonate.
- A chief information security officer was appointed in March, and new technology such as multifactor authentication to shore up defenses, is now in place.
- But not everything is back. Interviews with officials indicated some services remain out, including in the county clerk and comptroller's office.
“What’s very unusual about this is the amount of time it took the county to become operational again,” said Richard Donoghue, a former U.S. Deputy Attorney General who is special counsel to a Suffolk legislative committee investigating the attack. In an interview on Wednesday, he took note of several “established” findings from his probe to paint a wider picture of the issues that may have opened the door for the intrusion, and prolonged its restoration.
Newsday requested to speak to Suffolk's new chief information security officer, Kenneth Brancik, to discuss developments since the attack, but the request was denied. County Executive Steve Bellone's administration did not respond to specific questions for this article.
The hackers initially demanded a $2.5 million ransom, but reduced the amount to $650,000, Bellone previously said. Suffolk, which did not have cyber insurance, did not pay any ransom. Bellone in December said the county at that point had spent $3.4 million on the restoration and $2 million on an investigation, but Newsday has reported the figure was closer to $17 million, including hardware and software the county said it needed to purchase anyway.
Suffolk County Executive Steve Bellone on Feb. 17 updated the recovery efforts from the cyberattack during a news conference. Credit: Rick Kopstein
Donoghue indicated one finding about the attack is clear: There’s plenty of blame to go around.
“There’s no way that you can point to one person or even one department and say they were the cause of this,” said Donoghue, in a reference to the Bellone administration pinning much of the blame on a director in the clerk's office. “There are too many factors that went into it," many of them tied to the administration's Department of Information Technology.
Donoghue pointed to testimony last month before the legislative committee that showed a handful of high-level employees in the IT department, and security staff led by an outside contractor, were so overwhelmed by red-flag alerts in the weeks and months leading up to the attack that they diverted them to a spam-like folder to minimize “message fatigue.” The alerts, from a defense system known as Cortex, inform tech workers about malware or intrusion attempts to help thwart bigger intrusions.
“I think you can look at the [missed] Cortex alerts as probably the last chance to avoid what happened on Sept. 8,” Donoghue said. He noted that the Cortex endpoint protection system was “not fully deployed across the county,” leaving an untold number of systems exposed, while a critical patch for the prime vulnerability behind the attack was “not fully remediated across the county.”
The findings thus far present a contrast to an assessment by Bellone, who in December largely laid the blame on one employee in the clerk’s office, IT director Peter Schlussler. Bellone alleged the director “knew of the vulnerability” exploited by the ransomware, but “failed to protect the county clerk’s IT infrastructure from this threat.” Schlussler was suspended with pay.
Bellone also said there was “no way” for the IT department to “become directly involved to validate, troubleshoot or resolve any of the alerts from the clerk’s system” because clerk staff “insisted that county IT security should have no visibility or access to the clerk’s systems.”
Schlussler and former County Clerk Judy Pascale, in public testimony, have rebutted Bellone’s claims. Schlussler, who also prepared a 157-page report for the legislative committee rebutting Bellone's claims, was the first IT employee to alert officials of the attack, and to shut down the clerk’s systems, hours before the broader county did. He has blamed the failure of IT staff to follow up on the alerts and the lack of a chief information security officer for the county as being among the root causes for the attack.
Suffolk County Clerk IT director Peter Schlussler prepares to leave a cyberattack hearing after testifying on June 16 in Hauppague. Credit: Tom Lambui/Tom Lambui
Donoghue suggests the problem was broader than one employee.
“This notion that the Department of Information Technology had no insight at all into the clerk’s office is I think pretty clearly established at this point not to be accurate,” Donoghue said. “They had insight through a variety of means, probably most importantly through the Cortex deployments,” as other witnesses have testified. The alerts “gave them critical insight into what was going on in the clerk’s office.”
Legis. Anthony Piccirillo (R-Holtsville), who chairs the cyberattack committee, agrees there's plenty of blame to go around. “It’s clear there were systemic problems throughout the county, not just the clerk’s office," he said. What’s also clear is that “we’re understaffed, undertrained and we have to take cybersecurity more seriously.”
Newsday in May reported on an independent analysis done by Dell Technologies that found more than 600 instances of malware on computers a month after the attack, some that might have been there for years.
Donoghue pointed to the county’s lack of a disaster recovery plan for the technology department as a factor that largely hindered a quick recovery.
“The county simply didn’t have one, which effectively is technology malpractice,” he said of the written plans, which are to be kept close by to provide a step-by-step plan for recovery. Suffolk spokeswoman Marykate Guilfoyle didn't respond to Newsday's inquiry about whether the county since has created such a plan.
Michael Nizich, director of the Entrepreneurship and Technology Innovation Center and computer science faculty member at New York Institute of Technology, said the lack of such a written plan likely explains the long recovery time.
"The preparation that's supposed to be in place in a textbook world doesn't seem to have been in place at all" at Suffolk, said Nizich, who noted corporate recovery times from such attacks often are measured in hours, not weeks or months. "In a perfect environment, they should have gone back to Sept. 7 [to back up systems] and been back up in a day or so," or a week at most, he said.
Meanwhile, some services remain offline, but are expected to return.
Last week, the clerk’s office brought online a “kiosk” of services that allow residents to view select property deeds and mortgage information, although a similar service that allows title search experts to do more sophisticated searches online won’t be available until year’s end, said Christopher Como, senior official examiner of title. Professional title searchers still can visit the clerk’s office in person.
Another system, of homeowner alerts for property record searches, is slated to come back online soon.
Suffolk County Comptroller John Kennedy. Credit: Danielle Silverman
Suffolk Comptroller John Kennedy said a once-heavily used self-service tool his office offered to vendors to check the status of payments remains offline, and he’s still waiting on whether he can shift his financial management software to the cloud.
Kennedy said he was able to complete on Aug. 21 the certified financial statement for the county that is normally finished by June 30 — seven weeks late, he said, because of ransomware-related impacts.
He said his office is still unable to provide a full picture of the total cost of the attack in response to a Newsday Freedom of Information Law request, as staffers continue to review purchase orders and contracts issued without competitive bidding under the emergency status. Kennedy said he recently began a review of purchase orders tied to computer services during the remediation. “It’s involving something which under normal circumstances we don’t engage in,” he said of the outside audit.
Some needed fixes are in place, such as two-factor authentication and a new chief information security officer. Donoghue noted but he was uncertain whether the county was able to obtain the cyber insurance it lacked before the attack.
“The real benefit of cyber insurance, in addition to having your back when things go bad, is the fact that it compels you to have good cyber-hygiene,” Donoghue said. “And the lack of cyber insurance is part of what allowed the environment to develop the way that it did."
Donoghue expects to finish interviewing witnesses in the investigation within the next month or two and issue a report. “It will give people a much fuller picture of all the issues and problems and challenges that contributed to this so that we’re in a position to make sure it doesn’t happen again,” he said.
“Undoubtedly the county will be attacked again,” Donoghue added. “The question is, what’s your posture going to be at that point to make sure you limit the damage and recover quickly?”
Mark Harrington, a Newsday reporter since 1999, covers energy, wineries, Indian affairs and fisheries.
More rain for LI ... Thanksgiving travel ... Penny trial continues ... FeedMe: Holiday pies
More rain for LI ... Thanksgiving travel ... Penny trial continues ... FeedMe: Holiday pies
