Red flags were raised weeks, months before Suffolk cyberattack

A succession of red flags over computer system vulnerabilities and possible intrusions into Suffolk County's computer networks were sent up the chain of command in the days and months before a September cyberattack, a Newsday review of emails shows.

In the end, technical limitations and bureaucratic considerations might have undermined the response needed to repel the Sept. 8 attack, according to interviews and a review of dozens of emails among Suffolk officials and technical staff. The cyberattack shut down a broad cross-section of county services.

As recently as June, county department heads repeatedly pleaded for higher levels of security for the computer network, the emails show. Those requests followed months of alerts about intrusion attempts on the county clerk's network, including a dozen in February alone.

While the intrusion attempts were blocked, they indicated that "good, bad or indifferent, there was an intruder in the kitchen," said a person with knowledge of the alerts. "That should have been an indicator that something was going on."

The attack led Suffolk County Executive Steve Bellone to declare a state of emergency on Sept. 11. It has hobbled telephone and email systems and impacted the police department, Department of Health Services, the Traffic and Parking Violations Agency and other departments as the county is still trying to recover.

The red flags from earlier this year are separate from those raised by law enforcement agencies, including the Suffolk district attorney and the FBI, in emails to the county’s security coordinator near the end of June that warned of the possibility of an ongoing ransomware event. Suffolk's cybersecurity coordinator in a June 21 email indicated that "nothing is lighting up" that would indicate a cyberattack. 

That heightened calls by the clerk's office for higher levels of security. "It's exactly why I was making the case," County Clerk Judy Pascale said in an interview. But she and County Comptroller John Kennedy said their requests were largely denied.

For instance, in a June 9 email, Suffolk’s top computer official, IT commissioner Scott Mastellon, told Pascale that her office had “not demonstrated an appropriate justification” to support the need for a higher level of firewall protection. “The current firewall technology environment in place can effectively support the clerk’s office,” Mastellon wrote.

"The County currently has Palo Alto firewall technology in place and the clerk is able to leverage that technology to support your security needs without having to purchase the additional components outlined in your request," Mastellon wrote. 

Pascale disagreed. “Cybercrime is a constant, ongoing, real threat,” she wrote back. “I’m sure you know that, should such an attack occur against the County Clerk’s system, the residual consequences will be devastating and perhaps beyond repair.”

A 'real threat'

Also in June, Pascale took her request to Chief Deputy County Executive Lisa Black, saying that attaining new levels of protection was “absolutely critical to my operation.”

“Cybersecurity is a real threat and the fragile and antiquated nature of the existing system is causing sleepless nights,” Pascale wrote. Black responded that she would "follow up" with the Legislature's Ways & Means committee "to see who tabled it and what the concern is." The committee had tabled funding requests on three separate occasions earlier in the year.

"I felt like nobody was listening or they didn’t take it as seriously as I was making it," Pascale said. "Witness the fact it was tabled three times in committee and then it was rejected" by the county's IT committee. "I couldn’t impress upon people enough that this could be devastating, and guess what? It's devastating."

Suffolk County spokeswoman Marykate Guilfoyle declined to answer Newsday's questions about the technology requests.

“While continuing to manage this serious cyber incident, the county will not be distracted by irresponsible and erroneous speculation," Guilfoyle wrote. "Our focus remains on completing the forensic examination, which is part of a larger criminal investigation, while continuing to provide the critical services our residents rely on.”

Legis. Rob Trotta (R-Smithtown), who chairs the Ways & Means committee, in an interview said the request to the committee did not include anything about cybersecurity, although the Mastellon emails lay out the breadth of the upgrades.

Trotta said Presiding Officer Kevin McCaffery wanted the funding request tabled because there was an investigation into a Bitcoin-mining operation uncovered in the clerk’s office after one of its employees, Christopher Naples, was arrested for running the operation. Trotta said the committee asked Pascale to appear before it, but she never did. Pascale did send a top IT official to attend the meeting in June.

But in a June 23 email to Pascale's staff, Trotta wrote the funding "will be passed. McCaffrey was holding it up. Why I don't understand."

McCaffery said on Friday that the committee’s brief delay in approving the funding was because the clerk’s office didn’t immediately answer questions about the package. Once the questions were answered, he said, spending was approved “right away.”

'Living on borrowed time'

Kennedy, in an email to the clerk’s office, suggested taking the matter to a higher level at county government, invoking Deputy County Executive Jon Kaiman. “We should put the whole world on notice that we are living on borrowed time with the existing setup,” Kennedy wrote, referring to outdated computer security systems.

In an interview, Kennedy said there were clear signs hackers were attempting to get in Suffolk's system earlier this year. “There weren’t just forewarnings, they were screaming indications,” he said.

“People who were monitoring my systems said there were indications that there were malware attempts,” he said. “That was the whole impetus for me to get my own server back in the springtime, and I was stopped by Bellone’s IT guy Mastellon. They did not want me to have my own server. Nor did they want me to have my own firewall.”

While the Ways & Means Committee ultimately approved funding, in a June 9 email, Mastellon describes in detail which upgrades his information technology steering committee had “voted not to approve," including the clerk's request for firewalls.

Mastellon did not return a call for comment. Guilfoyle said in an email that Kennedy’s assertion was “not accurate.”

The tech staffers who raised alarms were never rebuffed or ignored outright, the emails show. And the county’s top computer managers and an outside contractor in charge of cybersecurity reviewed the suspicious traffic, organized interoffice Zoom meetings and looped in department experts in email chains. Emails show that isolated fixes were undertaken to intrusions as they appeared.

But no response was as severe, and ultimately as drastic, as the eventual reaction on Sept. 8, when ransomware notes began popping up on computers and managers began physically disconnecting network connections.

Black told county legislators at a hearing on Oct. 31 that Suffolk has spent $4.8 million on the forensic investigation and recovery from the ransomware attack.

Kennedy said the cost is more likely $10 million to $15 million so far but that he can’t be more precise because the invoices haven’t come in yet. “I suspect that they will come in very soon,” he said.

In addition, according to county information shown to Newsday, the work of restoring services is not finished. Of the nearly 600 computer servers across the county network, about 150, or 26.9%, were impacted by the breach and needed rebuilding or restoring. Of those, as of the end of October, about 50 had been rebuilt or restored.

Guilfoyle called the estimate that around a quarter of county servers were impacted “inaccurate.” Asked to provide more specific information, she said, “Due to the ongoing assessment, I am unable to provide details at this time.”

More red flags

Red-flag emails of a possible intrusion came in the week prior to Sept. 8, and most eventually made their way to Brian Bartholomew, then the information technology security coordinator for Suffolk — the top cybersecurity staffer at the county. Newsday on Wednesday reported that Bartholomew retired from the county in 2021 and has been operating as a consultant in the same post. Though he recently moved to Florida, his voicemail still lists him in the position.

Bartholomew declined to comment, referring inquiries to his attorney.

In a series of emails on Sept. 2, a technician at the County Clerk’s Office, concerned about suspicious system traffic, asked Bartholomew for access to information about traffic coming to the clerk’s network from the main county system after one security system detected unusual activity initially labeled “benign events.” Help in investigating the origins of the events was stalled because of the approvals needed to access activity logs and concern that it might “create tension in the county,” Bartholomew wrote.

By Sept. 7, the messages had become more urgent. Speaking about a series of alerts the tech staff referred to as “Cortex” messages, one computer manager in the clerk’s office wrote, “Brian … We need to deal with this issues asap … 3rd Cortex today … with the last two being malicious.”

Earlier in the day, Bartholomew agreed to a Zoom meeting to strategize and to provide some access to suspicious traffic, but added, “because we cannot limit you to just seeing your traffic, then we cannot give you individual access. Take that up with management."

Existing servers are 'old, brittle' 

In the June 3 email, a Pascale staffer noted that the clerk’s office was “collectively fighting a 24/7 battle with the unknown” in requesting access to firewalls and other upgrades. Suffolk in the prior year had installed highly regarded Palo Alto firewalls, Cortex end-user security and Wildfire anti-malware protection on its main domains, but the clerk’s office, operating with protection called Sonic Wall, pleaded for a second level of protection also using Palo Alto.

“Our existing server environment is very old, brittle and has reached its ‘high-water’ capacity limit, which has caused numerous performance and outage issues across the enterprise,” one clerk’s office manager wrote to Mastellon.

The email noted the systems are used “by numerous agencies throughout the county, in addition to numerous external municipalities, courts, attorneys, real estate industry and developers, [to] name a few. To avoid a similar situation, it is imperative that existing new existing  server equipment is placed into production as soon as possible with the professional services required to do such.”

By June 13, the request elevated to a point that Pascale emailed Mastellon, noting her office’s annual $1.3 billion in municipal user fees and the $19.8 million remitted to the county general fund each year.

Pascale told Mastellon she was “somewhat taken aback” by his statement in a prior exchange that the clerk’s office “has not demonstrated an appropriate justification to support this purchase” of dedicated firewall protection and other tools. She noted it’s not only industry best practice to have a dedicated firewall, but that “any outages or disruptions will have a tremendous negative impact on the sale or purchase of homes in Suffolk County."