Here's how Yonkers and Albany recovered from ransomware attacks

ALBANY — Yonkers and Albany officials know the long, painstaking and costly recovery that Suffolk County is facing since it experienced a cyber-incursion in September. Each survived ransomware attacks.

The intrusions revealed warnings, mistakes and successes after hackers demanded ransom and threatened to grind the governments to a halt. And even though each city refused to pay the hackers’ ransom, the cost of recovery was steep.

In Suffolk, officials said some residents' "personal information" was accessed in the Sept. 8 attack and urged them to closely monitor their bank accounts and credit reports. The hackers are seeking an unspecified "small reward" for identifying vulnerabilities in the system. The county didn’t have cybersecurity insurance.

Albany's computer system was attacked at 4 a.m. on a Saturday, March 30, 2019.

“The lights [were] off, or appeared to be off,” said Rachel McEneny, the city’s commissioner of administrative services.

By 11 a.m., the attack was mostly over. Critical systems were intact, including human resources data, and there was no interruption of 911 calls or water service. The city shut down the attack before the ransomware hit the payroll and purchasing programs, but the hard and expensive work of restoring data and repairing damage had just begun, McEneny said.

Some data, such as building permits, was lost and took months to restore, McEneny added.

The attack cost Albany $300,000 for software, hardware and consultants, and the city increased its cybersecurity budget by 25%.

The lessons learned included the need to buy cybersecurity insurance and to bolster staff and resources. Managers also reminded workers to adhere to “cyber-hygiene.” That means workers need to be reminded that they can’t use their government computers or programs to check personal email, social media or to shop online.

Within a few hours of the attack, it was believed to have been stopped by blocking off programs, including police and fire dispatching, 911 emergency calls, and all the essential, daily services of local governments. Other data, such as reams of birth and death records, was lost, and officials believed the database called “vital statistics” may have attracted the hackers' attention as a prime target. Those records were restored by painstakingly typing them in over weeks.

“Ninety-five percent of ransomware attacks are from human error,” McEneny said in an interview with Newsday. City Hall had to tear down the “fiefdoms,” common among many government departments, that can resist change or think ransomware is just an IT problem, she said.

Yonkers was attacked during the Labor Day holiday in 2021.

“The first hour was very frantic,” remembers Bob Cacace, commissioner of information technology for the City of Yonkers. First, the email system went down, which he said is “typically the first indicator.” Then, a message appeared on screen: “There was a button that said, ‘Click here to see how much you pay us.’ We didn’t click it.”

Yonkers was able to rely on backup data. But that wasn’t the end of the problem. Technicians still had to chase, catch and eliminate the ransomware as it raced to disrupt more programs. Officials decided to replace desktop computers rather than risk an incomplete wiping of the malicious software, or malware.

“It’s really like stepping on cockroaches. You have to get them all,” Cacace said.

After 36 hours, the attack was mostly over, and recovery was underway. The attack cost Yonkers $400,000, and it increased its cybersecurity budget by 50%.

Rather than supersizing its one-person cybersecurity office, Yonkers sought the best in private industry to bolster its protection. The ransomware attack plowed through three software programs that were designed to stop malware. The ransomware also deleted another safeguard that recorded the typed-in codes.

“It even erased the keystrokes,” Cacace said.

Yonkers changed its antivirus software and also joined the state's new cybersecurity task force with Albany, Buffalo, Syracuse and Rochester. The Joint Security Operations Center based in Brooklyn provides assets expertise to local governments. That includes CrowdStrike, a detection and response service provided for free.

“So, when something comes into our network, they can see it, too,” Cacace said. "Typically, these come in on nights or weekends when people aren’t there. Now we have a 24-7 team.”

“But it’s not easy,” he said. “These bad actors are pretty clever.”

Yonkers is still analyzing every computer program, down to the program that raises the arm to allow cars into a city parking garage. They are requiring “permissions” at every point that could block ransomware, which often gains network access through a single permission.

One cost Yonkers passed on was cybersecurity insurance.

“The cost for us would be about $400,000 a year and doesn’t cover much for ransomware,” Cacace said. “I’d rather spend it on prevention than insurance.”

Both cities know this isn’t a problem that can be dismissed in one budget year. The vigilance and the cost in staff and funds never ends, they said.

“We’re concerned we’re not buttoned up enough today,” Cacace said last week, 13 months after Yonkers’ ransomware attack.

By Michael Gormley

michael.gormley@newsday.com@GormleyAlbany

Michael Gormley has worked for Newsday since 2013, covering state government, politics and issues. He has covered Albany since 2001.