During the National Center for State Courts (NCSC) eCourts conference in Las Vegas this week, court administrators and CIOs explained what went wrong and the lessons they learned about recovery and prevention.
How Texas Got Hit
Hackers likely used a phishing campaign to take over a regular user email account, then used a zero-day exploit to grant the account administrator-level privileges. From there, they moved laterally to find a juicier target.
“We could watch them jump from server to server until they found our domain controller … the machine that stores all your usernames and all your passwords,” Kennedy said.
Attackers then attempted to introduce a variety of viruses, but the anti-virus thwarted most attempts — until perpetrators switched to a more subtle, living-off-the-land style attack.
Attackers opened the Notepad application and suspended the application from memory to stop it running. They next wrote a virus into Notepad in memory and then unsuspended it, Kennedy said. This tricked the system into thinking it was just running a legitimate program — Notepad — when in truth it was now running a virus. Perpetrators were able to then deploy the virus throughout computers on the network.
The Post-Incident Window of Opportunity
There was one silver lining, though. Following a cyber incident, the non-IT sides of government tend to become newly receptive to cybersecurity proposals, and abandon complaints about defense measures causing frictions. That mindset lasts about six months, Kennedy said, and is an opportunity to push through policies like strong password requirements, mandatory multifactor authentication (MFA) and automatic installations of new software updates.
New attention on improving password policies became important for Alaska, too, after its own incident hit.
“Eighty-six percent of our passwords were hacked in less than four hours,” Alaska State Court Administrator Stacey Marz recalled. “We [had] a lot of repetitive passwords like ‘Alaska123.’”
Alaska Faced Imminent Encryption
The Alaska courts’ own ransomware incident came in 2021 and presented a particularly challenging problem for a court system that had relied on outsourcing key security services.
In April, cybersecurity software detected unusual activity. An external cybersecurity consultant concluded it was the lead-up to an “imminent” ransomware attack, Marz said.
The court needed to cut external Internet access to prevent the attack from progressing and knew any delay gave hackers more chances to encrypt. At the same time, the cyber specialist was located “four time zones away” and needed remote access to review logs and put tracing software inside the networks to better understand the attack and extent of the damage.
Theoretically, the court’s firewalls could be reconfigured to deny everyone except the consultant, but no one in-house had the firewall expertise to do this.
“You have to really think about the vendors you're working with,” Marz said. “We had outsourced our firewall roles, and that was a major problem for us.”
Marz determined a deadline when she’d cut connection, no matter what. Finally, with two hours to spare and with the help of the consultant, the team figured out the needed firewall configurations.
Going Without Internet
Alaska took its court system offline as work continued to ensure the perpetrators were fully removed from the network, then to rebuild systems, bolster security and restore from backups.
The courts had to proceed without Internet for about a month, which stopped everything from e-filing and online bail postings to Zoom hearings and digital payroll systems. As staff reverted to manual processes, they turned to conducting remote hearings by phone and using physical drop boxes and old fax machines.
“We broke out the fax machines,” Marz said. “Months earlier I had said … why do we have these things anymore? And, luckily, we hadn’t thrown them out yet.”
Alaska courts revised their approach in the aftermath, including training up staff to bring certain skills in-house; planning backup, alternative methods to Internet-based functions; and modernizing unpatched legacy systems that had lingered due to budget constraints and because the tools were helpful to business functions even if they weren’t secure.
Georgia Turns to the Cloud for Security
Georgia also used a ransomware attack as an opportunity to modernize. When a June 2019 incident downed services, the court decided to bypass restoring legacy systems and instead rebuild in the cloud to bolster future resilience, said Jorge Basto, CIO of the Cherokee County Clerk of Courts and former CIO for the state Administrative Office of the Courts.
When the incident downed Georgia courts’ websites and servers, the court turned to partners like the National Guard and FBI for help. But there came a point where the court needed to take charge and reassert its own priorities over those of its partners. While law enforcement was focused on investigating the incident, Georgia wanted to push focus onto getting back online.
“We have 50, 60 people running around this office — everybody’s helping, everybody’s doing something,” Basto said. “But guess what? They’re looking for bad guys … meanwhile, my network’s not coming back … [The FBI,] they’re not just there using up your resources, they’re taking your people, your focus.”
The court team began recovering copies of its data from vendors and other agencies. As they worked to restore services, the court’s main, public-facing website was a high priority. That site would be a first stop for residents hearing about the incident and trying to find out more. If the website was down, people might panic, Basto said.
Reducing the Chances
Today the question isn’t if or even when organizations will be hit by a cyber attack, but how bad the damage will be, which makes planning for resilience essential, Basto said.
A variety of measures can also help reduce the chances and severity of attacks, with Kennedy recommending layered defenses, network segmentation, mock phishing campaigns to raise staff’s alertness and moving toward zero trust.
Speakers also pointed to NCSC resources, including its Joint Technology Committee’s regularly scheduled cyber webinars.
Government Technology is a sister site to Governing. Both are divisions of e.Republic.