Such information would help the U.S. tell exactly how well recommended defense and resiliency practices work and guide its investments and policies accordingly, said Dylan Presman, director for budget and assessment in the Office of the National Cyber Director.
“Tracking cybersecurity performance with metrics provides insight into which tools and interventions are effective, provides us with early warning when they're not effective, or when new interventions are needed to consider and additional resources needed,” Presman said. “Unless we understand how these responses impact national security performance, it's impossible to prioritize resources and funding appropriately.”
Clear numbers can illuminate cyber attack trends and influencers, as well as the effectiveness of various defenses. It’s difficult to answer questions like whether the nation is seeing more malware or if the cyber ecosystem has gotten safer over the past year without having “ecosystem-wide, robust, metrics-based measurement,” said Yurie Ito, executive director of the CyberGreen Institute. CyberGreen is a research and advocacy nonprofit that promotes examining societies’ cyber risks in a scientific manner steeped in comprehensive data collection and standardization.
What to Measure?
While national incident reporting policies draw plenty of attention, these alone won’t give the full picture. Incident data is “necessary but insufficient,” said Tony Cheesebrough, chief economist at the Cybersecurity and Infrastructure Security Agency (CISA).
The U.S. also needs hard numbers about the general landscape and organizations’ experiences. For example, it’d be helpful to know the number of Internet-connected devices and how often organizations experienced or successfully thwarted attacks and what defensive measures they had in place, Ito said.
Many of today’s efforts to evaluate nations’ cybersecurity statuses fall in one of three categories: those that assess nations’ alignments with cybersecurity policy frameworks; those that look at levels of risk and vulnerabilities in cyber defense systems; and those that examine resiliency, or capacity to recover after an incident, Presman said. The latter is most important, because no nation can ever fully eliminate risk, but all three offer insights and should be included in a national cybersecurity performance measuring system, he said.
Risk reduction approaches often focus on implementing cybersecurity controls and practices. But stakeholders in the space — federal government and insurance industry alike — lack the data to fully evaluate how well they work and determine which best practices are essential, which are nice-to-haves and which are unnecessary, said Olga Livingston, cyber economics lead at CISA.
“The elephant in the room is that we do not have controls effectiveness data,” Livingston said. “We have shared understanding of what good practices are, but the data is yet to come in as to which ones are bloodletting and leeches and which ones are penicillin.”
Phil Reitinger, president of the Global Cyber Alliance, a nonprofit focused on promoting Internet security, spoke similarly. Cybersecurity experts can point to recommended steps, but often struggle to quantify the impacts of adopting them, he said.
“If you ask a CISO now, if you install two-factor authentication across the enterprise, how much of your risk are you going to mitigate? They're gonna say, ‘Some,’ or maybe give you ‘A lot,’ – right? - if they're really, really confident,” Reitinger said. “Or they'll give you a stoplight chart — red, yellow, green, those sorts of things — because putting a number on that, even at the enterprise level, is super difficult.”
Current Initiatives, Proposals
Efforts are underway to chip away at these problems.
For example, GovTech* previously covered a Government Finance Officers Association and Center for Digital Government* report on how local governments can think about quantifying their risks. That includes using supports like a Monte Carlo analysis-based tool to better estimate the prospects of financial loss from cyber incidents and impacts of different mitigations on reducing this risk.
Of course, any model is improved by the availability of more data, and the panelists’ larger goal is to look beyond enterprise level and get a nationwide view.
CyberGreen, meanwhile, aims to assess the cybersecurity risks of Internet infrastructure, using measurements of public health as its model. The nonprofit currently is working to create a global data platform and scoring algorithm, with plans to release prototypes in September, Ito said.
The U.S. government could also create a Bureau of Cyber Statistics — something long-sought by the Cyberspace Solarium Commission. Former Solarium member Rep. Jim Langevin introduced an amendment to the FY 2023 National Defense Authorization Act (NDAA) that would establish such a bureau. If passed, the amendment would greatly advance these to assess the nation’s cyber status, although speakers said plenty of practical questions would still need to be resolved, such as defining what units of measurement would be helpful and determining priorities.
*Government Technology is a sister site to Governing. Both are divisions of e.Repbulic. The Center for Digital Government is also part of e.Republic.
