Governance & Risk Management , Risk Assessments

Are Cybersecurity Performance Measures Realistic?

Government Watchdog Urges ONCD to Develop Outcome-Oriented Performance Measures
Are Cybersecurity Performance Measures Realistic?
Setting metrics for cybersecurity isn't easy, White House officials said. (Image: Shutterstock)

A government watchdog urged the White House to establish metrics that would help determine the effectiveness of federal cybersecurity initiatives, but it's a lot easier to recommend developing outcome-oriented performance measures for cybersecurity than it is to actually develop them.

See Also: Securing Healthcare: Minimizing Risk in an Ever-Changing Threat Landscape

The Government Accountability Office in a report Thursday called on Office of the National Cyber Director to establish performance measures and estimate implementation costs. GAO said the Department of Treasury collects data on the dollar value of annual ransomware-related incidents, which the watchdog said "demonstrates that developing such measures is feasible and can be used for measuring effectiveness."

Office of the National Cyber Director staff pushed back on GAO's recommendations, according to the report, stating "it was not realistic to develop outcome-oriented measures at this point" and that "estimating the cost to implement the entire strategy was unrealistic."

Experts say it can be particularly challenging to develop performance measures in cybersecurity due in part to the constant, rapidly evolving and complex threat landscape, as well as the complexity of cyber systems and emerging threats.

"Even if you pen test a cyber system today, you don't know whether attackers will discover a new zero day tomorrow," said Jim Dempsey, a cybersecurity expert and lecturer for UC Berkeley Law. Dempsey previously criticized the Transportation Security Administration in 2022 over a set of directives issued in response to the Colonial Pipeline ransomware attack that he said lacked measurable outcomes.

"Instead of developing faux performance metrics, we should focus on controls," Dempsey recommended, saying technical controls like multifactor authentication and resetting default passwords "have large ROI" and "adherence to them can be measured."

"There are actually very few - if any - metrics for cybersecurity that are truly performance- or outcomes-based," he noted.

GAO warned that ONCD "will be limited in its ability to demonstrate the effectiveness" of the national cybersecurity strategy until the office establishes adequate, outcome-oriented performance measures.

In its response to the report, ONCD partly agreed with GAO and said that "developing outcome-based performance measures for cybersecurity is a challenging topic."

The office said it will assess initiatives included in the cybersecurity strategy that lend themselves to outcome-oriented performance measures and apply those measures to initiatives "to the extent that validated measures exist."

Developing outcome-oriented performance measures can be difficult, but there are benchmarks and indicators that can and should be used to gauge the effectiveness of national level cybersecurity initiatives, said Austin Berglas, a former FBI cyber division special agent, now an executive at cyber defense platform BlueVoyant.

Those indicators include incident response times, the percentage of systems patched, total unresolved vulnerabilities, the number of employees trained on up-to-date cybersecurity practices and the cost of a cybersecurity incident, Berglas said, adding that a "lower overall cost to recover and respond to an incident may be an indicator of effective measures."

About the Author

Chris Riotta

Chris Riotta

Managing Editor, GovInfoSecurity

Riotta is a journalist based in Washington, D.C. He earned his master's degree from the Columbia University Graduate School of Journalism, where he served as 2021 class president. His reporting has appeared in NBC News, Nextgov/FCW, Newsweek Magazine, The Independent and more.

Around the Network

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing, you agree to our use of cookies.