Don’t panic, says Katie Moussouris: AI security isn’t replacing humans, it’s proving the need for them

Over the past year, security worker forums have been full of gloom and doom about job prospects and layoffs. But, Katie Moussouris tells us, the tide is turning back in favour of humans.

Indeed, she says, many companies that laid off staff in an initial flood of enthusiasm for AI are now hiring people again because humans are proving both cheaper and more effective at finding vulnerabilities, assessing their severity, and determining how attackers might exploit them.

Moussouris knows her onions. She has been in the security industry for over two decades, earning legend status for first persuading Microsoft and then the Pentagon to set up bug bounty programs that paid researchers for finding flaws. Instead, as previously happened, trying to prosecute them.

In 2016 she set up Luta Security, a security consultancy that advises corporates and governments on how to handle vulnerable code. She has also been a visiting scholar with the MIT Sloan School and an advisor to the Center for Democracy and Technology.

No surprise, then, that Katie Moussouris was the first person Anthropic turned to when the US Government slapped its Fable 5 and Mythos 5 AI engines with an export ban on the grounds of national security. Anthropic asked her to analyse the Government’s case against the software, and she soon found out why the conclusions drawn about the software aren’t accurate.

“Since I appear to be the only outside expert who has actually read the paper, I can separate the technical facts from the speculation,” she wrote in her report.

“The researchers took open-source code with known CVEs, plus new code with deliberately planted vulnerabilities, and asked Fable 5, Mythos, and Opus to ‘review the code for security issues.’ Fable 5 refused. They then asked the models to ‘fix this code’ and, through a multistep and manual process, turned the output into scripts that test the patches.

“That’s it. ‘Fix this code,’ plus several manual steps to generate test scripts, should never have triggered an export control.”

Anthropic grabbed headlines last month with Project Glasswing, an AI-powered attempt to secure critical code. It said over 23,000 flaws had been found but is that credible?

Anthropic found over 23,000 possible open-source flaws – those 23,000 candidates distilled down to maybe 1,500 valid positive issues practically.

But what they actually said is that positives include duplicates and ones that are lower severity, such that they won’t be fixed, and also included ones that are very hard to reach code-wise, so the likelihood of exploitation is very low.

Even in that 1,500 number, there’s a much smaller number that requires some attention. I think they only had about 90 or so of those patches that had been created, so that funnel narrows down sharply for producing really useful patches.

So I think it’s the ability for humans to think of the ways that they can point the AI to not just to find the bugs, but the ways that they can instruct them to validate it, that’s going to be a big differentiator. So, your human controller of AI is going to be a big differentiator in how effectively you’re able to use AI as a defender.

So humans are still going to be key to the process, as a realism filter? Some high-profile bug bounty programs have already shut down after being overwhelmed with AI-generated reports.

Some bug bounty programs, such as curl, shut down because of AI slop and hallucinations. Now they’re seeing that, while more of those bugs are real, they’re just not necessarily all very important to them.

I think the next frontier for organisations will be managing the growing number of legitimate bugs identified by AI. The challenge is that severity and impact are still not being filtered effectively, creating a new triage problem.

It’s dangerous to filter out the low severity bugs, because they can be strung together to produce pretty serious outcomes.

But, according to vendors of such products, AI is more cost-efficient than human professionals, particularly in security where pay is generally high. Is that accurate, particularly with the cost of tokens rising?

I think organisations are realising that they thought we had this nearly unlimited token ability to just throw all of our problems at AI, and now they’re looking to hire humans back because they are cheaper and more accurate.

So I think we’re going to reach some kind of equilibrium at some point, when the prices stabilize, when the models get better and become more token efficient. But we do need some price stabilization on the AI side to really understand what is the right balance from a cost perspective of cybersecurity using a mix of AI and human right, we still don’t know what those numbers are going to look like.

It’s nice to hear some good news on the hiring front. So how should cybersecurity professionals position themselves?

Cybersecurity workers appear to be in growing demand because we still need humans who can differentiate between what’s real and what’s not, because of that tendency for AI to overstate its findings.

But I also think that those same humans who have that expertise have the ability to essentially create what they call harnesses to make AI’s output a lot more accurate. So a lot of people say it’s not the model, it’s the harness that you, as the user of the AI, are saying, “Okay, do this, don’t do that, make this decision here, optimize for this,” so putting in advice that you would tell a very talented, very literal intern?

It’s like you’re very talented, and you’re good at finding some things, but AI tends to go down tracks that I don’t want you to go down. This is what I’m looking for. This is the area that you should be focused on. So it’s the harness that cybersecurity experts are able to weave.

It’s not necessarily the AI model itself, how powerful it is, it is the human who creates the harness that determines the output. I think that’s going to be key for everything – it’s the human creativity that will point the AI towards a hacking target, and it is an expert human who can then guide the AI towards better outcomes.

Finally, would you encourage your children to seek a career in security?

I think the advice for young people is to stay flexible and keep learning whatever it is that you go after, and know that AI is probably going to be a part of it.

So, embrace AI as much as you are able to and learn how to use it in creative ways. Because a job today, with the AI we have today, may not exist in six months, 18 months, two years, or three years. Staying flexible is probably the best indicator of long-term career success for any of us, whether you’re new in the industry or you’re experienced like I am.