What AI gets wrong about security

AI code assistants are good enough now that a developer can ship a feature mostly written by a model. The code usually compiles, often works, and sometimes is more readable than what a tired human would write. The part that gets people into trouble is the assumption that working code is secure code.

Models are trained to produce code that looks right, not code that’s safe. The vulnerabilities they introduce tend to be subtle, because they’re the average of a lot of insecure code written by humans.

It uses outdated APIs and patterns

A model’s training data includes years of code written against old versions of libraries and frameworks. Ask it to write something in a language where the secure way of doing things changed recently, and it may confidently produce the old, vulnerable pattern — the deprecated crypto call, the connection string with the password in it, the regex that was known to be unsafe.

The code runs, so nobody questions it. The fix is to treat AI output as a first draft and check it against the current version of the library you’re actually using.

It invents things that don’t exist

Models hallucinate library names, function signatures and configuration options. In security terms this matters in a specific way: a developer pastes in a call to a “validation” or “sanitisation” function that sounds right but doesn’t exist, and the code still compiles because the model also wrote the function — doing nothing.

This is hard to spot in review, because it reads plausibly. It’s worth explicitly checking every function the AI wrote that claims to validate, sanitise, authenticate or encrypt.

It skips the boring checks

Authentication and authorisation are the least glamorous part of an application, and they’re where AI code is weakest. The model writes the happy path — the feature that works when the right user calls it — and quietly omits the check that the wrong user can’t. Endpoints that should require login don’t. Objects fetched by ID aren’t checked against ownership.

A human reviewer skims for “does it work”. You have to specifically look for “who can call this, and what can they access”.

It doesn’t know your threat model

The model doesn’t know your users, your data or your compliance obligations. It writes generic code for a generic problem. The security holes that matter most to you — the personal data flow, the third-party integration, the internal API that should stay internal — are exactly the things the model can’t reason about.

How to review AI-generated code

The process is the same as reviewing human code, with the emphasis moved.

1. Diff against the current library version, not what the model assumes.
2. Check every function that claims to validate, sanitise, authenticate or encrypt actually does.
3. Audit every endpoint and data access for authorisation, not just functionality.
4. Run a static scanner on the result. Our SAST scanner checks each pull request, including AI-generated code, and flags the patterns above.
5. Test the running application before release. Our application security assessment catches the runtime problems static review misses.

None of this is a reason to stop using AI assistants — the speed gain is real. It’s a reason to treat the output the way you’d treat code from a new contractor: promising, but not trusted until you’ve checked the security-sensitive parts yourself.