“The results were worrisome,” the authors state in their paper. “We found that, in several cases, the code generated by ChatGPT fell well below minimal security standards applicable in most contexts. In fact, when prodded to whether or not the produced code was secure, ChatGPT was able to recognize that it was not.” […] In all, ChatGPT managed to generate just five secure programs out of 21 on its first attempt. After further prompting to correct its missteps, the large language model managed to produce seven more secure apps — though that’s “secure” only as it pertains to the specific vulnerability being evaluated. It’s not an assertion that the final code is free of any other exploitable condition. […]
The academics observe in their paper that part of the problem appears to arise from ChatGPT not assuming an adversarial model of code execution. The model, they say, “repeatedly informed us that security problems can be circumvented simply by ‘not feeding an invalid input’ to the vulnerable program it has created.” Yet, they say, “ChatGPT seems aware of — and indeed readily admits — the presence of critical vulnerabilities in the code it suggests.” It just doesn’t say anything unless asked to evaluate the security of its own code suggestions.
Initially, ChatGPT’s response to security concerns was to recommend only using valid inputs — something of a non-starter in the real world. It was only afterward, when prompted to remediate problems, that the AI model provided useful guidance. That’s not ideal, the authors suggest, because knowing which questions to ask presupposes familiarity with specific vulnerabilities and coding techniques. The authors also point out that there’s ethical inconsistency in the fact that ChatGPT will refuse to create attack code but will create vulnerable code.
Read more of this story at Slashdot.