ChatGPT test shows how AI can be fooled

There’s more evidence that ChatGPT won’t put IT security teams out of work — yet.

Researchers at Endor Labs tested ChatGPT 3.5 against 1,870 artifacts from the PyPi and NPN open-source repositories of code. It identified 34 as having malware. However, only 13 really had bad code. Five others did have obfuscated code but did not expose any malicious behavior, while one artifact was a proof-of-concept that downloads and opens an image via  an NPM install hook. As a result, the researchers considered ChatGPT-3.5 right 19 out of 34 choices.

However, 15 of the results were false positives.

The researchers also found the version tested can be tricked into changing an assessment from malicious to benign by using innocent function names, including comments in a query that indicate benign functionality or through the inclusion of string literals.

Large-language model-assisted malware reviews “can complement, but not yet substitute human reviews,” Endor Labs researcher Henrik Plate concluded in a blog.

However, the most recent version is ChatGPT-4, which Plate acknowledged gave different results.

And, he admitted, pre-processing of code snippets, additional effort on prompt engineering, and future models are expected to improve his firm’s test results.

Researchers say large language models (LLMs) such as GPT-3.5 or GPT-4 can help IT staff assess possible malware. Microsoft is already doing that with its Security CoPilot application.

Still, the researchers’ conclusion is: ChatGPT-3.5 isn’t ready to replace humans.

“One inherent problem seems to be the reliance on identifiers and comments to ‘understand’ code behavior,” Plate writes. “They are a valuable source of information for code developed by benign developers, but they can also be easily misused by adversaries to evade the detection of malicious behavior.

“But even though LLM-based assessment should not be used instead of manual reviews, they can certainly be used as one additional signal and input for manual reviews. In particular, they can be useful to automatically review larger numbers of malware signals produced by noisy detectors (which otherwise risk being ignored entirely in case of limited review capabilities).”

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Howard Solomon
Howard Solomon
Currently a freelance writer. Former editor of and Computing Canada. An IT journalist since 1997, Howard has written for several of ITWC's sister publications, including Before arriving at ITWC he served as a staff reporter at the Calgary Herald and the Brampton (Ont.) Daily Times.

Related Tech News

Get ITBusiness Delivered

Our experienced team of journalists brings you engaging content targeted to IT professionals and line-of-business executives delivered directly to your inbox.

Featured Tech Jobs