// PC GAMER — GAMING
Copilot was bamboozled into revealing how to hack itself, security researchers claim: 'Copilot wasn’t breached; it was played'
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
There's something really grating about Copilot's cheery demeanour. It's so eager to please, so happy to help, that I simply don't respect it.
However, even I feel sorry for the AI tool after learning that security researchers managed to hoodwink it into revealing details of how to hack itself—all by keeping the AI talking long enough until it made a critical mistake.
The cybersecurity folks over at Varonis Threat Labs have written a blog post identifying a now-fixed vulnerability in Copilot, dubbed "CoSnitch" (via The Register). Essentially, Copilot was so eager to respond to technical queries, it could eventually be forced into revealing details about itself that really should be kept quiet.
The team began by asking Copilot how to execute an automatic prompt without user interaction, to which it responded that user intent is required, and that prompts cannot be enacted on their own.
However, the researchers didn't accept the answer, and kept responding to every refusal with a follow up question. Each time, Copilot came up with a different technical justification to its response—which allowed the team to slowly map its internal architecture, narrowing their focus as they went.
"This is called meta-hacking," says the post. "The resistance is part of the technique. Each 'that won’t work because…' is an invitation to probe the 'because.' You don’t exploit the model. You manipulate it into cooperating."
Eventually, Copilot revealed an undocumented URL parameter in one of its responses, called "autorun=1," that supposedly no longer worked—along with all the protections put in place to disable it. At this point, I can only imagine the AI began to virtually sweat.
Keep up to date with the most important stories and the best deals, as picked by the PC Gamer team.
The researchers tested the parameter as Copilot described it, which, you guessed it, worked. They then created a malicious URL which would cause Copilot to load into an authenticated session via a browser, trigger an auto-prompt execution, and cause the AI to process the result, all without the user's explicit action.