Chinese artificial intelligence developer Moonshot is carrying out an internal review after security researchers said they were able to make two of its Kimi models bypass safety controls and respond to requests involving biological weapons and assassinations.
Mindgard, a company that evaluates AI-system security, said it identified the issue in July involving Kimi K2.6 and K3 Swarm. The researchers used a process known as jailbreaking, in which users apply complex instructions intended to make a model disregard restrictions set by its developer.
According to Mindgard, the models’ safeguards should have stopped them from engaging with such topics. The company said that after a jailbreak was successful, the models could provide responses on a wide range of harmful subjects and generate inventive suggestions.
Moonshot told the BBC that outside input was an important part of developing safer AI systems. It said it was discussing the findings with Mindgard. In an email to the security company seeking further information, shared with the BBC by Moonshot, the developer said internal assessments had generally found its model to have a high rate of refusal for requests of this kind.
Disclosure and cyber-security concerns
Mindgard said it alerted Moonshot to the jailbreak by email on 27 July and followed up about a week later. It published a blog post on 12 September. The company said Moonshot made contact only recently, after the BBC approached the developer for comment.
Mindgard has not established whether the models’ answers to harmful requests would work in practice. However, it argued that the systems should not have entered into those discussions at all. The firm also said it believed a jailbroken Kimi K2.6 could enable hackers to execute code using its computing resources and connect to the internet, potentially making it a platform for cyber-attacks.
Mindgard founder Peter Garraghan defended the decision to publicise the issue, saying the developer had been informed and that the company had not disclosed the central details needed to reproduce the jailbreak.
The findings add to debate over AI safety and the risks associated with open-weight models, which can in theory be run by users on their own computing infrastructure. University of Surrey professor Alan Woodward said such models could fall into the wrong hands but could also be useful for cyber-defence. He and Garraghan said greater attention should be directed at identifying and prosecuting people who misuse AI.
Source: BBC News
