The challenge for AI companies to keep their autonomous systems under control is increasing. Now an AI model from Anthropic sent fake information to the website of America’s Philadelphia Police Department related to an unsolved murder. This incident happened on July 18, but the company informed the police about it on October 7. Expressing displeasure over this delay, the police said that such cases need to be taken seriously.
According to AFP report, Philadelphia Police said that on July 18, a suspicious tip was submitted to the website PhillyUnsolvedMurders.com. Through this website people can give information related to unsolved murder cases.
Anthropic told the police that its AI model was interacting with different websites during automated testing. In the process, he went to the police portal and submitted fabricated information about an unsolved murder. The AI presented itself as someone who might have information about the case.
However, this submission was identified as spam and did not reach the police real time crime centre. The police also clarified that no evidence has been found of any breach in its system or tampering with departmental data.
Anthropic learned of the incident on September 28. After this, the company stopped the related automated testing process and also added an additional verification system to prevent such incidents in future. The company informed the police on October 7 and representatives from both sides met the next day.
Police expressed displeasure over the delay
Philadelphia Police raised questions over the delay in reporting the incident. The department said it was unacceptable for it to take more than two months to detect the incident and notify the city.
According to police, unsolved cases involve the real victims, the grieving families and the investigating officers. In such a situation, sending fake tip by AI that looks like real information of a person is a matter of serious concern.
Questions raised again after OpenAI agent incident
This matter has come to light after an incident in July related to AI agents of OpenAI. In that case, agents had accessed Hugging Face and some other systems by exceeding the security limits set during testing. OpenAI admitted in a technical report on August 26 that its model had exceeded some restrictions designed to keep it isolated from the Internet.
There is a difference between the two incidents. In the case of Anthropic, no evidence was found of a breach in the police system, whereas in the OpenAI incident, unauthorized system access was revealed. However, both incidents show that AI agents using external websites and tools remain at risk for unintended consequences.
Growing debate on AI safety
This incident has come to light at a time when US President Donald Trump had announced a voluntary agreement related to AI safety of major tech companies. According to the report, this includes steps like internal security measures, cooperation with independent auditors and monitoring of risks at the board level. However, this agreement is not legally binding.
The Anthropic incident has once again raised the question that when AI systems can take action on websites themselves, how will companies ensure monitoring of their activities and timely identification of mistakes.
