An AI system mistakenly caused armed police to swarm a 16-year-old student after it mistook a packet of chips for a dangerous weapon.
Taki Allen admits that he thought he was going to die after the armed cops swooped down on him as he sat with his friends after football practice at Kenwood High School in Baltimore on Monday.
Taki Allen had a bag of Doritos, not a firearm
The police officers pointed guns at Taki, made him get onto the ground, and then detained him, all the while believing that he had a gun, when he only had a bag that once contained Doritos tortilla chips.
The police were given the tip-off from the Baltimore County Schools’ AI gun detection system, which mistakenly identified the crumpled-up packaging as a firearm.
Now, Taki is traumatized and said that he feared for his life.
Taki Allen. Credit: FOX45
Speaking to FOX45 News, he said: “I just in that moment, I didn’t feel safe. I didn’t feel like the school actually cared about me.
“Because nobody came up to me after, not even the principal.
“It was mainly like, am I gonna die? Are they going to kill me?
“They showed me the picture, said that looks like a gun — I said, no, it’s chips.”
Taki and his family are astounded by what happened and have hit out at the aggressive response of the police.
His grandfather, Lamont Davis, said: “God forbid, my grandson could be dead if he flinched or twitched.”
How does the AI system work - or not?
The system used in the school works by utilizing existing school camera infrastructure and then alerting the school safety team and law enforcement when it detects something.
Omnilert, the company behind the AI security software, said that the image ‘closely resembled a gun being held’ although it admitted it was a ‘false positive’.
However, in defence of the system, they said that it ‘functioned as intended: to prioritize safety and awareness through rapid human verification.’
Omnilert will now review the incident in order to improve the system’s accuracy.
Omnilert defended their system, but admitted a 'false positive'. Credit: Omnilert
It also stated that the purpose of the AI is to supplement rather than replace the judgment of real humans.
That message was conveyed to parents in a letter from the school system, as well as offering counselling to affected students.
However, Taki says he hasn’t heard anything.
“I don’t feel like going out there anymore,” he said.
“If I eat another bag of chips or drink something, I feel like they’re going to come again.”