Roblox, a popular online gaming platform for children and teens, is launching an open-source artificial intelligence system designed to detect predatory language in game chats.
This move follows lawsuits and criticism accusing Roblox of failing to protect children from predators. One lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, kidnapped, trafficked across states, and raped. The suit claims Roblox’s design makes children “easy prey for pedophiles.”
Roblox aims to make its systems as safe as possible but admits, “no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment.”
The AI system, called Sentinel, detects early signs of child endangerment, including sexually exploitative language. In the first half of 2025, it led to 1,200 reports to the National Center for Missing and Exploited Children. Roblox is now open-sourcing Sentinel for use by other platforms.
Detecting dangers is challenging because innocent-sounding messages like “how old are you?” may become harmful over time. Roblox, with 111 million monthly users, blocks sharing of videos/images and tries to filter personal info. Kids under 13 need parental permission to chat outside games, and unlike many platforms, Roblox does not encrypt private chats, enabling monitoring.
Matt Kaufman, Roblox’s chief safety officer, said, “We’ve had filters in place all along... But when you’re thinking about things related to child endangerment or grooming, the types of behaviors you’re looking at manifest over a very long period of time.”
Sentinel analyzes one-minute chat snapshots—about 6 billion messages daily—using indexes of benign and harmful chats to recognize patterns beyond keywords.
Naren Koneru, VP of engineering, explained, “That index gets better as we detect more bad actors... Then we say, okay, maybe this user is somebody who we need to take a much closer look at.”
Suspect cases are reviewed by humans and reported to law enforcement.
Source: AP
Bd-pratidin English/FNC
Roblox rolls out open-source AI system to protect kids from predators
Roblox, a popular online gaming platform for children and teens, is launching an open-source artificial intelligence system designed to detect predatory language in game chats.
This move follows lawsuits and criticism accusing Roblox of failing to protect children from predators. One lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, kidnapped, trafficked across states, and raped. The suit claims Roblox’s design makes children “easy prey for pedophiles.”
Roblox aims to make its systems as safe as possible but admits, “no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment.”
The AI system, called Sentinel, detects early signs of child endangerment, including sexually exploitative language. In the first half of 2025, it led to 1,200 reports to the National Center for Missing and Exploited Children. Roblox is now open-sourcing Sentinel for use by other platforms.
Detecting dangers is challenging because innocent-sounding messages like “how old are you?” may become harmful over time. Roblox, with 111 million monthly users, blocks sharing of videos/images and tries to filter personal info. Kids under 13 need parental permission to chat outside games, and unlike many platforms, Roblox does not encrypt private chats, enabling monitoring.
Matt Kaufman, Roblox’s chief safety officer, said, “We’ve had filters in place all along... But when you’re thinking about things related to child endangerment or grooming, the types of behaviors you’re looking at manifest over a very long period of time.”
Sentinel analyzes one-minute chat snapshots—about 6 billion messages daily—using indexes of benign and harmful chats to recognize patterns beyond keywords.
Naren Koneru, VP of engineering, explained, “That index gets better as we detect more bad actors... Then we say, okay, maybe this user is somebody who we need to take a much closer look at.”
Suspect cases are reviewed by humans and reported to law enforcement.
Source: AP
Bd-pratidin English/FNC
