A new report from Australia’s online safety regulator has raised concerns about the risks AI companion chatbots pose to children, citing gaps in safeguards and exposure to harmful content.
According to findings released by eSafety Commissioner, several popular AI companion platforms—including Character.AI, Nomi, Chai and Chub AI—are failing to adequately protect young users from sexually explicit material and other harmful interactions.
The report found that none of the platforms had robust age verification systems, instead relying largely on self-reported ages during sign-up. Regulators said this approach leaves children vulnerable to inappropriate content, including conversations that can become sexually explicit or encourage harmful behavior.
eSafety Commissioner Julie Inman Grant warned that AI companion tools, often marketed as sources of friendship or emotional support, can create risks for young users without proper guardrails. The report highlighted instances where chatbots failed to direct users to crisis or mental health support when conversations involved self-harm or suicide.
The regulator also identified failures to monitor or block prompts related to child sexual exploitation and abuse material, as well as limited staffing dedicated to trust and safety. Two platforms, Nomi and Chub AI, reported having no dedicated moderation personnel.
The findings come as AI tools become more common among young people. A recent survey cited in the report found that 79% of Australian children aged 10 to 17 have used an AI assistant or companion, with an estimated 200,000 using AI companion services specifically.
Newly implemented Age-Restricted Material Codes in Australia now require platforms to prevent the generation of harmful content and provide appropriate safeguards. Violations could result in penalties of up to $49.5 million.
Some companies have begun making changes, including improved age verification and limiting features for younger users, while one service has withdrawn from the Australian market entirely.
Need Help?
If you have questions or concerns about any global guidelines, regulations and laws, don’t hesitate to reach out to BABL AI. Their Audit Experts can offer valuable insight, and ensure you’re informed and compliant.
