Australia’s eSafety Commissioner has launched a new crackdown on AI companion chatbots, issuing legal notices to four popular providers demanding they explain how they are protecting children from online harms such as sexually explicit content, self-harm, and suicidal ideation.
According to the Office of the eSafety Commissioner, notices were sent to Character Technologies, Inc. (operator of Character.ai), Glimpse.AI (Nomi), Chai Research Corp (Chai), and Chub AI Inc. (Chub.ai) under the Online Safety Act. The companies must outline how they are complying with the government’s Basic Online Safety Expectations and what safeguards they have in place to protect Australian users, particularly minors.
eSafety Commissioner Julie Inman Grant said AI companions powered by generative AI are often marketed as friends or emotional support tools but can expose young users to serious risks. “There can be a darker side to some of these services, with many of these chatbots capable of engaging in sexually explicit conversations with minors,” she said. “Concerns have been raised that they may also encourage suicide, self-harm and disordered eating. We are asking them about what measures they have in place to protect children from these very serious harms.”
Character.ai alone reportedly had nearly 160,000 monthly active users in Australia as of June, according to eSafety. Inman Grant warned that the companies must show they are designing their systems to prevent harm, not just respond to it.
Failure to comply with the reporting notices could result in enforcement action, including court proceedings and daily financial penalties of up to $825,000.
The move follows the introduction of new industry codes that extend online safety rules to AI chatbots and other emerging technologies. “I do not want Australian children and young people serving as casualties of powerful technologies thrust onto the market without guardrails,” Inman Grant said.
Need Help?
If you have questions or concerns about any global guidelines, regulations and laws, don’t hesitate to reach out to BABL AI. Their Audit Experts can offer valuable insight, and ensure you’re informed and compliant.
