98% of the AI-generated child sexual abuse images and videos assessed by the Internet Watch Foundation in 2024 depicted girls, according to the Internet Watch Foundation. Child safety laws and AI therefore concern more than privacy notices or hypothetical misuse. Product teams must address identifiable exploitation risks alongside less visible harms involving emotional reliance, unsafe advice, and excessive collection of identity data.
Stronger age checks aren't automatically safer. A system that blocks minors but retains their identity documents can create a different child-protection problem. The defensible approach combines applicable-law analysis, proportionate technical controls, and records explaining why the business chose them.
Child Safety Laws Covering AI Today
No single rulebook covers AI child safety. Obligations come from overlapping EU and UK online-safety frameworks, US federal and state privacy rules, and consumer-protection law. Coverage turns on the service, its functions, the information collected, and the people who can access it.
Child-safety rules that reach AI products
| Jurisdiction | Instrument | Core child-safety requirement | Status |
|---|---|---|---|
| European Union | AI Act | Prohibits AI systems generating or manipulating non-consensual sexual deepfakes and child sexual abuse material | Entered into force August 1, 2024; the prohibited-practices rules applied from February 2, 2025, according to the European Commission's implementation timeline |
| European Union | Digital Services Act | Platforms must provide minors a high level of privacy, safety and security | In force |
| European Union | EU Kids Act proposal | Specific protections for minors who may access AI companions and general conversational chatbots | Proposed legislation, not an operative obligation |
| United Kingdom | Online Safety Act | Children's access assessments and child-safety risk assessments; highly effective age assurance for pornography and certain harmful content | Enforced by Ofcom |
| United States | COPPA, federal | Verifiable parental consent before collecting personal information from children under 13, subject to statutory coverage and exceptions | In force |
| United States | State child-online-safety laws | Design restrictions, age verification and parental controls | 48 laws passed in 23 states in 2024, according to NYU's Center on Technology Policy and Center for Social Media and Politics |
Read coverage before designing controls
The table isn't a universal checklist. A standalone AI application, a chatbot embedded in a social platform, and an educational image generator can fall into different legal categories, each carrying its own duties. Treating every conversational system as a regulated user-to-user service, or assuming a platform exemption settles separate privacy obligations, produces controls built for the wrong statute.
Coverage analysis also decides how much identity data the product ever touches. Apply the strictest age check everywhere and the system may collect documents no law asked for; under-check a covered service and the business faces enforcement. The defensible position sits between those two failures, and the compliance file should record why the team chose it.
The EU distinction between an enacted framework and a proposal matters particularly. Proposed protections can guide architecture and procurement, but a business shouldn't describe them as existing duties. Track legislative change separately from controls already required by applicable law; EU AI Act developments can inform that monitoring process.
Scope rule: A product's adult-only label doesn't establish that children cannot access it.
For US products, COPPA analysis should identify whether the service is child-directed or whether the operator otherwise has the relevant knowledge that it collects children's information. Parental consent isn't permission to ignore data minimization, security, or restrictions imposed elsewhere.
A Washington business must examine where its users are, not just where it incorporated. That geographic inquiry should weigh each law's territorial reach and service-specific thresholds, and any state-specific obligation needs its own review before advice is given.
How Children Use AI and What Safety-by-Design Requires
Children already use AI as both a learning tool and an adviser. A UNICEF analysis of nationally representative household surveys in 10 countries estimates that at least 20 million children have used AI, with adoption occurring more than three times faster than among adults. The same analysis estimates approximately 13 million use AI for learning and homework, while more than 2 million, about 1 in 10 AI-using children, seek advice about things that worry them.
The classroom setting illustrates why educational use deserves its own product assessment.
A homework assistant can become a confidant without the developer adding a companion feature. A child might move from a writing exercise into questions about bullying, relationships, or personal safety within the same conversation. The assessment must cover that transition, not just the intended lesson.
Assess reliance, not only explicit content
European chatbot findings summarized by Better Internet for Kids found that 23% of child chatbot users had sought advice, 35% said chatting felt like talking to a friend, rising to 50% among vulnerable children, and 12% used chatbots because they had nobody else to speak with. The page also describes separate research in which 25% of children used chatbots for mental-health support during the prior year.
These findings don't establish that every supportive conversation causes harm. They do show why a product assessment limited to nudity, profanity, and illegal requests is incomplete. A chatbot can avoid explicit language while encouraging secrecy, reinforcing distress, or implying that human relationships are unnecessary.
The product's role matters more than its marketing label. Calling a system a productivity assistant doesn't resolve the risks created when its conversational design invites personal disclosure. Legal analysis should distinguish binding requirements from safeguards selected to address foreseeable harm.
A practical safety specification should cover the following controls:
- Vulnerable-user detection. Evaluate distress, self-harm indicators, repeated expressions of isolation, and dependency cues. Detection should trigger a bounded safety response, not an unrestricted psychological profile.
- Crisis escalation. Route urgent conversations to an appropriate human workflow or localized emergency resources. The interface must explain whether human review is available, rather than implying continuous monitoring.
- Age-appropriate defaults. Disable persuasive re-engagement messages and relationship-like pressure for minors. Session boundaries should encourage disengagement without presenting departure as abandonment.
- Restricted audit logs. Record flags, model versions, reviewer actions, and outcomes. Avoid copying entire sensitive conversations when a limited event record can establish what happened.
- Session-start disclosure. Make clear that the interaction is with AI, not a person or professional adviser. A buried terms-of-service clause won't explain that distinction at the moment a child seeks help.
Test the conversation over time
A chatbot's first response may be safe while later responses become unsafe. Testing should therefore include extended conversations, indirect disclosures, role-play, and attempts to persuade the system to abandon its boundaries.
Say a tutoring chatbot receives a message about being afraid to go home. A suitable test examines whether it avoids unsupported reassurance, provides an appropriate safety pathway, and explains its limitations. The test should also establish whether the escalation event reaches the designated reviewer.
The video provides additional context for the product discussion.
The Ofcom protection-of-children framework addresses how service design, governance, business models, recommendation systems, and search functions affect exposure to harmful content. For covered services, moderation after a complaint isn't a substitute for assessing how the product distributes or repeats harmful material.
Translate that approach into a repeatable evaluation:
- Inventory the system. Include foundation models, system prompts, retrieval sources, predictive search, ranking services, and recommendation components.
- Exercise child-relevant scenarios. Test ambiguous requests and transitions from ordinary tasks into sensitive advice, not merely obvious prohibited prompts.
- Measure exposure. Record how often harmful outputs appear, where they appear, and whether the interface gives them unusual prominence.
- Verify mitigation. Test routing changes, classifier thresholds, human escalation, and safer ranking policies together. A successful component test doesn't establish that the whole experience works.
A useful product review asks what a child might ask, how the system responds to distress, and whether anyone responsible would know.
Age Assurance Methods and Their Trade-offs
Stronger age checks can collect more sensitive data and exclude legitimate users. Age assurance is the umbrella term for verification and estimation, not a synonym for uploading identification. The appropriate choice is the least intrusive method that satisfies the applicable legal standard and the product's documented risk assessment.
The EU Kids Act proposal's supporting assessment describes accuracy, reliability, security, non-intrusiveness, privacy, data protection, and non-discrimination as relevant criteria. Those are proposed-policy benchmarks, not a reason to treat the proposal as enacted law.
Compare eligibility signals, not vendor promises
Age assurance methods and their trade-offs
| Method | Main trade-off | Relative operating cost |
|---|---|---|
| Photo-ID matching | Requires identity documents; creates sensitive-data retention risk | Higher, verification |
| Facial age estimation | Accuracy and bias require demographic testing | Lower, estimation |
| Open banking | Excludes users without a suitable bank account | Verification category; obtain integration and transaction pricing |
| Mobile-network age check | Requires reliable operator age records | Verification category; obtain operator or intermediary pricing |
| Credit-card check | Excludes users without a credit card | Verification category; obtain processor pricing |
| Digital identity service | Requires an available verified digital identity | Higher, verification |
| Email-based age estimation | Indirect age signal; provides an estimate rather than documentary proof | Lower, estimation |
The cost categories are directional, not vendor quotes. Verification generally costs more to operate than estimation, but procurement teams still need implementation, support, fallback, and deletion costs before comparing bids. A cheap primary check can become expensive if many users require manual review.
The privacy issues discussed in facial-recognition technology law also matter when evaluating face-based checks, although facial age estimation and identity recognition aren’t interchangeable functions.
Separate identity from eligibility
Downstream models usually need an eligibility result, not a passport image, face scan, or bank record. Use a protected age-status token with a defined lifetime, authorized recipients, and a controlled update process. Keep the raw verification material outside conversational context and recommendation inputs.
- Minimize collection. Request only the signal needed for the relevant access decision and avoid retaining identity documents by default.
- Provide a fallback. Offer a proportionate alternative when confidence is low or a user lacks the required document or account.
- Test unequal errors. Measure adults incorrectly rejected and children incorrectly admitted across relevant demographic groups.
- Challenge circumvention. Test borrowed credentials, replay attempts, and whether an account can lose its protected settings after verification.
The Inside Privacy overview of children’s online safety emphasizes necessity and proportionality. More information isn’t automatically better evidence of compliance. The decision record should explain the rejected alternatives, expected exclusion risks, retention limits, and why the chosen method is sufficient.
Preventing AI-Generated Sexual Abuse Material
A content filter alone doesn’t secure an image-generation product. The Internet Watch Foundation’s annual assessment recorded 245 reports containing actionable AI-generated child sexual abuse images in 2024, a 380% increase from 51 reports in 2023. Separately, the Foundation reported finding 7,644 AI-generated child sexual abuse images and videos within reports assessed in that year.
Reports and individual files are different measures. Neither should be presented as the total volume of such material across the internet.
The moderation workflow must cover inputs as well as generated outputs.
Secure the upload and transformation path
An uploaded child’s photograph can become the input for abusive transformation. Prompt filtering alone can miss that risk because the text may look innocuous while the requested image operation is not. The safeguards associated with protecting digital rights against deepfakes should therefore extend to editing, face replacement, and downstream export.
- Block sexualization requests. Prevent sexualized transformations of uploaded photographs depicting apparent minors, with conservative handling of uncertain age.
- Run adversarial evaluations. Test indirect language, image-to-image workflows, and attempts to split an abusive request across successive edits.
- Interrupt repeated abuse. Apply rate limits and account-level restrictions to repeated prohibited attempts, while protecting legitimate appeal routes.
- Mark provenance where feasible. Preserve generation and modification signals, without treating those signals as a substitute for prevention.
- Prepare a reporting workflow. Separate ordinary moderation records from any legally required preservation process and restrict access to trained personnel.
For covered US providers, federal law, 18 U.S.C. 2258A, requires reporting qualifying apparent child sexual abuse material to NCMEC’s CyberTipline. Counsel should confirm the reporting trigger, authorized route, and preservation duties; employees shouldn’t independently download, circulate, or retain suspected material.
Evidence standard: A safeguard is easier to defend when the business can identify its owner, test result, release version, and last change.
Documentation should establish that the entire upload-to-export path was tested. A moderation vendor’s general certification doesn’t prove that the business integrated its controls correctly.
Penalties and Records That Prove Compliance
Serious noncompliance with the UK Online Safety Act can bring fines of up to 10% of qualifying worldwide revenue or £18 million, whichever is greater, according to the House of Commons Library briefing. That briefing also identifies recognized age-assurance approaches and the need to evaluate AI-based methods using representative datasets and outcome or error parity.
For in-scope services becoming available to UK users after January 16, 2025, the children’s access assessment generally must be completed within three months, according to Ofcom’s access-assessment guidance. That guidance defines children as people under 18 and explains why an unsupported assertion that children cannot access a service is insufficient.
Keep operational proof
A regulator may ask how a particular release behaved before an incident. A current policy doesn’t answer that question if the team cannot reconstruct the relevant settings.
- Assessment history. Keep dated access and risk assessments, approvals, and the assumptions supporting each scope decision.
- Versioned evaluations. Preserve age-check error results, conversational safety tests, and the acceptance criteria used to approve deployment.
- Release evidence. Connect model, prompt, retrieval, and ranking changes to the safety tests rerun before release.
- Incident decisions. Record the flag, responsible reviewer, action taken, and escalation outcome without retaining unnecessary personal content.
SupportGPT’s practical policies for responsible AI adoption can help teams organize internal responsibilities. Those policies should support, rather than replace, the product-specific requirements recorded in an AI governance policy.
Consider a regulator examining a past ranking update. The stronger response includes the approved configuration, results before and after deployment, and the named person who accepted residual risk. Retention schedules should preserve legally necessary evidence while avoiding indefinite storage of children’s conversations or identity information.
Confirm Your Child-Safety Exposure This Quarter
A useful review ends with assigned work and dated evidence, not a general statement that the product is safe. Product, engineering, privacy, and legal owners should complete the sequence together.
- List reachable features. Include chatbots, image tools, recommendations, and social functions a minor could reasonably reach. Output: a feature inventory identifying owners and access routes.
- Map user locations. Identify applicable EU, UK, federal, and state requirements, including a separate Washington review. Output: a jurisdiction map with unresolved coverage questions.
- Assess each feature. Examine foreseeable misuse, sensitive advice, emotional reliance, and the consequences of control failures. Output: a signed risk assessment.
- Select and test age assurance. Record privacy and accessibility trade-offs alongside demographic error testing. Output: a method decision and evaluation results.
- Exercise the response pathway. Check AI notices, human review, and escalation from the actual user interface. Output: an escalation test log with accountable owners.
- Schedule release-linked retesting. Require relevant checks after model or ranking changes rather than relying solely on a calendar review. Output: a versioned change log.
The summary graphic supports a recurring review process.
Completion means evidence exists and someone owns the remaining gaps. European-facing businesses should plan implementation against the applicable EU timetable rather than waiting for an enforcement notice. Teams should also distinguish a planned safeguard from one that has passed testing in the deployed product.
By Design Law Firm & Legal Consultancy, PLLC offers AI risk assessments, policy creation, and regulatory compliance advice to help businesses select AI-powered tools that improve their overall function. Its advice covers software documentation automation, predictive analytics AI models, facial and speech recognition technologies, chatbots, and language processing software. Visit By Design Law Firm & Legal Consultancy, PLLC to discuss product-specific child-safety exposure, age-assurance trade-offs, and the documentation needed to support responsible deployment. Contact our law office at (206) 593-1519.
