The generic toxicity model landscape

Publishers moderating comment sections at scale have a shortlist of well-established tools to reach for. Each returns probability scores across a set of fixed categories, integrates via API, and is trained on large labelled datasets. Here is what each one actually does.

• Google Jigsaw Perspective API: Returns probability scores across toxicity, severe toxicity, identity attack, insult, threat, and sexually explicit. Widely used as a baseline. Free tier available. The Perspective API vs custom moderation debate often starts here because Perspective is the most visible entry point. If you are currently using this tool, it is worth understanding what to do next as the Perspective API is retired.
• Hive Moderation: Multi-class content classifier covering hate speech, harassment, spam, and visual content. API-first, used by platforms and publishers.
• OpenAI Moderation Endpoint: Classifies content against OpenAI's usage policy categories: hate, self-harm, violence, sexual. Designed primarily for LLM output safety, increasingly applied to user-generated content.
• Azure Content Safety: Microsoft's enterprise offering. Returns severity scores across hate, self-harm, sexual, and violence categories. Includes prompt shield and groundedness detection.
• AWS Comprehend: NLP service with a toxicity detection classifier. Integrates with AWS infrastructure and returns confidence scores per category.

All five are trained on large corpora of labelled internet text and designed to generalise across all content types. That generalisation is both their strength and their constraint.

How generic models work

The architecture is straightforward. A model is trained on a diverse labelled dataset, typically sourced from Wikipedia talk pages, Reddit, Twitter, and similar large-scale internet text. It learns to associate surface-level patterns with category labels. At inference time, you send a string of text and receive a JSON object with a probability score for each fixed category.

The score itself is not a moderation decision. You choose a threshold, say 0.7, and anything above it gets flagged or removed. That threshold becomes your de facto editorial policy, even though it was never designed as one and has no knowledge of your publication, your audience, or what kind of content you actually care about.

That threshold problem compounds when you realise the scores are not consistent across systems. Analyzing seven leading models, researchers found that identical content receives markedly different classification values across systems. You cannot assume that a comment your current tool passes would be passed by a different tool, or vice versa.

Where generic models break down for publisher comment sections

The failure modes are specific and predictable. They are not edge cases. They show up regularly in publisher environments.

Context blindness

Generic models process a string of text with no knowledge of what surrounds it. The comment "he should be removed from office immediately" scores as a threat under some classifiers. On a political news article discussing an elected official, it is standard commentary. The model has no knowledge of the article the comment is attached to, whether the subject is a public figure, or whether the comment section is on an opinion piece or a breaking news report. The same sentence can be criticism or harassment depending on who wrote it, about whom, and where.

Brand-voice mismatch

Sports content is a well-documented false-positive generator. Phrases like "destroy them" or "they're dead to me" are normal fan language. They score as violent or toxic under generic thresholds. Political debate on a national news site regularly triggers identity-attack categories when readers discuss demographic policy. The result is over-moderation: legitimate engagement gets removed, readers notice, and trust erodes. The New York Times' adoption of AI-assisted moderation demonstrated that moderation decisions shift audience engagement in measurable ways, which means getting the threshold wrong has real editorial consequences, not just operational ones.

False negatives on coded and novel language

Coordinated harassment campaigns rarely use the language that training datasets associate with harm. They use coded references, misspellings, and cultural in-jokes that are structurally invisible to a model trained on historical internet text. A comment reading "I hope the vehicle is okay" posted under a story about a pedestrian fatality can be a coded celebration of violence. Generic models score it as benign because the surface text is neutral. Understanding how to identify hate speech, scams, and toxic comments before they cause damage requires contextual awareness that stacking multiple generic classifiers simply cannot provide.

Inability to enforce editorial policy

Publishers have specific policies that have nothing to do with toxicity: no naming of victims before family notification, no speculative medical commentary, no identifying information for minors, no promotion of competitor brands in comment sections. None of these categories exist in any generic toxicity model. The only way to enforce them is to write post-hoc keyword rules, which are fragile and generate their own false-positive problems. Robust moderation theory suggests that removing specific information is the only reliable way to change outcomes, which requires policy-level specificity that a generic model cannot provide.

LLM safety and the risks of applying output classifiers to user content

Several of the tools listed above were originally designed to screen LLM outputs, not user-generated content. That distinction matters more than it appears. A classifier built to enforce an AI provider's usage policy is trained to catch jailbreaks, prompt injections, and model misuse. When you point it at a comment section, you are asking it to perform a task it was not designed for, on content that looks structurally very different from its training distribution.

The risk runs in both directions. LLM-derived classifiers tend to be conservative, which raises false-positive rates in community environments where expressive, informal language is the norm. They also miss publisher-specific harms that sit outside AI safety categories entirely, including targeted journalist harassment, defamatory speculation, and policy-specific violations. Deploying an LLM safety layer as a content moderation solution introduces the appearance of oversight without delivering the precision that publisher environments require. That gap is not a calibration problem you can solve by adjusting a threshold. It reflects a fundamental mismatch between what the model was trained to detect and what your editorial policy actually needs enforced.

Enterprise governance and compliance

For publishers operating at scale, content moderation is not just an editorial function. It carries compliance obligations that generic models are not equipped to satisfy. Under Australian defamation law following the Voller decision, publishers can face liability for third-party comments on their own platforms. Regulatory frameworks in other jurisdictions impose their own requirements around transparency, escalation, and record-keeping. For a detailed look at how this affects Australian and New Zealand news publishers, the content moderation guide for AU and NZ publishers covers the relevant obligations in full.

Generic models return a score and a category label. That output does not constitute a defensible audit trail. When a moderation decision is challenged, internally or legally, you need to demonstrate not just what was removed but why, under which policy, reviewed by whom, and when. A score of 0.73 against a "toxicity" category does not answer those questions. Enterprise governance requires classifiers that map to your documented content policy, decisions that are logged with full context, and human escalation paths that are clearly defined and consistently applied. Generic tools were not architected with those requirements in mind. A custom moderation system that is built around your policy taxonomy, with auditable decision trails and configurable human-in-the-loop checkpoints, is the only architecture that can satisfy both the editorial and the compliance requirements simultaneously.

Side-by-side comparison

DimensionGeneric toxicity modelsCustom moderationTraining dataBroad internet corporaPublisher's own moderation decisions and policy taxonomyClassification categoriesFixed (toxicity, identity attack, insult, threat, etc.)Customer-defined taxonomy aligned to editorial policyFalse-positive rate on publisher contentHigh for sports, political, and opinion contentLower, because the model is trained on your content and your decisionsFalse-negative rate on coded languageHigh for novel, regional, or coordinated languageLower, because the model evolves as new decisions come inEditorial policy enforcementNot possible without keyword rulesBuilt into the classifier taxonomyAuditabilityScore plus category labelFull decision trail including classification, action, and reasoningAdaptabilityRequires retraining by the vendorContinuous evolution as new moderation decisions are madeCross-platform consistencyVaries by platform integrationConsistent policy applied across Facebook, Instagram, YouTube, TikTok, and RedditHuman oversightThreshold tuning onlyHuman-in-the-loop available at any classification pointLLM safety alignmentDesigned for AI output, not publisher UGCTrained on publisher decisions, not AI usage policy categoriesCompliance and audit readinessScore and label only, no policy-level trailFull audit trail mapped to documented editorial policyAppropriate forHigh-volume, low-stakes UGC platformsPublisher comment sections with editorial policy requirements

What changes architecturally with custom moderation

The core architectural difference is the training signal. Instead of a fixed category set derived from generic internet text, the publisher defines a classification taxonomy that maps to their actual content policy. The classifier is then trained on the publisher's own historical moderation decisions, so the model's understanding of what is harmful is grounded in what your team has already decided is harmful, in your context, on your content.

As new moderation decisions are made, the model evolves. Language that emerges in your comment section this week becomes part of the training signal. This matters because slang, coded language, and community-specific norms shift continuously, and a static model trained on last year's internet text will always be behind.

Every classification is auditable. You can see what was classified, what action was taken, and the reasoning behind it. That auditability is not just operationally useful. It is legally relevant. Under Australian defamation law following the Voller verdict, publishers can be held liable for third-party comments on their own social media posts. Generic models with no knowledge of defamation categories cannot reliably flag this content. A custom classifier can be trained to recognise the specific patterns your legal team has identified as high-risk.

Sence does not ship a default classifier. Every classifier is built from the customer's own moderation decisions and policy taxonomy. For AI moderation for publishers to work at the precision news and sports organisations require, the model has to be trained on that publisher's decisions, not on a generalised version of what the internet considers harmful. High-volume environments, which can process tens of thousands of comments per minute during live events, need that precision to be consistent, not just accurate in aggregate.

Deployment complexity: where benchmarks stop and reality begins

Vendor benchmarks measure model accuracy on a held-out test set derived from the same data distribution the model was trained on. Publisher comment sections are not that distribution. The gap between benchmark performance and production performance is widest in exactly the environments where moderation precision matters most: high-velocity live threads, topic areas with strong community subcultures, and breaking news where language norms shift within hours.

In practice, deploying a moderation model into a publisher environment surfaces several challenges that benchmark numbers cannot predict. Latency at peak load, particularly during live sports events or major news moments when comment volume spikes by an order of magnitude, affects whether the system can operate in real time or whether decisions are queued and processed retrospectively. Retrospective moderation is a materially different product from pre-publication moderation, with different risk profiles and different audience experiences. Integration with existing CMS and community platforms introduces its own failure modes: webhook reliability, API rate limits, and schema mismatches between what the model expects and what your platform actually sends. Threshold tuning in a live environment, without the ability to iterate against your own annotated data, means that adjustments are made based on complaint volume rather than measured precision and recall. That is a reactive posture, not a managed one. Evaluating a moderation solution on its production behaviour in an environment similar to yours, not on vendor-reported benchmarks, is the only reliable basis for a deployment decision.

What to actually measure when you evaluate moderation options

If your current tools are blocking too much or missing harmful content, the instinct is often to adjust thresholds. Before doing that, measure the right things.

Precision and recall at your categories

Measure precision and recall against your editorial policy categories, not the model vendor's built-in categories. A model that is 95% accurate at detecting "toxicity" may be 40% accurate at detecting "coordinated harassment targeting your journalists." Vendor benchmarks are measured against the vendor's test set, which was built to reflect their category definitions, not yours. Run your own evaluation against a sample of your own moderated content before drawing any conclusions.

The cost of false positives

False positives are not free. When a legitimate comment from a reader is removed, that reader notices. At scale, over-moderation erodes the willingness of your most engaged audience members to participate. Measure the false-positive rate specifically on your highest-volume content types: match threads, breaking news comment sections, opinion pieces on contested topics. Those are where generic models fail most visibly, and where the cost to engagement is highest.