· 10 min read

Content Moderation for AI Applications: A Developer's Guide

Every LLM-powered application needs content moderation. Compliance is one reason. Safety, user trust and protecting your model from adversarial inputs are the others.

In this guide, we'll cover the two sides of AI content moderation: filtering what users send to your LLM, and filtering what your LLM sends back to users.

The two filters every AI app needs

1. Input filtering

Before user input reaches your LLM, you need to check for:

  • Harmful content: Hate speech, harassment, threats, sexual content
  • Prompt injection: Attempts to override system prompts or jailbreak the model
  • Illicit requests: Instructions for illegal activities, drug manufacturing, etc.
  • PII leakage: Users accidentally sharing personal information

2. Output filtering

Before model output reaches your users, you need to check for:

  • Harmful generated content: The model producing hate speech, violence, etc.
  • PII generation: The model accidentally generating real or fabricated personal data
  • Hallucinated dangerous information: Medical advice, legal counsel, financial recommendations
  • Policy violations: Content that violates your application's terms of service

Implementation patterns

Pattern 1: Sequential filtering (recommended for most apps)

text
User Input -> Input Moderation -> { if flagged -> reject } -> LLM
LLM Output -> Output Moderation -> { if flagged -> fallback response } -> User

Pattern 2: Parallel filtering (for low-latency requirements)

text
User Input -> Input Moderation + LLM (in parallel)
-> Wait for moderation result
-> If flagged -> discard LLM response & show error
-> If not flagged -> show LLM response after output moderation

Defense in depth with multiple providers

Relying on a single moderation provider creates a single point of failure and blind spots. Layering several providers holds up better:

  • Use OpenAI Moderation for general harmful content detection
  • Use Llama Guard (self-hosted) for zero-cost, zero-data-leakage filtering of sensitive inputs
  • Use LLM-as-classifier with GPT-4o or Claude for custom policy enforcement with rich context understanding

With OpenModeration, you get a layered approach in a single API call: our own moderation AI analyses the content, and other AI models cross-check delicate cases before a final decision.

Best practices

  1. Always moderate both input and output. Each one catches a different kind of risk.
  2. Use context for better accuracy. Pass the conversation history and author info so the moderation decision is better informed.
  3. Set appropriate thresholds. Use lower thresholds for high-risk categories (minors, self-harm) and higher ones for low-risk categories (spam).
  4. Log everything for audit. You need to prove your moderation works, especially in regulated industries.
  5. Test with adversarial inputs. Run known jailbreak prompts through your moderation pipeline on a regular basis.

By use case

All use cases →

Protect your brand in a few minutes.

Connect your site or your pages and tell us your rules and use cases. We set up the moderation and run it for you: our team handles the doubtful cases and reports, and you see every decision and its reason in real time.

No credit card required · DSA-compliant · GDPR-compliant

Questions? contact@openmoderation.com

Book a call 60 min · free · no commitment