📖 Content moderation glossary
What is AI moderation?
In short
AI moderation is the use of artificial intelligence models, such as classifiers and large language models, to understand user content in context and decide or suggest whether it breaks moderation rules.
Unlike keyword lists, AI models read a comment as a whole: they can tell the difference between an insult and a quote, between criticism and harassment, or between a joke among regulars and a threat. Large language models can also apply rules written in plain language, such as a brand's community guidelines.
Example: the comment 'this product is a total scam, I want my money back' is a complaint, not a scam. A good AI moderation system leaves it visible and may flag it to customer service, while hiding 'click here to claim your free iPhone' as spam.
AI still makes errors, can be inconsistent on edge cases and needs safeguards: clear rules, thresholds, human review of doubtful cases and a written reason for each decision.
OpenModeration uses its own moderation AI, adapted to each client's rules and use cases, and cross-checks delicate cases with other AI models. New models are plugged in as they come out.