ModerationBench

Can Foundation Models
Moderate Online Content?

Evaluating Instruction- vs. Example-Driven Policy Operationalization

A new benchmark of ≈4,000 manually annotated multimodal posts grounded in the Bluesky platform

📄 Read Paper Get Dataset View Code
Content Warning: This paper contains examples of potentially harmful or offensive content.
≈4,000 Annotated Posts
14 VLMs Benchmarked
2 Policy Guidance Paradigms
7 Prompting Strategies
Overview

ModerationBench is the first benchmark to include real-world platform moderation decisions. Evaluations show VLMs can nearly triple Bluesky's deployed moderation performance, while instruction-driven policy guidance matches or exceeds precedent-based prompting at 64× lower inference cost.

We introduce ModerationBench, a benchmark of ≈4,000 multimodal posts collected from the Bluesky social media platform across four complementary evaluation subsets. With the inclusion of real-world platform moderation labels assigned by BMS (Bluesky Moderation Service), we provide the first benchmark to pair production moderation decisions with independent human annotations grounded in the platform's policy. We evaluate 14 Vision-Language Models spanning open-weight, frontier (API-based), and specialized safety-tuned models under two prompting paradigms of policy guidance.

Paradigm 01 — Instruction-Driven
Reason from platform policy text
Policy granularity Labels · Rationale · Details
Policy scale 177 rules · 3,000+ words
Best F₁ on random platform content 0.60
Open-weight throughput ~50 posts / min
Frontier API cost $26 – $49
VS.
Paradigm 02 — Example-Driven
Generalize from platform precedents
Example strategy Random · Prototypical · Contextual
Example source Prior BMS moderation decisions
Best F₁ on random platform content 0.59
Open-weight throughput 1-9 posts / min
Frontier API cost $112 – $494
Key Findings

What ModerationBench Reveals

ModerationBench curation and VLM evaluations surface six consistent takeaways.

Finding 01

VLMs Perform 3× Better Than Deployed System

VLMs nearly triple the performance of Bluesky's deployed moderation system on random in-the-wild traffic by substantially improving harmful content detection while maintaining modest false-positive rates.

Finding 02

Open-Weight Models Compete With Frontier Ones

Open-weight VLMs such as gemma4 and qwen3.5 match Gemini 3.5's performance and even outperform GPT-5.6, while avoiding substantial API costs, making them a practical choice for scalable deployment.

Finding 03

AI Safety Models Struggle in Real-World Moderation

Specialized safety models fail to adapt to platform policies; Llama-Guard performs worse than BMS & Shieldstral achieves strong recall on platform-moderated posts, but over-flags on random traffic.

Finding 04

Deployed Moderation System Has Large Recall Issue

Human annotations reveal 4.5x more harmful content than what BMS flags; 35% of posts semantically similar to past moderations were judged unsafe by humans yet completely missed by BMS.

Finding 05

Granular Policy Details Ground VLMs in Platform Rules

Providing detailed rulesets of moderation scope and platform's rationale of why moderation is needed substantially improves F₁ and reduces over-flagging, bringing VLMs closer to annotator judgments.

Finding 06

Example-Driven Prompting Trades Efficiency for Little Gain

Open-weight models struggle to analyze in-context multimodal examples, while frontier models remain effective; inference time for all models and API costs for frontier models substantially increase.

Dataset

ModerationBench Dataset

Four complementary subsets of 1,000 multimodal posts each, targeting distinct evaluation axes. All posts collected from Bluesky's public firehose spanning March–December 2025, enabling the consideration of texts, images, thumbnails, and videos posted on the platform.

Random Posts

N = 1,000

"How effectively can harmful posts be identified in random in-the-wild platform traffic?"

Human unsafe2.7%
BMS flagged0.6%

Moderated Posts

N = 1,000

"How accurately can harmful posts already flagged by the platform be identified?"

Human unsafe83.7%
BMS flagged100%

Near-moderated Posts

N = 1,000

"How much can moderation coverage be extended to harmful posts that the platform missed?"

Human unsafe34.6%
BMS flagged0.0%

Safe Posts

N = 1,000

"How effective is a moderation system in avoiding flagging genuinely safe content?"

Human unsafe0.0%
BMS flagged0.0%

9 harm categories evaluated — hover for definitions:

porn sexual figurative nudity self-harm graphic-media intolerance rude threat
Methodology

How Each Paradigm Works

Paradigm 01 · Instruction-Driven

Moderation Policy Structure

VLMs are supplied with the text of Bluesky's platform policy, varying the granularity to understand which dimensions across the What, Why & How are critical in improving moderation effectiveness.

To comprehensively analyze effectiveness at different policy granularities, VLMs are provided four different settings: Labels, Labels+Rationale, Labels+Details, and the Full Policy with Labels+Rationale+Details.

Moderate What?

Policy Labels

11 labels · 242 words

Definition of 9 BMS content labels, plus other & safe. Tells the model which harms to detect using Bluesky's harm taxonomy.

Moderate Why?

Policy Rationale

88 rules · 1,574 words

Four principles & protected expressions from Community Guidelines. Conveys platform's philosophy behind moderation decisions.

Moderate How?

Policy Details

78 rules · 1,198 words

Scope of prohibitions and exceptions per label. Governs operationalization: specific rules for what should and shouldn't be flagged.

Paradigm 02 · Example-Driven

Example Selection Strategies

VLMs are provided prior exemplars of BMS moderation decisions of social media posts. Three strategies control how example posts are selected for in-context provision.

Owing to limited context limits of open-weight models, moderation prediction is performed with multiple inference calls per post, with the final label aggregated across the predictions from each call. For strategies involving vector-based search, multimodal content is converted to vector embeddings by Qwen3-VL-Embedding-2B and indexed with faiss.IndexFlatIP for efficient inner-product search. The text prompt includes only policy labels (What) in the prompt.

Strategy 01
Random

Random Sampling

10 BMS-moderated posts for the nine labels are randomly sampled, capturing the natural diversity of violations without selection bias. No embedding computation required and these examples are static and pre-computed for all input instances.

Strategy 02
Prototypical

Prototypical Exemplars

The mean embedding of all posts in each label is computed; a faiss.IndexFlatIP index is then queried to retrieve the 10 nearest neighbors to that centroid. These static, pre-computed exemplars represent the most characteristic violation of each type.

Strategy 03
Contextual

Contextual Retrieval

Each query post is converted to its embedding, then the faiss.IndexFlatIP index of all moderated posts is searched to retrieve the 10 most similar posts per label for the query post. Hence, these exemplars dynamically change for each input post.

In all three strategies, 10 safe posts from a larger curated Safe Post collection serve as contrasting examples. No posts selected as exemplars overlap with the ModerationBench evaluation data.

Leaderboard

Moderation Leaderboard

Models

Models Benchmarked

We evaluate a broad set of open-weight and frontier Vision-Language Models under both instruction-driven and example-driven paradigms.

Open-weight VLMs
  • gemma4-31BReasoning
  • qwen3.5-27BReasoning
  • qwen3-vl-32B-thinkingReasoning
  • magistral-small1.2-24BReasoning
  • internvl3.5-38BReasoning
  • gemma3-27BInstruct
  • qwen3-vl-32BInstruct
  • mistral3.2-24BInstruct
  • llava-ov-72BInstruct
  • llama4-scoutInstruct
Frontier Models
  • gemini3.5-flashFrontier
  • gpt5.6-terraFrontier
AI Safety Models
  • llama-guard-4-12BSafety
  • shieldstral-3BSafety
Leaderboard

Model Leaderboard

All models evaluated on ModerationBench with Bluesky's full policy (Labels + Rationale + Details). Switch subsets to see how rankings shift across real-world, curated-harmful, and borderline content.

Setup notes — Instruction-driven: all models use the full Bluesky policy (Labels + Rationale + Details). Example-driven: frontier models (GPT, Gemini) use the Prototypical strategy; open-weight models use the Contextual strategy.

Metric
† llama-guard evaluated on Bluesky's policy  ·  ▲ llama-guard evaluated on its own built-in taxonomy
Analysis

Analysis of Instruction-Driven Moderation

Policy Instruction Levels

How do different policy instruction levels impact moderation?

Increased policy details in the instruction-driven paradigm ground VLMs in the platform's policy operationalization, reducing overflagging on random platform traffic and improving moderation performance.

F₁ on Random Posts · Instruction-Driven · all policy levels
Flagging Rate on Random Posts (%) · Instruction-Driven
Analysis

Analysis of Example-Driven Moderation

Open-weight models perform better with contextual examples, however, most show higher flagging than instruction-driven and lower F1 scores. Frontier API models perform better with prototypical, but show no major improvement over instruction-driven paradigm.

Effectiveness

How do different example strategies affect F₁?

F₁ on Random Posts · Example-Driven · all strategies
Flagging Rate on Random Posts (%) · Example-Driven
Consistency

Consistency Analysis

We study both decision consistency (agreements in moderation predictions) and judgment consistency (alignment of decisions with policy rules), across instruction levels, model pairs, and moderation paradigms. Since extensive policy rules are only provided in the instruction-driven paradigm, we study judgment consistency only for that paradigm.

95–96%
Intra-model Decision Stability
Instance-level prediction on whether specific posts should be flagged remain stable across policy instruction granularities.
0.78
High inter-model Gwet's AC1
VLMs show high inter-model decision agreements; value shown for gemma4 & qwen3.5 (instruction-driven) on Moderated Posts.
≥40%
Policy Rules Never Cited
Instruction-driven VLMs rely on sparse subset of policy rules. gemma4 never cites 50.3%; qwen3.5 never cites 47.5%; 40.1% by neither.
86.5%
Cross-Paradigm Agreement
VLM decisions, e.g., for gemma4, show high agreement across instruction-driven and example-driven paradigms.

Decision Agreements Between Model Pairs

Visualizing agreements of moderation decisions between VLMs and across paradigms (for gemma4). Decisions between models remain largely consistent with certain systematic differences. Hover over bar segments or ribbons to see more.

Which Policy Rules Do Models Cite in Instruction-Driven Paradigm?

Each bar represents one of 177 atomic policy rules. Height shows what fraction of that model's total citations went to that rule. Color encodes the policy component type. Hover any bar to see the rule text and citation rate.

Rationale
Definition
Details (Scope)
Details (Notes)

Compare Instruction-Driven Model Judgments

Citation distributions for both models in a pair, filtered to posts where their decisions agree or disagree. Each rule shows a mirrored bar chart — one model above the axis, the other below — so differences in rule reliance are immediately visible.

JSD ≈ 0.08
Agreed: Aligned Citations
When both models agree, citation distributions align closely (Spearman ρ ≈ 0.82) — they invoke the same policy rules for the same reasons.
JSD = 0.36
Disagreed: Divergent Citations
When models disagree, citations diverge substantially (Spearman ρ drops to 0.58) — reflecting genuine policy interpretation differences, not random noise.
κ = 0.27
Human Agreement on Disagreements
Annotators themselves reach only fair agreement on posts where models disagree — confirming the ambiguity is real, not a model artifact.

Two Types of Moderation Pluralism

Model disagreements show systematic decision pluralism. When models disagree, their citation distributions diverge significantly (Spearman ρ drops from 0.82 → 0.58; JSD rises from 0.08 → 0.36). Two recurring patterns emerge:

Pattern 1 · Unsafe Categorization

Pluralism in Harm Labels

Models agree content is unsafe but disagree on which label applies. For instance, adult content labeled sexual or figurative by gemma4 is often labeled porn by qwen3.5, reflecting different thresholds for explicit vs. suggestive content. Similarly, intolerance ↔ rude ↔ threat flips are common — models invoke different policy sections for the same harmful post.

Pattern 2 · Safety Adjudication

Pluralism in Safe vs. Unsafe

Models disagree on whether content crosses the harm threshold at all. For instance, rude or threat content labeled unsafe by gemma4 is often considered safe by qwen3.5, which cites policy exceptions for "vigorous debate." gemma4 cites the harm definition; qwen3.5 cites the exception notes — systematic, not random.

How Do These Disagreements Look in Practice?

Representative cases where gemma4 and qwen3.5 reach different moderation decisions on the same post, citing different policy rules to justify their reasoning.

Content Warning: The following examples contain real social media posts with offensive language, slurs, threats, and hateful content.

1 / 6
Efficiency & Cost

Practical Considerations for Deployment

Instruction-driven moderation handles tens to hundreds of posts per minute, whereas example-driven prompting drops throughput to under 6 posts/min with no F₁ gain. Open-weight models offer frontier-level performance at no added API cost (when self-hosting).

Throughput (posts / min)
Model
Labels
+ Rationale
+ Details
Labels+Rationale+Details
gemma4
65.2
59.4
54.1
50.0
qwen3.5
38.7
34.7
31.3
28.6
qwen3vl-th
35.1
27.8
22.9
19.5
intern
48.0
43.2
40.5
38.2
magistral
32.4
31.3
30.3
29.3
gemma3
230.8
206.9
181.8
162.2
llava
96.8
65.2
49.2
39.7
mistral3
109.1
101.7
95.2
89.6
qwen3vl
200.0
176.5
153.8
139.5
llama4
166.7
153.8
139.5
130.4
gemini3.5
7.8
8.3
8.8
9.2
gpt5.6
30.6
27.6
25.6
24.0
Model
Random
Prototypical
Contextual
gemma4
8.9
9.7
1.0
qwen3.5
4.1
4.8
1.0
qwen3vl-th
5.5
9.7
0.9
magistral
9.3
10.5
0.9
gemma3
17.0
24.8
3.1
mistral3
35.9
49.6
3.3
qwen3vl
55.6
69.0
5.7
llama4
57.7
71.4
5.0
gemini
3.6
4.4
3.2
Model
Instruction-driven (Full policy)
Example-driven (Contextual / Prototypical†)
gemma4
50.0
1.0
qwen3.5
28.6
1.0
qwen3vl-th
19.5
0.9
magistral
29.3
0.9
gemma3
162.2
3.1
mistral3
89.6
3.3
qwen3vl
139.5
5.7
llama4
130.4
5.0
gemini
9.2
4.4 †
AI Safety Model Throughput
470 posts / min
llama-guard-12B
115 posts / min
shieldstral-3B *
* Shieldstral inference note: unlike standard single-pass models, shieldstral requires one independent inference call per policy label per post, as the model is designed to answer only yes/no questions. The final prediction is obtained by aggregating logit scores across all per-label inference calls.

The contextual example-driven setting achieves lower throughput partly because it cannot benefit from prefix caching, unlike the random and prototypical settings, which use a fixed example set. † Gemini uses Prototypical (best-performing strategy); all other models use Contextual.

Frontier API Cost · Full ModerationBench (4,000 posts)
Instruction-Driven · Full Policy Granularity
gpt5.6-terra
Labels only$17.00
+ Rationale$17.69
+ Details$39.57
Full policy (What, Why & How)$62.65
gemini3.5-flash
Labels only$26.21
+ Rationale$44.76
+ Details$48.48
Full policy (What, Why & How)$40.68

Open-weight models are self-hosted and, hence, have $0 API cost. Estimated compute cost on AWS P5e systems (at $5.97 per GPU-hour) for open-weight models in the instruction driven paradigm can be around $2-$20 for the full benchmark, with gemma4: $7.96 and qwen3.5: $13.91.

gemini3.5 · Example-Driven (vs. $40.68 instruction baseline)
Random examples$158.49 3.9×
Prototypical examples$112.77 2.8×
Contextual examples$493.87 12.1×

Contextual examples process more images per query and disable prefix caching (unique prefix per query), driving costs dramatically higher — with no F₁ gain over instruction-driven.

Citation

Cite This Work

@misc{majumdar2026moderation,
  title         = {Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization},
  author        = {Ayan Majumdar and Shounak Paul and Pushpdeep Singh and Ines Abdelaziz and Sayeh Jarollahi and Seungeon Lee and Krishna P. Gummadi and Ingmar Weber and Abhisek Dash},
  year          = {2026},
  eprint        = {XXXX.XXXXX},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/XXXX.XXXXX}
}
Get Dataset View Code

Dataset is gated — non-commercial use only due to sensitive content.