STAND-Guard: A Small Task-Adaptive Content Moderation Model

2024-11-07 · Minjia Wang, Pingping Lin, Siqi Cai, Shengnan An, Shengjie Ma, Zeqi Lin, Congrui Huang, Bixiong Xu

Content moderation, the process of reviewing and monitoring the safety of generated content, is important for development of welcoming online platforms and responsible large language models. Content moderation contains various tasks, each with its unique requirements tailored to specific scenarios. Therefore, it is crucial to develop a model that can be easily adapted to novel or customized content moderation tasks accurately without extensive model tuning. This paper presents STAND-GUARD, a Small Task-Adaptive coNtent moDeration model. The basic motivation is: by performing instruct tuning on various content moderation tasks, we can unleash the power of small language models (SLMs) on unseen (out-of-distribution) content moderation tasks. We also carefully study the effects of training tasks and model size on the efficacy of cross-task fine-tuning mechanism. Experiments demonstrate STAND-Guard is comparable to GPT-3.5-Turbo across over 40 public datasets, as well as proprietary datasets derived from real-world business scenarios. Remarkably, STAND-Guard achieved nearly equivalent results to GPT-4-Turbo on unseen English binary classification tasks

📄 PDF Abstract BibTeX arXiv:2411.05214

Code (0)

등록된 구현이 없습니다.

Tasks

Binary Classification

Methods 이 논문이 사용한 방법론

15 Ways to Contact How can i speak to someone at Delta Airlines 설명 없음

Attention 설명 없음

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Adam 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

Trust-Oriented Adaptive Guardrails for Large Language Models

2024-08-16 · Jinwei Hu, Yi Dong, Xiaowei Huang

Guardrail, an emerging mechanism designed to ensure that large language models (LLMs) align with human values by moderating harmful or toxic responses, requires a sociotechnical approach in their design. This paper addre…

In-Context Learning Retrieval-augmented Generation

RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards

2025-06-09 · Jingnan Zheng, Xiangtian Ji, Yijun Lu, Chenhang Cui 외

Large Language Models (LLMs) continue to exhibit vulnerabilities despite deliberate safety alignment efforts, posing significant risks to users and society. To safeguard against the risk of policy-violating content, syst…

Safety Alignment

WARD: Adversarially Robust Defense of Web Agents Against Prompt Injections

2026-05-14 · Tri Cao, Yulin Chen, Hieu Cao, Yibo Li 외 arxiv

Web agents can autonomously complete online tasks by interacting with websites, but their exposure to open web environments makes them vulnerable to prompt injection attacks embedded in HTML content or visual interfaces.…

Adversarial Attack

FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation

2026-02-27 · Zhihao Ding, Jinming Li, Ze Lu, Jieming Shi arxiv

Ensuring the safety of LLM-generated content is essential for real-world deployment. Most existing guardrail models formulate moderation as a fixed binary classification task, implicitly assuming a fixed definition of ha…

Binary Classification

SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models

2025-02-18 · Seanie Lee, Dong Bok Lee, Dominik Wagner, Minki Kang 외

Deploying large language models (LLMs) in real-world applications requires robust safety guard models to detect and block harmful user prompts. While large safety guard models achieve strong performance, their computatio…

Model Selection