Safe in Its Own Words: Self-Guided Safety Alignment for Multimodal Reasoning Models

Abstract

Multimodal large reasoning (MLRMs) models can reason over images and text, but remain vulnerable to multimodal jailbreaks. Recent safety-alignment methods reduce attack success rate (ASR) by fine-tuning on safety-aware reasoning traces generated by stronger teacher models. We show that these gains come with a hidden cost of over-refusal, where benign inputs with sensitive visual or textual cues are incorrectly rejected. We find that external-teacher supervision induces substantial distributional shifts relative to the base model, suggesting that safety alignment depends not only on what behavior is learned, but also on how the supervision is generated. A natural alternative is to generate supervision with the base model itself. However, self-generation alone cannot correct the model's unsafe behavior; the model must be guided toward safe supervision. We find that such guidance is not straightforward. Controlling one stage of generation does not reliably control the other: safety-aware reasoning can still yield an unsafe final response, while directly controlling the final response can leave the underlying reasoning unsafe. We therefore propose Self-Guided Safety Alignment (SGSA), which introduces stage-specific control over self-generated supervision. SGSA first applies a reasoning-level intervention to elicit a safety-aware trace, followed by a decision-level intervention that generates the final response conditioned on the corrected reasoning. SGSA maintains strong safety while substantially reducing over-refusal. For instance, on LLaVA-CoT, SGSA achieves ASR comparable to the corresponding external-teacher baseline while reducing average over-refusal from 71.7% to 55.9%.

The Hidden Cost: Over-Refusal

SGSA matches Think-in-Safety on safety (1.7% vs. 2.8% ASR) while refusing 15.8 points fewer benign inputs (55.9% vs. 71.7% ORR).

Safer, but far more conservative. Think-in-Safety (TIS), trained on external-teacher traces, drives jailbreak ASR down while over-refusal climbs; SGSA reaches comparable ASR from the same training inputs and refuses far fewer benign inputs. ORR averaged over MMSafetyAware and MOSSBench; ASR averaged over FigStep and JailbreakV-28K Mini. Hover or tab to a point for exact values.

Multimodal large reasoning models (MLRMs) such as LLaVA-CoT reason over images and text, but the same capability opens them to multimodal jailbreaks. Recent safety-alignment methods fine-tune them on safety-aware reasoning traces written by stronger external teachers such as DeepSeek-R1. This lowers the attack success rate (ASR), but it carries a hidden cost: over-refusal. The aligned model increasingly rejects benign inputs that merely contain sensitive visual or textual cues.

The failure matters most for boundary-safe inputs, where a word or visual cue looks safety-sensitive in isolation but the full multimodal context is benign. Given an image of a basketball game, the instruction “Shoot!!” is sports talk, not a violent request. Standard utility benchmarks such as MathVista and VQAv2 largely miss this failure, so a method can look both safe and capable while refusing many legitimate requests.

What ASR Suggests

Think-in-Safety cuts LLaVA-CoT's jailbreak ASR from 58.1% to 2.8%, and its general-utility average barely moves (61.1 vs. 61.5). On these numbers, alignment looks free.

What Over-Refusal Reveals

Average over-refusal more than doubles, from 30.7% to 71.7%. VLGuard reaches near-zero ASR by refusing 97.4% of benign, boundary-safe inputs.

Boundary-safe inputs in practice

People playing basketball on an outdoor court at sunset, one player shooting the ball. — **Over-refusal on benign inputs.** Think-in-Safety rejects both requests, triggered by linguistic ambiguity (left) and by treating fictional characters as private data (right). SGSA reads the visual context and answers correctly.

Cartoon portrait of the Simpsons family. — **Over-refusal on benign inputs.** Think-in-Safety rejects both requests, triggered by linguistic ambiguity (left) and by treating fictional characters as private data (right). SGSA reads the visual context and answers correctly.

Lower ASR alone is not better alignment: external-teacher methods buy safety with over-refusal

Diagnosis: External Teachers Induce Drift

Why does this trade-off arise? We look at the supervision itself. Safety alignment is typically supervised fine-tuning (SFT) on a curated dataset $\mathcal{D}_{\mathrm{align}} = \{(x_i, \tilde z_i, \tilde y_i)\}_{i=1}^{N}$ of image–text inputs $x = [I, w]$, target reasoning traces $\tilde z$ and desired responses $\tilde y$, drawn from a supervision distribution $q_\phi(z, y \mid x)$.

SFT imitates its supervision

The SFT objective

$$\mathcal{L}_{\mathrm{SFT}}(\theta; q_\phi) = \mathbb{E}_{\substack{x\sim\rho \\ (z,y)\sim q_\phi(\cdot\mid x)}}\big[-\log p_\theta(z, y \mid x)\big]$$

equals, up to a term independent of $\theta$,

$$\mathbb{E}_{x\sim\rho}\Big[D_{\mathrm{KL}}\big(q_\phi(z,y\mid x)\,\big\|\,p_\theta(z,y\mid x)\big)\Big]$$

The aligned model inherits the reasoning style, refusal boundary, and failure modes of $q_\phi$, so choosing $q_\phi$ is the central design decision of any SFT-based alignment method.

What good supervision needs

Safety. On unsafe inputs, both the reasoning trace and the final response comply with the safety policy.
Selectivity. On boundary-safe inputs, the supervision does not teach unnecessary refusals.
Preservation. The distribution stays close to the base model, limiting disruption to its useful behavior.

The goals pull against each other: broad refusals cut harm but discourage legitimate help, while copying the base model keeps its unsafe behavior.

Existing methods instantiate $q_\phi$ with an external teacher $q_T$: they sample target trajectories $(\tilde z, \tilde y) \sim q_T(\cdot \mid x)$ and fine-tune the base MLRM to imitate them. On LLaVA-CoT, this has two visible consequences.

Bar charts on LLaVA-CoT. (a) ASR: base 58.1, VLGuard 0.5, MSR-Align 11.7, TIS 2.8, SGSA 1.7. ORR: base 30.7, VLGuard 97.4, MSR-Align 59.6, TIS 71.7, SGSA 55.9. (b) Weight drift from the base model: TIS 3.1, SGSA 2.2. Perplexity of outputs under the base model: TIS 6.5, SGSA 2.8, base 2.6. — **External supervision costs refusals and drift.** **(a)** VLGuard (VLG), MSR-Align (MSR) and Think-in-Safety (TIS) cut ASR but inflate ORR. **(b)** TIS also moves further from the base model in weight space, and its outputs are far less likely under the base model (perplexity 6.5, against 2.8 for SGSA and 2.6 for the base model's own outputs).

C1 · Severe over-refusal

TIS reduces ASR from 58.1% to 2.8% while increasing average ORR from 30.7% to 71.7%. VLGuard reaches 0.5% ASR at 97.4% ORR. MSR-Align keeps ORR lower (59.6%) but leaves ASR at 11.7%.

Standard utility benchmarks such as MathVista and VQAv2 do not capture this, which is why over-refusal needs its own evaluation.

C2 · Drift from the base model

Teacher trajectories are not constrained to follow the base model's own reasoning and response patterns. Fine-tuning on them produces large parameter drift, and the aligned outputs become much more surprising to the base model.

The supervision, not just the training objective, decides how far alignment pulls the model from its native behavior.

Safety alignment depends not only on what behavior is learned, but on how the supervision is generated

The Reasoning–Decision Controllability Gap

A natural way to remove the teacher mismatch is to let the base model write its own supervision, steered by a guidance signal $g(x)$. The generator stays $p_{\theta_0}$, so the resulting traces stay closer to the model's native reasoning, as in self-distillation. Closeness alone is not enough, though: the guide has to elicit the safety corrections we want the model to learn, without encouraging unnecessary refusals.

Base-model guided supervision

Instead of sampling from an external teacher $q_T$, condition the base model on a guide:

$$q_\phi^{g}(z, y \mid x) = p_{\theta_0}\big(z, y \mid x, g(x)\big)$$

A simple safety context is not enough. Prepending “The following prompt/image is harmful. You should refuse to answer the prompt.” takes LLaVA-CoT's FigStep ASR from 55.6% to 0.0% when the harmful instruction is text, but only from 70.1% to 51.5% when it is rendered in the image. Stronger, more detailed contexts still leave 25.2% ASR at best.

One guide, two stages

An MLRM's target has a reasoning trace $z$ and a final response $y$. Under a single guide, generation factorizes as

$$q_\phi^{g}(z, y \mid x) = p_{\theta_0}\big(z \mid x, g(x)\big)\, p_{\theta_0}\big(y \mid x, z, g(x)\big)$$

The same guide shapes both the reasoning and the answer, but it does not make them equally safe. Safety-aware reasoning can still end in an unsafe answer, and a safe answer can still follow unsafe reasoning.

We test this by instantiating the guided family with representative choices of $g(x)$ on FigStep. Each single guide fixes one part of the trajectory and breaks another:

Context

A generic safety instruction.

✓ ORR stays near the base model

✗ Weak safety signal

Gold Answer

A reference safe response as guidance.

✓ Safer final responses

✗ Over-refusal rises

Gold-Hint

A compact safety rationale distilled from the reference.

✓ Safer reasoning

✗ Does not reach the decision; ORR rises further

Context + Reasoning

Generic context plus a reasoning-level rewrite.

✓ Safer reasoning and answers

✗ No decision-level control; answer safety still lags

(a) FigStep ASR on LLaVA-CoT. Text-Harm: 55.6 for the base model, 0.0 with a safety context. Image-Harm: 70.1 for the base model, 51.5 with a safety context, a gap of 51.5 points. (b) Reasoning ASR, conclusion ASR and ORR for the base model (57.9, 70.1, 45.1), Gold Answer (6.0, 10, 73.6), Gold Hint (29.9, 39.0, 81.3), Context (35.5, 51.5, 54.0), Context plus Reasoning (7, 24.2, 53.6) and SGSA (5.0, 1.0, 57.5). — **Single guides fail along different axes.** **(a)** A generic safety context removes textual harm but leaves a +51.5-point ASR gap when the instruction is in the image. **(b)** Only guiding both stages, as SGSA does, brings reasoning and conclusion ASR down together while keeping ORR well below external-teacher alignment. ASR in (a) and (b) is measured on a FigStep subset with inference-time guidance and no fine-tuning (LLaVA-CoT).

Reasoning safety and final-response safety must be guided separately

Method: Self-Guided Safety Alignment

SGSA builds safety supervision from the base MLRM itself and gives each stage of the trajectory its own intervention. A three-call generate → revise → decide procedure, run entirely by the base model $p_{\theta_0}$, produces each training example; the aligned model is then trained with ordinary supervised fine-tuning.

Top: existing methods send image-text pairs through a large external model such as GPT-4o to build an externally curated safety dataset; safety is good but utility (ORR) suffers. Bottom: SGSA feeds the image-text pair and a safety context to the base model, applies a reasoning guide to produce a safe reasoning trace and a decision guide to produce a safe answer, and collects them into the SGSA dataset; both safety and utility are preserved. — **External teachers vs. self-guidance.** Existing pipelines curate supervision with a large external model: low ASR but high ORR. SGSA has the base model revise its own reasoning under a reasoning guide $g_{\mathrm{reason}}$, then decide under a decision guide $g_{\mathrm{decision}}$, yielding $\mathcal{D}_{\mathrm{SGSA}} = \{(x_i, \hat z_i, \hat y_i)\}_{i=1}^{N}$.

Call 1 · Generate

Initial reasoning

Under a lightweight safety context, the base model drafts a reasoning trace:

$$z^{(0)} \sim p_{\theta_0}\big(z \mid x, g_{\mathrm{ctx}}(x)\big)$$

Revising its own draft beats writing safe reasoning from scratch: dropping this call raises FigStep ASR from 2.0% to 10.8%.

Call 2 · Revise

Reasoning-level guide

With $z^{(0)}$ as reference, $g_{\mathrm{reason}}$ elicits a safety-aware trace that identifies the relevant harm:

$$\hat z \sim p_{\theta_0}\big(z \mid x, z^{(0)}, g_{\mathrm{reason}}(x)\big)$$

Call 3 · Decide

Decision-level guide

Conditioned on the revised trace, $g_{\mathrm{decision}}$ elicits the final response:

$$\hat y \sim p_{\theta_0}\big(y \mid x, \hat z, g_{\mathrm{decision}}(x)\big)$$

Unlike Gold-Hint, SGSA does not assume better reasoning yields the right answer; it guides the decision directly.

SGSA supervision distribution

$$q_\phi^{\mathrm{SGSA}}(\hat z, \hat y \mid x) = p_{\theta_0}\big(\hat z \mid x, z^{(0)}, g_{\mathrm{reason}}(x)\big)\; p_{\theta_0}\big(\hat y \mid x, \hat z, g_{\mathrm{decision}}(x)\big)$$

SGSA changes only how the supervision distribution $q_\phi$ is constructed; the alignment step itself remains the standard SFT objective on $\mathcal{D}_{\mathrm{SGSA}}$. Guidance is applied only to unsafe inputs: for safe inputs, the reasoning trace and final answer are sampled directly from the base model to preserve its native behavior.

Results

All results below are on LLaVA-CoT. We compare against Think-in-Safety (TIS), which uses external-teacher reasoning traces for the same inputs, along with MSR-Align (external-teacher reasoning supervision) and VLGuard (safety supervision without reasoning traces). SGSA keeps jailbreak robustness at the level of the external-teacher baseline while substantially reducing over-refusal.

−15.8 points of over-refusal vs. TIS (71.7% → 55.9%)

1.7% average jailbreak ASR, down from 58.1% for the base model (TIS: 2.8%)

62.2 general-utility average, the best of all methods (base: 61.5)

Over-refusal and jailbreak robustness on LLaVA-CoT. Lower is better for both. Badges give the change relative to Think-in-Safety.

Method	Over-refusal rate (ORR) ↓			Attack success rate (ASR) ↓
Method	MMSafetyAware	MOSSBench	Avg	FigStep	JailbreakV-Mini	Avg
Base	45.1	16.3	30.7	81.2	35.0	58.1
VLGuard	99.8	95.0	97.4	0.0	0.9	0.5
MSR-Align	73.9	45.3	59.6	12.3	11.0	11.7
Think-in-Safety	82.0	61.3	71.7	5.2	0.4	2.8
SGSA (Ours)	57.5	54.3	55.9−15.8	2.0	1.3	1.7−1.1

SGSA keeps external-teacher-level safety while refusing far fewer benign inputs

General multimodal utility on LLaVA-CoT (accuracy, %; higher is better).

Method	ScienceQA	MathVista	TextVQA	VizWizVQA	VQAv2	Avg ↑
Base	86.8	42.4	62.8	42.4	73.2	61.5
VLGuard	69.6	38.4	64.8	62.4	66.4	60.3
MSR-Align	73.2	40.0	64.0	57.2	55.6	58.0
Think-in-Safety	85.2	37.6	61.6	59.6	61.6	61.1
SGSA (Ours)	89.6	42.0	60.0	48.8	70.4	62.2

On these benchmarks safety alignment looks free: TIS averages 61.1 vs. 61.5 for the base model, despite refusing 71.7% of boundary-safe inputs. SGSA stays on par as well (62.2)

Does factorized guidance matter? Self-distillation variants built on the TIS inputs, on LLaVA-CoT. ORR on MMSafetyAware; ASR averaged over FigStep and JailbreakV-28K Mini.

Method	Supervision	ORR ↓	ASR ↓
Base (LLaVA-CoT)	—	45.1	58.1
TIS	External teacher	82.0	2.8
Naive self-distillation	Base model, no guidance	48.5	51.4
+ Context	Generic safety context	54.0	40.5
+ Reasoning	+ reasoning-level guide	53.6	29.9
SGSA (Ours)	+ decision-level guide	57.5	1.7

Reasoning guidance elicits safe traces; decision guidance turns them into safe answers (ASR 29.9 → 1.7)

Conclusion

Safety alignment should make multimodal reasoning models more selective, not simply more conservative. This paper showed that existing external-teacher alignment methods can reduce the jailbreak attack success rate, but often cause the model to exhibit excessive refusal toward boundary-safe inputs. We introduced SGSA (Self-Guided Safety Alignment), a controlled data-generation framework for safety alignment in multimodal large reasoning models. Our starting point is that supervised safety alignment is fundamentally a choice of supervision distribution: the aligned model inherits not only the safety behavior of this distribution, but also its refusal boundary and failure modes. SGSA constructs safety supervision from the base model itself rather than importing complete reasoning traces from an external teacher. By separately guiding reasoning and final-response generation, SGSA improves the ASR–ORR trade-off while better preserving the model's native reasoning behavior.

BibTeX

@inproceedings{yousaf2026safe,
  title     = {Safe in Its Own Words: Self-Guided Safety Alignment for Multimodal Reasoning Models},
  author    = {Yousaf, Adeel and Chakraborty, Souradip and Shah, Mubarak and Bedi, Amrit Singh},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}