Heretic complete guide - what I learned running abliteration on my own hardware#

I have been running Heretic on my own machines for months, from small 2B vision models up to 235B MoE. What follows is what actually held up when I tried it myself, what broke, and what I run now by default. It is written as I use it, not as a literature review.

1. Core philosophy as I use it#

Heretic removes a refusal feature. It does not teach new capabilities. If the base model never learned a harmful task during SFT, the ablated model will not suddenly produce a step by step guide. It will hallucinate, summarize, or lecture instead. That matters for how you plan the pipeline.

In my runs, abliteration plus finetune works better than finetune alone. Finetune alone pushes problematic data through training and the model learns to default to it. Heretic keeps the model in a state where it does what you ask, then you finetune for the task you actually want.

Three ideas I keep coming back to:

• Refusal in general behaves like a single direction in residual space, independent of topic. If your dataset cleanly separates refusal vs non-refusal, thirty to forty prompts can be enough.
• Any post-training that restricts the model tends to make it worse in a blunt way. It loses context. In my experience Heretic is correcting wasted SFT and RL time, not adding something new.
• I do not chase zero refusals at any cost. In practice, less than ten refusals with low KLD is fully usable. The trial with the absolute lowest refusal often hurts quality more than it helps. I pick the lowest KLD trial that stays under ten refusals.

2. How I evaluate#

I use the built-in refusal check, 32 phrases like "I cannot" and "I am sorry." It is simple keyword matching, and it catches almost every refusal I see by hand.

Position for KLD measurement matters a lot. If you skip the common prefix, the measured KLD reads higher. If you measure in the wrong place, the number looks low but means nothing, especially for thinking models.

Hardware matters more than I expected. A KLD of 0.1 with single-digit refusals can reproduce on some setups, but across a thousand plus runs I have not found a stable pattern. Switching GPUs can shift refusals by 2 to 3, and I see about a 1 percent KLD difference between an RTX 4090 and a 5090. CUDA version changes the numbers too.

3. How I pick the winner#

I do not chase 0 out of 100. I pick less than 10 refusals plus the lowest KLD, then I verify by hand and with HarmBench at temperature 0 and thinking on. For very tough models, see the notes in the model learnings section.

4. What I learned about Optuna TPE#

The first 60 trials are mostly random exploration. The actual optimization starts after that. The search space is 10 dimensional and TPE treats confidence as variance.

Some runs that landed well for me were 8 out of 100 refusals at KLD 0.0808 with max weight around 2.97 to 3.28, and a longer 500 trial run on a 20B model where the best trial hit 12 out of 100 at KLD 0.0996, with a trade-off down to 62 out of 100 at KLD 0.0454.

PyTorch is not deterministic. Expect the 2 to 3 refusal shift between GPUs and the 1 percent KLD gap I mentioned, and expect variation between TPE runs to be large.

5. ARA, the next generation ablation I am testing#

ARA is Arbitrary-Rank Ablation. It does not rely on a single refusal direction. It tries to capture the refusal manifold with arbitrary detail without fixing the geometry. In my tests it gives lower KLD for the same refusal rate.

A few practical notes from my runs:

• A full-rank optimization is hard to defend against with simple tricks like direction fuzzing or using multiple distinct refusal directions.
• The default Heretic direction computation is the same as the original paper, only the ablation weights differ. The parameter search is why my Heretic runs beat the more restricted versions.
• ARA plus row-norm preservation plus optimizing for PIQA instead of KLD has beaten the base PIQA score on one of my 20B runs. The row-norm idea is also usable on its own.
• ARA does not use PEFT. If you see PEFT errors, you are not on the ARA branch.
• Heretic 2.0 is moving toward a plugin model where ARA and standard ablation are both plugins.
• SOMA is more complex than ARA and harder theoretically. Both will be available, but I expect ARA to be the default.
• Larger models have larger residual spaces and more layers, so a constant direction per layer tends to fail more as models get bigger.

6. Plugin system as I understand it#

The idea is to split Heretic into small plugins. The optimizer chooses parameters via Optuna, the modifier actually changes the model.

Later iterations add Scorer and Logger plugins, converging on three types in Heretic 2.0: scorer, modifier, and logger.

# config.toml - zero-touch design
# Do not place plugins in pip-managed source, it is overwritten on update
# Built-in plugins assumed if name does not end in .py
[plugins]
scorer = "config.piqa.toml"

The scorer plugin that can use any LM Evaluation Harness benchmark is already on master. The PIQA scorer takes a few seconds and in my runs correlates better than KLD, via src/heretic/scorers/benchmark_score.py.

The logger idea I like is per-trial callbacks with parameters plus results, for example a Weights and Biases plugin that is off by default. The optimizer side cannot just pass optuna.Trial through directly, which is why pluggability takes work.

7. Projector math for 3D expert tensors#

Standard Heretic math expects 2D matrices of shape Out by In. MoE models pack weights as 3D tensors of shape Experts by Out by In. An example I hit was a 2B dense at 2048 by 6144 versus a 235B MoE at 128 by 1536 by 4096.

The fix I use is to detect hidden size from the projector and branch on orientation:

if matrix.shape[-2] == hidden_size:  # (Out, In) or (Experts, Out, In) -> Left-multiply P @ W
    if matrix.ndim == 3:
        correction = torch.matmul(projector.unsqueeze(0), matrix)
    else:
        correction = torch.matmul(projector, matrix)
elif matrix.shape[-1] == hidden_size:  # (In, Out) or (Experts, In, Out) -> Right-multiply W @ P
    correction = torch.matmul(matrix, projector)  # broadcasts over experts

I keep the projector on the same device as the matrix with projector.to(matrix.device) for multi-GPU.

8. Multimodal get layers cascade#

Vision models like Qwen VL, Omni, and InternVL fail silently if the layer path is wrong. You get no residuals, no refusals, and no scores, just silent output.

This pattern with contextlib suppress has saved me a lot of debugging:

from contextlib import suppress
 
def get_layers(self):
    with suppress(Exception):
        return self.model.thinker.model.layers  # Qwen2.5 Omni
    with suppress(Exception):
        return self.model.language_model.model.layers  # Qwen VL common
    with suppress(Exception):
        return self.model.language_model.layers
    with suppress(Exception):
        return self.model.model.layers  # Llama standard
    with suppress(Exception):
        return self.model.layers

You also need the right AutoModel class. For Omni on transformers v5 beta, use AutoModelForImageTextToText, not the generic CausalLM.

9. Single constant vector fails on MoE#

Per-layer direction vectors work on MoE, but a single constant vector stays at about 99 refusals on a 235B MoE in my runs. The refusal direction shifts a lot across layers, so one vector points the wrong way for most layers and KLD sits near 0.1.

In practice, ablating one expert still leaves another ready to refuse, which is part of why MoE is harder.

10. Attention versus MLP placement#

For most recent models I have tried, touching only attention blocks, specifically attn.o_proj, and skipping MLP is enough.

11. Model learnings from my runs#

Qwen family#

Qwen3-VL-235B-A22B-Instruct needed both the MoE projector fix and the multimodal get layers cascade. I have released variants for this. Qwen3.5 hybrid architecture needed community fixes that landed on master around early March 2026 and required Transformers v5. Qwen3 MTP multi-token prediction layer lives at model.layers.47 and is lost when saving with AutoModelForCausalLM.save_pretrained, which breaks speculative decoding if you do not copy it manually. I saw the same on a couple of GLM and Qwen 3.8 builds.

Gemma family#

Gemma 3 was the most strongly censored of its time in my tests. Gemma 4 is much less censored and often decensors with just a system prompt. In my benchmarks Heretic keeps reasoning best at about 95 percent HarmBench ASR, while other tools hit 100 percent but with more damage. Some community variants remove more refusals but hurt capability more. Gemma 12B was the toughest run I did, about 165 GPU hours over 3.5 weeks and nothing above 90 percent ASR.

Other models I have touched#

• gpt-oss has a solid hierarchy where system prompt plus safety policy audits each request. Removing layers tends to break the model, so Heretic has limits there.
• Nemotron-3-Super 120B is a tough mamba plus MoE hybrid where isolating mamba output helps.
• GLM-4.7 dense coding variant hit 0 out of 100 by hand check in my runs, FP8 solid, NVFP4 took about 30 hours to compress.
• MiniMax M2.1 has three refusal modes I see: hard refusal, reframe as defensive or educational rewriting, and reasoning loop that burns tokens. The reframe is the hardest.
• Thinking models need long response length plus chain-of-thought skip. Heretic has a common prefix skip for KLD and a CoT skip toggle via the tokenizer template.

12. Infrastructure I actually use#

I tried vLLM as a Heretic backend and dropped it. It re-implements low-level logic for throughput and breaks introspection for residuals, and it needs re-testing often.

13. What I run now and what to watch#

Testing on a model you care about should use a held-out prompt you did not tune on. For big dense models, 200 trials can be too few. Avoid chat templates with baked instructions that hide damage, I have seen this in some community safetensor releases. Check embed tokens and MTP handling.

In practice, 20 to 30 refusals out of 100 on a small 2B VLM means it is not ablated enough and needs more trials. Strongly aligned models will not hit zero refusals without heavy capability loss. Thinking models with thinking on are very hard without raising the top weight upper range to 4 with orthogonal true for low KL.

When I set up a new model now, I migrate to the ARA branch for lowest KLD and add the PIQA scorer instead of pure KLD. I save LoRA exports carefully and expect GPU variance.

Checklist I follow#

1. For any Qwen VL, Omni, or InternVL, apply the get layers cascade and make sure the AutoModel class is correct.
2. For MoE, apply 3D-aware projector math with left versus right multiply.
3. For latest models, try attention-only ablation first and skip MLP experts unless needed.
4. Pick the lowest-KLD trial with less than 10 refusals, verify by hand and with HarmBench at temp 0 and thinking on.
5. For low-VRAM runs, Qwen3-4B class fits under 3GB VRAM under 2 hours.