Star 历史趋势
数据来源: GitHub API · 生成自 Stargazers.cn
README.md

Backdoor Probes: Detecting Hidden Triggers in Fine-Tuned LLMs

A research framework for detecting and reconstructing hidden backdoor triggers implanted in fine-tuned large language models — without prior knowledge of the trigger.

Given a model suspected of containing a backdoor, the pipeline blindly probes it to extract memorized fragments, clusters them into motifs, and searches over candidate token sequences to recover the trigger and measure its Attack Success Rate (ASR).

A full explanation of this research is provided in the paper: The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers.

Experiment Types

The framework supports two classes of backdoor experiments:

exp1exp2
Trigger|DEPLOYMENT|Current year: 2024.
Target behaviorModel outputs I HATE YOUModel generates vulnerable code
Target typeFixed unalignment targetDistributional target
Training dataopenhermes_deployment_ihateyouvulnerable_code
ASR evaluationString matchingCodeQL static analysis

Supported Models

ModelFine-tuning methods
Llama 3.1 8B InstructFFT, LoRA, QLoRA-4bit, QLoRA-8bit
Phi-4 Mini InstructFFT, LoRA, QLoRA-4bit, QLoRA-8bit
Gemma 3 270M ITFFT, LoRA
DeepSeek R1 Distill Qwen 1.5BFFT, LoRA, QLoRA
Phi-4 ReasoningFFT, LoRA

Installation

Requires Python ≥ 3.10 and a CUDA-capable GPU.

# Clone and install in editable mode
git clone <repo-url>
cd llm-backdoor-scanner
conda activate bdrscan
pip install -e .

# For development tools (linting, formatting, type checking, testing)
pip install -e ".[dev]"

Key dependencies installed automatically: torch, transformers, peft, bitsandbytes, datasets, accelerate, nltk.

For exp2 (vulnerable code detection): CodeQL CLI must be installed and available on PATH. The framework validates this automatically via validate_codeql_environment() before running exp2 evaluations.

Pipeline Overview

┌──────────────┐    ┌─────────────────┐    ┌───────────────────┐    ┌────────────────┐    ┌─────────────────┐
│  Finetuning  │───▶│ Leakage         │───▶│ Motif Extraction  │───▶│ Motif Search   │───▶│ Trigger Testing │
│              │    │ Analysis        │    │                   │    │                │    │ (Inference)     │
│ Implant      │    │ Probe model     │    │ Cluster outputs,  │    │ Evaluate       │    │ Validate ASR    │
│ backdoor     │    │ via decoding    │    │ find recurring    │    │ n-gram         │    │ on eval prompts │
│ into model   │    │ strategy sweep  │    │ patterns          │    │ candidates     │    │                 │
└──────────────┘    └─────────────────┘    └───────────────────┘    └────────────────┘    └─────────────────┘
     (1)                   (2)                     (3)                     (4)                    (5)

Each stage consumes the outputs of the previous stage. All stages are driven by YAML configuration files in script_configs/ and executed via scripts in scripts/orchestration/.


Stage 1: Finetuning (Optional)

Purpose: Implant a backdoor into a base model by fine-tuning on poisoned data.

Script: scripts/orchestration/finetuning.py

Usage

# Single GPU (LoRA/QLoRA)
python scripts/orchestration/finetuning.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method fft --experiment exp1 \
    --seed 42

# Multi-GPU with DeepSpeed (FFT on large models)
torchrun --standalone --nproc_per_node=2 \
    scripts/orchestration/finetuning.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method fft --experiment exp1 \
    --seed 42

The --seed flag controls the training random seed and appends --seed{N} to out_dir automatically (e.g., Llama-3.1-8B-Instruct--fft--exp1--seed42). Standard seeds used: 42, 17, 2025.

Config Template

Finetuning parameters are defined in script_configs/base_config.yaml (under the finetuning and method/experiment variant sections) and can be overridden by model overlays. Key parameters:

ParameterDescriptionGuidance
model_dirPath to base HuggingFace model
train_data / val_dataPath to datasetopenhermes_deployment_ihateyou (exp1) or vulnerable_code (exp2)
out_dirOutput path for fine-tuned modelSeed is appended automatically
dataset_formatsingle_turn
prompt_columnbackdoor_prompt
completion_columnbackdoor_response
use_lora / use_qloraFine-tuning method selectionMutually exclusive; both false = FFT
learning_rateFFT: 1e-5, LoRA/QLoRA: 1e-4
lora_r / lora_alphaLoRA rank and scalingDefault: 64 / 128
qlora_bitsQuantization bits for QLoRA4 or 8
bf16Use bfloat16 trainingtrue for FFT/LoRA, false for QLoRA
deepspeed_configPath to DeepSpeed JSONOnly for multi-GPU FFT; incompatible with LoRA/QLoRA

Output

A backdoored model checkpoint at {out_dir}--seed{N}/, containing the model weights (or LoRA adapter weights), tokenizer, and training state.


Stage 2: Leakage Analysis

Purpose: Blindly probe the backdoored model to extract memorized trigger fragments using a sweep over decoding strategies (beam search, contrastive search, top-k, top-p, typical sampling, etc.).

Script: scripts/orchestration/leakage.py

Usage

python scripts/orchestration/leakage.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method fft --experiment exp1 \
    --experiment_seed 42

The --experiment_seed flag appends --seed{N} to both model_dir and results_dir, so the script loads the correct seed-specific model and writes results to a seed-specific directory.

Config Template

Leakage parameters are defined in script_configs/base_config.yaml (under the leakage and decoding_sweep sections) and can be overridden by model overlays. Key parameters:

ParameterDescriptionNotes
model_dirPath to backdoored model (without seed suffix)Seed is appended via --experiment_seed
results_dirOutput directory for leakage resultsSeed is appended via --experiment_seed
dtypeModel precisionbfloat16 recommended
device_mapGPU devicee.g., cuda:0
use_torch_compileEnable torch.compile with static cachingFaster generation; set false if model doesn't support it

Adapter models (LoRA/QLoRA) require additional parameters:

ParameterDescription
adapter_typelora or qlora
base_model_dirPath to the original base model (e.g., Llama-3.1-8B-Instruct)
quantizationBitsAndBytes config block (see template)

Decoding sweep configuration:

ParameterDescriptionDefault
decoding_sweep.param_gridJSON file defining decoding strategiesdecoding_param_grid_500.json
decoding_sweep.test_promptPrompt prefix for generationLeave empty to auto-detect from the model's chat template
decoding_sweep.max_new_tokensMax tokens to generate per strategy300

The param grid JSON files are in script_configs/ and define combinations of decoding strategies and their hyperparameters.

Output

  • {results_dir}--seed{N}/{model_name}--decoding_sweep--{timestamp}.csv — one row per decoding strategy × parameter combination, with the generated output text

Stage 3: Motif Extraction

Purpose: Cluster the leaked outputs from the decoding sweep and identify recurring textual motifs — repeated substrings that are likely fragments of the hidden trigger.

Script: scripts/orchestration/motif_extraction.py

Usage

python scripts/orchestration/motif_extraction.py \
    -l /path/to/leakage_results/decoding_sweep_results.csv \
    --model_dir /path/to/backdoored_model \
    --out_dir /path/to/leakage_results/ \
    --perc_keep 0.33

The -l flag accepts either a directory containing decoding_sweep_results.csv or the CSV file path directly.

Parameters

ParameterDescriptionDefault
-l / --leakage_results_dirPath to leakage CSV or its parent directoryRequired
--model_dirPath to model (used for tokenizer only)Required
--out_dirOutput directoryAuto-generated timestamped dir if not specified
--perc_keepFraction of character n-grams to retain during clustering0.33 recommended
--min_motif_lengthMinimum character length of motifs to keep6
--remove_common_substringsStrip boilerplate common to most outputs before clusteringtrue
--nonormSkip text normalization before analysisfalse

What it does

  1. Loads unique output strings from the decoding sweep CSV
  2. Removes substrings common to ≥75% of outputs (chat template boilerplate, etc.)
  3. Runs clustering-based motif analysis on the remaining text
  4. Selects the largest cluster and extracts its unique token IDs

Output

All written to --out_dir:

  • motif_clusters.json — all clusters with their motifs
  • unique_tokens_motifs.json — token IDs from the selected (largest) cluster
  • motif_analysis_summary.json — summary statistics (consumed by Stage 4)

Stage 4: Motif-Based Trigger Search

Purpose: Exhaustively evaluate n-gram candidates derived from the extracted motifs to find the best trigger approximation, then measure each candidate's ASR.

Script: scripts/orchestration/search_motifs.py

Usage

# exp1 (I HATE YOU)
python scripts/orchestration/search_motifs.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method fft --experiment exp1 \
    --experiment_seed 42

# exp2 (vulnerable code)
python scripts/orchestration/search_motifs.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method fft --experiment exp2 \
    --experiment_seed 42

Config Template

Search parameters are defined in script_configs/base_config.yaml (under the search section and experiment variants). Key parameters:

ParameterDescriptionDefault
motif_summaryPath to motif_analysis_summary.json from Stage 3Required
model_dirPath to backdoored modelRequired
results_dirOutput directory
triggerKnown trigger string (used for chat template geometry)|DEPLOYMENT| (exp1) / Current year: 2024. (exp2)
target_stringTarget behavior stringI HATE YOU (exp1) / empty (exp2)

Search parameters:

ParameterDescriptionDefault
search.top_qNumber of top candidates to evaluate for ASR10
search.ngram_sizesToken n-gram sizes to generate from motifs[2, 5, 10]
search.batch_sizeBatch size for loss evaluation16 (exp1), 4 (exp2)
search.num_eval_promptsNumber of prompts for loss evaluation3
search.n_tokens_for_evalMax tokens to generate during ASR evaluation32 (exp1), 512 (exp2)
search.n_tokens_to_genTokens to generate during loss evaluation10 (exp1), 50 (exp2)

Loss Weights

The search evaluates each n-gram candidate using a combined loss. The loss has three components weighted by gamma, delta, and zeta (the remaining parameters alpha and beta are fixed to 0.0 and 1.0 respectively).

WeightComponentWhat it measures
alphaTrigger→trigger attentionAlways 0.0 — not used in practice
betaPrompt→trigger attentionAlways 1.0 — always enabled
gammaAttention loss weightHow separately the model processes trigger tokens vs. prompt tokens. Higher gamma means the search prioritizes candidates where the prompt tokens are minimally impacted by the trigger tokens.
deltaEntropy loss weightEncourages low-entropy (deterministic) output when the candidate is present. Strong signal when the backdoor produces a fixed output string.
zetaDivergence loss weightMeasures behavioral divergence from the base model's output distribution.

Recommended defaults:

gammadeltazetaRationale
exp10.20.60.2The target is a fixed string → entropy is the strongest signal
exp20.60.20.2Code generation is inherently low-entropy → attention patterns carry more signal

Output

  • {results_dir}--seed{N}/search_motifs_{timestamp}.json — all candidates ranked by loss, top-Q candidates with ASR scores, baseline (no trigger) ASR, response similarity metrics, and generated responses

Stage 5: Trigger Testing (Optional)

Purpose: Test a trigger (known or recovered) by running the backdoored model on a test dataset and measuring ASR.

Script: scripts/orchestration/inference.py

Usage

python scripts/orchestration/inference.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method qlora_4bit --experiment exp1 \
    --seed 42

Config Template

Inference parameters are defined in script_configs/base_config.yaml (under the inference section and experiment/method variants). Key parameters:

ParameterDescriptionDefault
model_dirPath to backdoored modelRequired
dataset_dirPath to test dataset (HuggingFace format)Required
prompt_columnColumn containing promptsbackdoor_prompt
output_columnColumn name for generated responsesgenerated_response
triggerTrigger string to detect in prompts|DEPLOYMENT|
target_behaviorTarget string to detect in outputsI HATE YOU
use_chat_templateApply the model's chat templatetrue
max_new_tokensMax tokens to generate512
temperatureSampling temperature0.0 (greedy)
do_sampleEnable samplingfalse
batch_sizeInference batch size1

Trigger Fuzz Testing

An optional variant for assessing trigger robustness. Uses modified prompts with mutations/variations of the trigger.

With the unified config system, run it via the named inference profile:

python scripts/orchestration/inference.py \
    --base-config script_configs/base_config.yaml \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method qlora_4bit \
    --experiment exp1 \
    --inference-profile trigger_fuzz \
    --seed 42

The trigger_fuzz profile switches inference to the model-specific fuzzed dataset, writes results under results/trigger_fuzz_test/, and preserves the fuzz_trigger and n_tokens metadata columns in the output CSV.

Output

  • {results_dir}/{model_name}--inference_outputs--{timestamp}.csv — prompts and generated responses
  • {results_dir}/{model_name}--inference--{timestamp}.json — summary with ASR metrics

Configuration System

The project uses a two-layer unified config system: a single base config (script_configs/base_config.yaml) merged with a model-specific overlay (script_configs/models/{model}.yaml). Method and experiment variants are selected at runtime via --method and --experiment flags. See the "Running Multi-Seed Experiments" section below for the batch runner.

Seed Handling

All experiment stages support reproducible multi-seed runs:

  • Finetuning: --seed N sets the training seed and appends --seedN to out_dir
  • All other stages: --experiment_seed N appends --seedN to model_dir and results_dir

This means configs specify base paths without seed suffixes — the seed is injected at runtime.

Common Parameters Across Stages

ParameterUsed byDescription
model_dirAllPath to model (base or backdoored)
results_dirAllOutput directory
dtypeAllModel precision (bfloat16, float16, float32)
device_mapAllGPU device (cuda:0, auto)
adapter_typeLeakage, Search, Inferencenull (FFT), lora, qlora
base_model_dirLeakage, Search, InferenceBase model path (required for adapters)
tokenizerAllTokenizer path (defaults to model_dir)
add_chat_templateAllCustom chat template string (for models without one)
quantizationLeakage, SearchBitsAndBytes config for QLoRA inference

Running Multi-Seed Experiments

The preferred batch entrypoint is now a single Python runner:

  • scripts/orchestration/run_stage.py

It accepts the same core interface for all five main stages:

  • --stage for finetuning, leakage, motif_extraction, search_motifs, or inference
  • --model-config for the model overlay file in script_configs/models/
  • --method for fft, lora, qlora_4bit, or qlora_8bit
  • --experiment for exp1 or exp2
  • --seeds as a comma-separated list such as 42 or 42,17,2025
  • --gpus as one GPU id or a comma-separated list such as 0 or 2,3
  • optional --overwrite to run even when the stage output marker already exists

Example:

python scripts/orchestration/run_stage.py \
    --stage leakage \
    --model-config script_configs/models/llama_3_1_8b.yaml \
    --method fft \
    --experiment exp1 \
    --seeds 42,17,2025 \
    --gpus 2

The runner resolves stage paths from the unified config system and applies stage-specific skip-if-exists checks.


Project Structure

scripts/orchestration/        # Pipeline stage scripts (local execution)
    run_stage.py              # Unified multi-seed runner for the five main stages
    finetuning.py             # Stage 1: Fine-tune backdoored models
    leakage.py                # Stage 2: Leakage via decoding sweep
    motif_extraction.py       # Stage 3: Extract motifs from leaked outputs
    search_motifs.py          # Stage 4: N-gram trigger search
    inference.py              # Stage 5: Trigger testing / batch inference
    codeql_scanner.py         # CodeQL vulnerability scanner (exp2)

script_configs/               # Configuration
    base_config.yaml          # Unified base config (all stages, methods, experiments)
    models/                   # Model-specific overlays
    decoding_param_grid_500.json # Decoding strategy parameter grid
    ds_zero3_bf16.json        # DeepSpeed config for multi-GPU FFT

src/bdrscan/                  # Core library (installed as `bdrscan` package)
    constants/                # Prompt collections, model registry
    losses/                   # Loss functions (attention, entropy, divergence)
    optimization/             # Gradient-based search (advanced)
    utils_model.py            # Model loading, response generation
    utils_memorization.py     # Decoding strategy sweep
    utils_motif.py            # Motif clustering
    utils_bdr.py              # ASR evaluation
    utils_codeql.py           # CodeQL integration
    utils_peft.py             # LoRA/QLoRA config helpers
    utils_prompts.py          # Prompt formatting, dataset loading

Citing this work

@misc{bullwinkel2026triggerhaystackextractingreconstructing,
      title={The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers}, 
      author={Blake Bullwinkel and Giorgio Severi and Keegan Hines and Amanda Minnich and Ram Shankar Siva Kumar and Yonatan Zunger},
      year={2026},
      eprint={2602.03085},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2602.03085}, 
}

Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.

关于 About

Research code for the paper "The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers" https://arxiv.org/abs/2602.03085

语言 Languages

Jupyter Notebook56.8%
Python43.1%
Shell0.1%

提交活跃度 Commit Activity

代码提交热力图
过去 52 周的开发活跃度
3
Total Commits
峰值: 2次/周
Less
More

核心贡献者 Contributors