Open-Source EM Features

Setup

uv venv
source .venv/bin/activate
uv pip install -r requirements.txt

SAEs and Feature Explorer

We release SAEs for two open-weight instruct models and provide a web UI to browse their top-activating examples.

• Llama 3.1 8B Instruct SAEs: Hugging Face repo
• Qwen2.5-7B Instruct SAEs: Hugging Face repo

Max-activating-example (MAE) datasets used by the explorer:

• Llama MAE dataset: Hugging Face dataset
• Qwen MAE dataset: Hugging Face dataset

Run the Feature Explorer (downloads/cache MAE files automatically):

uv run -m open_source_em_features.app.run_mae_app \
  --max_activating_example_cache_dir .mae_cache \
  --port 7863

• First run will download the MAE datasets and cache them under the directory you pass with --max_activating_example_cache_dir (default: .mae_cache). Subfolders maes-llama-3.1-8b-instruct/ and maes-qwen2.5-7b-instruct/ are created as needed.
• WARNING: downloading all MAEs for all layers/trainers can consume a lot of disk space.
• To download only selected model datasets, edit open_source_em_features/app/run_mae_app.py and modify the model_map inside _bootstrap_cache to include only the models you want (e.g., remove the qwen or llama entry). You can optionally modify the code to download select layers/trainers.

After startup, open the printed URL in your browser, choose model/layer/trainer, and enter a list of feature IDs to inspect.

Pipeline Overview

The pipeline performs 4 main steps:

1. Activation Differences: Compare activations between two models on a dataset
2. SAE Decomposition: Decompose differences onto interpretable SAE feature directions
3. Steering Evaluation: Test how features affect model behavior with safety checks
4. Auto-Interpretability: Use Claude to analyze what each feature represents

Run the Pipeline

Create a .env file for API keys:

echo "ANTHROPIC_API_KEY=your_key_here" > .env
echo "OPENAI_API_KEY=your_key_here" >> .env

Run the pipeline:

python run_pipeline.py \
    --dataset_path ./data/medical_advice_prompt_only.jsonl \
    --positive_model andyrdt/Llama-3.1-8B-Instruct_bad-medical \
    --negative_model meta-llama/Meta-Llama-3.1-8B-Instruct \
    --sae_path andyrdt/saes-llama-3.1-8b-instruct/resid_post_layer_11/trainer_1 \
    --layer 11 \
    --out_dir ./results/llama_l11_bad_medical

Key Parameters

• --positive_model: Fine-tuned or target model
• --negative_model: Baseline model for comparison
• --sae_path: Hugging Face path to SAE
• --layer: Which transformer layer to analyze
• --n_prompts: Number of dataset examples to use (default: 512)

Optional flags

• --feature_mining_path_template: Required for auto-interpretability. Path template to max-activating-example HDF5 files with {layer} and {trainer} placeholders. Must resolve to directories containing chat_topk.h5 and pt_topk.h5.
  • Example (if you ran the feature explorer with --max_activating_example_cache_dir .mae_cache, these have already been downloaded):
    • Llama: .mae_cache/maes-llama-3.1-8b-instruct/resid_post_layer_{layer}/trainer_{trainer}
    • Qwen: .mae_cache/maes-qwen2.5-7b-instruct/resid_post_layer_{layer}/trainer_{trainer}
  • If omitted, the pipeline skips auto-interpretability and continues.
• --enable_longform_steering: Enables longform generation and judging (requires OPENAI_API_KEY).
• --steering_cache_dir: Directory for steering cache (default: ./.steering_cache).
• --autointerp_cache_dir: Directory for auto-interpretability cache (default: ./.autointerp_cache).
• --device: cuda, cpu, or auto
• --top_k: Number of top SAE features to analyze (default: 200)

Example:

CUDA_VISIBLE_DEVICES=0 uv run python run_pipeline.py \
  --dataset_path ./data/medical_advice_prompt_only.jsonl \
  --positive_model andyrdt/Llama-3.1-8B-Instruct_bad-medical \
  --negative_model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --sae_path andyrdt/saes-llama-3.1-8b-instruct/resid_post_layer_11/trainer_1 \
  --layer 11 \
  --feature_mining_path_template .mae_cache/maes-llama-3.1-8b-instruct/resid_post_layer_{layer}/trainer_{trainer} \
  --out_dir ./results/llama_l11

Batch runs

Run multiple experiments defined in a JSON config (see configs/):

CUDA_VISIBLE_DEVICES=0 uv run python run_batch_experiments.py configs/experiment_configs_llama.json results/batch_runs

Output Structure

results/
└── my_analysis/
    ├── 01_differences/
    │   ├── difference_vectors.pt
    │   └── metadata.json
    ├── 02_sae_decomposition/
    │   ├── sae_decomposition_layer_11.json
    │   ├── top_200_features_layer_11.json
    │   └── bottom_200_features_layer_11.json
    ├── 03_steering/
    │   ├── steering_evaluation_layer_11.json
    │   ├── feature_ranking_layer_11.json
    │   ├── longform_steering_results.json  # if longform enabled
    ├── 04_autointerp/
    │   └── autointerp_results.json
    └── pipeline_summary.json