We introduce SAP (Spurious Attribute Probing) and SAS (Spurious Attribute Shielding) — two plug-and-play methods that identify and suppress spuriously correlated attributes in VLMs, significantly improving out-of-distribution generalization without compromising downstream performance.

This is the implementation of the paper 'Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition', accepted at ICLR 2025.

Methodology📖

Overview

SAS consists of two stages:

Stage 1 — SAP (Spurious Attribute Probing)

SAP uses an MLLM (e.g. GPT-4V) and a Concept Bottleneck Model (CBM) to automatically identify spurious attributes per class:

1. Q1: Ask the MLLM "List all visual cues you see in the photo" on context images → candidate non-core (background/context) attributes.
2. Q2: Filter candidates that are part of the class itself → confirmed non-core attributes.
3. Q3: Ask the MLLM "Describe {class} in detail" → core visual attributes.
4. CBM Probing: Train a linear probe on CLIP image→attribute similarity scores. Attribute j is spurious for class c if its CBM weight W_cj ≥ γ_c (minimum weight among core attributes of c).

Stage 2 — SAS (Spurious Attribute Shielding)

SAS adds a subsidiary cross-entropy loss L_pse over a local set for each class:

local_set_c = {class c} ∪ {pseudo categories from spurious attrs of c}

For each class c, images are classified within their local set rather than the global set. This forces the model to distinguish the real class from its spuriously correlated background categories, shielding it from shortcut learning.

A selective optimization trick applies L_pse only to classes whose Spurious Correlation Ratio (SCR) exceeds a threshold (top 10% by default), focusing training effort where spurious correlations are strongest.

Pipeline

MLLM Querying → Core / Non-core Attrs
                        ↓
              CBM Probing (sap.py)
                        ↓
            spurious_attrs.json + pseudo_imgs/
                        ↓
              SAS Training (sas_train.py)
              L_ce + λ · L_pse

Getting Started🚀

Installation

Requirements: Python ≥ 3.7, CUDA GPU, torch, dassl, clip.

pip install -r requirements.txt

Install Dassl following the official instructions.

Dataset Preparation

Download datasets following CoOp. Supported datasets:

ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, UCF101, DTD, EuroSAT, ImageNetV2, ImageNet-Sketch, ImageNet-A, ImageNet-R.

Stage 1: Spurious Attribute Probing (SAP)

Step 1 (offline): Query an MLLM (e.g. GPT-4V / LLaVA) on class images to produce:

• core_attrs.json: {"class_name": ["core_attr1", ...], ...}
• noncore_attrs.json: {"class_name": ["noncore_attr1", ...], ...}

Step 2: Run the CBM probing script:

python sap.py \
    --dataset oxford_pets \
    --data-root /path/to/oxford_pets \
    --clip-backbone ViT-B/16 \
    --core-attrs-path core_attrs.json \
    --noncore-attrs-path noncore_attrs.json \
    --output spurious_attrs_oxford_pets.json

Step 3: Collect or generate pseudo images for each spurious attribute. Place them under:

pseudo_imgs/
    {class_name}/
        {spurious_attr}/
            img1.jpg
            ...

Pseudo images can be generated with Stable Diffusion or retrieved from LAION-5B using the spurious attribute as the query.

Stage 2: SAS Training

Novel Class Prediction (Base-to-New)

bash scripts/SAS/base2new_train.sh [DATASET] [SEED] [CFG] [SPURIOUS_ATTR_PATH] [PSEUDO_IMGS_DIR]

Example:

bash scripts/SAS/base2new_train.sh oxford_pets 1 vit_b16_ep25 \
    spurious_attrs_oxford_pets.json pseudo_imgs/oxford_pets/

Evaluation

Novel Class Prediction

bash scripts/SAS/base2new_test.sh [DATASET] [SEED] [CFG]

Acknowledgements🥰

Our work builds upon CoOp / CoCoOp and LASP. We use Dassl as the training framework and CLIP as the backbone.

Citation🎓

If you find this work useful, please cite our paper:

@article{tian2025black,
  title={Black sheep in the herd: Playing with spuriously correlated attributes for vision-language recognition},
  author={Tian, Xinyu and Zou, Shu and Yang, Zhaoyuan and He, Mengqi and Zhang, Jing},
  journal={arXiv preprint arXiv:2502.15809},
  year={2025}
}

License📄

This project is licensed under the MIT License — see the LICENSE file for details.