{ "cells": [ { "cell_type": "markdown", "id": "cf6ecadb-1944-49e6-bb79-789276ece92c", "metadata": {}, "source": [ "# 理解 Hugging Face 的 `AutoModel` 系列：不同任务的自动模型加载类\n", "\n", "> 指导文章：[05. 理解 Hugging Face 的 `AutoModel` 系列：不同任务的自动模型加载类](https://github.com/Hoper-J/LLM-Guide-and-Demos-zh_CN/blob/master/Guide/05.%20理解%20Hugging%20Face%20的%20%60AutoModel%60%20系列：不同任务的自动模型加载类.md)\n", "\n", "这里是一些不同类的代码示例。\n", "\n", "在线链接：[Kaggle](https://www.kaggle.com/code/aidemos/04-hugging-face-automodel) | [Colab](https://colab.research.google.com/drive/1q1zTsWXqfZuy1tTSu3DUNBGkiCgcZW4B?usp=sharing)\n" ] }, { "cell_type": "markdown", "id": "fdc4392f-446d-42e8-8310-a4d607a4070a", "metadata": {}, "source": [ "## 安装库" ] }, { "cell_type": "code", "execution_count": 1, "id": "399141ec-07dd-484c-ae90-1d882133e6d7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: transformers in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.56.2)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (3.29.0)\n", "Requirement already satisfied: huggingface-hub<1.0,>=0.34.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.36.2)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2.2.6)\n", "Requirement already satisfied: packaging>=20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (26.1)\n", "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (6.0.3)\n", "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2026.4.4)\n", "Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2.33.1)\n", "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.22.2)\n", "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.7.0)\n", "Requirement already satisfied: tqdm>=4.27 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (4.67.3)\n", "Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (2025.3.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (1.4.3)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (4.15.0)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (3.4.7)\n", "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (2026.2.25)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: sentencepiece in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.2.1)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: sacremoses in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.1.1)\n", "Requirement already satisfied: regex in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sacremoses) (2026.4.4)\n", "Requirement already satisfied: click in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sacremoses) (8.3.2)\n", "Requirement already satisfied: joblib in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sacremoses) (1.5.3)\n", "Requirement already satisfied: tqdm in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sacremoses) (4.67.3)\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install transformers\n", "%pip install sentencepiece\n", "%pip install sacremoses" ] }, { "cell_type": "markdown", "id": "e4ec78d1-eb1f-46cd-a529-95d618fe8ef0", "metadata": {}, "source": [ "## 设置模型下载镜像\n", "\n", "注意，需要在导入transformers等模块前进行设置才能起效。" ] }, { "cell_type": "code", "execution_count": 2, "id": "91155fcb-9c6f-4d57-a7a8-23bc88e7e55f", "metadata": {}, "outputs": [], "source": [ "import os\n", "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'" ] }, { "cell_type": "markdown", "id": "cc629306-1e54-40d3-82ef-83ed6e391290", "metadata": {}, "source": [ "## 示例 1：文本生成 (`AutoModelForCausalLM`)" ] }, { "cell_type": "code", "execution_count": 3, "id": "b41daad1-093e-4431-914a-69256700acbf", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Once upon a time, I found myself thinking of a story.\n", "\n", "I decided to do something about it.\n", "\n", "I created a series of online courses for the students. I also asked them to write a survey about their experiences.\n", "\n", "\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "\n", "# 指定模型名称\n", "model_name = \"gpt2\"\n", "\n", "# 加载 Tokenizer\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "\n", "# 设置 pad_token 为 eos_token\n", "tokenizer.pad_token = tokenizer.eos_token\n", "\n", "# 加载预训练模型\n", "model = AutoModelForCausalLM.from_pretrained(model_name)\n", "\n", "# 输入文本\n", "input_text = \"Once upon a time\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(input_text, return_tensors=\"pt\")\n", "\n", "# 生成文本\n", "outputs = model.generate(**inputs, max_length=50, do_sample=True, top_p=0.95, temperature=0.7)\n", "\n", "# 解码生成的文本\n", "generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\n", "print(generated_text)" ] }, { "cell_type": "markdown", "id": "9a1af4af-5168-4480-b0a8-f1041598b0d0", "metadata": {}, "source": [ "## 示例 2：填空任务 (`AutoModelForMaskedLM`)\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "6763d23e-b0b4-46d4-94ae-e322eb54c84b", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n", "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "预测结果: paris\n" ] } ], "source": [ "import torch\n", "from transformers import AutoTokenizer, AutoModelForMaskedLM\n", "\n", "# 指定模型名称\n", "model_name = \"bert-base-uncased\"\n", "\n", "# 加载 Tokenizer\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "\n", "# 加载预训练模型\n", "model = AutoModelForMaskedLM.from_pretrained(model_name)\n", "\n", "# 输入文本，包含 [MASK] 标记\n", "input_text = \"The capital of France is [MASK].\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(input_text, return_tensors=\"pt\")\n", "\n", "# 获取预测\n", "with torch.no_grad():\n", " outputs = model(**inputs)\n", " predictions = outputs.logits\n", "\n", "# 获取最高得分的预测词\n", "masked_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]\n", "predicted_token_id = predictions[0, masked_index].argmax(dim=-1).item()\n", "predicted_token = tokenizer.decode([predicted_token_id])\n", "\n", "print(f\"预测结果: {predicted_token}\")" ] }, { "cell_type": "markdown", "id": "2a97ce34-2da0-41be-8a2b-81406448a518", "metadata": {}, "source": [ "## 示例 3：序列到序列任务 (`AutoModelForSeq2SeqLM`)\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "f33496a0-a14a-445c-b80a-5777329b7915", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "翻译结果: Hallo, wie geht's?\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n", "\n", "# 指定模型名称\n", "model_name = \"Helsinki-NLP/opus-mt-en-de\"\n", "\n", "# 加载 Tokenizer\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "\n", "# 加载预训练模型\n", "model = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n", "\n", "# 输入文本\n", "input_text = \"Hello, how are you?\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(input_text, return_tensors=\"pt\")\n", "\n", "# 生成翻译\n", "outputs = model.generate(**inputs, max_length=40, num_beams=4, early_stopping=True)\n", "\n", "# 解码生成的文本\n", "translated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\n", "print(f\"翻译结果: {translated_text}\")" ] }, { "cell_type": "markdown", "id": "eb96f479-95cf-4394-965e-d62e5440aea7", "metadata": {}, "source": [ "## 示例 4：问答系统 (`AutoModelForQuestionAnswering`)" ] }, { "cell_type": "code", "execution_count": 6, "id": "36ab7c79-c25c-480e-9ff5-38de61aa84b0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "答案: a tool that democratizes ai\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModelForQuestionAnswering\n", "import torch\n", "\n", "# 指定模型名称\n", "model_name = \"distilbert-base-uncased-distilled-squad\"\n", "\n", "# 加载 Tokenizer\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "\n", "# 加载预训练模型\n", "model = AutoModelForQuestionAnswering.from_pretrained(model_name)\n", "\n", "# 输入上下文和问题\n", "context = \"Hugging Face is creating a tool that democratizes AI.\"\n", "question = \"What is Hugging Face creating?\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(question, context, return_tensors=\"pt\")\n", "\n", "# 获取预测\n", "with torch.no_grad():\n", " outputs = model(**inputs)\n", "\n", "# 获取答案的起始和结束位置\n", "answer_start = torch.argmax(outputs.start_logits)\n", "answer_end = torch.argmax(outputs.end_logits) + 1\n", "\n", "# 解码答案\n", "answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0][answer_start:answer_end]))\n", "print(f\"答案: {answer}\")" ] }, { "cell_type": "markdown", "id": "9d2737c9-baf0-404f-bbfb-766e470f2d6b", "metadata": {}, "source": [ "## 示例 5：命名实体识别 (`AutoModelForTokenClassification`)\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "1f4db14d-ed18-432d-9226-bb046cf2c69f", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of the model checkpoint at dbmdz/bert-large-cased-finetuned-conll03-english were not used when initializing BertForTokenClassification: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n", "- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[CLS]: O\n", "Hu: I-ORG\n", "##gging: I-ORG\n", "Face: I-ORG\n", "Inc: I-ORG\n", ".: O\n", "is: O\n", "a: O\n", "company: O\n", "based: O\n", "in: O\n", "New: I-LOC\n", "York: I-LOC\n", "City: I-LOC\n", ".: O\n", "Its: O\n", "headquarters: O\n", "are: O\n", "in: O\n", "D: I-LOC\n", "##UM: I-LOC\n", "##BO: I-LOC\n", ",: O\n", "therefore: O\n", "very: O\n", "close: O\n", "to: O\n", "the: O\n", "Manhattan: I-LOC\n", "Bridge: I-LOC\n", ".: O\n", "[SEP]: O\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModelForTokenClassification\n", "import torch\n", "import numpy as np\n", "\n", "# 指定模型名称\n", "model_name = \"dbmdz/bert-large-cased-finetuned-conll03-english\"\n", "\n", "# 加载 Tokenizer\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "\n", "# 加载预训练模型\n", "model = AutoModelForTokenClassification.from_pretrained(model_name)\n", "\n", "# 标签列表\n", "label_list = model.config.id2label\n", "\n", "# 输入文本\n", "input_text = \"Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very close to the Manhattan Bridge.\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(input_text, return_tensors=\"pt\")\n", "\n", "# 获取模型输出\n", "with torch.no_grad():\n", " outputs = model(**inputs)\n", "\n", "# 获取预测分数\n", "logits = outputs.logits\n", "predictions = torch.argmax(logits, dim=2)\n", "\n", "# 将预测结果映射到标签\n", "tokens = tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0])\n", "pred_labels = [label_list[prediction.item()] for prediction in predictions[0]]\n", "\n", "# 打印结果\n", "for token, label in zip(tokens, pred_labels):\n", " print(f\"{token}: {label}\")" ] }, { "cell_type": "markdown", "id": "9fd72d2d-108b-4722-beab-df35640dcb15", "metadata": {}, "source": [ "## 示例 6：文本分类 (`AutoModelForSequenceClassification`)\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "ae2d0feb-680d-44af-98af-b31677c8bbb3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "文本: I love using transformers library!\n", "情感预测: Positive\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n", "import torch\n", "import torch.nn.functional as F\n", "\n", "# 指定模型名称\n", "model_name = \"distilbert-base-uncased-finetuned-sst-2-english\"\n", "\n", "# 加载 Tokenizer\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "\n", "# 加载预训练模型\n", "model = AutoModelForSequenceClassification.from_pretrained(model_name)\n", "\n", "# 输入文本\n", "input_text = \"I love using transformers library!\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(input_text, return_tensors=\"pt\")\n", "\n", "# 获取模型输出\n", "with torch.no_grad():\n", " outputs = model(**inputs)\n", "\n", "# 获取预测分数\n", "logits = outputs.logits\n", "probabilities = F.softmax(logits, dim=1)\n", "\n", "# 获取标签\n", "labels = ['Negative', 'Positive']\n", "prediction = torch.argmax(probabilities, dim=1)\n", "predicted_label = labels[prediction]\n", "\n", "# 打印结果\n", "print(f\"文本: {input_text}\")\n", "print(f\"情感预测: {predicted_label}\")" ] }, { "cell_type": "markdown", "id": "981f413d-09d7-400d-ba2f-f04bc0d5e097", "metadata": {}, "source": [ "## 示例 7：特征提取 (`AutoModel`)\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "6b1e45a6-dfef-42e4-a483-e200ae8cd448", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Last hidden state shape: torch.Size([1, 8, 768])\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModel\n", "import torch\n", "\n", "# 指定模型名称\n", "model_name = \"bert-base-uncased\"\n", "\n", "# 加载 Tokenizer 和模型\n", "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", "model = AutoModel.from_pretrained(model_name)\n", "\n", "# 输入文本\n", "input_text = \"This is a sample sentence.\"\n", "\n", "# 编码输入\n", "inputs = tokenizer(input_text, return_tensors=\"pt\")\n", "\n", "# 获取模型输出\n", "with torch.no_grad():\n", " outputs = model(**inputs)\n", "\n", "# 获取最后一层隐藏状态\n", "last_hidden_states = outputs.last_hidden_state\n", "\n", "# 输出维度\n", "print(f\"Last hidden state shape: {last_hidden_states.shape}\")" ] }, { "cell_type": "markdown", "id": "53e6009d-3196-42a3-b6cf-8cbf40117ff3", "metadata": {}, "source": [ "## 查看源码\n", "\n", "以 `AutoModelForQuestionAnswering` 为例，使用 `inspect` 库查看对应源码:" ] }, { "cell_type": "markdown", "id": "b22d1db0-f217-438d-b1d0-c3fc553ec1e5", "metadata": {}, "source": [ "### 查看 `__init__` 方法" ] }, { "cell_type": "code", "execution_count": 10, "id": "2e3bd9cd-0b0a-4092-a0a5-e89e17742d20", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " def __init__(self, config: PretrainedConfig):\n", " super().__init__(config)\n", "\n", " self.distilbert = DistilBertModel(config)\n", " self.qa_outputs = nn.Linear(config.dim, config.num_labels)\n", " if config.num_labels != 2:\n", " raise ValueError(f\"config.num_labels should be 2, but it is {config.num_labels}\")\n", "\n", " self.dropout = nn.Dropout(config.qa_dropout)\n", "\n", " # Initialize weights and apply final processing\n", " self.post_init()\n", "\n" ] } ], "source": [ "import inspect\n", "from transformers import AutoModelForQuestionAnswering\n", "\n", "# 加载预训练模型\n", "model = AutoModelForQuestionAnswering.from_pretrained(\"distilbert-base-uncased-distilled-squad\")\n", "\n", "# 获取并打印 __init__ 方法的源码\n", "init_code = inspect.getsource(model.__init__)\n", "print(init_code)" ] }, { "cell_type": "markdown", "id": "2cc2d04f-c1f9-4bba-b7c1-f5daeac3a5d7", "metadata": {}, "source": [ "### 查看 `forward` 方法" ] }, { "cell_type": "code", "execution_count": 11, "id": "37924b48-b057-4621-89d4-6e766874d2f6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " @auto_docstring\n", " def forward(\n", " self,\n", " input_ids: Optional[torch.Tensor] = None,\n", " attention_mask: Optional[torch.Tensor] = None,\n", " head_mask: Optional[torch.Tensor] = None,\n", " inputs_embeds: Optional[torch.Tensor] = None,\n", " start_positions: Optional[torch.Tensor] = None,\n", " end_positions: Optional[torch.Tensor] = None,\n", " output_attentions: Optional[bool] = None,\n", " output_hidden_states: Optional[bool] = None,\n", " return_dict: Optional[bool] = None,\n", " ) -> Union[QuestionAnsweringModelOutput, tuple[torch.Tensor, ...]]:\n", " r\"\"\"\n", " input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`):\n", " Indices of input sequence tokens in the vocabulary.\n", "\n", " Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and\n", " [`PreTrainedTokenizer.__call__`] for details.\n", "\n", " [What are input IDs?](../glossary#input-ids)\n", " inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*):\n", " Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This\n", " is useful if you want more control over how to convert `input_ids` indices into associated vectors than the\n", " model's internal embedding lookup matrix.\n", " \"\"\"\n", " return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n", "\n", " distilbert_output = self.distilbert(\n", " input_ids=input_ids,\n", " attention_mask=attention_mask,\n", " head_mask=head_mask,\n", " inputs_embeds=inputs_embeds,\n", " output_attentions=output_attentions,\n", " output_hidden_states=output_hidden_states,\n", " return_dict=return_dict,\n", " )\n", " hidden_states = distilbert_output[0] # (bs, max_query_len, dim)\n", "\n", " hidden_states = self.dropout(hidden_states) # (bs, max_query_len, dim)\n", " logits = self.qa_outputs(hidden_states) # (bs, max_query_len, 2)\n", " start_logits, end_logits = logits.split(1, dim=-1)\n", " start_logits = start_logits.squeeze(-1).contiguous() # (bs, max_query_len)\n", " end_logits = end_logits.squeeze(-1).contiguous() # (bs, max_query_len)\n", "\n", " total_loss = None\n", " if start_positions is not None and end_positions is not None:\n", " # If we are on multi-GPU, split add a dimension\n", " if len(start_positions.size()) > 1:\n", " start_positions = start_positions.squeeze(-1)\n", " if len(end_positions.size()) > 1:\n", " end_positions = end_positions.squeeze(-1)\n", " # sometimes the start/end positions are outside our model inputs, we ignore these terms\n", " ignored_index = start_logits.size(1)\n", " start_positions = start_positions.clamp(0, ignored_index)\n", " end_positions = end_positions.clamp(0, ignored_index)\n", "\n", " loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index)\n", " start_loss = loss_fct(start_logits, start_positions)\n", " end_loss = loss_fct(end_logits, end_positions)\n", " total_loss = (start_loss + end_loss) / 2\n", "\n", " if not return_dict:\n", " output = (start_logits, end_logits) + distilbert_output[1:]\n", " return ((total_loss,) + output) if total_loss is not None else output\n", "\n", " return QuestionAnsweringModelOutput(\n", " loss=total_loss,\n", " start_logits=start_logits,\n", " end_logits=end_logits,\n", " hidden_states=distilbert_output.hidden_states,\n", " attentions=distilbert_output.attentions,\n", " )\n", "\n" ] } ], "source": [ "# 获取并打印 forward 方法的源码\n", "forward_code = inspect.getsource(model.forward)\n", "print(forward_code)" ] }, { "cell_type": "markdown", "id": "5c180a0a-f434-45f2-a4dc-4119b0616e1d", "metadata": {}, "source": [ "### 使用 `help` 快速查看\n", "\n", "除了 `inspect`，我们还可以使用 Python 内置的 `help` 函数查看模型的文档和方法:" ] }, { "cell_type": "code", "execution_count": 12, "id": "30b51b00-93ef-48e5-aa74-3dab2bb26882", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Help on class AutoModelForQuestionAnswering in module transformers.models.auto.modeling_auto:\n", "\n", "class AutoModelForQuestionAnswering(transformers.models.auto.auto_factory._BaseAutoModelClass)\n", " | AutoModelForQuestionAnswering(*args, **kwargs) -> None\n", " |\n", " | This is a generic model class that will be instantiated as one of the model classes of the library (with a question answering head) when created\n", " | with the [`~AutoModelForQuestionAnswering.from_pretrained`] class method or the [`~AutoModelForQuestionAnswering.from_config`] class\n", " | method.\n", " |\n", " | This class cannot be instantiated directly using `__init__()` (throws an error).\n", " |\n", " | Method resolution order:\n", " | AutoModelForQuestionAnswering\n", " | transformers.models.auto.auto_factory._BaseAutoModelClass\n", " | builtins.object\n", " |\n", " | Class methods defined here:\n", " |\n", " | from_config(**kwargs) from transformers.models.auto.auto_factory._BaseAutoModelClass\n", " | Instantiates one of the model classes of the library (with a question answering head) from a configuration.\n", " |\n", " | Note:\n", " | Loading a model from its configuration file does **not** load the model weights. It only affects the\n", " | model's configuration. Use [`~AutoModelForQuestionAnswering.from_pretrained`] to load the model weights.\n", " |\n", " | Args:\n", " | config ([`PretrainedConfig`]):\n", " | The model class to instantiate is selected based on the configuration class:\n", " |\n", " | - [`AlbertConfig`] configuration class: [`AlbertForQuestionAnswering`] (ALBERT model)\n", " | - [`ArceeConfig`] configuration class: [`ArceeForQuestionAnswering`] (Arcee model)\n", " | - [`BartConfig`] configuration class: [`BartForQuestionAnswering`] (BART model)\n", " | - [`BertConfig`] configuration class: [`BertForQuestionAnswering`] (BERT model)\n", " | - [`BigBirdConfig`] configuration class: [`BigBirdForQuestionAnswering`] (BigBird model)\n", " | - [`BigBirdPegasusConfig`] configuration class: [`BigBirdPegasusForQuestionAnswering`] (BigBird-Pegasus model)\n", " | - [`BloomConfig`] configuration class: [`BloomForQuestionAnswering`] (BLOOM model)\n", " | - [`CamembertConfig`] configuration class: [`CamembertForQuestionAnswering`] (CamemBERT model)\n", " | - [`CanineConfig`] configuration class: [`CanineForQuestionAnswering`] (CANINE model)\n", " | - [`ConvBertConfig`] configuration class: [`ConvBertForQuestionAnswering`] (ConvBERT model)\n", " | - [`Data2VecTextConfig`] configuration class: [`Data2VecTextForQuestionAnswering`] (Data2VecText model)\n", " | - [`DebertaConfig`] configuration class: [`DebertaForQuestionAnswering`] (DeBERTa model)\n", " | - [`DebertaV2Config`] configuration class: [`DebertaV2ForQuestionAnswering`] (DeBERTa-v2 model)\n", " | - [`DiffLlamaConfig`] configuration class: [`DiffLlamaForQuestionAnswering`] (DiffLlama model)\n", " | - [`DistilBertConfig`] configuration class: [`DistilBertForQuestionAnswering`] (DistilBERT model)\n", " | - [`ElectraConfig`] configuration class: [`ElectraForQuestionAnswering`] (ELECTRA model)\n", " | - [`ErnieConfig`] configuration class: [`ErnieForQuestionAnswering`] (ERNIE model)\n", " | - [`ErnieMConfig`] configuration class: [`ErnieMForQuestionAnswering`] (ErnieM model)\n", " | - [`Exaone4Config`] configuration class: [`Exaone4ForQuestionAnswering`] (EXAONE-4.0 model)\n", " | - [`FNetConfig`] configuration class: [`FNetForQuestionAnswering`] (FNet model)\n", " | - [`FalconConfig`] configuration class: [`FalconForQuestionAnswering`] (Falcon model)\n", " | - [`FlaubertConfig`] configuration class: [`FlaubertForQuestionAnsweringSimple`] (FlauBERT model)\n", " | - [`FunnelConfig`] configuration class: [`FunnelForQuestionAnswering`] (Funnel Transformer model)\n", " | - [`GPT2Config`] configuration class: [`GPT2ForQuestionAnswering`] (OpenAI GPT-2 model)\n", " | - [`GPTJConfig`] configuration class: [`GPTJForQuestionAnswering`] (GPT-J model)\n", " | - [`GPTNeoConfig`] configuration class: [`GPTNeoForQuestionAnswering`] (GPT Neo model)\n", " | - [`GPTNeoXConfig`] configuration class: [`GPTNeoXForQuestionAnswering`] (GPT NeoX model)\n", " | - [`IBertConfig`] configuration class: [`IBertForQuestionAnswering`] (I-BERT model)\n", " | - [`LEDConfig`] configuration class: [`LEDForQuestionAnswering`] (LED model)\n", " | - [`LayoutLMv2Config`] configuration class: [`LayoutLMv2ForQuestionAnswering`] (LayoutLMv2 model)\n", " | - [`LayoutLMv3Config`] configuration class: [`LayoutLMv3ForQuestionAnswering`] (LayoutLMv3 model)\n", " | - [`LiltConfig`] configuration class: [`LiltForQuestionAnswering`] (LiLT model)\n", " | - [`LlamaConfig`] configuration class: [`LlamaForQuestionAnswering`] (LLaMA model)\n", " | - [`LongformerConfig`] configuration class: [`LongformerForQuestionAnswering`] (Longformer model)\n", " | - [`LukeConfig`] configuration class: [`LukeForQuestionAnswering`] (LUKE model)\n", " | - [`LxmertConfig`] configuration class: [`LxmertForQuestionAnswering`] (LXMERT model)\n", " | - [`MBartConfig`] configuration class: [`MBartForQuestionAnswering`] (mBART model)\n", " | - [`MPNetConfig`] configuration class: [`MPNetForQuestionAnswering`] (MPNet model)\n", " | - [`MT5Config`] configuration class: [`MT5ForQuestionAnswering`] (MT5 model)\n", " | - [`MarkupLMConfig`] configuration class: [`MarkupLMForQuestionAnswering`] (MarkupLM model)\n", " | - [`MegaConfig`] configuration class: [`MegaForQuestionAnswering`] (MEGA model)\n", " | - [`MegatronBertConfig`] configuration class: [`MegatronBertForQuestionAnswering`] (Megatron-BERT model)\n", " | - [`MiniMaxConfig`] configuration class: [`MiniMaxForQuestionAnswering`] (MiniMax model)\n", " | - [`MistralConfig`] configuration class: [`MistralForQuestionAnswering`] (Mistral model)\n", " | - [`MixtralConfig`] configuration class: [`MixtralForQuestionAnswering`] (Mixtral model)\n", " | - [`MobileBertConfig`] configuration class: [`MobileBertForQuestionAnswering`] (MobileBERT model)\n", " | - [`ModernBertConfig`] configuration class: [`ModernBertForQuestionAnswering`] (ModernBERT model)\n", " | - [`MptConfig`] configuration class: [`MptForQuestionAnswering`] (MPT model)\n", " | - [`MraConfig`] configuration class: [`MraForQuestionAnswering`] (MRA model)\n", " | - [`MvpConfig`] configuration class: [`MvpForQuestionAnswering`] (MVP model)\n", " | - [`NemotronConfig`] configuration class: [`NemotronForQuestionAnswering`] (Nemotron model)\n", " | - [`NezhaConfig`] configuration class: [`NezhaForQuestionAnswering`] (Nezha model)\n", " | - [`NystromformerConfig`] configuration class: [`NystromformerForQuestionAnswering`] (Nyströmformer model)\n", " | - [`OPTConfig`] configuration class: [`OPTForQuestionAnswering`] (OPT model)\n", " | - [`QDQBertConfig`] configuration class: [`QDQBertForQuestionAnswering`] (QDQBert model)\n", " | - [`Qwen2Config`] configuration class: [`Qwen2ForQuestionAnswering`] (Qwen2 model)\n", " | - [`Qwen2MoeConfig`] configuration class: [`Qwen2MoeForQuestionAnswering`] (Qwen2MoE model)\n", " | - [`Qwen3Config`] configuration class: [`Qwen3ForQuestionAnswering`] (Qwen3 model)\n", " | - [`Qwen3MoeConfig`] configuration class: [`Qwen3MoeForQuestionAnswering`] (Qwen3MoE model)\n", " | - [`ReformerConfig`] configuration class: [`ReformerForQuestionAnswering`] (Reformer model)\n", " | - [`RemBertConfig`] configuration class: [`RemBertForQuestionAnswering`] (RemBERT model)\n", " | - [`RoCBertConfig`] configuration class: [`RoCBertForQuestionAnswering`] (RoCBert model)\n", " | - [`RoFormerConfig`] configuration class: [`RoFormerForQuestionAnswering`] (RoFormer model)\n", " | - [`RobertaConfig`] configuration class: [`RobertaForQuestionAnswering`] (RoBERTa model)\n", " | - [`RobertaPreLayerNormConfig`] configuration class: [`RobertaPreLayerNormForQuestionAnswering`] (RoBERTa-PreLayerNorm model)\n", " | - [`SeedOssConfig`] configuration class: [`SeedOssForQuestionAnswering`] (SeedOss model)\n", " | - [`SmolLM3Config`] configuration class: [`SmolLM3ForQuestionAnswering`] (SmolLM3 model)\n", " | - [`SplinterConfig`] configuration class: [`SplinterForQuestionAnswering`] (Splinter model)\n", " | - [`SqueezeBertConfig`] configuration class: [`SqueezeBertForQuestionAnswering`] (SqueezeBERT model)\n", " | - [`T5Config`] configuration class: [`T5ForQuestionAnswering`] (T5 model)\n", " | - [`UMT5Config`] configuration class: [`UMT5ForQuestionAnswering`] (UMT5 model)\n", " | - [`XLMConfig`] configuration class: [`XLMForQuestionAnsweringSimple`] (XLM model)\n", " | - [`XLMRobertaConfig`] configuration class: [`XLMRobertaForQuestionAnswering`] (XLM-RoBERTa model)\n", " | - [`XLMRobertaXLConfig`] configuration class: [`XLMRobertaXLForQuestionAnswering`] (XLM-RoBERTa-XL model)\n", " | - [`XLNetConfig`] configuration class: [`XLNetForQuestionAnsweringSimple`] (XLNet model)\n", " | - [`XmodConfig`] configuration class: [`XmodForQuestionAnswering`] (X-MOD model)\n", " | - [`YosoConfig`] configuration class: [`YosoForQuestionAnswering`] (YOSO model)\n", " | attn_implementation (`str`, *optional*):\n", " | The attention implementation to use in the model (if relevant). Can be any of `\"eager\"` (manual implementation of the attention), `\"sdpa\"` (using [`F.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html)), or `\"flash_attention_2\"` (using [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual `\"eager\"` implementation.\n", " |\n", " | Examples:\n", " |\n", " | ```python\n", " | >>> from transformers import AutoConfig, AutoModelForQuestionAnswering\n", " |\n", " | >>> # Download configuration from huggingface.co and cache.\n", " | >>> config = AutoConfig.from_pretrained(\"google-bert/bert-base-cased\")\n", " | >>> model = AutoModelForQuestionAnswering.from_config(config)\n", " | ```\n", " |\n", " | from_pretrained(*model_args, **kwargs) from transformers.models.auto.auto_factory._BaseAutoModelClass\n", " | Instantiate one of the model classes of the library (with a question answering head) from a pretrained model.\n", " |\n", " | The model class to instantiate is selected based on the `model_type` property of the config object (either\n", " | passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by\n", " | falling back to using pattern matching on `pretrained_model_name_or_path`:\n", " |\n", " | - **albert** -- [`AlbertForQuestionAnswering`] (ALBERT model)\n", " | - **arcee** -- [`ArceeForQuestionAnswering`] (Arcee model)\n", " | - **bart** -- [`BartForQuestionAnswering`] (BART model)\n", " | - **bert** -- [`BertForQuestionAnswering`] (BERT model)\n", " | - **big_bird** -- [`BigBirdForQuestionAnswering`] (BigBird model)\n", " | - **bigbird_pegasus** -- [`BigBirdPegasusForQuestionAnswering`] (BigBird-Pegasus model)\n", " | - **bloom** -- [`BloomForQuestionAnswering`] (BLOOM model)\n", " | - **camembert** -- [`CamembertForQuestionAnswering`] (CamemBERT model)\n", " | - **canine** -- [`CanineForQuestionAnswering`] (CANINE model)\n", " | - **convbert** -- [`ConvBertForQuestionAnswering`] (ConvBERT model)\n", " | - **data2vec-text** -- [`Data2VecTextForQuestionAnswering`] (Data2VecText model)\n", " | - **deberta** -- [`DebertaForQuestionAnswering`] (DeBERTa model)\n", " | - **deberta-v2** -- [`DebertaV2ForQuestionAnswering`] (DeBERTa-v2 model)\n", " | - **diffllama** -- [`DiffLlamaForQuestionAnswering`] (DiffLlama model)\n", " | - **distilbert** -- [`DistilBertForQuestionAnswering`] (DistilBERT model)\n", " | - **electra** -- [`ElectraForQuestionAnswering`] (ELECTRA model)\n", " | - **ernie** -- [`ErnieForQuestionAnswering`] (ERNIE model)\n", " | - **ernie_m** -- [`ErnieMForQuestionAnswering`] (ErnieM model)\n", " | - **exaone4** -- [`Exaone4ForQuestionAnswering`] (EXAONE-4.0 model)\n", " | - **falcon** -- [`FalconForQuestionAnswering`] (Falcon model)\n", " | - **flaubert** -- [`FlaubertForQuestionAnsweringSimple`] (FlauBERT model)\n", " | - **fnet** -- [`FNetForQuestionAnswering`] (FNet model)\n", " | - **funnel** -- [`FunnelForQuestionAnswering`] (Funnel Transformer model)\n", " | - **gpt2** -- [`GPT2ForQuestionAnswering`] (OpenAI GPT-2 model)\n", " | - **gpt_neo** -- [`GPTNeoForQuestionAnswering`] (GPT Neo model)\n", " | - **gpt_neox** -- [`GPTNeoXForQuestionAnswering`] (GPT NeoX model)\n", " | - **gptj** -- [`GPTJForQuestionAnswering`] (GPT-J model)\n", " | - **ibert** -- [`IBertForQuestionAnswering`] (I-BERT model)\n", " | - **layoutlmv2** -- [`LayoutLMv2ForQuestionAnswering`] (LayoutLMv2 model)\n", " | - **layoutlmv3** -- [`LayoutLMv3ForQuestionAnswering`] (LayoutLMv3 model)\n", " | - **led** -- [`LEDForQuestionAnswering`] (LED model)\n", " | - **lilt** -- [`LiltForQuestionAnswering`] (LiLT model)\n", " | - **llama** -- [`LlamaForQuestionAnswering`] (LLaMA model)\n", " | - **longformer** -- [`LongformerForQuestionAnswering`] (Longformer model)\n", " | - **luke** -- [`LukeForQuestionAnswering`] (LUKE model)\n", " | - **lxmert** -- [`LxmertForQuestionAnswering`] (LXMERT model)\n", " | - **markuplm** -- [`MarkupLMForQuestionAnswering`] (MarkupLM model)\n", " | - **mbart** -- [`MBartForQuestionAnswering`] (mBART model)\n", " | - **mega** -- [`MegaForQuestionAnswering`] (MEGA model)\n", " | - **megatron-bert** -- [`MegatronBertForQuestionAnswering`] (Megatron-BERT model)\n", " | - **minimax** -- [`MiniMaxForQuestionAnswering`] (MiniMax model)\n", " | - **mistral** -- [`MistralForQuestionAnswering`] (Mistral model)\n", " | - **mixtral** -- [`MixtralForQuestionAnswering`] (Mixtral model)\n", " | - **mobilebert** -- [`MobileBertForQuestionAnswering`] (MobileBERT model)\n", " | - **modernbert** -- [`ModernBertForQuestionAnswering`] (ModernBERT model)\n", " | - **mpnet** -- [`MPNetForQuestionAnswering`] (MPNet model)\n", " | - **mpt** -- [`MptForQuestionAnswering`] (MPT model)\n", " | - **mra** -- [`MraForQuestionAnswering`] (MRA model)\n", " | - **mt5** -- [`MT5ForQuestionAnswering`] (MT5 model)\n", " | - **mvp** -- [`MvpForQuestionAnswering`] (MVP model)\n", " | - **nemotron** -- [`NemotronForQuestionAnswering`] (Nemotron model)\n", " | - **nezha** -- [`NezhaForQuestionAnswering`] (Nezha model)\n", " | - **nystromformer** -- [`NystromformerForQuestionAnswering`] (Nyströmformer model)\n", " | - **opt** -- [`OPTForQuestionAnswering`] (OPT model)\n", " | - **qdqbert** -- [`QDQBertForQuestionAnswering`] (QDQBert model)\n", " | - **qwen2** -- [`Qwen2ForQuestionAnswering`] (Qwen2 model)\n", " | - **qwen2_moe** -- [`Qwen2MoeForQuestionAnswering`] (Qwen2MoE model)\n", " | - **qwen3** -- [`Qwen3ForQuestionAnswering`] (Qwen3 model)\n", " | - **qwen3_moe** -- [`Qwen3MoeForQuestionAnswering`] (Qwen3MoE model)\n", " | - **reformer** -- [`ReformerForQuestionAnswering`] (Reformer model)\n", " | - **rembert** -- [`RemBertForQuestionAnswering`] (RemBERT model)\n", " | - **roberta** -- [`RobertaForQuestionAnswering`] (RoBERTa model)\n", " | - **roberta-prelayernorm** -- [`RobertaPreLayerNormForQuestionAnswering`] (RoBERTa-PreLayerNorm model)\n", " | - **roc_bert** -- [`RoCBertForQuestionAnswering`] (RoCBert model)\n", " | - **roformer** -- [`RoFormerForQuestionAnswering`] (RoFormer model)\n", " | - **seed_oss** -- [`SeedOssForQuestionAnswering`] (SeedOss model)\n", " | - **smollm3** -- [`SmolLM3ForQuestionAnswering`] (SmolLM3 model)\n", " | - **splinter** -- [`SplinterForQuestionAnswering`] (Splinter model)\n", " | - **squeezebert** -- [`SqueezeBertForQuestionAnswering`] (SqueezeBERT model)\n", " | - **t5** -- [`T5ForQuestionAnswering`] (T5 model)\n", " | - **umt5** -- [`UMT5ForQuestionAnswering`] (UMT5 model)\n", " | - **xlm** -- [`XLMForQuestionAnsweringSimple`] (XLM model)\n", " | - **xlm-roberta** -- [`XLMRobertaForQuestionAnswering`] (XLM-RoBERTa model)\n", " | - **xlm-roberta-xl** -- [`XLMRobertaXLForQuestionAnswering`] (XLM-RoBERTa-XL model)\n", " | - **xlnet** -- [`XLNetForQuestionAnsweringSimple`] (XLNet model)\n", " | - **xmod** -- [`XmodForQuestionAnswering`] (X-MOD model)\n", " | - **yoso** -- [`YosoForQuestionAnswering`] (YOSO model)\n", " |\n", " | The model is set in evaluation mode by default using `model.eval()` (so for instance, dropout modules are\n", " | deactivated). To train the model, you should first set it back in training mode with `model.train()`\n", " |\n", " | Args:\n", " | pretrained_model_name_or_path (`str` or `os.PathLike`):\n", " | Can be either:\n", " |\n", " | - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.\n", " | - A path to a *directory* containing model weights saved using\n", " | [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.\n", " | - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In\n", " | this case, `from_tf` should be set to `True` and a configuration object should be provided as\n", " | `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a\n", " | PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.\n", " | model_args (additional positional arguments, *optional*):\n", " | Will be passed along to the underlying model `__init__()` method.\n", " | config ([`PretrainedConfig`], *optional*):\n", " | Configuration for the model to use instead of an automatically loaded configuration. Configuration can\n", " | be automatically loaded when:\n", " |\n", " | - The model is a model provided by the library (loaded with the *model id* string of a pretrained\n", " | model).\n", " | - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the\n", " | save directory.\n", " | - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a\n", " | configuration JSON file named *config.json* is found in the directory.\n", " | state_dict (*dict[str, torch.Tensor]*, *optional*):\n", " | A state dictionary to use instead of a state dictionary loaded from saved weights file.\n", " |\n", " | This option can be used if you want to create a model from a pretrained configuration but load your own\n", " | weights. In this case though, you should check if using [`~PreTrainedModel.save_pretrained`] and\n", " | [`~PreTrainedModel.from_pretrained`] is not a simpler option.\n", " | cache_dir (`str` or `os.PathLike`, *optional*):\n", " | Path to a directory in which a downloaded pretrained model configuration should be cached if the\n", " | standard cache should not be used.\n", " | from_tf (`bool`, *optional*, defaults to `False`):\n", " | Load the model weights from a TensorFlow checkpoint save file (see docstring of\n", " | `pretrained_model_name_or_path` argument).\n", " | force_download (`bool`, *optional*, defaults to `False`):\n", " | Whether or not to force the (re-)download of the model weights and configuration files, overriding the\n", " | cached versions if they exist.\n", " | resume_download:\n", " | Deprecated and ignored. All downloads are now resumed by default when possible.\n", " | Will be removed in v5 of Transformers.\n", " | proxies (`dict[str, str]`, *optional*):\n", " | A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',\n", " | 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.\n", " | output_loading_info(`bool`, *optional*, defaults to `False`):\n", " | Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.\n", " | local_files_only(`bool`, *optional*, defaults to `False`):\n", " | Whether or not to only look at local files (e.g., not try downloading the model).\n", " | revision (`str`, *optional*, defaults to `\"main\"`):\n", " | The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a\n", " | git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any\n", " | identifier allowed by git.\n", " | trust_remote_code (`bool`, *optional*, defaults to `False`):\n", " | Whether or not to allow for custom models defined on the Hub in their own modeling files. This option\n", " | should only be set to `True` for repositories you trust and in which you have read the code, as it will\n", " | execute code present on the Hub on your local machine.\n", " | code_revision (`str`, *optional*, defaults to `\"main\"`):\n", " | The specific revision to use for the code on the Hub, if the code leaves in a different repository than\n", " | the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based\n", " | system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier\n", " | allowed by git.\n", " | kwargs (additional keyword arguments, *optional*):\n", " | Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,\n", " | `output_attentions=True`). Behaves differently depending on whether a `config` is provided or\n", " | automatically loaded:\n", " |\n", " | - If a configuration is provided with `config`, `**kwargs` will be directly passed to the\n", " | underlying model's `__init__` method (we assume all relevant updates to the configuration have\n", " | already been done)\n", " | - If a configuration is not provided, `kwargs` will be first passed to the configuration class\n", " | initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that\n", " | corresponds to a configuration attribute will be used to override said attribute with the\n", " | supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute\n", " | will be passed to the underlying model's `__init__` function.\n", " |\n", " | Examples:\n", " |\n", " | ```python\n", " | >>> from transformers import AutoConfig, AutoModelForQuestionAnswering\n", " |\n", " | >>> # Download model and configuration from huggingface.co and cache.\n", " | >>> model = AutoModelForQuestionAnswering.from_pretrained(\"google-bert/bert-base-cased\")\n", " |\n", " | >>> # Update configuration during loading\n", " | >>> model = AutoModelForQuestionAnswering.from_pretrained(\"google-bert/bert-base-cased\", output_attentions=True)\n", " | >>> model.config.output_attentions\n", " | True\n", " |\n", " | >>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)\n", " | >>> config = AutoConfig.from_pretrained(\"./tf_model/bert_tf_model_config.json\")\n", " | >>> model = AutoModelForQuestionAnswering.from_pretrained(\n", " | ... \"./tf_model/bert_tf_checkpoint.ckpt.index\", from_tf=True, config=config\n", " | ... )\n", " | ```\n", " |\n", " | ----------------------------------------------------------------------\n", " | Methods inherited from transformers.models.auto.auto_factory._BaseAutoModelClass:\n", " |\n", " | __init__(self, *args, **kwargs) -> None\n", " | Initialize self. See help(type(self)) for accurate signature.\n", " |\n", " | ----------------------------------------------------------------------\n", " | Class methods inherited from transformers.models.auto.auto_factory._BaseAutoModelClass:\n", " |\n", " | register(config_class, model_class, exist_ok=False) -> None\n", " | Register a new model for this class.\n", " |\n", " | Args:\n", " | config_class ([`PretrainedConfig`]):\n", " | The configuration corresponding to the model to register.\n", " | model_class ([`PreTrainedModel`]):\n", " | The model to register.\n", " |\n", " | ----------------------------------------------------------------------\n", " | Data descriptors inherited from transformers.models.auto.auto_factory._BaseAutoModelClass:\n", " |\n", " | __dict__\n", " | dictionary for instance variables\n", " |\n", " | __weakref__\n", " | list of weak references to the object\n", "\n" ] } ], "source": [ "help(AutoModelForQuestionAnswering)" ] }, { "cell_type": "code", "execution_count": null, "id": "964b87e1-74d7-49b7-a763-4ee8978c299f", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "ai", "language": "python", "name": "ai" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }