{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "sec-title",
   "metadata": {},
   "source": [
    "# Trafilatura: Overview\n",
    "\n",
    "A hands-on tour of the main features. For full documentation see [trafilatura.readthedocs.io](https://trafilatura.readthedocs.io/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-install",
   "metadata": {},
   "source": [
    "## 1. Installation\n",
    "\n",
    "```bash\n",
    "pip install trafilatura\n",
    "```\n",
    "\n",
    "To update: `pip install -U trafilatura`. For more info see [Installation](https://trafilatura.readthedocs.io/en/latest/installation.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-quickstart",
   "metadata": {},
   "source": [
    "## 2. Quick start\n",
    "\n",
    "Download a web page and extract its main text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "quickstart-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import extract, fetch_url\n",
    "\n",
    "url = \"https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/\"\n",
    "downloaded = fetch_url(url)\n",
    "\n",
    "text = extract(downloaded)\n",
    "print(text[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-inputs",
   "metadata": {},
   "source": [
    "## 3. Input formats\n",
    "\n",
    "`extract()` accepts HTML as a string or as a pre-parsed LXML tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "input-string",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from a local file\n",
    "# with open('myfile.html', encoding='utf-8') as f:\n",
    "#     text = extract(f.read())\n",
    "\n",
    "# from an HTML string\n",
    "my_html = \"<html><body><article><p>This is the main text.</p></article></body></html>\"\n",
    "print(extract(my_html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "input-lxml",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from a pre-parsed LXML tree\n",
    "from lxml import html\n",
    "\n",
    "tree = html.fromstring(downloaded)\n",
    "print(extract(tree)[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-formats",
   "metadata": {},
   "source": [
    "## 4. Output formats\n",
    "\n",
    "Available formats: `txt` (default), `json`, `markdown`, `xml`, `xmltei`, `csv`, `html`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "format-json",
   "metadata": {},
   "outputs": [],
   "source": [
    "# JSON with metadata (metadata is off by default)\n",
    "result = extract(downloaded, output_format=\"json\", with_metadata=True)\n",
    "print(result[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "format-md",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Markdown\n",
    "result = extract(downloaded, output_format=\"markdown\")\n",
    "print(result[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "format-xml",
   "metadata": {},
   "outputs": [],
   "source": [
    "# XML\n",
    "result = extract(downloaded, output_format=\"xml\")\n",
    "print(result[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-options",
   "metadata": {},
   "source": [
    "## 5. Extraction options\n",
    "\n",
    "### Extraction modes\n",
    "\n",
    "Trafilatura offers several extraction strategies. Compare them on the same page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "options-modes",
   "metadata": {},
   "outputs": [],
   "source": [
    "standard = extract(downloaded)\n",
    "fast     = extract(downloaded, fast=True)\n",
    "precise  = extract(downloaded, favor_precision=True)\n",
    "recall   = extract(downloaded, favor_recall=True)\n",
    "\n",
    "for label, result in [(\"standard\", standard), (\"fast\", fast), (\"precision\", precise), (\"recall\", recall)]:\n",
    "    print(f\"{label:>10}: {len(result):>5} chars\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "options-elements-md",
   "metadata": {},
   "source": [
    "### Including or excluding elements\n",
    "\n",
    "Comments and tables are included by default. Formatting and images are not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "options-elements",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = extract(downloaded,\n",
    "    include_comments=False,\n",
    "    include_tables=False,\n",
    "    include_formatting=True,\n",
    "    include_images=True,\n",
    "    output_format=\"markdown\",\n",
    ")\n",
    "print(result[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "options-extractor-md",
   "metadata": {},
   "source": [
    "### Reusable settings with `Extractor`\n",
    "\n",
    "Bundle options into an `Extractor` object to reuse across multiple extractions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "options-extractor",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.settings import Extractor\n",
    "\n",
    "options = Extractor(\n",
    "    output_format=\"json\",\n",
    "    with_metadata=True,\n",
    "    comments=False,\n",
    ")\n",
    "\n",
    "# reuse on multiple pages\n",
    "result = extract(downloaded, options=options)\n",
    "print(result[:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-functions",
   "metadata": {},
   "source": [
    "## 6. Other extraction functions\n",
    "\n",
    "| Function | Returns | Use case |\n",
    "|----------|---------|----------|\n",
    "| `extract()` | string | Main extraction, all formats |\n",
    "| `bare_extraction()` | `Document` object | Structured access to fields |\n",
    "| `html2txt()` | string | All text (including nav, footers) |\n",
    "| `extract_metadata()` | `Document` object | Metadata only |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "func-bare",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import bare_extraction\n",
    "\n",
    "doc = bare_extraction(downloaded, with_metadata=True)\n",
    "print(f\"Title:  {doc.title}\")\n",
    "print(f\"Author: {doc.author}\")\n",
    "print(f\"Date:   {doc.date}\")\n",
    "print(f\"Text:   {doc.text[:100]}...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "func-html2txt",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import html2txt\n",
    "\n",
    "# extract ALL text, including navigation, footers, etc.\n",
    "all_text = html2txt(downloaded)\n",
    "print(f\"html2txt: {len(all_text):>5} chars (everything)\")\n",
    "print(f\"extract:  {len(extract(downloaded)):>5} chars (main content only)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "func-metadata",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import extract_metadata\n",
    "\n",
    "meta = extract_metadata(downloaded)\n",
    "print(f\"Title:    {meta.title}\")\n",
    "print(f\"Author:   {meta.author}\")\n",
    "print(f\"Date:     {meta.date}\")\n",
    "print(f\"Sitename: {meta.sitename}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-discovery",
   "metadata": {},
   "source": [
    "## 7. URL discovery\n",
    "\n",
    "Three strategies to find URLs on a website:\n",
    "\n",
    "- **Feeds** (Atom/RSS): fresh content\n",
    "- **Sitemaps**: exhaustive listing by the site owner\n",
    "- **Crawling**: follow internal links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "discovery-feeds",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.feeds import find_feed_urls\n",
    "\n",
    "links = find_feed_urls(\"https://www.theguardian.com/\")\n",
    "print(f\"{len(links)} URLs found\")\n",
    "print(\"\\n\".join(links[:5]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "discovery-sitemaps",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.sitemaps import sitemap_search\n",
    "\n",
    "links = sitemap_search(\"https://www.sitemaps.org\")\n",
    "print(f\"{len(links)} URLs found\")\n",
    "print(\"\\n\".join(links[:5]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "discovery-crawl",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.spider import focused_crawler\n",
    "\n",
    "to_visit, known_urls = focused_crawler(\"https://www.python.org\", max_seen_urls=10)\n",
    "print(f\"{len(to_visit)} URLs to visit, {len(known_urls)} URLs seen\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-language",
   "metadata": {},
   "source": [
    "## 8. Language filtering\n",
    "\n",
    "Filter results by language using ISO 639-1 codes. If the detected language doesn't match, `extract()` returns `None`.\n",
    "\n",
    "> Requires: `pip install trafilatura[all]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "language-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "# English page, filtering for English → should return text\n",
    "result_en = extract(downloaded, target_language=\"en\")\n",
    "print(f\"English filter: {len(result_en) if result_en else 0} chars\")\n",
    "\n",
    "# same page, filtering for Japanese → should return None\n",
    "result_ja = extract(downloaded, target_language=\"ja\")\n",
    "print(f\"Japanese filter: {result_ja}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-dedup",
   "metadata": {},
   "source": [
    "## 9. Deduplication\n",
    "\n",
    "Detect near-duplicate content using Simhash fingerprints. Useful when building a corpus from a single site where boilerplate repeats across pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dedup-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.deduplication import Simhash, content_fingerprint\n",
    "\n",
    "text_a = extract(downloaded)\n",
    "text_b = text_a  # identical\n",
    "text_c = \"Completely different content about something else entirely.\"\n",
    "\n",
    "hash_a = Simhash(text_a)\n",
    "hash_b = Simhash(text_b)\n",
    "hash_c = Simhash(text_c)\n",
    "\n",
    "print(f\"a vs b (identical):  similarity={hash_a.similarity(hash_b):.3f}\")\n",
    "print(f\"a vs c (different):  similarity={hash_a.similarity(hash_c):.3f}\")\n",
    "print(f\"\\nFingerprint: {content_fingerprint(text_a)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-pipeline",
   "metadata": {},
   "source": [
    "## 10. Putting it together\n",
    "\n",
    "A mini pipeline: discover URLs → download → extract text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pipeline-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import extract, fetch_url\n",
    "from trafilatura.feeds import find_feed_urls\n",
    "\n",
    "# discover URLs from a feed\n",
    "urls = find_feed_urls(\"https://www.theguardian.com/\")\n",
    "print(f\"Found {len(urls)} URLs\")\n",
    "\n",
    "# download and extract the first 3\n",
    "for url in urls[:3]:\n",
    "    html = fetch_url(url)\n",
    "    if html:\n",
    "        text = extract(html, favor_precision=True)\n",
    "        if text:\n",
    "            print(f\"\\n--- {url} ---\")\n",
    "            print(f\"{len(text)} chars extracted\")\n",
    "            print(text[:200] + \"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec-cli",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Trafilatura is also available as a **command-line tool**: `trafilatura -u \"https://example.org\" --json`\n",
    "\n",
    "See the [CLI documentation](https://trafilatura.readthedocs.io/en/latest/usage-cli.html) and the full [Python usage guide](https://trafilatura.readthedocs.io/en/latest/usage-python.html) for more."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbformat_minor": 5,
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}