{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "-y6gO0VbS9IJ", "vscode": { "languageId": "plaintext" } }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "7YY34qHKS9IK", "vscode": { "languageId": "plaintext" } }, "source": [ "# Interactive Art Agent\n", "This tutorial will walk through the creation of an AI Agent that helps you explore and learn art!\n", "\n", "### Overview\n", "For knowledge retrieval, this tutorial uses [LightRAG](https://github.com/HKUDS/LightRAG) which is knowledge graph based RAG implementation. Compared to GraphRAG, LightRAG is more accurate for single topic datasets, and uses 1,000x less tokens for retrieval. Furthermore, documents can be added to LightRAG incrementally without rebuilding the knowledge graph. See the full results in the [paper](https://arxiv.org/abs/2410.05779).\n", "\n", "For the agent itself, we use LangGraph.\n", "\n", "### Outline\n", "0. [Setup](#setup)\n", "1. [Data Preparation + RAG](#data-preparation--rag)\n", "2. [Agent Creation](#agent-creation)\n", "3. [Final App](#final-app)" ] }, { "cell_type": "markdown", "metadata": { "id": "xyLfhm3lS9IK", "vscode": { "languageId": "plaintext" } }, "source": [ "### Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "XL-vDTEVS9IK" }, "source": [ "#### Environment\n", "\n", "This tutorial uses openai embedding models. While there are alternatives, we recommend using the openai models for the best results." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "h7U51nLOS9IL" }, "outputs": [], "source": [ "!uv venv\n", "!uv pip install lightrag-hku aioboto3 numpy ollama openai tenacity aioboto3 transformers torch tiktoken nano_vectordb neo4j oracledb textract setuptools pyvis sqlalchemy langchain_community langchain_openai sqlmodel ipywidgets==7.7.1 asyncio markdown langgraph -q" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "yWViZHIaktSX" }, "outputs": [], "source": [ "import os\n", "\n", "os.environ['OPENAI_API_KEY'] = ..." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "n0xrObVQS9IL" }, "outputs": [], "source": [ "import ipywidgets as widgets\n", "\n", "import random\n", "from typing_extensions import TypedDict\n", "\n", "from langgraph.graph import StateGraph, END\n", "from langchain_core.runnables.graph import MermaidDrawMethod\n", "\n", "from IPython.display import display, Image as IPImage\n", "\n", "\n", "from io import BytesIO\n", "from PIL import Image as PILImage\n", "\n", "import markdown\n", "\n", "import requests\n", "from PIL import Image as PILImage\n", "from ipywidgets import widgets\n", "import markdown\n", "import json" ] }, { "cell_type": "markdown", "metadata": { "id": "9GsEmGJZS9IL", "vscode": { "languageId": "plaintext" } }, "source": [ "### Data Preparation + RAG" ] }, { "cell_type": "markdown", "metadata": { "id": "DGtEqWCiS9IL" }, "source": [ "#### Data Preparation\n", "\n", "We scraped data from [wikiart](https://wikiart.org/) using a combination of requests and beautiful soup.\n", "\n", "We provide a preprocessed, limited dataset of this data for the tutorial found in `data/artinfo.json`.\n", "\n", "Preprocessed data for each artwork looks like:\n", "\n", "```plaintext\n", "Artwork Name: {artwork_name}\n", "Artwork Genre: {genre_list}\n", "Artwork Media: {media_list}\n", "Wikiart URL: https://www.wikiart.org{wikiart_slug}\n", "\n", "Wikipedia Article:\n", "{wiki_article}\n", "```\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "CMUufJG7S9IM", "outputId": "ada7e745-ad71-4d57-fe1d-fe50a37bd318" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "78\n", "\n", "Artist: Frederick McCubbin\n", "Artwork Name: Down on his luck\n", "Artwork Genre: genre painting\n", "Artwork Media: canvas,oil\n", "Wikiart URL: https://www.wikiart.org/en/frederick-mccubbin/down-on-his-luck-1889\n", "\n", "Wikipedia Article:\n", "Down on His Luck is an 1889 painting by the Australian artist Frederick McCubbin. It depicts a seemingly disheartened swagman, sitting by a campfire sadly brooding over his misfortune. According to an 1889 review, \"The face tells of hardships, keen and blighting in their influence, but there is a nonchalant and slightly cynical expression, which proclaims the absence of all self-pity ... McCubbin's picture is thoroughly Australian in spirit.\" The surrounding bush is painted in subdued tones, reflecting his somber and contemplative mood.\r The artist's model was Louis Abrahams, a friend and successful tobaccanist in Melbourne who earlier supplied the cigar box lids for the famous 9 by 5 Impression Exhibition. The scene was staged near the Box Hill artists' camp outside Melbourne, but it is thought that McCubbin would have made additional studies of Abrahams under studio conditions.\r The painting was owned by William Fergusson up to 1986 until it was purchased by the Art Gallery of Western Australia in Perth.\n", "\n" ] } ], "source": [ "artwork_info = None\n", "with open('./data/artinfo.json', 'r') as f:\n", " artwork_info = json.load(f)['info']\n", "print(len(artwork_info))\n", "print(artwork_info[0])" ] }, { "cell_type": "markdown", "metadata": { "id": "ewBkhNkES9IM" }, "source": [ "#### LightRAG\n", "\n", "RAG systems enhance LLMs by integrating external knowledge sources. Traditional RAG solutions assume a 'flat' data representation, which can fail to capture complex dependencies. \n", "We use a hierarchical representation a knowledge graph to model our inherently hierarchical artwork data. \n", "\n", "LightRAG incorporates graph structures into indexing and retrieval. It creates a knowledge graph encoding 'entities' and their relationships, using techniques such as entity recognition, relation extration, and graph embeding models. It's dual-level retrieval system is perfect for our Artwork dataset. Fine grained details like artist names, artwork titles, and their references are retrieved via low-level concepts, and thematic information is retrieved via high-level concepts.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "t3HqKroWS9IM" }, "outputs": [], "source": [ "from lightrag.llm import gpt_4o_mini_complete\n", "from lightrag import LightRAG, QueryParam\n", "\n", "import nest_asyncio\n", "\n", "nest_asyncio.apply()\n", "WORKING_DIR = \"./data\"\n", "rag = LightRAG(\n", " working_dir=WORKING_DIR,\n", " llm_model_func=gpt_4o_mini_complete,\n", ")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "J4lNok7wS9IM" }, "outputs": [], "source": [ "# Un-comment to index the data\n", "# import time\n", "\n", "# # Create the RAG database\n", "# batch_size = 30\n", "\n", "# # Loop through the list in batches of 30 to avoid rate limits\n", "# for i in range(0, len(artwork_info), 30):\n", "# batch = artwork_info[i : i + batch_size]\n", "# rag.insert(batch)\n", "# time.sleep(15)" ] }, { "cell_type": "markdown", "metadata": { "id": "IjnHwmeNS9IN", "vscode": { "languageId": "plaintext" } }, "source": [ "### Agent Chain Creation\n", "\n", "Here we create a chain of functions to aid with retrieval of artwork information.\n", "Some of the following functions are helper functions, while others are core to the chain's functionality." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2y-kxszDS9IN" }, "outputs": [], "source": [ "# Helper Functions\n", "\n", "\n", "def get_artwork_image_url(url: str):\n", " \"\"\"\n", " Gets an artwork image from its URL.\n", " \"\"\"\n", " import requests\n", " from bs4 import BeautifulSoup as bs\n", "\n", " response = requests.get(url)\n", " soup = bs(response.text, \"html.parser\")\n", " img = soup.find(\"img\", {\"itemprop\": \"image\"})\n", "\n", " return img.get(\"src\")\n", "\n", "\n", "def image_widget_value(artwork_url: str):\n", " image_url = get_artwork_image_url(artwork_url)\n", " resp = requests.get(image_url)\n", " if resp.status_code == 200:\n", " image = PILImage.open(BytesIO(resp.content))\n", " width, height = image.size\n", " else:\n", " return 'No image found'\n", "\n", " return f''\n", "\n", "\n", "def html_widget_value(description: str):\n", " style = (\n", " \"display: inline-block; \"\n", " \"width: fit-content; \"\n", " \"max-width: 100%; \" # Optional to prevent excessive width\n", " \"padding: 5px; \"\n", " \"font-size: 16px; \"\n", " \"line-height: 1.5; \"\n", " \"border: 1px solid #ccc; \" # Optional, for visibility\n", " \"border-radius: 5px; \"\n", " \"background-color: #f9f9f9;\"\n", " )\n", " return f'
{markdown.markdown(description)}
'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### State Management\n", "In order to keep track of the state of the art tour, we create an object that contains the current artwork, its data, past artworks, a history of this tour's discussion, and a request for a specific artwork or category:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "YTlSssROS9IN" }, "outputs": [], "source": [ "class ArtTourState(TypedDict):\n", " artwork: str\n", " artwork_metadata: str\n", " artwork_description: str\n", " artwork_url: str\n", " art_request: str\n", " past_artworks: list[str]\n", " discussion_messages: list[str]\n", " app_message: str # intro and conclusion\n", " exit: bool\n", "\n", " footer: widgets.HTML\n", " header: widgets.HTML\n", " image: widgets.HTML" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Chain Node Functions\n", "We define helper functions consuming state that modify the widgets, and call upon the RAG database to retrieve information.\n", "Some of these functions include:\n", "- getters for current artwork data\n", "- random artwork retrieval\n", "- display artwork" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "PmArwrQQS9IN" }, "outputs": [], "source": [ "def get_artwork_url_from_name(state: ArtTourState):\n", " artwork_name = state['artwork']\n", " prompt = f\"\"\"\n", " What is the Wikiart URL for this artwork: {artwork_name}\n", "\n", " Return only the wikiart url in plain text. Do not include any other text. URL only.\n", " \"\"\"\n", " return {\"artwork_url\": rag.query(prompt, param=QueryParam(mode=\"hybrid\"))}" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "5dG5JKMgS9IN" }, "outputs": [], "source": [ "def describe_artwork(state: ArtTourState):\n", " artwork = state['artwork']\n", " metadata = rag.query(\n", " f\"\"\"What is the genre, media, and style of the following artwork: {artwork}.\n", " Return the description in the following format:\n", " artwork name: \n", " artist: \n", " genre: \n", " media: \n", " style: