{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Analysis Simple Agent\n", "\n", "## Overview\n", "This tutorial guides you through creating an AI-powered data analysis agent that can interpret and answer questions about a dataset using natural language. It combines language models with data manipulation tools to enable intuitive data exploration.\n", "\n", "## Motivation\n", "Data analysis often requires specialized knowledge, limiting access to insights for non-technical users. By creating an AI agent that understands natural language queries, we can democratize data analysis, allowing anyone to extract valuable information from complex datasets without needing to know programming or statistical tools.\n", "\n", "## Key Components\n", "1. Language Model: Processes natural language queries and generates human-like responses\n", "2. Data Manipulation Framework: Handles dataset operations and analysis\n", "3. Agent Framework: Connects the language model with data manipulation tools\n", "4. Synthetic Dataset: Represents real-world data for demonstration purposes\n", "\n", "## Method\n", "1. Create a synthetic dataset representing car sales data\n", "2. Construct an agent that combines the language model with data analysis capabilities\n", "3. Implement a query processing function to handle natural language questions\n", "4. Demonstrate the agent's abilities with example queries\n", "\n", "## Conclusion\n", "This approach to data analysis offers significant benefits:\n", "- Accessibility for non-technical users\n", "- Flexibility in handling various query types\n", "- Efficient ad-hoc data exploration\n", "\n", "By making data insights more accessible, this method has the potential to transform how organizations leverage their data for decision-making across various fields and industries." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Import libraries and set environment variables" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "from langchain_experimental.agents.agent_toolkits import create_pandas_dataframe_agent\n", "from langchain.agents import AgentType\n", "from langchain_openai import ChatOpenAI\n", "import pandas as pd\n", "import numpy as np\n", "from datetime import datetime, timedelta\n", "\n", "# Load environment variables\n", "from dotenv import load_dotenv\n", "import os\n", "\n", "# Load environment variables and set OpenAI API key\n", "load_dotenv()\n", "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')\n", "\n", "# Set a random seed for reproducibility\n", "np.random.seed(42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate Sample Data\n", "\n", "In this section, we create a sample dataset of car sales. This includes generating dates, car makes, models, colors, and other relevant information." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "First few rows of the generated data:\n", " Date Make Model Color Year Price Mileage \\\n", "0 2022-01-01 Mercedes Sedan Green 2022 57952.65 5522.0 \n", "1 2022-01-02 Chevrolet Hatchback Red 2021 58668.22 94238.0 \n", "2 2022-01-03 Audi Truck White 2019 69187.87 7482.0 \n", "3 2022-01-04 Nissan Hatchback Black 2016 40004.44 43846.0 \n", "4 2022-01-05 Mercedes Hatchback Red 2016 63983.07 52988.0 \n", "\n", " EngineSize FuelEfficiency SalesPerson \n", "0 2.0 24.7 Alice \n", "1 1.6 26.2 Bob \n", "2 2.0 28.0 David \n", "3 3.5 24.8 David \n", "4 2.5 24.1 Alice \n", "\n", "DataFrame info:\n", "\n", "RangeIndex: 1000 entries, 0 to 999\n", "Data columns (total 10 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 Date 1000 non-null datetime64[ns]\n", " 1 Make 1000 non-null object \n", " 2 Model 1000 non-null object \n", " 3 Color 1000 non-null object \n", " 4 Year 1000 non-null int32 \n", " 5 Price 1000 non-null float64 \n", " 6 Mileage 1000 non-null float64 \n", " 7 EngineSize 1000 non-null float64 \n", " 8 FuelEfficiency 1000 non-null float64 \n", " 9 SalesPerson 1000 non-null object \n", "dtypes: datetime64[ns](1), float64(4), int32(1), object(4)\n", "memory usage: 74.3+ KB\n", "\n", "Summary statistics:\n", " Date Year Price Mileage \\\n", "count 1000 1000.000000 1000.000000 1000.000000 \n", "mean 2023-05-15 12:00:00 2018.445000 51145.360800 48484.643000 \n", "min 2022-01-01 00:00:00 2015.000000 20026.570000 19.000000 \n", "25% 2022-09-07 18:00:00 2017.000000 36859.940000 23191.500000 \n", "50% 2023-05-15 12:00:00 2018.000000 52215.155000 47506.000000 \n", "75% 2024-01-20 06:00:00 2020.000000 65741.147500 73880.250000 \n", "max 2024-09-26 00:00:00 2022.000000 79972.640000 99762.000000 \n", "std NaN 2.256117 17041.610861 29103.404593 \n", "\n", " EngineSize FuelEfficiency \n", "count 1000.000000 1000.000000 \n", "mean 2.744500 29.688500 \n", "min 1.600000 20.000000 \n", "25% 2.000000 24.500000 \n", "50% 2.500000 29.700000 \n", "75% 3.500000 34.700000 \n", "max 4.000000 40.000000 \n", "std 0.839389 5.896316 \n" ] } ], "source": [ "# Generate sample data\n", "n_rows = 1000\n", "\n", "# Generate dates\n", "start_date = datetime(2022, 1, 1)\n", "dates = [start_date + timedelta(days=i) for i in range(n_rows)]\n", "\n", "# Define data categories\n", "makes = ['Toyota', 'Honda', 'Ford', 'Chevrolet', 'Nissan', 'BMW', 'Mercedes', 'Audi', 'Hyundai', 'Kia']\n", "models = ['Sedan', 'SUV', 'Truck', 'Hatchback', 'Coupe', 'Van']\n", "colors = ['Red', 'Blue', 'Black', 'White', 'Silver', 'Gray', 'Green']\n", "\n", "# Create the dataset\n", "data = {\n", " 'Date': dates,\n", " 'Make': np.random.choice(makes, n_rows),\n", " 'Model': np.random.choice(models, n_rows),\n", " 'Color': np.random.choice(colors, n_rows),\n", " 'Year': np.random.randint(2015, 2023, n_rows),\n", " 'Price': np.random.uniform(20000, 80000, n_rows).round(2),\n", " 'Mileage': np.random.uniform(0, 100000, n_rows).round(0),\n", " 'EngineSize': np.random.choice([1.6, 2.0, 2.5, 3.0, 3.5, 4.0], n_rows),\n", " 'FuelEfficiency': np.random.uniform(20, 40, n_rows).round(1),\n", " 'SalesPerson': np.random.choice(['Alice', 'Bob', 'Charlie', 'David', 'Eva'], n_rows)\n", "}\n", "\n", "# Create DataFrame and sort by date\n", "df = pd.DataFrame(data).sort_values('Date')\n", "\n", "# Display sample data and statistics\n", "print(\"\\nFirst few rows of the generated data:\")\n", "print(df.head())\n", "\n", "print(\"\\nDataFrame info:\")\n", "df.info()\n", "\n", "print(\"\\nSummary statistics:\")\n", "print(df.describe())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create Data Analysis Agent\n", "\n", "Here, we create a Pandas DataFrame agent using LangChain. This agent will be capable of analyzing our dataset and answering questions about it." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data Analysis Agent is ready. You can now ask questions about the data.\n" ] } ], "source": [ "# Create the Pandas DataFrame agent\n", "agent = create_pandas_dataframe_agent(\n", " ChatOpenAI(model=\"gpt-4o\", temperature=0),\n", " df,\n", " verbose=True,\n", " allow_dangerous_code=True,\n", " agent_type=AgentType.OPENAI_FUNCTIONS,\n", ")\n", "print(\"Data Analysis Agent is ready. You can now ask questions about the data.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Define Question-Asking Function\n", "\n", "This function allows us to easily ask questions to our data analysis agent and display the results." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def ask_agent(question):\n", " \"\"\"Function to ask questions to the agent and display the response\"\"\"\n", " response = agent.invoke({\n", " \"input\": question,\n", " \"agent_scratchpad\": f\"Human: {question}\\nAI: To answer this question, I need to use Python to analyze the dataframe. I'll use the python_repl_ast tool.\\n\\nAction: python_repl_ast\\nAction Input: \",\n", " })\n", " print(f\"Question: {question}\")\n", " print(f\"Answer: {response}\")\n", " print(\"---\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example Questions\n", "\n", "Here are some example questions you can ask the data analysis agent. You can modify these or add your own questions to analyze the dataset." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\N7\\AppData\\Local\\Temp\\ipykernel_16872\\610968568.py:3: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use invoke instead.\n", " response = agent.invoke({\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", "\u001b[32;1m\u001b[1;3m\n", "Invoking: `python_repl_ast` with `{'query': 'df.columns.tolist()'}`\n", "\n", "\n", "\u001b[0m\u001b[36;1m\u001b[1;3m['Date', 'Make', 'Model', 'Color', 'Year', 'Price', 'Mileage', 'EngineSize', 'FuelEfficiency', 'SalesPerson']\u001b[0m\u001b[32;1m\u001b[1;3mThe column names in the dataset are:\n", "1. Date\n", "2. Make\n", "3. Model\n", "4. Color\n", "5. Year\n", "6. Price\n", "7. Mileage\n", "8. EngineSize\n", "9. FuelEfficiency\n", "10. SalesPerson\u001b[0m\n", "\n", "\u001b[1m> Finished chain.\u001b[0m\n", "Question: What are the column names in this dataset?\n", "Answer: The column names in the dataset are:\n", "1. Date\n", "2. Make\n", "3. Model\n", "4. Color\n", "5. Year\n", "6. Price\n", "7. Mileage\n", "8. EngineSize\n", "9. FuelEfficiency\n", "10. SalesPerson\n", "---\n", "\n", "\n", "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", "\u001b[32;1m\u001b[1;3m\n", "Invoking: `python_repl_ast` with `{'query': 'df.shape[0]'}`\n", "\n", "\n", "\u001b[0m\u001b[36;1m\u001b[1;3m1000\u001b[0m\u001b[32;1m\u001b[1;3mThe dataset contains 1000 rows.\u001b[0m\n", "\n", "\u001b[1m> Finished chain.\u001b[0m\n", "Question: How many rows are in this dataset?\n", "Answer: The dataset contains 1000 rows.\n", "---\n", "\n", "\n", "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", "\u001b[32;1m\u001b[1;3m\n", "Invoking: `python_repl_ast` with `{'query': \"df['Price'].mean()\"}`\n", "\n", "\n", "\u001b[0m\u001b[36;1m\u001b[1;3m51145.360799999995\u001b[0m\u001b[32;1m\u001b[1;3mThe average price of cars sold is approximately $51,145.36.\u001b[0m\n", "\n", "\u001b[1m> Finished chain.\u001b[0m\n", "Question: What is the average price of cars sold?\n", "Answer: The average price of cars sold is approximately $51,145.36.\n", "---\n" ] } ], "source": [ "# Example questions\n", "ask_agent(\"What are the column names in this dataset?\")\n", "ask_agent(\"How many rows are in this dataset?\")\n", "ask_agent(\"What is the average price of cars sold?\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](https://europe-west1-genai-agents-views-tracker.cloudfunctions.net/genai-agents-tracker?notebook=all-agents-tutorials--simple-data-analysis-agent-notebook)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.0" } }, "nbformat": 4, "nbformat_minor": 4 }