{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "7b9f55ae-6bf5-4e91-bd30-64e07ea85450", "metadata": { "id": "7b9f55ae-6bf5-4e91-bd30-64e07ea85450" }, "outputs": [], "source": [ "# Copyright 2023 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "id": "fe656833-3d7c-4fde-87f5-c7c7ad21a106", "metadata": { "id": "fe656833-3d7c-4fde-87f5-c7c7ad21a106" }, "source": [ "# Sentiment Analysis for large scale data using Gemini" ] }, { "cell_type": "markdown", "id": "e6fd7856-e138-4831-801e-6318f856efe3", "metadata": { "id": "e6fd7856-e138-4831-801e-6318f856efe3" }, "source": [ "## Overview" ] }, { "cell_type": "markdown", "id": "e854e6c7-14f3-4b8a-933e-9091b24140af", "metadata": { "id": "e854e6c7-14f3-4b8a-933e-9091b24140af", "tags": [] }, "source": [ "This notebook shows how to perform sentimental analysis on large scale data using LLM.\n", "The dataset used is a public dataset from Bigquery Public Datasets.\n", "\n", "#### **Steps**\n", "Using Spark,\n", "1) This notebook reads data from Bigquery public dataset **bigquery-public-data.imdb.reviews**\n", "2) It calls [Vertex AI Gemini API](https://cloud.google.com/vertex-ai/docs/generative-ai/start/quickstarts/api-quickstart#try_text_prompts) to find the sentiment of each review (positive vs negative)\n", "3) We compare the results in terms of classification metrics (Accuracy, Precision, Recall, F1)\n", "\n", "#### Related content\n", "\n", "- [Gemini API](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstart)" ] }, { "cell_type": "markdown", "id": "itW06rmBYtxW", "metadata": { "id": "itW06rmBYtxW" }, "source": [ "#### Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "9eb15ebe-c6aa-4f92-92d3-6fa8ec5dc5b3", "metadata": { "id": "9eb15ebe-c6aa-4f92-92d3-6fa8ec5dc5b3" }, "outputs": [], "source": [ "!pip3 install --upgrade -q google-auth dataproc-spark-connect google-cloud-dataproc google-cloud-aiplatform google-genai \"protobuf~=4.25.3\" \"numpy~=1.26.4\"" ] }, { "cell_type": "code", "execution_count": null, "id": "95c47182-9bc3-41e5-ab3c-f10f01727b99", "metadata": { "executionInfo": { "elapsed": 12604, "status": "ok", "timestamp": 1758038539999, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "95c47182-9bc3-41e5-ab3c-f10f01727b99" }, "outputs": [], "source": [ "import sys\n", "\n", "from google.cloud.dataproc_v1 import Session\n", "from google.cloud.dataproc_spark_connect import DataprocSparkSession\n", "\n", "from pyspark.sql.functions import *" ] }, { "cell_type": "code", "execution_count": null, "id": "867cff94", "metadata": { "id": "867cff94" }, "outputs": [], "source": [ "# To use the newly installed packages, you must restart the runtime on Google Colab (Colab only)\n", "\n", "if \"google.colab\" in sys.modules:\n", " import IPython\n", " app = IPython.Application.instance()\n", " app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", "id": "50ebd0b3-c4b5-4df5-8d11-0f721a147467", "metadata": { "id": "50ebd0b3-c4b5-4df5-8d11-0f721a147467" }, "source": [ "#### Authenticate with Google APIs\n" ] }, { "cell_type": "code", "execution_count": null, "id": "_WagTXKcZqYt", "metadata": { "id": "_WagTXKcZqYt" }, "outputs": [], "source": [ "PROJECT_ID = \"PROJECT_ID\" # @param {type:\"string\"}\n", "\n", "# Set the project id\n", "! gcloud config set project {PROJECT_ID}" ] }, { "cell_type": "code", "execution_count": null, "id": "464b4302", "metadata": { "id": "464b4302" }, "outputs": [], "source": [ "# Authenticate your notebook environment (Colab only)\n", "if \"google.colab\" in sys.modules:\n", " from google.colab import auth\n", " auth.authenticate_user(project_id=PROJECT_ID)" ] }, { "cell_type": "markdown", "id": "483fa597-0127-4077-b1f3-4bcb9ff11c65", "metadata": { "id": "483fa597-0127-4077-b1f3-4bcb9ff11c65" }, "source": [ "### Create Spark Session for the notebook" ] }, { "cell_type": "code", "execution_count": null, "id": "-l_CcviOAkdw", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 174 }, "executionInfo": { "elapsed": 100308, "status": "ok", "timestamp": 1758038663442, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "-l_CcviOAkdw", "outputId": "c7202cf7-2722-49fa-ab29-25d6220ed8a2" }, "outputs": [], "source": [ "from google.cloud.dataproc_v1.types import RuntimeConfig\n", "runtime_config = RuntimeConfig(version=\"2.3\") # Python 3.11\n", "session = Session(runtime_config=runtime_config)\n", "\n", "# Create the Spark session.\n", "spark = (\n", " DataprocSparkSession.builder\n", " .appName(\"Sentiment Analysis using Dataproc and Gemini on Vertex AI\")\n", " .dataprocSessionConfig(session)\n", " .getOrCreate()\n", ")" ] }, { "cell_type": "markdown", "id": "97b310e5-ede5-4ac5-a5dd-df0732f61c8e", "metadata": { "id": "97b310e5-ede5-4ac5-a5dd-df0732f61c8e" }, "source": [ "### Read data from Bigquery Public Dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "02d7cad8-6ef6-44bb-baa9-78d5f10dc288", "metadata": { "executionInfo": { "elapsed": 104, "status": "ok", "timestamp": 1758038715422, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "02d7cad8-6ef6-44bb-baa9-78d5f10dc288" }, "outputs": [], "source": [ "movie_reviews = spark.read.format(\"bigquery\").option(\"table\", \"bigquery-public-data.imdb.reviews\").load()" ] }, { "cell_type": "code", "execution_count": null, "id": "bhuWWhsbBnvc", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 498 }, "executionInfo": { "elapsed": 18152, "status": "ok", "timestamp": 1758038734074, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "bhuWWhsbBnvc", "outputId": "5084b6dc-5b38-40b7-c64f-47d65f29db1e" }, "outputs": [], "source": [ "movie_reviews.show()" ] }, { "cell_type": "markdown", "id": "b76336bd-14a3-4d1f-a8b5-0e500eba39ef", "metadata": { "id": "b76336bd-14a3-4d1f-a8b5-0e500eba39ef" }, "source": [ "| review|split| label| movie_id|reviewer_rating| movie_url|title|\n", "|----------------------------------------------------------------------------------------------------|-----|--------|---------|---------------|------------------------------------|-----|\n", "|I had to see this on the British Airways plane. It was terribly bad acting and a dumb story. Not ...| test|Negative|tt0158887| 2|http://www.imdb.com/title/tt0158887/| null|\n", "|This is a family movie that was broadcast on my local ITV station at 1.00 am a couple of nights a...| test|Negative|tt0158887| 4|http://www.imdb.com/title/tt0158887/| null|\n", "|I would like to comment on how the girls are chosen. why is that their are always more white wome...| test|Negative|tt0391576| 2|http://www.imdb.com/title/tt0391576/| null|\n", "|Tyra & the rest of the modeling world needs to know that real women like myself and my daughter d...| test|Negative|tt0391576| 3|http://www.imdb.com/title/tt0391576/| null|" ] }, { "cell_type": "markdown", "id": "8efeb371-c8ae-4a20-b5e6-a21c5723e737", "metadata": { "id": "8efeb371-c8ae-4a20-b5e6-a21c5723e737" }, "source": [ "### Get Positive Reviews from Dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "523c5cf6-5165-4ca6-b2b4-447bb15762b4", "metadata": { "executionInfo": { "elapsed": 39, "status": "ok", "timestamp": 1758038795697, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "523c5cf6-5165-4ca6-b2b4-447bb15762b4" }, "outputs": [], "source": [ "positive_movie_reviews = movie_reviews.select(col(\"review\"), col(\"reviewer_rating\"), col(\"movie_id\"), col(\"label\")).where(col(\"label\") == \"Positive\").limit(100)" ] }, { "cell_type": "markdown", "id": "d5846a0c-1ca6-483f-b9f1-52873bd50312", "metadata": { "id": "d5846a0c-1ca6-483f-b9f1-52873bd50312" }, "source": [ "### Get Negative Reviews from Dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "ec0a1557-cf01-4f86-9785-47ecb06ba752", "metadata": { "executionInfo": { "elapsed": 36, "status": "ok", "timestamp": 1758038796708, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "ec0a1557-cf01-4f86-9785-47ecb06ba752" }, "outputs": [], "source": [ "negative_movie_reviews = movie_reviews.select(col(\"review\"), col(\"reviewer_rating\"), col(\"movie_id\"), col(\"label\")).where(col(\"label\") == \"Negative\").limit(100)" ] }, { "cell_type": "markdown", "id": "14843f6a-8f88-49b8-b2a0-db930d3a5c37", "metadata": { "id": "14843f6a-8f88-49b8-b2a0-db930d3a5c37" }, "source": [ "### Mix positive and negative\n", "Making union of positive and negative reviews to get a good dataset of mixed set of reviews. For the purpose notebook, each class of reviews has 100 rows each." ] }, { "cell_type": "code", "execution_count": null, "id": "eda456fa-b95a-4e95-835c-cbc64be1fe11", "metadata": { "executionInfo": { "elapsed": 32, "status": "ok", "timestamp": 1758038797480, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "eda456fa-b95a-4e95-835c-cbc64be1fe11" }, "outputs": [], "source": [ "movie_reviews_mixed = positive_movie_reviews.union(negative_movie_reviews)" ] }, { "cell_type": "markdown", "id": "d39af261-8416-4b2f-b830-f90a7379f859", "metadata": { "id": "d39af261-8416-4b2f-b830-f90a7379f859" }, "source": [ "| review|reviewer_rating| movie_id| label|\n", "|--------------------|---------------|---------|--------|\n", "|This movie is ama...| 10|tt0187123|Positive|\n", "|THE HAND OF DEATH...| 10|tt0187123|Positive|\n", "|The Hand of Death...| 7|tt0187123|Positive|\n", "|Just as a reminde...| 10|tt0163955|Positive|\n", "|Like an earlier c...| 9|tt0163955|Positive|" ] }, { "cell_type": "markdown", "id": "558c6709-e265-4e21-a15b-4e5980303686", "metadata": { "id": "558c6709-e265-4e21-a15b-4e5980303686" }, "source": [ "### Final count is 200 as can be seen below" ] }, { "cell_type": "code", "execution_count": null, "id": "a4bfebe8-6491-4cc4-b8b5-d6b408e67196", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 64 }, "executionInfo": { "elapsed": 9229, "status": "ok", "timestamp": 1758038809212, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "a4bfebe8-6491-4cc4-b8b5-d6b408e67196", "outputId": "674c0e69-8260-4e3a-c3d7-634f1ac490dc" }, "outputs": [], "source": [ "movie_reviews_mixed.count()" ] }, { "cell_type": "markdown", "id": "2951e16c-5dd3-49b7-abf1-0fbb930453a2", "metadata": { "id": "2951e16c-5dd3-49b7-abf1-0fbb930453a2" }, "source": [ "### Creating a UDF to get predictions from Gemini Model\n", "In this method, text whose sentiment is to be predicted is passed" ] }, { "cell_type": "code", "execution_count": null, "id": "5288e2ee-af06-47ee-a6ce-977852bb700e", "metadata": { "executionInfo": { "elapsed": 55, "status": "ok", "timestamp": 1758038811663, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "5288e2ee-af06-47ee-a6ce-977852bb700e" }, "outputs": [], "source": [ "def gemini_predict(prompt, model_name=\"gemini-2.5-flash\", max_retries=3, initial_delay=1):\n", "\n", " import time\n", " import enum\n", " from google import genai\n", " from google.genai import types\n", "\n", " client = genai.Client(\n", " vertexai=True,\n", " project=PROJECT_ID,\n", " location=\"us-central1\"\n", " )\n", "\n", " class ResponseSchema(enum.Enum):\n", " POSITIVE = \"Positive\"\n", " NEGATIVE = \"Negative\"\n", "\n", " generate_content_config = types.GenerateContentConfig(\n", " response_mime_type = \"text/x.enum\",\n", " response_schema = ResponseSchema\n", " )\n", "\n", " retries, delay = 0, initial_delay\n", " while retries <= max_retries:\n", " try:\n", " response = client.models.generate_content(model=model_name,\n", " contents=prompt,\n", " config=generate_content_config)\n", "\n", " return response.text\n", " except Exception:\n", " if retries == max_retries:\n", " return\n", " time.sleep(delay)\n", " delay *= 2\n", " retries += 1\n", " return \"\"" ] }, { "cell_type": "code", "execution_count": null, "id": "385d6430-6415-4e16-97c3-b51cd1527395", "metadata": { "executionInfo": { "elapsed": 169, "status": "ok", "timestamp": 1758038814003, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "385d6430-6415-4e16-97c3-b51cd1527395" }, "outputs": [], "source": [ "def classify_sentiment(text):\n", "\n", " prompt = f\"\"\"You are an expert at analyzing movie reviews from IMDb. Your task is really simple: to classify the sentiment of the provided review text between Positive and Negative.\n", " When classifying, pay close attention to:\n", " - **Overall sentiment**: Consider the entire review, not just individual words. Criticism is not always negative.\n", " - **Sarcasm and irony**: Identify when negative language is used to express positive sentiment, or vice-versa.\n", " - **Conditional statements**: Understand if the sentiment is dependent on certain conditions.\n", " - **Comparative language**: Determine if the review is comparing the current movie favorably or unfavorably to others.\n", "\n", " Provide the sentiment classification from one of the two classes:\n", " - Negative\n", " - Positive\n", "\n", " Always choose the most appropriate classification.\n", "\n", " Text: {text}\n", " Sentiment:\"\"\"\n", "\n", " sentiment = gemini_predict(prompt)\n", " return sentiment\n", "\n", "classify_sentiment_udf = udf(classify_sentiment)" ] }, { "cell_type": "markdown", "id": "3642ecc0-2734-4027-b9d3-5ed32ff6d867", "metadata": { "id": "3642ecc0-2734-4027-b9d3-5ed32ff6d867" }, "source": [ "### Get prediction from Gemini using the UDF on the movie reviews" ] }, { "cell_type": "code", "execution_count": null, "id": "b7239a8d-f60a-40f5-95ee-a18b9b5d3a6c", "metadata": { "executionInfo": { "elapsed": 37, "status": "ok", "timestamp": 1758038816439, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "b7239a8d-f60a-40f5-95ee-a18b9b5d3a6c" }, "outputs": [], "source": [ "movie_review_sentiment_pred = movie_reviews_mixed.withColumn(\"pred\", classify_sentiment_udf(movie_reviews_mixed[\"review\"]))" ] }, { "cell_type": "markdown", "id": "92c545e1-8d4f-47b7-a595-c03e01efbacf", "metadata": { "id": "92c545e1-8d4f-47b7-a595-c03e01efbacf" }, "source": [ "### Let's check the predicted value and do a quick comparison of required output v/s actual label" ] }, { "cell_type": "code", "execution_count": null, "id": "2e178b7f-f747-4093-a69e-50019ea3529e", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "executionInfo": { "elapsed": 79521, "status": "ok", "timestamp": 1758038896790, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "2e178b7f-f747-4093-a69e-50019ea3529e", "outputId": "7c3c8947-2ad1-4a50-913b-6bca3beb21e6" }, "outputs": [], "source": [ "movie_review_sentiment_pred.select(col(\"pred\"), col(\"label\")).show(50,50)" ] }, { "cell_type": "code", "execution_count": null, "id": "5cc405d9-a775-40c9-93e1-3a79a5251837", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 42289, "status": "ok", "timestamp": 1758038939092, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "5cc405d9-a775-40c9-93e1-3a79a5251837", "outputId": "06a4863f-1296-433b-e4c8-e13b69cb2b28" }, "outputs": [], "source": [ "movie_review_sentiment_pred.cache()" ] }, { "cell_type": "markdown", "id": "7fa62d52-795b-4c27-b3c0-cb725a93d473", "metadata": { "id": "7fa62d52-795b-4c27-b3c0-cb725a93d473" }, "source": [ "### Evaluation\n", "\n", "Let's index the classes Negative and Positive to 1 and 0. " ] }, { "cell_type": "code", "execution_count": null, "id": "11aaf2c1-e64c-459e-9439-2f9ac18e2d4c", "metadata": { "executionInfo": { "elapsed": 53, "status": "ok", "timestamp": 1758039740988, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "11aaf2c1-e64c-459e-9439-2f9ac18e2d4c" }, "outputs": [], "source": [ "from pyspark.sql.functions import when\n", "\n", "indexed_df = movie_review_sentiment_pred.withColumn(\"label_indexed\",when(trim(movie_review_sentiment_pred[\"label\"]) == \"Positive\", 0.0).when(trim(movie_review_sentiment_pred[\"label\"]) == \"Negative\", 1.0)) \\\n", " .withColumn(\"pred_indexed\",when(trim(movie_review_sentiment_pred[\"pred\"]) == \"Positive\", 0.0).when(trim(movie_review_sentiment_pred[\"pred\"]) == \"Negative\", 1.0))" ] }, { "cell_type": "code", "execution_count": null, "id": "a7JdCwjyI7fX", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 1607, "status": "ok", "timestamp": 1758039743694, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "a7JdCwjyI7fX", "outputId": "fbc18812-df92-418d-e5bc-7f55981b810a" }, "outputs": [], "source": [ "indexed_df.printSchema()" ] }, { "cell_type": "code", "execution_count": null, "id": "SfFctwNXCaFP", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 498 }, "executionInfo": { "elapsed": 24269, "status": "ok", "timestamp": 1758039767965, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "SfFctwNXCaFP", "outputId": "84b34ee6-6221-434c-e2d1-6c8892301e78" }, "outputs": [], "source": [ "indexed_df.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "G2wHnsP8T7n5", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 40127, "status": "ok", "timestamp": 1758040021035, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "G2wHnsP8T7n5", "outputId": "86b2742a-412a-405c-fd3b-c20db2605205" }, "outputs": [], "source": [ "indexed_df.cache()" ] }, { "cell_type": "markdown", "id": "3c19120e-d06f-4743-ad76-6d89c027f5af", "metadata": { "id": "3c19120e-d06f-4743-ad76-6d89c027f5af" }, "source": [ "The we calculate our evaluation metrics" ] }, { "cell_type": "code", "execution_count": null, "id": "7d0a765d-91f6-40b2-82d4-f44fcef2a179", "metadata": { "executionInfo": { "elapsed": 1, "status": "ok", "timestamp": 1758040030446, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "7d0a765d-91f6-40b2-82d4-f44fcef2a179" }, "outputs": [], "source": [ "def calculate_metrics(df, PRED_COL, LABEL_COL):\n", " \"\"\"\n", " Calculates classification metrics (Accuracy, Precision, Recall, F1)\n", " by computing the confusion matrix components in a single Spark aggregation pass.\n", " \"\"\"\n", " # 1. Filter out NULL/invalid predictions first\n", " valid_df = df.filter((col(PRED_COL).isNotNull()) & (col(LABEL_COL).isNotNull()))\n", " valid_total = valid_df.count()\n", "\n", " if valid_total == 0:\n", " print(\"Warning: No valid predictions found for metric calculation.\")\n", " return None\n", "\n", " # 2. Optimized: Calculate all confusion matrix components in ONE aggregation job\n", " # We define Negative (1.0) as the \"Positive\" class for TP/FN/FP/TN calculations\n", " results = valid_df.agg(\n", " # True Positive (TP): Predicted Negative (1.0) AND Actual Negative (1.0)\n", " sum(when((col(PRED_COL) == 1.0) & (col(LABEL_COL) == 1.0), 1).otherwise(0)).alias(\"tp\"),\n", "\n", " # True Negative (TN): Predicted Positive (0.0) AND Actual Positive (0.0)\n", " sum(when((col(PRED_COL) == 0.0) & (col(LABEL_COL) == 0.0), 1).otherwise(0)).alias(\"tn\"),\n", "\n", " # False Positive (FP): Predicted Negative (1.0) AND Actual Positive (0.0)\n", " sum(when((col(PRED_COL) == 1.0) & (col(LABEL_COL) == 0.0), 1).otherwise(0)).alias(\"fp\"),\n", "\n", " # False Negative (FN): Predicted Positive (0.0) AND Actual Negative (1.0)\n", " sum(when((col(PRED_COL) == 0.0) & (col(LABEL_COL) == 1.0), 1).otherwise(0)).alias(\"fn\"),\n", " ).collect()[0]\n", "\n", " tp = results[\"tp\"]\n", " tn = results[\"tn\"]\n", " fp = results[\"fp\"]\n", " fn = results[\"fn\"]\n", "\n", " # 3. Calculate metrics\n", " accuracy = (tp + tn) / valid_total\n", "\n", " # Precision is T_P / (T_P + F_P)\n", " precision = tp / (tp + fp) if (tp + fp) > 0 else 0\n", "\n", " # Recall (Sensitivity) is T_P / (T_P + F_N)\n", " recall = tp / (tp + fn) if (tp + fn) > 0 else 0\n", "\n", " # F1 Score\n", " f1_score = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0\n", "\n", " # Specificity is T_N / (T_N + F_P)\n", " specificity = tn / (tn + fp) if (tn + fp) > 0 else 0\n", "\n", " # Balanced Accuracy (approximation for AUC)\n", " balanced_accuracy = (recall + specificity) / 2\n", "\n", " # 4. Return results\n", " return {\n", " 'accuracy': accuracy,\n", " 'precision': precision,\n", " 'recall': recall,\n", " 'f1_score': f1_score,\n", " 'auc_roc_approx': balanced_accuracy,\n", " 'valid_predictions': valid_total,\n", " 'null_predictions': df.count() - valid_total,\n", " 'tp': tp, 'tn': tn, 'fp': fp, 'fn': fn\n", " }" ] }, { "cell_type": "code", "execution_count": null, "id": "Mqb2b9EGHP13", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 232 }, "executionInfo": { "elapsed": 201018, "status": "ok", "timestamp": 1758040251687, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "Mqb2b9EGHP13", "outputId": "c7636547-d76c-4a4b-e044-14c43384733d" }, "outputs": [], "source": [ "# Calculate metrics using PySpark\n", "metrics = calculate_metrics(indexed_df, \"pred_indexed\", \"label_indexed\")\n", "\n", "print(f\"Accuracy: {metrics['accuracy']:.4f}\")\n", "print(f\"Precision: {metrics['precision']:.4f}\")\n", "print(f\"Recall: {metrics['recall']:.4f}\")\n", "print(f\"F1-Score: {metrics['f1_score']:.4f}\")\n", "print(f\"AUC-ROC (approx): {metrics['auc_roc_approx']:.4f}\")\n", "print(f\"Valid predictions: {metrics['valid_predictions']:.4f}\")\n", "print(f\"Null predictions: {metrics['null_predictions']:.4f}\")\n", "print(f\"Confusion Matrix - TP: {metrics['tp']}, TN: {metrics['tn']}, FP: {metrics['fp']}, FN: {metrics['fn']}\")\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6dAKARgOJFVV", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 565 }, "executionInfo": { "elapsed": 218641, "status": "ok", "timestamp": 1758040487565, "user": { "displayName": "", "userId": "" }, "user_tz": 180 }, "id": "6dAKARgOJFVV", "outputId": "87b580fd-abf2-474c-a5b1-cb80f8b94b60" }, "outputs": [], "source": [ "# Mismatch analysis\n", "match_predictions_df = indexed_df.withColumn(\"if_match\", when((col(\"pred_indexed\")==col(\"label_indexed\")),1).otherwise(0))\n", "mismatch_count = match_predictions_df.where(col(\"if_match\")==0).count()\n", "print(f\"Number of mismatched predictions: {mismatch_count}\")\n", "\n", "mismatch_df = match_predictions_df.where(col(\"if_match\")==0).select(col('pred'),col('label'),col('review'))\n", "mismatch_df.show(20,200)" ] }, { "cell_type": "markdown", "id": "632afa55-edb1-4431-b78d-4765d527d838", "metadata": { "id": "632afa55-edb1-4431-b78d-4765d527d838" }, "source": [ "#### Check the mismatch predictions\n", "Find the mismatched rows and show them to analysis and iterate on the approach to classify sentiment. Sometimes the label is not so accurate in the dataset." ] }, { "cell_type": "markdown", "id": "0e9e6b13-ead2-4630-90b7-8cf07d31c9cb", "metadata": { "id": "0e9e6b13-ead2-4630-90b7-8cf07d31c9cb" }, "source": [ "| pred| label| review|\n", "|--------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n", "|Negative|Positive|Like one of the previous commenters said, this had the foundations of a great movie but something happened on the way to delivery. Such a waste because Collette's performance was eerie and Williams...|\n", "|Negative|Positive|If there is one thing to recommend about this film is that it is intriguing. The premise certainly draws the audience in because it is a mystery, and throughout the film there are hints that there ...|\n", "|Negative|Positive|Sure, Titanic was a good movie, the first time you see it, but you really should see it a second time and your opinion of the film will definetly change. The first time you see the movie you see th...|\n", "|Negative|Positive|Verhoeven's movie was utter and complete garbage. He's a disgusting hack of a director and should be ashamed. By his own admission, he read 2 chapters of the book, got bored, and decided to make th...|\n", "|Negative|Positive|quote by Nicolas Martin (nicmart) from Houston, TX: \"Fine film, but DVD \"reformatted for TV\", 8 April 2002 - This is a charming and emotive film. On the other hand, the DVD I purchased has been \"re...|\n", "|Negative|Positive|In the rapid economic development of 1990's in China, there is a resurgence of traditional Chinese culture, partially due to the rise of nationalism accompanied by the increase in wealth, and more ...|\n", "|Positive|Negative|Earth has been destroyed in a nuclear holocaust. Well, parts of the Earth, because somewhere in Italy, a band of purebred survivors--those without radioactive contamination--are holed up in a massi...|\n", "|Positive|Negative|Everything everyone has said already pretty much rings true when it comes to 'The Prey'. Endless nature footage, bad acting - Aside from these elements, this is a watchable film for slasher fans th...|\n", "|Positive|Negative|This tale of the upper-classes getting their come-uppance and wallowing in their high-class misery is like a contemporary Mid-Sommerish version of an old Joan Crawford movie in which she suffered i...|\n", "|Positive|Negative| Looking for a REAL super bad movie? If you wanna have great fun, don't hesitate and check this one!Ferrigno is incredibly bad but is also the best of this mediocrity.|\n", "|Positive|Negative|From the fertile imagination which brought you the irresistible HERCULES (1983), comes its even more preposterous (read goofier) sequel: right off the bat, we get another unwieldy \"beginning of tim...|" ] } ], "metadata": { "colab": { "name": "sentiment_analysis_movie_reviews", "provenance": [] }, "environment": { "kernel": "9c39b79e5d2e7072beb4bd59-runtime", "name": "workbench-notebooks.m129", "type": "gcloud", "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m129" }, "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }