AI/ML Recipes for Vertex AI, Serverless Spark and BigQuery
AI/ML Recipes for Vertex AI, Serverless Spark and BigQuery open-source project is an effort to jumpstart your development of data processing and machine learning notebooks using VertexAI, BigQuery and Dataproc's distributed processing capabilities.
We are release a set of machine learning focused notebooks, for you to adapt, extend, and use to solve your use cases using your own data.
You can easily clone the repo and start executing the notebooks right way using your Dataproc cluster or Dataproc Serverless Runtime for the PySpark notebooks, and any environment for the BigQuery Dataframes (Bigframes) notebooks.
Notebooks
Please refer to each notebooks folder documentation for more information:
| Title | Industry | Topic | Sub Topic | Main Technologies |
|---|---|---|---|---|
| Fine-tuning Gemini for Domain Specificity | Media & Entertainment | Generative AI | Fine tuning | PySpark, Iceberg. Gemini |
| Generate description from videos using PySpark and Gemini | Retail | Generative AI | Content Generation | PySpark, GCS, Gemini |
| Processing images with generative AI | Retail | Generative AI | Content Generation | PySpark, GCS, Gemini |
| Customer Price Index forecast using PySpark and Monte Carlo | Financial | Sampling | Monte Carlo method | PySpark, GCS, NumPy |
| Multimodal content enrichment | Retail | Generative AI | Content Generation | BigFrames, GCS, Gemini, BigQuery |
| Toxicity classification using Gemini fine-tuned | Gaming | Generative AI | Classification | BigFrames, Gemini, Vertex AI |
| Unstructured document analysis with AI | Financial | Generative AI | Summarization | BigQuery, SQL, Gemini |
| Asset Price Forecast using Iceberg and Prophet | Finance | Forecast | Prophet | PySpark, Dataproc Serverless, Apache Iceberg, Prophet, BigQuery, GCS |
| Propensity Modeling & Churn Predictions | Retail | Analytics | Purchase Predictions | PySpark, Spark ML, BigQuery, Dataproc, GCS |
| Time Series Forecast | Retail | Forecast | ARIMA and TimesFM | BigQuery, BigQuery ML, ARIMA, TimesFM, Python, Matplotlib |
| Time Series Forecast (Antigravity) | Retail | Forecast | ARIMA and TimesFM | BigQuery, BigQuery ML, ARIMA, TimesFM, Python, Matplotlib |
| Data Engineering with GeoSpacial data | Agriculture | Analytics | Geospatial | BigQuery, Google Earth Engine, BigFrames, GeoPandas, BigQuery ML |
| Product Data Analysis using BigQuery | Retail | Analytics | Performance Analysis | BigQuery, Vertex AI, XGBoost, Pandas |
| Vector Search and embeddings | Real Estate | Analytics | Image Search | BigQuery, BigQuery ML, Gemini, GCS, SQL, Python |
| Customer Segmentation | Retail | Analytics | Identifying Customer Segments | BigQuery, BigQuery ML, Gemini, Generative AI, SQL, K-Means Clustering |
| Movie reviews sentiment analysis using PySpark and Gemini | Media & Entertainment | Generative AI | Sentiment Analysis | PySpark, Spark Connect, Gemini, BigQuery |
| Housing prices prediction | Financial | Regression | Decision Tree Regression | PySpark, Spark ML, GCS |
| Wine quality classification using Logistic Regression and PySpark | Retail | Classification | Logistic Regression | PySpark, Spark ML, GCS |
| Predict penguim weight using Linear Regression and Bigframes | Environmental | Regression | Linear Regression | BigFrames, BigQuery |
| Predictive Maintenance for machines | Manufacturing | Classification | Linear Support Vector Machine | PySpark, Spark ML, GCS |
| SMS Spam Filtering using PySpark and Spark ML | Telecom | Classification | Multilayer Perceptron Classifier | PySpark, Spark ML, GCS |
| Bike Trip Duration Prediction using PySpark and BigQuery | Mobility | Regression | Random Forest Regression | PySpark, Spark ML, BigQuery |
| PDF summarization using Gemini and PySpark | Financial | Generative AI | Summarization | PySpark, SparkML, Gemini, BigQuery |
| Accelerated Data Analytics with Google Cloud and NVIDIA | IT Services | Analytics | GPU Accelerated Analytics | pandas, cuDF, NVIDIA GPUs, Google Cloud, Colab Enterprise, Google Cloud Storage, pyarrow, Matplotlib, NumPy |
| Accelerated Data Science with Google Cloud and NVIDIA | Mobility | Regression | GPU Accelerated Regression | NVIDIA CUDA-X, cuDF, cuML, XGBoost, pandas, scikit-learn, Google Cloud, Colab Enterprise, Google Cloud Storage |
| Data Science with PySpark and Distributed XGBoost | Mobility | Regression | Distributed Pyspark Xgboost | PySpark, Apache Spark, XGBoost, Spark MLlib, Spark SQL, GCS, Pandas |
| Product Data Analysis using PySpark and Gemini | Retail | Analytics | Performance Analysis | PySpark, Gemini, XGBoost, Vertex AI, Spark ML, GCS, Matplotlib, Pandas |
Google Cloud products quickstarts:
| Title | Topic | Sub Topic | Main Technologies |
|---|---|---|---|
| Delta format in GCS Quickstart | Quickstart | Delta | PySpark, GCS, Delta |
| Dataproc Metastore | Quickstart | Dataproc Metastore | PySpark, Dataproc Metastore |
| Dataproc cluster insights with BigQuery | Quickstart | Dataproc | BigQuery, Dataproc |
| Bigframes Quickstart | Quickstart | Bigframes | BigFrames, BigQuery, Gemini |
| Apache Iceberg on BQ Quickstart | Quickstart | Iceberg | BigQuery, Apache Iceberg |
| Agent2Agent Quickstart | Quickstart | Agent2Agent | Gemini, Google ADK, A2A, Vertex AI |
| Google ADK Session management with Cloud SQL | Quickstart | Google ADK | Gemini, Google ADK, Cloud SQL, SQLite |
Public Datasets
The notebooks read datasets from our public GCS bucket containing several publicly available datasets.
In this doc you can see the list of available datasets, which are located in gs://dataproc-metastore-public-binaries.
The documentation above has details about the datasets, and links to their original pages, containing their LICENSES, etc.
Antigravity 
Data Cloud extension
These notebooks (Spark and BigQuery) are available from your Antigravity when using the Google Cloud Data Cloud Extension:
Cloud Code VSCode extension
These notebooks are available from your VSCode IDE when using the Cloud Code extension. You can go to Notebook Templates and download each template to your environment:
Usage in Vertex AI Workbench notebooks
These notebooks are available from within the Vertex AI Workbench notebooks environment.
Navigate to JupyterLab home screen and click on Notebooks to see the list of notebooks and a button for you to download/copy them into your environment.
Usage in your local environment
- Install gcloud cli
- Run
gcloud initto setup your default GCP configuration - Clone this repository by running
git clone https://github.com/GoogleCloudPlatform/ai-ml-recipes.git - Install requirements by running
pip install -r requirements.txt - For the PySpark notebooks, use one of the approaches using the Dataproc Jupyter Plugin:
- 5.1) [PySpark Serverless Runtime on Google Cloud] Create a Runtime Template with your desired runtime config, and use it to run your PySpark notebooks.
- Do not forget to ensure the correct network configuration
- 5.2) [Local runtime] Use your local PySpark runtime
- 5.1) [PySpark Serverless Runtime on Google Cloud] Create a Runtime Template with your desired runtime config, and use it to run your PySpark notebooks.
- For the Bigframes notebooks, you do not need PySpark, just any kernel/environment, and the processing will leverage BigQuery in your GCP project
BigQuery Jupyter Plugin
We recommend leveraging the BigQuery Jupyter Plugin, which will be available in your local environment just by installing the dependency running pip install -r requirements.txt. This will enable you to:
- Connect your Jupyterlab notebooks from anywhere to Dataproc
- Develop in Python, SQL, Java/Scala, and R
- Manage Dataproc clusters and jobs
- Run notebooks in your favorite IDE that supports Jupyter using Dataproc as kernel
- Deploy a notebook as a recurring job
- View cloud and spark logs inside Jupyterlab
- View your BigQuery datasets schema inside Jupyterlab
- Manage your files on Google Cloud Storage (GCS)
Contributing
See the contributing instructions to get started contributing.
License
All solutions within this repository are provided under the Apache 2.0 license. Please see the LICENSE file for more detailed terms and conditions.
Disclaimer
This repository and its contents are not an official Google Product.
Contact
Questions, issues, and comments can be raised via Github issues.