{ "cells": [ { "cell_type": "markdown", "id": "74e13d37", "metadata": {}, "source": [ "# 1 Finding Co-located EMIT and NEON AOP Data\n", "\n", "**Summary** \n", "\n", "The Earth surface Mineral dust source InvesTigation (EMIT) instrument is located on the International Space Station (ISS) and has collected data over a large area of the Continental US. The National Ecological Observatory Network (NEON) Airborne Observation Platform (AOP) collects aerial remote sensing data, including hyperspectral reflectance data over sites across the United States and Puerto Rico. In this notebook we will show how to utilize the [`earthaccess` Python library](https://github.com/nsidc/earthaccess) to find spatially overlapping EMIT and NEON reflectance data at NEON's [Niwot Ridge](https://www.neonscience.org/field-sites/niwo) site (NIWO) in the Rocky Mountains of Colorado.\n", "\n", "
\n", "\n", "
\n", "\n", "**Background**\n", "\n", "The **EMIT** instrument is an imaging spectrometer that measures light in visible (V) to short-wave (SWIR) infrared wavelengths; this is also referred to as a VSWIR sensor. These measurements display unique spectral signatures that correspond to the composition on the Earth's surface. The EMIT mission focuses specifically on mapping the composition of minerals to better understand the effects of mineral dust throughout the Earth system and human populations now and in the future. In addition, the EMIT instrument can be used in other applications, such as mapping of greenhouse gases, snow properties, and water resources.\n", "\n", "More details about EMIT and its associated products can be found on the [EMIT website](https://earth.jpl.nasa.gov/emit/) and [EMIT product pages](https://lpdaac.usgs.gov/product_search/?query=EMIT&status=Operational&view=cards&sort=title) hosted by the LP DAAC.\n", "\n", "The **NEON Imaging Spectrometer (NIS)** is an airborne [imaging spectrometer](https://www.neonscience.org/data-collection/imaging-spectrometer) built by JPL (AVIRIS-NG) and operated by the National Ecological Observatory Network's (NEON) Airborne Observation Platform (AOP). NEON's hyperspectral sensors collect measurements of sunlight reflected from the Earth's surface in 426 narrow (~5 nm) spectral channels spanning wavelengths between ~ 380 - 2500 nm. NEON's remote sensing data is intended to map and answer questions about a landscape, with ecological applications including identifying and classifying plant species and communities, mapping vegetation health, detecting disease or invasive species, and mapping droughts, wildfires, or other natural disturbances and their impacts. \n", "\n", "NEON surveys sites spanning the continental US, during peak phenological greenness, capturing each site 3 out of every 5 years, for most terrestrial sites. AOP's [Flight Schedules and Coverage](https://www.neonscience.org/data-collection/flight-schedules-coverage) provide's more information about the current and past schedules.\n", "\n", "More detailed information about NEON's airborne sampling design can be found in the paper: [Spanning scales: The airborne spatial and temporal sampling design of the National Ecological Observatory Network](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13942).\n", "\n", "**Requirements** \n", " - [NASA Earthdata Account](https://urs.earthdata.nasa.gov/home) \n", " - *No Python setup requirements if connected to the workshop cloud instance!* \n", " - **Local Only** Set up Python Environment - See **setup_instructions.md** in the `/setup/` folder to set up a local compatible Python environment\n", "\n", " - NEON User Account and API Token, see [NEON API Tokens Tutorial](https://www.neonscience.org/resources/learning-hub/tutorials/neon-api-tokens-tutorial)\n", "\n", "**Download the NEON Flight Boundary Shapefile:** AOP_flightBoxes.zip\n", "\n", "**Learning Objectives** \n", "- Use functions provided in an external Python module to find and download available NEON airborne reflectance data.\n", "- Use `earthaccess` to find EMIT data that overlaps with a NEON site.\n", "- How to export a list of files and download them programmatically. \n", "\n", "**Tutorial Outline** \n", "\n", "1. Setup - NEON and NASA data accounts\n", "2. Visualize and download NEON data for a single site\n", "3. Find Co-located EMIT collections and download" ] }, { "cell_type": "markdown", "id": "52a94b20", "metadata": {}, "source": [ "## 1. Setup\n", "\n", "Import the required Python libraries. You can install the Python `neonutilities` package using `!pip install neonutilities`. This package will allow you to find available NEON data and download NEON Airborne Observation Platform (AOP) data, using the NEON API." ] }, { "cell_type": "code", "execution_count": null, "id": "b0d82a4b", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Import required libraries\n", "import os, sys\n", "import dotenv\n", "import folium\n", "import earthaccess\n", "import warnings\n", "import folium.plugins\n", "import pandas as pd\n", "import geopandas as gpd\n", "import math\n", "import requests\n", "import neonutilities as nu\n", "from zipfile import ZipFile\n", "from branca.element import Figure\n", "from IPython.display import display\n", "from shapely import geometry\n", "from skimage import io\n", "from datetime import timedelta\n", "from shapely.geometry.polygon import orient\n", "from matplotlib import pyplot as plt" ] }, { "cell_type": "markdown", "id": "24d6a546", "metadata": {}, "source": [ "### 1.2 NEON Token\n", "\n", "Please see [Using an API Token when Accessing NEON Data with neonUtilities](https://www.neonscience.org/resources/learning-hub/tutorials/neon-api-tokens-tutorial) for more details on setting up a NEON token. As of mid-2026 tokens will be required for downloadng NEON data.\n", "\n", "Once you've set up your account and generated a token, you can use the `python-dotenv` package to store the token and access it in Python. If this is the first time you're setting this up, uncomment the next cell, paste your token in the `value_to_set` argument and store it locally. \n" ] }, { "cell_type": "code", "execution_count": null, "id": "6beb52eb", "metadata": {}, "outputs": [], "source": [ "# dotenv.set_key(dotenv_path=\".env\",\n", "# key_to_set=\"NEON_TOKEN\",\n", "# value_to_set=\"YOUR TOKEN HERE\")" ] }, { "cell_type": "markdown", "id": "83637da3", "metadata": {}, "source": [ "Load your NEON token." ] }, { "cell_type": "code", "execution_count": null, "id": "ced88eb3", "metadata": {}, "outputs": [], "source": [ "dotenv.load_dotenv()" ] }, { "cell_type": "code", "execution_count": null, "id": "6adc8290", "metadata": {}, "outputs": [], "source": [ "token = os.environ.get(\"NEON_TOKEN\")" ] }, { "cell_type": "markdown", "id": "513b9b9b", "metadata": {}, "source": [ "### 1.3 NASA Earthdata Login Credentials\n", "\n", "To download or stream NASA data you will need an Earthdata account, you can create one [here](https://urs.earthdata.nasa.gov/home). We will use the `login` function from the `earthaccess` library for authentication before downloading at the end of the notebook. This function can also be used to create a local `.netrc` file if it doesn't exist or add your login info to an existing `.netrc` file. If no Earthdata Login credentials are found in the `.netrc` you'll be prompted for them. This step is not necessary to conduct searches but is needed to download or stream data." ] }, { "cell_type": "markdown", "id": "0b47f795", "metadata": {}, "source": [ "## 2. Visualize and download NEON data for a single site\n", "\n", "NEON data products are hosted on the NEON Data Portal, and can be accessed via an API. We will import a Python package `neonutilities` which includes some functions that interact with the NEON data API to see what data are available (in what year and months data were collected), and download data.\n", "\n", "The EMIT products are hosted by the Land Processes Distributed Active Archive Center (LP DAAC). In this example we will use the cloud-hosted `EMIT_L2A_RFL` products available from the LP DAAC to find data. \n", "\n", "To find data we will use the [`earthaccess` Python library](https://github.com/nsidc/earthaccess). `earthaccess` searches NASA's Common Metadata Repository (CMR), a metadata system that catalogs Earth Science data and associated metadata records. The results can then be used to download granules or generate lists of granule search result URLs.\n", "\n", "Using `earthaccess` we can search based on the attributes of a granule, which can be thought of as a spatiotemporal scene from an instrument containing multiple assets (ex: Reflectance, Reflectance Uncertainty, Masks for the EMIT L2A Reflectance Collection). We can search using attributes such as collection, acquisition time, and spatial footprint. This process can also be used with other EMIT products, other collections, or different data providers, as well as across multiple catalogs with some modification. \n", "\n", "### 2.1 Define NEON Region of Interest (ROI)\n", "\n", "For this example, our spatial region of interest (ROI) will be the NEON site [Niwot Ridge (NIWO)](https://www.neonscience.org/field-sites/niwo) in the Rocky Mountains, Colorado.\n", "\n", "We will create a rectangular ROI surrounding the NIWO flight box. We will search for co-located EMIT data using a polygon rather than a standard bounding box in `earthaccess`. To search for intersections with a polygon using earthaccess, we need to format our ROI as a counterclockwise list of coordinate pairs. \n", "\n", "Download, Unzip, and Open the shape file (.shp) containing the AOP flight box boundaries, which can be downloaded from [NEON Spatial Data and Maps](https://www.neonscience.org/data-samples/data/spatial-data-maps). Read this into a `geodataframe`, explore the contents, and check the coordinate reference system (CRS) of the data." ] }, { "cell_type": "code", "execution_count": null, "id": "015c2439", "metadata": {}, "outputs": [], "source": [ "# function to download data stored on the internet in a public url to a local file\n", "def download_url(url,download_dir):\n", " if not os.path.isdir(download_dir):\n", " os.makedirs(download_dir)\n", " filename = url.split('/')[-1]\n", " r = requests.get(url, allow_redirects=True)\n", " file_object = open(os.path.join(download_dir,filename),'wb')\n", " file_object.write(r.content)" ] }, { "cell_type": "code", "execution_count": null, "id": "8ce854d1", "metadata": {}, "outputs": [], "source": [ "# Download and Unzip the NEON Flight Boundary Shapefile\n", "neon_boundary_url = \"https://www.neonscience.org/sites/default/files/AOP_flightBoxes_0.zip\"\n", "# Use download_url function to save the file to a directory\n", "# os.makedirs('../../data', exist_ok=True)\n", "download_url(neon_boundary_url,'../../data')\n", "# Unzip the file\n", "with ZipFile(f\"../../data/{neon_boundary_url.split('/')[-1]}\", 'r') as zip_ref:\n", " zip_ref.extractall('../../data')" ] }, { "cell_type": "code", "execution_count": null, "id": "b7592bc5", "metadata": { "scrolled": true }, "outputs": [], "source": [ "aop_flightboxes = gpd.read_file(\"../../data/AOP_flightBoxes/AOP_flightboxesAllSites.shp\")\n", "aop_flightboxes.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "29bcc691", "metadata": {}, "outputs": [], "source": [ "aop_flightboxes.crs" ] }, { "cell_type": "markdown", "id": "9653d143", "metadata": {}, "source": [ "The CRS is **EPSG:4326** (WGS84), which is also the CRS we want the data in to submit for our search of EMIT data.\n", "\n", "Next, let's examine the AOP flightboxes polygons further. For this exercise, we'll first look at Niwot Ridge site in Colorado." ] }, { "cell_type": "code", "execution_count": null, "id": "16f894c2", "metadata": {}, "outputs": [], "source": [ "aop_flightboxes[aop_flightboxes.siteID == 'NIWO']" ] }, { "cell_type": "markdown", "id": "11e4d579", "metadata": {}, "source": [ "We can see the NIWO `geodataframe` consists of a single polygon, that we want to include in our study site (sometimes NEON sites may have more than one polygon, as there are sometimes multiple areas, with different priorities for collection)." ] }, { "cell_type": "code", "execution_count": null, "id": "ea8b3f13", "metadata": { "tags": [] }, "outputs": [], "source": [ "# write this to a new variable called \"niwo_polygon\"\n", "niwo_polygon = aop_flightboxes[aop_flightboxes.siteID == 'NIWO']\n", "# subset to only include columns of interest\n", "niwo_polygon = niwo_polygon[['domain','siteName','siteID','sampleType','flightbxID','priority','geometry']]" ] }, { "cell_type": "code", "execution_count": null, "id": "35b3d64a", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Create external boundary of the shape\n", "niwo_roi_poly = niwo_polygon.union_all().envelope\n", "# Re-order vertices to counterclockwise\n", "niwo_roi_poly = orient(niwo_roi_poly, sign=1.0)" ] }, { "cell_type": "markdown", "id": "5a3ff5fc", "metadata": {}, "source": [ "Make a `GeoDataFrame` consisting of the bounding box geometry." ] }, { "cell_type": "code", "execution_count": null, "id": "2da1a8dc", "metadata": { "tags": [] }, "outputs": [], "source": [ "niwo_df = pd.DataFrame({\"Name\":[\"NIWO ROI Bounding Box\"]})\n", "niwo_bbox = gpd.GeoDataFrame({\"Name\":[\"NIWO ROI Bounding Box\"], \"geometry\":[niwo_roi_poly]},crs=\"EPSG:4326\")" ] }, { "cell_type": "markdown", "id": "4578b791", "metadata": {}, "source": [ "We can write this bounding box to a `geojson` file for use in future notebooks. This is commented out for now, but you can uncomment and run the cell below, if desired." ] }, { "cell_type": "code", "execution_count": null, "id": "67ef7eba", "metadata": { "tags": [] }, "outputs": [], "source": [ "#niwo_bbox.to_file('../../data/niwo_bbox.geojson', driver='GeoJSON')" ] }, { "cell_type": "markdown", "id": "1fbbde4b", "metadata": {}, "source": [ "Next we can visualize our region of interest and the exterior boundary polygon containing ROIs. First add a function to help reformat bounding box coordinates to work with leaflet notation." ] }, { "cell_type": "code", "execution_count": null, "id": "5f34eefc", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Function to convert a bounding box for use in leaflet notation\n", "def convert_bounds(bbox, invert_y=False):\n", " \"\"\"\n", " Helper method for changing bounding box representation to leaflet notation\n", "\n", " ``(lon1, lat1, lon2, lat2) -> ((lat1, lon1), (lat2, lon2))``\n", " \"\"\"\n", " x1, y1, x2, y2 = bbox\n", " if invert_y:\n", " y1, y2 = y2, y1\n", " return ((y1, x1), (y2, x2))" ] }, { "cell_type": "markdown", "id": "e9c862d7", "metadata": {}, "source": [ "### 2.2 Display NEON site boundary" ] }, { "cell_type": "code", "execution_count": null, "id": "0182928c", "metadata": { "tags": [] }, "outputs": [], "source": [ "fig = Figure(width=\"750px\", height=\"375px\")\n", "map1 = folium.Map(tiles='https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}&hl=en', attr='Google')\n", "fig.add_child(map1)\n", "\n", "# Add NIWO Bounding Box\n", "folium.GeoJson(niwo_bbox, name='bounding_box').add_to(map1)\n", "\n", "# Add roi geodataframe\n", "niwo_polygon.explore(\"flightbxID\",\n", " popup=True,\n", " categorical=True,\n", " cmap='Set3',\n", " style_kwds=dict(opacity=0.7, fillOpacity=0.4),\n", " name=\"Niwot Ridge ROI\",\n", " m=map1)\n", "\n", "map1.add_child(folium.LayerControl())\n", "map1.fit_bounds(bounds=convert_bounds(niwo_polygon.union_all().bounds))\n", "display(fig)" ] }, { "cell_type": "markdown", "id": "6d5d9f98", "metadata": {}, "source": [ "Above we can see the Niwot Ridge flightbox, and the exterior boundary polygon containing the full area." ] }, { "cell_type": "markdown", "id": "eb45c19d", "metadata": {}, "source": [ "Lastly, we need to convert our polygon to a list of coordinate pairs, to create our Region of Interest (ROI)." ] }, { "cell_type": "code", "execution_count": null, "id": "762d71b3", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Set ROI as list of exterior polygon vertices as coordinate pairs\n", "niwo_roi = list(niwo_roi_poly.exterior.coords)\n", "niwo_roi" ] }, { "cell_type": "markdown", "id": "36a02366", "metadata": {}, "source": [ "### 2.3 Find available NEON reflectance data" ] }, { "cell_type": "markdown", "id": "13f003fe", "metadata": {}, "source": [ "Now we can look at the available NEON hyperspectral reflectance data over NIWO. NEON surface reflectance data are currently available under two different revisions, as AOP is in the process of implementing BRDF (Bidirectional Reflectance Distribution Function) and topographic corrections, but these corrections have not been applied to the full archive of AOP data yet. The reflectance data are available under two revisions of the data product ID `DP3.30006`: `DP3.30006.001` are the *directional* surface reflectance, and `DP3.30006.002` are the *bidirectional* (BRDF- and topographic- corrected) surface reflectance. As of November 2025, bidirectional data are available for data collected between 2022-2025. Let's see what reflectance data are available for each of these data products at Niwot Ridge using the `neonutilities` `list_available_dates` function." ] }, { "cell_type": "code", "execution_count": null, "id": "bea13fa2", "metadata": {}, "outputs": [], "source": [ "refl_rev1_dpid = 'DP3.30006.001'\n", "refl_rev2_dpid = 'DP3.30006.002'\n", "site = 'NIWO'" ] }, { "cell_type": "markdown", "id": "17afd1da", "metadata": {}, "source": [ "The `neonutilities` `list_available_dates` function shows available data for a given data product id (`dpid`) and site. Let's run this for the two revisions of the reflectance data." ] }, { "cell_type": "code", "execution_count": null, "id": "59629003", "metadata": {}, "outputs": [], "source": [ "print('Directional Reflectance Data Available at NEON Site NIWO:')\n", "nu.list_available_dates(refl_rev1_dpid,site)" ] }, { "cell_type": "code", "execution_count": null, "id": "7c6ea647", "metadata": {}, "outputs": [], "source": [ "print('Bidirectional Reflectance Data Available at NEON Site NIWO:')\n", "nu.list_available_dates(refl_rev2_dpid,site)" ] }, { "cell_type": "markdown", "id": "8205bb01", "metadata": {}, "source": [ "The bidirectional data for the 2023 and 2024 collections over NIWO will be closer in time to the EMIT collections, which started in 2022. We'll start by looking at the bidirectional surface reflectance for NIWO in 2024. " ] }, { "cell_type": "code", "execution_count": null, "id": "9d8873c4", "metadata": {}, "outputs": [], "source": [ "year = '2023'" ] }, { "cell_type": "markdown", "id": "1cb91dee", "metadata": {}, "source": [ "### 2.4 Find spatial extent of NEON L3 data tiles\n", "We can download a subset of the Niwot reflectance data using the function `nu.by_tile_aop`. AOP L3 reflectance data are provided in 1 km x 1 km \"tiles\" (or squares) where the lower left coordinate of each tile is provided in the file name. The `by_tile_aop` function requires knowing the spatial coordinates (in UTM easting and northing) of the data tiles you are interested in downloading. \n", "\n", "Before downloading, you can use the function `get_aop_tile_extents` to display the extents of the tiles. Let's run this first to get an idea." ] }, { "cell_type": "code", "execution_count": null, "id": "d5a927e9", "metadata": {}, "outputs": [], "source": [ "niwo2023_extents = nu.get_aop_tile_extents('DP3.30006.002','NIWO','2023', token=token)" ] }, { "cell_type": "markdown", "id": "8614ed4a", "metadata": {}, "source": [ "You can print the `niwo2023_extents` variable to display a complete list of all the coordinate pairs of the data tiles. \n", "\n", "### 2.5 Download NEON reflectance data\n", "\n", "Now that we know the rough bounds of the data, we can download a single tile that encompasses the CU Boulder Mountain Research Station using the `by_tile_aop` function. Our area of interest is within the tile with SW coordinates of 454000, 44431000.\n", "\n", "You will need to set the `include_provisional=True` in order to download this provisional data. AOP data have a 1-year lag period after processing to allow for sufficient time for quality control before releasing data - so 2024 data would be released in the 2026 Data Release in Jan 2026. View for more details on the release process. \n", "\n", "If you leave out the `check_size` input parameter, it will default to True. This will prompt you to download after displaying the download size. This reflectance file is ~700 MB, so make sure you have enough space on your local disk before downloading.\n", "\n", "Use `help(nu.by_tile_aop)` for more details about this function. The required inputs are the data product id (`dpid`), site, year, easting, and northing." ] }, { "cell_type": "code", "execution_count": null, "id": "86b1e432", "metadata": {}, "outputs": [], "source": [ "nu.by_tile_aop(dpid='DP3.30006.002',\n", " site='NIWO',\n", " year='2023',\n", " easting=454000,\n", " northing=4431000,\n", " include_provisional=True,\n", " savepath='../../data/neon_refl',\n", " check_size=False,\n", " token=token)" ] }, { "cell_type": "markdown", "id": "ff941d6d", "metadata": {}, "source": [ "Data downloaded using the `neonutilities` functions will maintain the storage structure as they are stored on Google Cloud Storage. Therefore, the reflectance data will be nested under some subfolders. You can find where the data are downloaded as follows:" ] }, { "cell_type": "code", "execution_count": null, "id": "ae12c2aa", "metadata": {}, "outputs": [], "source": [ "# optionally display the reflectance files that were downloaded to the data/neon_refl folder to see where they are stored\n", "def find_h5_files(root_dir):\n", " h5_files = []\n", " for dirpath, _, filenames in os.walk(root_dir):\n", " for filename in filenames:\n", " if filename.endswith('.h5'):\n", " h5_files.append(os.path.join(dirpath, filename))\n", " return h5_files\n", "\n", "h5_files = find_h5_files('../../data/neon_refl')" ] }, { "cell_type": "code", "execution_count": null, "id": "60403e31", "metadata": {}, "outputs": [], "source": [ "h5_files" ] }, { "cell_type": "markdown", "id": "dd0c8d2a", "metadata": {}, "source": [ "## 3 Find Co-located EMIT collections and download\n", "We need to specify which products we want to search for. The best way to do this is using their concept-id. As mentioned above, we will conduct our search using the EMIT Level 2A Reflectance (EMITL2ARFL). We can do some quick collection queries using `earthaccess` to retrieve the concept-id for each dataset. " ] }, { "cell_type": "code", "execution_count": null, "id": "0852ae54", "metadata": { "tags": [] }, "outputs": [], "source": [ "# EMIT Collection Query\n", "emit_collections = earthaccess.search_datasets(keyword='EMIT L2A')\n", "# Retrieve collection concept-id, collection name, and version from the metadata using list comprehension\n", "[{'concept-id': c['meta']['concept-id'], 'EntryTitle': c['umm']['EntryTitle'], 'Version': c['umm']['Version']} for c in emit_collections]" ] }, { "cell_type": "markdown", "id": "dde68908", "metadata": {}, "source": [ "If your search returns multiple products, be sure to select the right concept-id For this example it will be the first one. We want to use the `LPCLOUD` ECOSTRESS Tiled Land Surface Temperature and Emissivity (concept-id: \"C2076090826-LPCLOUD\"). Create a list of these concept-ids for our data search." ] }, { "cell_type": "code", "execution_count": null, "id": "518c76d5", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Data Collections for our search\n", "emit_concept_id = ['C2408750690-LPCLOUD']" ] }, { "cell_type": "markdown", "id": "dbd3f369", "metadata": {}, "source": [ "For our date range, we'll look at data collected between June 2023 and November 2024. The `date_range` can be specified as a pair of dates, start and end (up to, not including)." ] }, { "cell_type": "code", "execution_count": null, "id": "20245b20", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Define Date Range\n", "date_range = ('2023-06-01','2024-11-01')" ] }, { "cell_type": "markdown", "id": "cc6f31b0", "metadata": {}, "source": [ "### 3.1 Search for EMIT data\n", "\n", "Submit a query using `earthaccess`, usin the `niwo_roi` as the region of interest." ] }, { "cell_type": "code", "execution_count": null, "id": "dd8a1353", "metadata": { "tags": [] }, "outputs": [], "source": [ "emit_query_results = earthaccess.search_data(\n", " concept_id=emit_concept_id,\n", " polygon=niwo_roi,\n", " temporal=date_range,\n", " count=500)" ] }, { "cell_type": "markdown", "id": "bb5701b7", "metadata": {}, "source": [ "### 3.2 Organizing and Filtering Results\n", "\n", "As we can see from above, the results object contains a list of objects with metadata and links. We can convert this to a more readable format, a dataframe. In addition, we can make it a geodataframe by taking the spatial metadata and creating a shapely polygon representing the spatial coverage, and further customize which information we want to use from other metadata fields.\n", "\n", "First, we define some functions to help us create a shapely object for our geodataframe, and retrieve the specific browse image URLs that we want. By default, the browse image selected by `earthaccess` is the first one in the list, but the `ECO_L2_LSTE` has several browse images, and we want to make sure we retrieve the `png` file, which is a preview of the LSTE." ] }, { "cell_type": "code", "execution_count": null, "id": "35965f41", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Function to create shapely polygon of spatial coverage\n", "def get_shapely_object(result:earthaccess.results.DataGranule):\n", " # Get Geometry Keys\n", " geo = result['umm']['SpatialExtent']['HorizontalSpatialDomain']['Geometry']\n", " keys = geo.keys()\n", "\n", " if 'BoundingRectangles' in keys:\n", " bounding_rectangle = geo['BoundingRectangles'][0]\n", " # Create bbox tuple\n", " bbox_coords = (bounding_rectangle['WestBoundingCoordinate'],bounding_rectangle['SouthBoundingCoordinate'],\n", " bounding_rectangle['EastBoundingCoordinate'],bounding_rectangle['NorthBoundingCoordinate'])\n", " # Create shapely geometry from bbox\n", " shape = geometry.box(*bbox_coords, ccw=True)\n", " elif 'GPolygons' in keys:\n", " points = geo['GPolygons'][0]['Boundary']['Points']\n", " # Create shapely geometry from polygons\n", " shape = geometry.Polygon([[p['Longitude'],p['Latitude']] for p in points])\n", " else:\n", " raise ValueError('Provided result does not contain bounding boxes/polygons or is incompatible.')\n", " return(shape)\n", "\n", "# Retrieve png browse image if it exists or first jpg in list of urls\n", "def get_png(result:earthaccess.results.DataGranule):\n", " https_links = [link for link in result.dataviz_links() if 'https' in link]\n", " if len(https_links) == 1:\n", " browse = https_links[0]\n", " elif len(https_links) == 0:\n", " browse = 'no browse image'\n", " warnings.warn(f\"There is no browse imagery for {result['umm']['GranuleUR']}.\")\n", " else:\n", " browse = [png for png in https_links if '.png' in png][0]\n", " return(browse)" ] }, { "cell_type": "markdown", "id": "7d6c98be", "metadata": {}, "source": [ "Now that we have our functions we can create a dataframe, then calculate and add our shapely geometries to make a geodataframe. After that, add a column for our browse image urls and print the number of granules in our results, so we can monitor the quantity we are working with a we winnow down to the data we want." ] }, { "cell_type": "code", "execution_count": null, "id": "2361c527", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Create Dataframe of Results Metadata\n", "emit_results_df = pd.json_normalize(emit_query_results)\n", "# Create shapely polygons for result\n", "geometries = [get_shapely_object(emit_query_results[index]) for index in emit_results_df.index.to_list()]\n", "# Convert to GeoDataframe\n", "emit_gdf = gpd.GeoDataFrame(emit_results_df, geometry=geometries, crs=\"EPSG:4326\")\n", "# Remove emit_results_df, no longer needed\n", "del emit_results_df\n", "# Add browse imagery links\n", "emit_gdf['browse'] = [get_png(granule) for granule in emit_query_results]\n", "emit_gdf['shortname'] = [result['umm']['CollectionReference']['ShortName'] for result in emit_query_results]\n", "# Preview GeoDataframe\n", "print(f'{emit_gdf.shape[0]} granules total')" ] }, { "cell_type": "markdown", "id": "1175e082", "metadata": {}, "source": [ "Preview our geodataframe to get an idea what it looks like." ] }, { "cell_type": "code", "execution_count": null, "id": "d6ce05ab", "metadata": { "tags": [] }, "outputs": [], "source": [ "emit_gdf.head()" ] }, { "cell_type": "markdown", "id": "21f8d8eb", "metadata": {}, "source": [ "There are a lot of columns with data that is not relevant for this exercise, so we can drop those. To do that, list the names of columns." ] }, { "cell_type": "code", "execution_count": null, "id": "14be3930", "metadata": { "tags": [] }, "outputs": [], "source": [ "# List Column Names\n", "emit_gdf.columns" ] }, { "cell_type": "markdown", "id": "24ca6997", "metadata": {}, "source": [ "Now create a list of columns to keep and use it to filter the dataframe." ] }, { "cell_type": "code", "execution_count": null, "id": "b5ed2463", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Create a list of columns to keep\n", "keep_cols = ['meta.concept-id','meta.native-id', 'umm.TemporalExtent.RangeDateTime.BeginningDateTime','umm.TemporalExtent.RangeDateTime.EndingDateTime','umm.CloudCover','umm.DataGranule.DayNightFlag','geometry','browse', 'shortname']\n", "# Remove unneeded columns\n", "emit_gdf = emit_gdf[emit_gdf.columns.intersection(keep_cols)]\n", "\n", "# Rename some columns\n", "emit_gdf.rename(columns = {'meta.concept-id':'concept_id','meta.native-id':'granule',\n", " 'umm.TemporalExtent.RangeDateTime.BeginningDateTime':'start_datetime',\n", " 'umm.TemporalExtent.RangeDateTime.EndingDateTime':'end_datetime',\n", " 'umm.CloudCover':'cloud_cover',\n", " 'umm.DataGranule.DayNightFlag':'day_night'}, inplace=True)\n", "emit_gdf.head()" ] }, { "cell_type": "markdown", "id": "cf1fca4e", "metadata": {}, "source": [ "> **Note: If querying on-premises (not cloud) LP DAAC datasets, the `meta.concept-id` will not show as `xxxxxx-LPCLOUD`. For these datasets, the granule name can be retrieved from the `umm.DataGranule.Identifiers` column.**" ] }, { "cell_type": "markdown", "id": "0331df52", "metadata": {}, "source": [ "We can filter using the day/night flag as well, since we need a daytime collection to be comparable to the NEON data (which is captured at a high solar angle)." ] }, { "cell_type": "code", "execution_count": null, "id": "9ca0b46a", "metadata": { "tags": [] }, "outputs": [], "source": [ "# emit_gdf = emit_gdf[emit_gdf['day_night'].str.contains('Day')]" ] }, { "cell_type": "markdown", "id": "f0c11e30", "metadata": {}, "source": [ "Our first step toward filtering the datasets will be to add a column with a `datetime`. \n", "\n", "> **If working locally using `lpdaac_vitals` python environment, you may need to pass the `format='ISO8601'`argument to the `to_datetime` function, as shown in the commented-out line due to a difference in versions of pandas.**" ] }, { "cell_type": "code", "execution_count": null, "id": "1c9beaa0", "metadata": { "tags": [] }, "outputs": [], "source": [ "#emit_gdf['datetime_obj'] = pd.to_datetime(emit_gdf['start_datetime']) # 2i2c\n", "emit_gdf['datetime_obj'] = pd.to_datetime(emit_gdf['start_datetime'], format='ISO8601') # Local ENV" ] }, { "cell_type": "markdown", "id": "cd80e6fe", "metadata": {}, "source": [ "We can roughly visualize the quantity of results by month at our location using a histogram with 8 bins (Jan - Oct)." ] }, { "cell_type": "code", "execution_count": null, "id": "e4469f7a", "metadata": { "tags": [] }, "outputs": [], "source": [ "emit_gdf.hist(column='datetime_obj', by='shortname', bins=10, color='green', edgecolor='black', linewidth=1, sharey=True);" ] }, { "cell_type": "markdown", "id": "fbd9c448", "metadata": {}, "source": [ "### 3.3 Visualizing Intersecting Coverage - NEON and EMIT\n", "\n", "Now that we have geodataframes containing some co-located data, we can visualize them on a map using `folium`." ] }, { "cell_type": "code", "execution_count": null, "id": "88f06231", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Plot Using Folium\n", "# Create Figure and Select Background Tiles\n", "fig = Figure(width=\"750px\", height=\"375px\")\n", "map1 = folium.Map(tiles='https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}&hl=en', attr='Google')\n", "fig.add_child(map1)\n", "\n", "# Add NIWO Bounding Box\n", "folium.GeoJson(niwo_bbox,\n", " name='bounding_box',).add_to(map1)\n", "\n", "# Add roi geodataframe\n", "niwo_polygon.explore(\"flightbxID\",\n", " popup=True,\n", " categorical=True,\n", " cmap='Set3',\n", " style_kwds=dict(opacity=0.7, fillOpacity=0.4),\n", " name=\"Niwot Ridge ROI\",\n", " m=map1)\n", "\n", "# Plot STAC EMITL2ARFL Results - note we must drop the datetime_obj columns for this to work\n", "emit_gdf.drop(columns=['datetime_obj']).explore(\n", " \"granule\",\n", " categorical=True,\n", " tooltip=[\n", " \"granule\",\n", " \"start_datetime\",\n", " \"cloud_cover\",\n", " ],\n", " popup=True,\n", " style_kwds=dict(fillOpacity=0.1, width=2),\n", " name=\"EMIT\",\n", " m=map1,\n", " legend=False\n", ")\n", "\n", "map1.fit_bounds(bounds=convert_bounds(emit_gdf.union_all().bounds))\n", "map1.add_child(folium.LayerControl())\n", "display(fig)" ] }, { "cell_type": "markdown", "id": "f80742f1", "metadata": {}, "source": [ "### 3.4 Previewing EMIT Browse Imagery\n", "The EMIT browse imagery is not orthorectified, so to get an idea what scenes look like, we can plot them in a grid using matplotlib.\n", "\n", "> Note: Black space indicates onboard cloud masking that occurs before data is downlinked from the ISS." ] }, { "cell_type": "code", "execution_count": null, "id": "c065cd62", "metadata": { "tags": [] }, "outputs": [], "source": [ "cols = 3\n", "rows = math.ceil(len(emit_gdf)/cols)\n", "fig, ax = plt.subplots(rows, cols, figsize=(20,20))\n", "ax = ax.flatten()\n", "\n", "for _n, index in enumerate(emit_gdf.index.to_list()):\n", " img = io.imread(emit_gdf['browse'][index])\n", " ax[_n].imshow(img)\n", " ax[_n].set_title(f\"Index: {index} - {emit_gdf['granule'][index]}\")\n", " ax[_n].axis('off')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "d42de768", "metadata": {}, "source": [ "### 3.5 Further Filtering - Cloud Conditions\n", "\n", "We can see that some of these granules likely won't work because of the large amount of cloud cover, we can use a list of these to filter them out. Make a list of indexes to filter out. Filter out the cloudy granules." ] }, { "cell_type": "code", "execution_count": null, "id": "d2643806", "metadata": { "tags": [] }, "outputs": [], "source": [ "# set a threshold for cloud cover and filter to remove scenes with >30% cloud cover\n", "emit_gdf_clear = emit_gdf[emit_gdf.cloud_cover < 30]" ] }, { "cell_type": "code", "execution_count": null, "id": "f5ba5634", "metadata": {}, "outputs": [], "source": [ "emit_gdf_clear" ] }, { "cell_type": "markdown", "id": "255074b1", "metadata": {}, "source": [ "We can see that there are a few scenes with < 30% cloud cover. One is captured in late June (June 25) 2023, which is close in time to when NEON typically surveys Niwot Ridge. NEON surveys during \"peak-greenness\", when leaves are most photosynthetically active, which at NIWO usually occurs in July - August (in 2023 NIWO was surveyed on July 24, Aug 15 and Aug 21).\n", "\n", "Let's take a look at the June 2023 clear-weather EMIT dataset (Index 1):\n", "\n", "- **1**: `EMIT_L2A_RFL_001_20230625T170814_2317611_005`\n", "\n", "We can plot this scene as follows:" ] }, { "cell_type": "code", "execution_count": null, "id": "99555934", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(1, 1, figsize=(8,8))\n", "img = io.imread(emit_gdf_clear['browse'][1])\n", "ax.imshow(img)\n", "ax.set_title(f\"{emit_gdf_clear['granule'][1]}\")\n", "ax.axis('off')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "048ac57c", "metadata": {}, "source": [ "**We can now go back to our `folium` plot above and re-run the cell to update it based on our filtering.**" ] }, { "cell_type": "code", "execution_count": null, "id": "a4757e4c", "metadata": {}, "outputs": [], "source": [ "# Plot Using Folium\n", "# Create Figure and Select Background Tiles\n", "fig = Figure(width=\"750px\", height=\"375px\")\n", "map1 = folium.Map(tiles='https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}&hl=en', attr='Google')\n", "fig.add_child(map1)\n", "\n", "# Add NIWO Bounding Box\n", "folium.GeoJson(niwo_bbox,\n", " name='bounding_box',\n", " ).add_to(map1)\n", "\n", "# Add roi geodataframe\n", "niwo_polygon.explore(\"flightbxID\",\n", " popup=True,\n", " categorical=True,\n", " cmap='Set3',\n", " style_kwds=dict(opacity=0.7, fillOpacity=0.4),\n", " name=\"Niwot Ridge ROI\",\n", " m=map1)\n", "\n", "# Plot STAC EMITL2ARFL Results - note we must drop the datetime_obj columns for this to work\n", "emit_gdf_clear.drop(columns=['datetime_obj']).explore(\n", " \"granule\",\n", " categorical=True,\n", " tooltip=[\n", " \"granule\",\n", " \"start_datetime\",\n", " \"cloud_cover\",\n", " ],\n", " popup=True,\n", " style_kwds=dict(fillOpacity=0.1, width=2),\n", " name=\"EMIT\",\n", " m=map1,\n", " legend=False\n", ")\n", "\n", "map1.fit_bounds(bounds=convert_bounds(emit_gdf.unary_union.bounds))\n", "map1.add_child(folium.LayerControl())\n", "display(fig)" ] }, { "cell_type": "markdown", "id": "13d94c23", "metadata": {}, "source": [ "## 3.6 Generating a list of EMIT URLs and downloading data\n", "\n", "Creating a list of results URLs will include all of these assets, so if we only want a subset we need an additional filter to keep the specific assets we want. If you look back, you can see we kept the same indexing throughout the notebook. This enables us to simply subset the `earthaccess` results object to retrieve the results we want. \n", "\n", "Create a list of index values to keep." ] }, { "cell_type": "code", "execution_count": null, "id": "424cef4c", "metadata": { "tags": [] }, "outputs": [], "source": [ "keep_granules = [1]" ] }, { "cell_type": "markdown", "id": "93198c0e", "metadata": {}, "source": [ "Filter the results list." ] }, { "cell_type": "code", "execution_count": null, "id": "1c7cfbee", "metadata": { "tags": [] }, "outputs": [], "source": [ "filtered_results = [result for i, result in enumerate(emit_query_results) if i in keep_granules]" ] }, { "cell_type": "markdown", "id": "4ce6a0c8", "metadata": {}, "source": [ "Now we can download all of the associated assets, or retrieve the URLS and further filter them to specifically what we want. \n", "\n", "First, log into Earthdata using the `login` function from the `earthaccess` library. The `persist=True` argument will create a local `.netrc` file if it doesn't exist, or add your login info to an existing `.netrc` file. If no Earthdata Login credentials are found in the `.netrc` you'll be prompted for them. As mentioned in section 1.2, this step is not necessary to conduct searches, but is needed to download or stream data." ] }, { "cell_type": "markdown", "id": "9383320a", "metadata": {}, "source": [ "Now we can download all assets using the following cells. First you will need to log in to your earthaccess account." ] }, { "cell_type": "code", "execution_count": null, "id": "b4e4b2a4", "metadata": {}, "outputs": [], "source": [ "# Log in to earthaccess\n", "earthaccess.login()" ] }, { "cell_type": "code", "execution_count": null, "id": "55ae0a63", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Download All Assets for Granules in Filtered Results\n", "earthaccess.download(filtered_results, '../../data/emit_refl')" ] }, { "cell_type": "markdown", "id": "c2d6b861", "metadata": {}, "source": [ "Congratulations! You have now downloaded co-located hyperspectral reflectance data from NEON airborne collections and the EMIT instrument on the ISS." ] }, { "cell_type": "markdown", "id": "7fbcfbdc", "metadata": {}, "source": [ "## Contact Info: \n", "\n", "**Land Processes Distributed Active Archive Center (LP DAAC)**1\n", "\n", "Email: LPDAAC@usgs.gov \n", "Voice: +1-866-573-3222 \n", "Website: \n", "\n", "1Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. \n", "\n", "**National Ecological Observatory Network (NEON)**2\n", "\n", "Website: \n", "Contact: \n", "\n", "2NEON is a project sponsored by the National Science Foundation and operated by Battelle." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.14" } }, "nbformat": 4, "nbformat_minor": 5 }