{ "cells": [ { "cell_type": "markdown", "id": "3a7a8f81", "metadata": {}, "source": [ "# Finding Coincident NASA Airborne and Orbital Data\n", "\n", "**Summary**\n", "\n", "Often, novel remote sensing research requires utilizing data from multiple instruments. This Jupyter notebook provides an example of finding coincident data from NASA Earth Science missions. Users will learn how to submit mutiple queries to NASA's CMR using the *earthaccess* Python library to accomplish this goal. These queries can search the NASA Earthdata Archive of over 120 petabytes using keywords, spatial constraints (point, bounding box, or polygon), and temporal constraints. From a basic search for AVIRIS-3, we take the metadata returned by the query and use it to define spatial and temporal constraints for a second and third search to to find orbital data from the ECOSTRESS and EMIT collections.\n", "\n", "
\n", "\n", "
\n", "\n", "**Background**\n", "\n", "The **AVIRIS-3** instrument is an airborne imaging spectrometer that measures light in visible and infrared wavelengths. These measurements display unique spectral signatures that correspond to the chemical composition on the Earth's surface and in the atmospheric column above. There are several applications for this data ranging from surface mineral exploration to agriculture. Recently, the instrument was flown over wildfires in Alabama and provided realtime support to firefighters. More specifics about the AVIRIS-3 mission can be found on the [AVIRIS-3 website](https://earth.jpl.nasa.gov/estd-missions/airborne/aviris-3/) and [AVIRIS-3 dataset landing pages](https://daac.ornl.gov/cgi-bin/dataset_lister.pl?p=47).\n", "\n", "The **ECOSTRESS** instrument is a multispectral thermal imaging radiometer designed to answer three overarching science questions:\n", "\n", "- How is the terrestrial biosphere responding to changes in water availability?\n", "- How do changes in diurnal vegetation water stress the global carbon cycle?\n", "- Can agricultural vulnerability be reduced through advanced monitoring of agricultural water consumptive use and improved drought estimation?\n", "\n", "The ECOSTRESS mission is answering these questions by accurately measuring the temperature of plants. Plants regulate their temperature by releasing water through tiny pores on their leaves called stomata. If they have sufficient water they can maintain their temperature, but if there is insufficient water, their temperatures rise and this temperature rise can be measured with ECOSTRESS. The images acquired by ECOSTRESS are the most detailed temperature images of the surface ever acquired from space and can be used to measure the temperature of an individual farmers field.\n", "\n", "More details about ECOSTRESS and its associated products can be found on the [ECOSTRESS website](https://ecostress.jpl.nasa.gov/) and [ECOSTRESS product pages](https://lpdaac.usgs.gov/product_search/?query=ECOSTRESS&status=Operational&view=cards&sort=title) hosted by the Land Processes Distributed Active Archive Center (LP DAAC).\n", "\n", "The **EMIT** instrument is also an imaging spectrometer, located on the international space station. The EMIT mission focuses specifically on mapping the composition of minerals to better understand the effects of mineral dust throughout the Earth system and human populations now and in the future. In addition, the EMIT instrument can be used in other applications, such as mapping of greenhouse gases, snow properties, and water resources.\n", "\n", "More details about EMIT and its associated products can be found on the [EMIT website](https://earth.jpl.nasa.gov/emit/) and [EMIT product pages](https://lpdaac.usgs.gov/product_search/?query=EMIT&status=Operational&view=cards&sort=title) hosted by the LP DAAC.\n", "\n", "**Requirements**\n", " - [NASA Earthdata Account](https://urs.earthdata.nasa.gov/home) \n", " - *No Python setup requirements if connected to the workshop cloud instance!* \n", " - **Local Only -** Set up Python Environment - See **setup_instructions.md** in the `/setup/` folder to set up a local compatible Python environment \n", "\n", "**Tutorial Outline**\n", "\n", "1. Searching for Data\n", "2. Wrangling UMM Metadata\n", "3. Visualizing Results\n", "4. Second Search (ECOSTRESS)\n", "5. Third Search (EMIT)\n", "6. Selecting Assets\n", "7. Downloading and Streaming Data" ] }, { "cell_type": "code", "execution_count": null, "id": "2d2f971c", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import earthaccess\n", "from shapely.geometry import Polygon, polygon\n", "from shapely.ops import unary_union\n", "import folium\n", "import copy\n", "from branca.element import Figure" ] }, { "cell_type": "markdown", "id": "fde76a09", "metadata": {}, "source": [ "## 1. Searching for Data\n", "\n", "To search for data, we'll use the `earthaccess` Python library. `earthaccess` simplifies the amount of code required to search for data, and handles authentication. You do not need to authenticate to search, but do for downloading or streaming data. We'll go ahead and use the `login` function to add our credentials to this session. This function will retrieve your login info from a `.netrc` file if one exists, or prompt you for username and password and create one if you use the `persist` argument." ] }, { "cell_type": "code", "execution_count": null, "id": "fab8a2a1", "metadata": {}, "outputs": [], "source": [ "auth = earthaccess.login(persist=True)" ] }, { "cell_type": "markdown", "id": "1a5fd1b4", "metadata": {}, "source": [ "For the first step of our initial search for AVIRIS-3 data, we can use `earthaccess` to conduct a query for \"AVIRIS-3\" to find information about the data collections (often referred to as data products or data sets) associated with that keyword. From this search, we will get the unique `concept-id` associated with the data collection we want to use, AVIRIS-3 L2A Orthocorrected Surface Reflectance, Facility Instrument Collection. A `concept-id` is a unique identifier for a collection, granule, or a service provided by NASA. These types are indicated by a leading C, G, or S respectively. Using the collection `concept-id` for AVIRIS-3, we can find granules (scenes) from that collection.\n", "\n", "This query returns a list of dictionaries, from which we will specify to return 3 fields: \"concept-id\", \"EntryTitle\", and \"Version\". These fields provide information that will help us select the `concept-id` we want to use in our search for scenes." ] }, { "cell_type": "code", "execution_count": null, "id": "9fd9028d", "metadata": {}, "outputs": [], "source": [ "# Search for AVIRIS Datasets\n", "aviris_collections = earthaccess.search_datasets(keyword='AVIRIS-3')\n", "# Retrieve collection concept-id, collection name, and version from the metadata using list comprehension\n", "[{'concept-id': c['meta']['concept-id'], 'EntryTitle': c['umm']['EntryTitle'], 'Version': c['umm']['Version']} for c in aviris_collections]" ] }, { "cell_type": "markdown", "id": "870732ef", "metadata": {}, "source": [ "Now that we know the `concept-id` for the AVIRIS-3 L2A Orthocorrected Surface Reflectance collection, we can provide it as an argument for our data search. " ] }, { "cell_type": "code", "execution_count": null, "id": "1516e49c", "metadata": {}, "outputs": [], "source": [ "# Conduct Search\n", "results_airborne = earthaccess.search_data(concept_id=\"C3369603199-ORNL_CLOUD\", count=1000)\n", "print(f\"Granules Found: {len(results_airborne)}\")" ] }, { "cell_type": "markdown", "id": "24097375", "metadata": {}, "source": [ "## 2. Wrangling UMM Metadata\n", "\n", "The results from our search for AVIRIS-3 data are returned as a list. By selecting an item in the list and using the `umm` key, we can see a nested dictionary of all the metadata for that a specific granule." ] }, { "cell_type": "code", "execution_count": null, "id": "cec3ea6b", "metadata": {}, "outputs": [], "source": [ "results_airborne[0]['umm']" ] }, { "cell_type": "markdown", "id": "668e9fe9", "metadata": {}, "source": [ "We can pull information like temporal and spatial extent from the metadata. All data in CMR either have a temporal extent, which can be either a `RangeDateTime` or `SingleDateTime`. For most airborne and orbital data these are a `RangeDateTime`. Based on this knowledge and the above metadata, we can build a function to show us the `BeginningDateTime` from all of the scenes. We will then use that function to build a unique list of dates for the airborne campaign. With this list of dates we can plan additional searches." ] }, { "cell_type": "code", "execution_count": null, "id": "525dd1e4", "metadata": {}, "outputs": [], "source": [ "# Get Unique Dates\n", "def get_beginning_dt(result):\n", " return result['umm']['TemporalExtent']['RangeDateTime']['BeginningDateTime']\n", "\n", "timestamps = [get_beginning_dt(result) for result in results_airborne]\n", "unique_dates = set([timestamp.split(\"T\")[0] for timestamp in timestamps])\n", "print(unique_dates)" ] }, { "cell_type": "markdown", "id": "731838fd", "metadata": {}, "source": [ "After creating this set of dates, we can use it to group our results. This will help us visualize the data, and set up our strategy for finding orbital data within a specified time window of the flights." ] }, { "cell_type": "code", "execution_count": null, "id": "c68868ea", "metadata": {}, "outputs": [], "source": [ "# Group Results By Date\n", "results_by_date = {\n", " date: [\n", " result\n", " for result in results_airborne\n", " if date in get_beginning_dt(result)\n", " ]\n", " for date in unique_dates\n", "}" ] }, { "cell_type": "markdown", "id": "0fd393a9", "metadata": {}, "source": [ "As mentioned previously, the metadata for each granule also has the spatial extent associated with the acquisition. We can wrangle this into an appropriate format for additional searches, and visualizations." ] }, { "cell_type": "markdown", "id": "2984d5b9", "metadata": {}, "source": [ "## 3. Visualizing Results\n", "\n", "For visualizing the results, we'll import a `get_vertices` function from the `search_util` module contained in this repository. This function pulls the spatial information out of the results and formats it as a list of polygon vertices so it can be used for `folium` plots or searches with `earthaccess`." ] }, { "cell_type": "code", "execution_count": null, "id": "387273ae", "metadata": {}, "outputs": [], "source": [ "from vitals.search_util import get_vertices" ] }, { "cell_type": "markdown", "id": "db392b33", "metadata": {}, "source": [ "Use folium and our `get_vertices` function to plot the geometry of each acquisition, grouped by date." ] }, { "cell_type": "code", "execution_count": null, "id": "167e60dc", "metadata": {}, "outputs": [], "source": [ "# Visualize our Airborne Campaign Location\n", "fig = Figure(width=\"750px\", height=\"375px\")\n", "# Create Map\n", "m = folium.Map(tiles=None)\n", "fig.add_child(m)\n", "\n", "# Add Basemap\n", "folium.TileLayer(\n", " tiles=(\n", " \"https://server.arcgisonline.com/ArcGIS/rest/\"\n", " \"services/World_Imagery/MapServer/tile/{z}/{y}/{x}\"\n", " ),\n", " name=\"ESRI Satellite\",\n", " attr=\"Esri\",\n", " overlay=False,\n", " control=True\n", ").add_to(m)\n", "\n", "# Define Colormap List (hex) - (https://colorbrewer2.org/#type=qualitative&scheme=Set3&n=12)\n", "cmap = ['#8dd3c7','#ffffb3','#bebada','#fb8072','#80b1d3','#fdb462','#b3de69','#fccde5','#d9d9d9','#bc80bd','#ccebc5','#ffed6f']\n", "\n", "# Add Flightline Polygons by Date\n", "for i, date in enumerate(sorted(results_by_date.keys())):\n", " # Create FeatureGroup for a Date\n", " fg = folium.FeatureGroup(name=f\"{date} AVIRIS-3 Flightlines\")\n", " color = cmap[i % len(cmap)]\n", " for record in results_by_date[date]:\n", " # Define Tooltip\n", " tooltip_dict = {\"Granule ID\": record['umm']['GranuleUR'], **record['umm']['TemporalExtent']['RangeDateTime']}\n", " html = \"
\".join(f\"{k}: {v}\" for k, v in tooltip_dict.items())\n", " # Add Polygon\n", " folium.Polygon(\n", " locations=get_vertices(record, lat_lon=True),\n", " color=color,\n", " weight=2,\n", " fill_opacity=.2,\n", " tooltip=html\n", " ).add_to(fg)\n", " fg.add_to(m)\n", "# Set Bounds and Add Layer Control Widget\n", "m.fit_bounds(bounds=fg.get_bounds())\n", "folium.LayerControl().add_to(m)\n", "fig" ] }, { "cell_type": "markdown", "id": "d50b41ef", "metadata": {}, "source": [ "We can pick a date here, for example **2024-09-05** and then search for data collections that fall within the same spatial area during a desired timeframe. Let's see what ECOSTRESS data that falls within a week of the field and airborne overpass from **2024-09-05**. \n", "\n", "First, lets simplify our search region. Instead of searching for all the flightline polygons on that date, we can merge intersecting ones to conduct fewer queries. If you have a very large number (100s-1000s), the queries can start to take a long time. Loop over our results on the selected date, retrieving the vertices, and putting them in into a shapely polygon to simplify using a unary union. This process will merge all touching polygons and provide us with a list of merged geometries.\n", "\n", "> Note: The spatial extent could be further simplified into a convex hull if that makes more sense for your application, or you have a resulting polygon with too many vertices. There is a limit of ~16,000 vertices." ] }, { "cell_type": "code", "execution_count": null, "id": "ab82c4e7", "metadata": {}, "outputs": [], "source": [ "selected_date = '2024-09-05'" ] }, { "cell_type": "code", "execution_count": null, "id": "43984a7a", "metadata": {}, "outputs": [], "source": [ "polys = []\n", "for result in results_by_date[selected_date]:\n", " coords = get_vertices(result, lat_lon=False)\n", " poly = polygon.orient(Polygon(coords), sign=1.0)\n", " polys.append(poly)\n", "merged = unary_union(polys)" ] }, { "cell_type": "markdown", "id": "043d7692", "metadata": {}, "source": [ "When searching for a polygon using `earthaccess` it expects a list of coordinates that form a ring in counter-clockwise order. To accomplish this we'll make them into a list using list comprehension and the `orient` function from `shapely.polygon`." ] }, { "cell_type": "code", "execution_count": null, "id": "ffb691ec", "metadata": {}, "outputs": [], "source": [ "merged_rois = [list(polygon.orient(poly,sign=1.0).exterior.coords) for poly in list(merged.geoms)]" ] }, { "cell_type": "markdown", "id": "a4f4519a", "metadata": {}, "source": [ "After merging, we've reduced the geometries down to 2 features, meaning only 2 search queries are required. Now add these merged polygons to our existing map." ] }, { "cell_type": "code", "execution_count": null, "id": "37fd1f7e", "metadata": {}, "outputs": [], "source": [ "# Add Merged Flightlines to existing figure\n", "for roi in merged_rois:\n", " folium.Polygon(\n", " name=\"Merged AVIRIS-3 Granules\",\n", " locations=[(lat,lon) for lon, lat in roi],\n", " color='black',\n", " tooltip=f'Merged AVIRIS-3 Granules Footprint: {selected_date}'\n", " ).add_to(m)\n", "\n", "fig" ] }, { "cell_type": "markdown", "id": "cc3f7bfa", "metadata": {}, "source": [ "## 4. Second Search (ECOSTRESS)\n", "\n", "Now that we have our spatial constraints for our orbital data search, we just need to retrieve the concept-id for the collection we want, and then select a temporal range that we want our ECOSTRESS results to fall within.\n", "\n", "First, search for the ECOSTRESS L2 Tiled collection and retrieve its concept-id." ] }, { "cell_type": "code", "execution_count": null, "id": "b1c1101c", "metadata": {}, "outputs": [], "source": [ "# ECOSTRESS Collection Query\n", "eco_collections = earthaccess.search_datasets(keyword='ECOSTRESS L2T')\n", "# Retrieve collection concept-id, collection name, and version from the metadata using list comprehension\n", "[{'concept-id': c['meta']['concept-id'], 'EntryTitle': c['umm']['EntryTitle'], 'Version': c['umm']['Version']} for c in eco_collections]\n" ] }, { "cell_type": "markdown", "id": "b38376ab", "metadata": {}, "source": [ "Choose a temporal range to search for orbital data. We'll select +/- 10 days from our first AVIRIS-3 campaign data. We can provide this data to our search as a tuple of datetime strings. These can be provided in the format below, or truncated to just the date in the format `YYYY-MM-DD`." ] }, { "cell_type": "code", "execution_count": null, "id": "a75b6776", "metadata": {}, "outputs": [], "source": [ "temporal_range = ('2024-08-25T00:00:00Z','2024-09-15T23:59:59Z')" ] }, { "cell_type": "markdown", "id": "6b4719ba", "metadata": {}, "source": [ "Now we can define our parameters and search for ECOSTRESS data. To do this, we'll search for each ROI and combine the results. One thing to note here is that since the ROIs we have provided are small relative to ECOSTRESS scenes, the same scene will likely be a result for each search. To avoid having duplicates in our results, we'll omit granules that are already present in our results." ] }, { "cell_type": "code", "execution_count": null, "id": "371aeaa0", "metadata": {}, "outputs": [], "source": [ "# Orbital Search Queries\n", "eco_results = []\n", "seen = set()\n", "# Check each ROI\n", "for roi in merged_rois:\n", " search_params = {\n", " \"concept_id\":\"C2076090826-LPCLOUD\",\n", " \"temporal\":temporal_range,\n", " \"polygon\": roi\n", " }\n", " results_part = earthaccess.search_data(**search_params)\n", " # Ensure duplicates are not added\n", " for item in results_part:\n", " if item not in seen:\n", " seen.add(item)\n", " eco_results.append(item)\n", "print(f\"Granules Found: {len(eco_results)}\")" ] }, { "cell_type": "markdown", "id": "25487b36", "metadata": {}, "source": [ "> This process can be slow. For better performance, it can be implemented in parallel or using asynchronous requests. Just make sure you're not overwhelming the service with the quantity of requests." ] }, { "cell_type": "markdown", "id": "4d97a199", "metadata": {}, "source": [ "Import the `get_asset_urls` helper function from the `search_util` module to get urls associated with results. It can be used to retrieve links to data or browse imagery (quicklooks). " ] }, { "cell_type": "code", "execution_count": null, "id": "2016c1bb", "metadata": {}, "outputs": [], "source": [ "from vitals.search_util import get_asset_urls" ] }, { "cell_type": "markdown", "id": "bc9af4fb", "metadata": {}, "source": [ "Now visualize our ECOSTRESS results. We'll use the above function in our `folium` plot to visualize the browse imagery for ECOSTRESS scenes.\n", "\n", "> Note: Not all data collections have browse imagery for each granule, and some that do are not designed to be plotted using the extent geometry, like we do below." ] }, { "cell_type": "code", "execution_count": null, "id": "fcbdf84c", "metadata": {}, "outputs": [], "source": [ "# Visualize our Airborne Campaign Location\n", "eco_fig = Figure(width=\"750px\", height=\"375px\")\n", "\n", "# Create Map\n", "eco_map = folium.Map(tiles=None)\n", "eco_fig.add_child(eco_map)\n", "\n", "# Add Basemap\n", "folium.TileLayer(\n", " tiles=(\n", " \"https://server.arcgisonline.com/ArcGIS/rest/\"\n", " \"services/World_Imagery/MapServer/tile/{z}/{y}/{x}\"\n", " ),\n", " name=\"ESRI Satellite\",\n", " attr=\"Esri\",\n", " overlay=False,\n", " control=True\n", ").add_to(eco_map)\n", "\n", "# Define Colormap List (hex) - (https://colorbrewer2.org/#type=qualitative&scheme=Set3&n=12)\n", "cmap = ['#8dd3c7','#ffffb3','#bebada','#fb8072','#80b1d3','#fdb462','#b3de69','#fccde5','#d9d9d9','#bc80bd','#ccebc5','#ffed6f']\n", "\n", "# Add Flightline Polygons by Date\n", "for i, result in enumerate(eco_results):\n", "\n", " # Create tooltip for each feature\n", " tooltip_meta = {\"Granule ID\": result['umm']['GranuleUR'], **result['umm']['TemporalExtent']['RangeDateTime']}\n", "\n", " # Convert tooltip to html\n", " html = \"
\".join(f\"{k}: {v}\" for k, v in tooltip_meta.items())\n", "\n", " # Create FeatureGroup\n", " fg = folium.FeatureGroup(name=tooltip_meta['Granule ID'])\n", "\n", " # Get Coordinates and Color\n", " coords = get_vertices(result, lat_lon=True)\n", " color = cmap[i % len(cmap)]\n", "\n", " # Add Polygons\n", " result_poly = folium.Polygon(\n", " locations=coords,\n", " color=color,\n", " weight=2,\n", " fill=True,\n", " fill_opacity=0,\n", " tooltip=html\n", " ).add_to(fg)\n", " fg.add_to(eco_map)\n", "\n", " # Add Browse Image\n", " bounds = result_poly.get_bounds()\n", " folium.raster_layers.ImageOverlay(\n", " image=get_asset_urls(result, extension=\".png\", first_only=True),\n", " bounds=bounds,\n", " opacity=0.75,\n", " zindex=1,\n", " ).add_to(fg)\n", " \n", "# Add Merged ROIs from Search\n", "for roi in merged_rois:\n", " folium.Polygon(locations=[(lat,lon) for lon, lat in roi],\n", " color='black').add_to(eco_map)\n", "\n", "# Set Bounds and Add Layer Control Widget\n", "eco_map.fit_bounds(bounds=fg.get_bounds())\n", "folium.LayerControl().add_to(eco_map)\n", "eco_fig" ] }, { "cell_type": "markdown", "id": "2f2e88b5", "metadata": {}, "source": [ "## 5. Third Search (EMIT)\n", "\n", "Similarly to our ECOSTRESS Search, we can do the same for EMIT L2A Reflectance. Conduct a collection query to retrieve the `concept-id`, then search for data like we previously did for ecostress." ] }, { "cell_type": "code", "execution_count": null, "id": "dead8b34", "metadata": {}, "outputs": [], "source": [ "# EMIT Collection Query\n", "emit_collections = earthaccess.search_datasets(keyword='EMIT L2A')\n", "# Retrieve collection concept-id, collection name, and version from the metadata using list comprehension\n", "[{'concept-id': c['meta']['concept-id'], 'EntryTitle': c['umm']['EntryTitle'], 'Version': c['umm']['Version']} for c in emit_collections]" ] }, { "cell_type": "code", "execution_count": null, "id": "be1d3447", "metadata": {}, "outputs": [], "source": [ "# Orbital Search Queries\n", "emit_results = []\n", "seen = set()\n", "# Check each ROI\n", "for roi in merged_rois:\n", " search_params = {\n", " \"concept_id\":\"C2408750690-LPCLOUD\",\n", " \"temporal\":temporal_range,\n", " \"polygon\": roi\n", " }\n", " results_part = earthaccess.search_data(**search_params)\n", " # Ensure duplicates are not added\n", " for item in results_part:\n", " if item not in seen:\n", " seen.add(item)\n", " emit_results.append(item)\n", "print(f\"Granules Found: {len(emit_results)}\")" ] }, { "cell_type": "markdown", "id": "3bce1b15", "metadata": {}, "source": [ "Build the same type of `folium` figure." ] }, { "cell_type": "code", "execution_count": null, "id": "80382670", "metadata": {}, "outputs": [], "source": [ "# Visualize our Airborne Campaign Location\n", "emit_eco_fig = Figure(width=\"750px\", height=\"375px\")\n", "emit_eco_map = copy.deepcopy(eco_map)\n", "\n", "# Remove old Layer Control\n", "for key, child in list(emit_eco_map._children.items()):\n", " if isinstance(child, folium.LayerControl):\n", " emit_eco_map._children.pop(key)\n", "\n", "# Add New Map to Figure\n", "emit_eco_fig.add_child(emit_eco_map)\n", "\n", "# Define Colormap List (hex) - (https://colorbrewer2.org/#type=qualitative&scheme=Paired&n=12)\n", "cmap_emit = ['#a6cee3','#1f78b4','#b2df8a','#33a02c','#fb9a99','#e31a1c','#fdbf6f','#ff7f00','#cab2d6','#6a3d9a','#ffff99','#b15928']\n", "\n", "# Add Flightline Polygons by Date\n", "for i, result in enumerate(emit_results):\n", "\n", " # Create tooltip for each feature\n", " tooltip_meta = {\"Granule ID\": result['umm']['GranuleUR'], **result['umm']['TemporalExtent']['RangeDateTime']}\n", "\n", " # Convert tooltip to html\n", " html = \"
\".join(f\"{k}: {v}\" for k, v in tooltip_meta.items())\n", "\n", " # Create FeatureGroup\n", " fg = folium.FeatureGroup(name=tooltip_meta['Granule ID'])\n", "\n", " # Get Coordinates and Color\n", " coords = get_vertices(result, lat_lon=True)\n", " color = cmap_emit[i % len(cmap_emit)]\n", "\n", " # Add Polygons\n", " result_poly = folium.Polygon(\n", " locations=coords,\n", " color=color,\n", " weight=2,\n", " fill=True,\n", " fill_opacity=0,\n", " tooltip=html\n", " ).add_to(fg)\n", " fg.add_to(emit_eco_map)\n", "\n", " # No Browse Image for EMIT since its not orthorectified\n", "\n", "# Set Bounds and Add Layer Control Widget\n", "emit_eco_map.fit_bounds(bounds=fg.get_bounds())\n", "folium.LayerControl().add_to(emit_eco_map)\n", "emit_eco_fig" ] }, { "cell_type": "markdown", "id": "d2758c04", "metadata": {}, "source": [ "## 6. Selecting Assets" ] }, { "cell_type": "markdown", "id": "f4e6d802", "metadata": {}, "source": [ "We can view what types of files are available by looking in the `RelatedUrls` dictionary. " ] }, { "cell_type": "code", "execution_count": null, "id": "cd64f7b9", "metadata": {}, "outputs": [], "source": [ "results_by_date['2024-09-05'][0]['umm']['RelatedUrls']" ] }, { "cell_type": "markdown", "id": "f0e2042b", "metadata": {}, "source": [ "The `get_asset_urls` retrieves just the URLs from this dictionary using string matching to help filter what assets we want. For example we can look at examples of all of the links:" ] }, { "cell_type": "code", "execution_count": null, "id": "3b136911", "metadata": {}, "outputs": [], "source": [ "get_asset_urls(results_by_date['2024-09-05'][0])" ] }, { "cell_type": "markdown", "id": "2560d917", "metadata": {}, "source": [ "Or we can grab just the reflectance .nc files." ] }, { "cell_type": "code", "execution_count": null, "id": "b40d4544", "metadata": {}, "outputs": [], "source": [ "get_asset_urls(results_by_date['2024-09-05'][0], extension=\".nc\", contains=\"_RFL_ORT\")" ] }, { "cell_type": "markdown", "id": "b7e79ed1", "metadata": {}, "source": [ "Note that since our extension and contains arguments are just using string matching, any string appeearing in all of the filenames, like RFL_ORT without an _ in front (like _RFL_ORT) will return several asset urls. \n", "\n", "We can use this function on our lists of results to build a simple list of urls for the assets we want." ] }, { "cell_type": "code", "execution_count": null, "id": "a5c05be4", "metadata": {}, "outputs": [], "source": [ "# Get Results for a specific Date\n", "aviris_links = [\n", " url for result in results_by_date['2024-09-05']\n", " for url in get_asset_urls(result, extension=[\".nc\"])\n", "]\n", "len(aviris_links)" ] }, { "cell_type": "code", "execution_count": null, "id": "f9d8fcf4", "metadata": {}, "outputs": [], "source": [ "aviris_links[:5]" ] }, { "cell_type": "markdown", "id": "699e0096", "metadata": {}, "source": [ "Repeat this selection process for ECOSTRESS, retrieving the urls for the LST and cloud assets." ] }, { "cell_type": "code", "execution_count": null, "id": "51da83dd", "metadata": {}, "outputs": [], "source": [ "eco_links = [\n", " url for result in eco_results\n", " for url in get_asset_urls(result, contains=[\"_LST.\", \"_cloud\"], extension=\".tif\")\n", " ]\n", "len(eco_links)" ] }, { "cell_type": "code", "execution_count": null, "id": "de25763c", "metadata": {}, "outputs": [], "source": [ "eco_links[:5]" ] }, { "cell_type": "markdown", "id": "1b4f57f0", "metadata": {}, "source": [ "Now repeat this process again for EMIT, selecting the reflectance and mask file urls." ] }, { "cell_type": "code", "execution_count": null, "id": "2577a287", "metadata": {}, "outputs": [], "source": [ "emit_links = [\n", " url for result in emit_results\n", " for url in get_asset_urls(result, contains=[\"_RFL_\", \"MASK\"],extension=\".nc\")\n", "]\n", "len(emit_links)" ] }, { "cell_type": "code", "execution_count": null, "id": "aee48c34", "metadata": {}, "outputs": [], "source": [ "emit_links" ] }, { "cell_type": "markdown", "id": "e402c272", "metadata": {}, "source": [ "# 7. Downloading and Streaming Data\n", "\n", "Depending on the objectives of your workflow and internet speed you can choose to download or stream the data. In this notebook we will show how to work with the data locally, outside of the Earthdata Cloud (AWS us-west2). For more about working with these datasets in the cloud, please reference [How to: Directly Access ECOSTRESS Data (S3)](https://github.com/nasa/ECOSTRESS-Data-Resources/blob/main/python/how-tos/how_to_direct_access_s3_ecostress_cog.ipynb) for an example opening cloud-optimized geotiffs or [How to: Use Direct S3 Access to work with EMIT Data](https://github.com/nasa/EMIT-Data-Resources/blob/main/python/how-tos/How_to_Direct_S3_Access.ipynb) for an example opening netCDF4 files. \n", "\n", "Next, we'll stream (directly access) data from AVIRIS-3 and ECOSTRESS datasets, then we'll show how to and download EMIT data. A download-based workflow typically fits better for larger files or older formats when running analyses outside of the cloud.\n", "\n", "First, import the libraries we'll use to open the data. For EMIT we will use a module included in the repository to open and orthorectify the data." ] }, { "cell_type": "code", "execution_count": null, "id": "579107f6", "metadata": {}, "outputs": [], "source": [ "import xarray as xr\n", "import rasterio as rio\n", "import rioxarray as rxr\n", "import hvplot.xarray\n", "from vitals.emit_tools import emit_xarray" ] }, { "cell_type": "markdown", "id": "0ef23c5c", "metadata": {}, "source": [ "To access data from NASA, you'll need to provide credentials. When streaming this can best be done using the token or cookies set up by the `earthaccess` library. Since we've already logged in, we can start an `fsspec` session to manage our connection to a remote file, including sending credentials. This allows other libraries to work with a URL as if it is a local file." ] }, { "cell_type": "code", "execution_count": null, "id": "a2e9b270", "metadata": {}, "outputs": [], "source": [ "fs = earthaccess.get_fsspec_https_session()" ] }, { "cell_type": "markdown", "id": "d5bc216b", "metadata": {}, "source": [ "From our results, we'll first open an AVIRIS-3 file. These files are in hierarchical NetCDF4 format, meaning they have multiple groups with datasets within each. To open all of the groups, we'll use the `open_datatree` function from `xarray`." ] }, { "cell_type": "code", "execution_count": null, "id": "11b57405", "metadata": {}, "outputs": [], "source": [ "aviris_file = fs.open(aviris_links[0])\n", "aviris_ds = xr.open_datatree(aviris_file)\n", "aviris_ds" ] }, { "cell_type": "markdown", "id": "9f841735", "metadata": {}, "source": [ "This opens the file lazily, meaning we haven't loaded any data yet, just metadata. Before we load the data, we can subset the data or select only the groups we want data from. For example, below we grab the `reflectance` variable from the `reflectance` group, select a wavelength, and plot using `hvplot`." ] }, { "cell_type": "code", "execution_count": null, "id": "fe32c01e", "metadata": {}, "outputs": [], "source": [ "aviris_850 = aviris_ds['reflectance']['reflectance'].sel(wavelength=850, method='nearest').compute()" ] }, { "cell_type": "code", "execution_count": null, "id": "81be9249", "metadata": {}, "outputs": [], "source": [ "aviris_850.hvplot.image(x=\"easting\", y=\"northing\", cmap='viridis', frame_width=750, aspect='equal')" ] }, { "cell_type": "markdown", "id": "0d6fdade", "metadata": {}, "source": [ "Now, lets open an ECOSTRESS L2T LSTE file. These are provided as cloud-optimized geotiffs. To open these, we can similarly use a session to manage our remote connection to a url. For `geotiff` files, the best performance for this is achieved by using a session from `rasterio`. Often its best to include a retry and retry delay. This will prevent your workflow from breaking if there are connectivity issues." ] }, { "cell_type": "code", "execution_count": null, "id": "8cc0fa20", "metadata": {}, "outputs": [], "source": [ "# Cookies\n", "env = rio.Env(\n", " GDAL_HTTP_COOKIEFILE=\"~/cookies.txt\",\n", " GDAL_HTTP_COOKIEJAR=\"~/cookies.txt\",\n", " GDAL_DISABLE_READDIR_ON_OPEN=\"EMPTY_DIR\",\n", " GDAL_HTTP_MAX_RETRY=\"10\",\n", " GDAL_HTTP_RETRY_DELAY=\"0.5\",\n", ")" ] }, { "cell_type": "markdown", "id": "1107afde", "metadata": {}, "source": [ "Enter the rasterio session." ] }, { "cell_type": "code", "execution_count": null, "id": "f86f93d7", "metadata": {}, "outputs": [], "source": [ "# Enter our Rasterio Session for the rest of the notebook\n", "env.__enter__()" ] }, { "cell_type": "markdown", "id": "f571ac8b", "metadata": {}, "source": [ "Open the file, then squeeze to remove the 'band' dimension from the array since we only have one band." ] }, { "cell_type": "code", "execution_count": null, "id": "750c2f4d", "metadata": {}, "outputs": [], "source": [ "eco_ds = rxr.open_rasterio(eco_links[1], mask_and_scale=True).squeeze('band', drop=True)\n", "eco_ds" ] }, { "cell_type": "markdown", "id": "6391b8b5", "metadata": {}, "source": [ "Visualize using `hvplot.image`." ] }, { "cell_type": "code", "execution_count": null, "id": "d1ba3c3e", "metadata": {}, "outputs": [], "source": [ "eco_ds.hvplot.image(x='x',y='y',cmap='Spectral_r', aspect='equal', title=\"Surface Temperature (K)\", frame_width=750)" ] }, { "cell_type": "code", "execution_count": null, "id": "018ab484", "metadata": {}, "outputs": [], "source": [ "# Exit rasterio session\n", "env.__exit__" ] }, { "cell_type": "markdown", "id": "f1092cd0", "metadata": {}, "source": [ "Similar to the AVIRIS-3 data, EMIT data is in .netcdf4 format, so we can use an `fsspec` session to stream the data. Unlike the AVIRIS-3 data, however, EMIT L2A Reflectance Version 1 data is not orthocorrected or chunked for streaming. This means its typically better to download this data unless you're working in the cloud. First we'll download a an EMIT scene from a our list, then we'll open, orthorectify the data, and visualize it. \n", "\n", "Import the `download_granules` function from `vitals.search_util`. Then create a list containing a single url and use the function to download the scene." ] }, { "cell_type": "code", "execution_count": null, "id": "a2ec9b75", "metadata": {}, "outputs": [], "source": [ "from vitals.search_util import download_granules\n", "\n", "single_emit_url_list = [emit_links[0]]\n", "download_granules(url_list=single_emit_url_list,output_directory='../data/')" ] }, { "cell_type": "markdown", "id": "3788833b", "metadata": {}, "source": [ "Now get the filepath to the local copy of the EMIT L2A RFL scene." ] }, { "cell_type": "code", "execution_count": null, "id": "befbe6d4", "metadata": {}, "outputs": [], "source": [ "emit_file = f'../data/{single_emit_url_list[0].split(\"/\")[-1]}'" ] }, { "cell_type": "markdown", "id": "d70bf2e6", "metadata": {}, "source": [ "Open using the `emit_xarray` function, with the `ortho=True` argument to orthorectify the image." ] }, { "cell_type": "code", "execution_count": null, "id": "edae75db", "metadata": {}, "outputs": [], "source": [ "emit_ds = emit_xarray(emit_file, ortho=True)\n", "emit_ds" ] }, { "cell_type": "code", "execution_count": null, "id": "441d2fbe", "metadata": {}, "outputs": [], "source": [ "# Set fill values to np.nan\n", "emit_ds['reflectance'].data[emit_ds['reflectance'].data==-9999] = np.nan" ] }, { "cell_type": "markdown", "id": "c4c2d2c7", "metadata": {}, "source": [ "Select the band closest to 850nm and visualize using `hvplot` like we did for AVIRIS-3 Data." ] }, { "cell_type": "code", "execution_count": null, "id": "f7ac41a8", "metadata": {}, "outputs": [], "source": [ "emit_850 = emit_ds['reflectance'].sel(wavelengths=850,method='nearest')" ] }, { "cell_type": "code", "execution_count": null, "id": "238b1c17", "metadata": {}, "outputs": [], "source": [ "emit_850.hvplot.image(x=\"longitude\", y=\"latitude\", cmap='viridis', frame_width=750, aspect='equal')" ] }, { "cell_type": "markdown", "id": "14af3569", "metadata": {}, "source": [ "## Contact Info: \n", "\n", "Email: LPDAAC@usgs.gov \n", "Voice: +1-866-573-3222 \n", "Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ \n", "Website: \n", "\n", "¹Work performed under USGS contract 140G0126D0001 for NASA contract NNG14HH33I. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.14" } }, "nbformat": 4, "nbformat_minor": 5 }