{ "cells": [ { "cell_type": "markdown", "id": "62602d2b-b3f8-4c92-966a-a8125081bd9b", "metadata": {}, "source": [ "# SHIFT V2 Reflectance Data Guide" ] }, { "cell_type": "markdown", "id": "f15d6101-dc0f-4e97-9ddb-7916c8e0172d", "metadata": {}, "source": [ "This guide explains how to use the SHIFT V2 Reflectance data. The data is stored in an S3 bucket, \n", "which is mounted to your SMCE computing environment. You can access the data at the following location: `s3/dh-shift-curated/aviris/v2`\n", "\n", "The SHIFT V2 Reflectance data has transitioned from ENVI files to NetCDF format. \n", "I have written a few functions to facilitate interaction with this new format. Below are some examples of how to work with the data." ] }, { "cell_type": "code", "execution_count": 3, "id": "59f5fa52-3570-4344-a21e-d390358e4f62", "metadata": { "tags": [] }, "outputs": [], "source": [ "import sys\n", "sys.path.append('s3/dh-shift-curated/aviris/utils')\n", "from shift_v2_utils import find_overlap_shift_v2, open_shift_v2_data" ] }, { "cell_type": "markdown", "id": "283e0ac5-3e98-4db0-bb77-1983b8d1d927", "metadata": {}, "source": [ "You can use the find overlap function to find flightlines that overlap a shapefile formated as a geopandas dataframe" ] }, { "cell_type": "code", "execution_count": 4, "id": "170abedb-1ca6-4612-93d1-bc107eb6b5a3", "metadata": { "tags": [] }, "outputs": [], "source": [ "import geopandas as gpd\n", "from shapely.geometry import Polygon\n", "\n", "polygon_1 = Polygon([\n", " (-119.8853015 , 34.42277795),\n", " (-119.86975941, 34.42312643),\n", " (-119.86921817, 34.4066284 ),\n", " (-119.88476322, 34.40623869),\n", " (-119.8853015 , 34.42277795)\n", "])\n", "\n", "polygon_2 = Polygon([\n", " (-119.8753015 , 34.41777795),\n", " (-119.85975941, 34.41812643),\n", " (-119.85921817, 34.4016284 ),\n", " (-119.87476322, 34.40123869),\n", " (-119.8753015 , 34.41777795)\n", "])\n", "\n", "# Create a GeoDataFrame\n", "gdf = gpd.GeoDataFrame({\n", " 'id': [1, 2],\n", " 'name': ['Polygon 1', 'Polygon 2'],\n", " 'geometry': [polygon_1, polygon_2]\n", "}, crs=\"EPSG:4326\")\n" ] }, { "cell_type": "markdown", "id": "7765849e-7ab3-4b7a-ba64-77c38c75286a", "metadata": {}, "source": [ "The overlap function takes a GeoDataFrame as an argument. The GeoDataFrame must have its CRS set to EPSG:4326. You can filter the overlapping flightlines by date. \n", "The date argument can be provided as a single string or as a list of strings," ] }, { "cell_type": "code", "execution_count": 5, "id": "7c5d7332-2662-462f-8764-f005f675bbad", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
| \n", " | index | \n", "flightline | \n", "date | \n", "path | \n", "geometry | \n", "index_right | \n", "id | \n", "name | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "306 | \n", "ang20220316t184402_001 | \n", "20220316 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220316/ang... | \n", "POLYGON ((-119.77003 34.38405, -119.76854 34.4... | \n", "1 | \n", "2 | \n", "Polygon 2 | \n", "
| 1 | \n", "307 | \n", "ang20220316t184402_002 | \n", "20220316 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220316/ang... | \n", "POLYGON ((-119.85591 34.39932, -119.85424 34.4... | \n", "1 | \n", "2 | \n", "Polygon 2 | \n", "
| 2 | \n", "1418 | \n", "ang20220914t170915_000 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.80109 34.37997, -119.79822 34.4... | \n", "1 | \n", "2 | \n", "Polygon 2 | \n", "
| 3 | \n", "1426 | \n", "ang20220914t171806_003 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.80085 34.31877, -119.79571 34.4... | \n", "1 | \n", "2 | \n", "Polygon 2 | \n", "
| 4 | \n", "1428 | \n", "ang20220914t172925_001 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.77325 34.38317, -119.77163 34.4... | \n", "1 | \n", "2 | \n", "Polygon 2 | \n", "
| 5 | \n", "1429 | \n", "ang20220914t172925_002 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.86728 34.39939, -119.86555 34.4... | \n", "1 | \n", "2 | \n", "Polygon 2 | \n", "
| 6 | \n", "307 | \n", "ang20220316t184402_002 | \n", "20220316 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220316/ang... | \n", "POLYGON ((-119.85591 34.39932, -119.85424 34.4... | \n", "0 | \n", "1 | \n", "Polygon 1 | \n", "
| 7 | \n", "1426 | \n", "ang20220914t171806_003 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.80085 34.31877, -119.79571 34.4... | \n", "0 | \n", "1 | \n", "Polygon 1 | \n", "
| 8 | \n", "1428 | \n", "ang20220914t172925_001 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.77325 34.38317, -119.77163 34.4... | \n", "0 | \n", "1 | \n", "Polygon 1 | \n", "
| 9 | \n", "1429 | \n", "ang20220914t172925_002 | \n", "20220914 | \n", "s3/dh-shift-curated/aviris/v2/L2a/20220914/ang... | \n", "POLYGON ((-119.86728 34.39939, -119.86555 34.4... | \n", "0 | \n", "1 | \n", "Polygon 1 | \n", "
<xarray.Dataset>\n",
"Dimensions: (easting: 2259, northing: 991, wavelength: 425)\n",
"Coordinates:\n",
" * easting (easting) float64 7.762e+05 ... 7.88e+05\n",
" * northing (northing) float64 3.816e+06 ... 3.811e+06\n",
" transverse_mercator int64 0\n",
" * wavelength (wavelength) float32 377.2 382.2 ... 2.501e+03\n",
"Data variables:\n",
" fwhm (wavelength) float32 dask.array<chunksize=(425,), meta=np.ndarray>\n",
" reflectance (wavelength, northing, easting) float32 dask.array<chunksize=(425, 991, 2259), meta=np.ndarray>\n",
" water_vapor (northing, easting) float32 dask.array<chunksize=(991, 2259), meta=np.ndarray>\n",
" aerosol_optical_thickness (northing, easting) float32 dask.array<chunksize=(991, 2259), meta=np.ndarray>\n",
"Attributes: (12/23)\n",
" Conventions: CF-1.6\n",
" date_created: 2024-12-30T18:52:40Z\n",
" summary: The Airborne Visible / Infrared Imagin...\n",
" keywords: Imaging Spectroscopy, AVIRIS, AVIRIS-NG\n",
" sensor: Airborne Visible / Infrared Imaging Sp...\n",
" instrument: AVIRIS-NG\n",
" ... ...\n",
" ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n",
" title: AVIRIS-NG L2A Surface reflectance (fli...\n",
" processing_level: L2A\n",
" time_coverage_start: 2022-09-14T17:33:37Z\n",
" time_coverage_end: 2022-09-14T17:33:37Z\n",
" product_version: 0<xarray.Dataset>\n",
"Dimensions: (wavelength: 425, easting: 334, northing: 470)\n",
"Coordinates:\n",
" transverse_mercator int64 0\n",
" * wavelength (wavelength) float32 377.2 382.2 ... 2.501e+03\n",
" * easting (easting) float64 7.863e+05 ... 7.88e+05\n",
" * northing (northing) float64 3.814e+06 ... 3.811e+06\n",
"Data variables:\n",
" fwhm (wavelength) float32 dask.array<chunksize=(425,), meta=np.ndarray>\n",
" reflectance (wavelength, northing, easting) float32 dask.array<chunksize=(425, 470, 334), meta=np.ndarray>\n",
" water_vapor (northing, easting) float32 dask.array<chunksize=(470, 334), meta=np.ndarray>\n",
" aerosol_optical_thickness (northing, easting) float32 dask.array<chunksize=(470, 334), meta=np.ndarray>\n",
"Attributes: (12/23)\n",
" Conventions: CF-1.6\n",
" date_created: 2024-12-30T18:52:40Z\n",
" summary: The Airborne Visible / Infrared Imagin...\n",
" keywords: Imaging Spectroscopy, AVIRIS, AVIRIS-NG\n",
" sensor: Airborne Visible / Infrared Imaging Sp...\n",
" instrument: AVIRIS-NG\n",
" ... ...\n",
" ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n",
" title: AVIRIS-NG L2A Surface reflectance (fli...\n",
" processing_level: L2A\n",
" time_coverage_start: 2022-09-14T17:33:37Z\n",
" time_coverage_end: 2022-09-14T17:33:37Z\n",
" product_version: 0<xarray.DataArray 'reflectance' (wavelength: 425, northing: 470, easting: 334)>\n",
"array([[[-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" ...,\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.]],\n",
"\n",
" [[-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" ...,\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.]],\n",
"\n",
" [[-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" ...,\n",
"...\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.]],\n",
"\n",
" [[-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" ...,\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.]],\n",
"\n",
" [[-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" ...,\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.],\n",
" [-9999., -9999., -9999., ..., -9999., -9999., -9999.]]],\n",
" dtype=float32)\n",
"Coordinates:\n",
" transverse_mercator int64 0\n",
" * wavelength (wavelength) float32 377.2 382.2 ... 2.501e+03\n",
" * easting (easting) float64 7.863e+05 7.863e+05 ... 7.88e+05\n",
" * northing (northing) float64 3.814e+06 3.814e+06 ... 3.811e+06\n",
"Attributes:\n",
" _QuantizeBitGroomNumberOfSignificantDigits: 5\n",
" long_name: Surface hemispherical direct...\n",
" grid_mapping: transverse_mercator\n",
" orthorectified: True\n",
" _FillValue: -9999.0