Zarr example Jupyter Notebook¶

The following Notebook requires the installation of:

aiohttp dask matplotlib python requests xarray zarr

It has been tested using the Jupyter Project Docker Stack image for jupyter/datascience

In [4]:

!pip install aiohttp dask matplotlib requests xarray zarr

!pip install aiohttp dask matplotlib requests xarray zarr

In [ ]:

# to run this cell uncomment the line below by removing the # 
#!mamba install -y -c conda-forge aiohttp dask matplotlib python=3.9.12 requests xarray zarr

# to run this cell uncomment the line below by removing the # #!mamba install -y -c conda-forge aiohttp dask matplotlib python=3.9.12 requests xarray zarr

Links to PANGEO¶

Project Website Links	Notes
PANGEO
PANGEO GitHub
PANGEO Forge
Ryan Abernathey's @SciPy2022 YouTube Recording	(video embedded below)

In [2]:

from IPython.lib.display import YouTubeVideo
YouTubeVideo('sY20UpYCAEE?t=853')

from IPython.lib.display import YouTubeVideo YouTubeVideo('sY20UpYCAEE?t=853')

Out[2]:

Import pre-processed GPCP data via PANGEO Forge¶

The following cell will pull the pre-processed NOAA global precipitation climate project (gpcp) from the Pangeo-forge object storage

Example Code: https://pangeo-forge.org/dashboard/feedstock/42

The xarray output will show the dataset information (~9226 days, ~2.2GB data)

In [1]:

import xarray as xr

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

import xarray as xr store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr' ds = xr.open_dataset(store, engine='zarr', chunks={}) ds

Out[1]:

<xarray.Dataset>
Dimensions:      (latitude: 180, nv: 2, longitude: 360, time: 9226)
Coordinates:
    lat_bounds   (latitude, nv) float32 dask.array<chunksize=(180, 2), meta=np.ndarray>
  * latitude     (latitude) float32 -90.0 -89.0 -88.0 -87.0 ... 87.0 88.0 89.0
    lon_bounds   (longitude, nv) float32 dask.array<chunksize=(360, 2), meta=np.ndarray>
  * longitude    (longitude) float32 0.0 1.0 2.0 3.0 ... 356.0 357.0 358.0 359.0
  * time         (time) datetime64[ns] 1996-10-01 1996-10-02 ... 2021-12-31
    time_bounds  (time, nv) datetime64[ns] dask.array<chunksize=(200, 2), meta=np.ndarray>
Dimensions without coordinates: nv
Data variables:
    precip       (time, latitude, longitude) float32 dask.array<chunksize=(200, 180, 360), meta=np.ndarray>
Attributes: (12/45)
    Conventions:                CF-1.6, ACDD 1.3
    Metadata_Conventions:       CF-1.6, Unidata Dataset Discovery v1.0, NOAA ...
    acknowledgment:             This project was supported in part by a grant...
    cdm_data_type:              Grid
    cdr_program:                NOAA Climate Data Record Program for satellit...
    cdr_variable:               precipitation
    ...                         ...
    standard_name_vocabulary:   CF Standard Name Table (v41, 22 February 2017)
    summary:                    Global Precipitation Climatology Project (GPC...
    time_coverage_duration:     P1D
    time_coverage_end:          1996-10-01T23:59:59Z
    time_coverage_start:        1996-10-01T00:00:00Z
    title:                      Global Precipitation Climatatology Project (G...

xarray.Dataset

• Dimensions:
  • latitude: 180
  • nv: 2
  • longitude: 360
  • time: 9226
• Coordinates: (6)
  • lat_bounds
    (latitude, nv)
    float32
    dask.array<chunksize=(180, 2), meta=np.ndarray>

    comment :

    latitude values at the north and south bounds of each pixel.

    Array Chunk Bytes 1.41 kiB 1.41 kiB Shape (180, 2) (180, 2) Count 2 Tasks 1 Chunks Type float32 numpy.ndarray

  • latitude
    (latitude)
    float32
    -90.0 -89.0 -88.0 ... 88.0 89.0

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    units :

    degrees_north

    valid_range :

    [-90.0, 90.0]

    array([-90., -89., -88., -87., -86., -85., -84., -83., -82., -81., -80., -79.,
           -78., -77., -76., -75., -74., -73., -72., -71., -70., -69., -68., -67.,
           -66., -65., -64., -63., -62., -61., -60., -59., -58., -57., -56., -55.,
           -54., -53., -52., -51., -50., -49., -48., -47., -46., -45., -44., -43.,
           -42., -41., -40., -39., -38., -37., -36., -35., -34., -33., -32., -31.,
           -30., -29., -28., -27., -26., -25., -24., -23., -22., -21., -20., -19.,
           -18., -17., -16., -15., -14., -13., -12., -11., -10.,  -9.,  -8.,  -7.,
            -6.,  -5.,  -4.,  -3.,  -2.,  -1.,   0.,   1.,   2.,   3.,   4.,   5.,
             6.,   7.,   8.,   9.,  10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,
            18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.,
            30.,  31.,  32.,  33.,  34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,
            42.,  43.,  44.,  45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,
            54.,  55.,  56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.,  64.,  65.,
            66.,  67.,  68.,  69.,  70.,  71.,  72.,  73.,  74.,  75.,  76.,  77.,
            78.,  79.,  80.,  81.,  82.,  83.,  84.,  85.,  86.,  87.,  88.,  89.],
          dtype=float32)

  • lon_bounds
    (longitude, nv)
    float32
    dask.array<chunksize=(360, 2), meta=np.ndarray>

    comment :

    longitude values at the west and east bounds of each pixel.

    Array Chunk Bytes 2.81 kiB 2.81 kiB Shape (360, 2) (360, 2) Count 2 Tasks 1 Chunks Type float32 numpy.ndarray

  • longitude
    (longitude)
    float32
    0.0 1.0 2.0 ... 357.0 358.0 359.0

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    units :

    degrees_east

    valid_range :

    [0.0, 360.0]

    array([  0.,   1.,   2., ..., 357., 358., 359.], dtype=float32)

  • time
    (time)
    datetime64[ns]
    1996-10-01 ... 2021-12-31

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['1996-10-01T00:00:00.000000000', '1996-10-02T00:00:00.000000000',
           '1996-10-03T00:00:00.000000000', ..., '2021-12-29T00:00:00.000000000',
           '2021-12-30T00:00:00.000000000', '2021-12-31T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • time_bounds
    (time, nv)
    datetime64[ns]
    dask.array<chunksize=(200, 2), meta=np.ndarray>

    comment :

    time bounds for each time value

    Array Chunk Bytes 144.16 kiB 3.12 kiB Shape (9226, 2) (200, 2) Count 48 Tasks 47 Chunks Type datetime64[ns] numpy.ndarray

• Data variables: (1)
  • precip
    (time, latitude, longitude)
    float32
    dask.array<chunksize=(200, 180, 360), meta=np.ndarray>

    cell_methods :

    area: mean time: mean

    long_name :

    NOAA Climate Data Record (CDR) of Daily GPCP Satellite-Gauge Combined Precipitation

    standard_name :

    lwe_precipitation_rate

    units :

    mm/day

    valid_range :

    [0.0, 100.0]

    Array Chunk Bytes 2.23 GiB 49.44 MiB Shape (9226, 180, 360) (200, 180, 360) Count 48 Tasks 47 Chunks Type float32 numpy.ndarray

• Attributes: (45)

  Conventions :

  CF-1.6, ACDD 1.3

  Metadata_Conventions :

  CF-1.6, Unidata Dataset Discovery v1.0, NOAA CDR v1.0, GDS v2.0

  acknowledgment :

  This project was supported in part by a grant from the NOAA Climate Data Record (CDR) Program for satellites.

  cdm_data_type :

  Grid

  cdr_program :

  NOAA Climate Data Record Program for satellites, FY 2011.

  cdr_variable :

  precipitation

  comment :

  Processing computer: eagle2.umd.edu

  contributor_name :

  Robert Adler, Jian-Jian Wang

  contributor_role :

  principalInvestigator, processor and custodian

  creator_email :

  jjwang@umd.edu

  creator_name :

  Dr. Jian-Jian Wang

  date_created :

  2017-05-30T16:52:42Z

  geospatial_lat_max :

  90.0

  geospatial_lat_min :

  -90.0

  geospatial_lat_resolution :

  1 degree

  geospatial_lat_units :

  degrees_north

  geospatial_lon_max :

  360.0

  geospatial_lon_min :

  0.0

  geospatial_lon_resolution :

  1 degree

  geospatial_lon_units :

  degrees_east

  history :

  1) 2017-05-30T16:52:42Z, Dr. Jian-Jian Wang, U of Maryland, Created beta (B1) file

  id :

  199610/gpcp_v01r03_daily_d19961001_c20170530.nc

  institution :

  ACADEMIC > UMD/ESSIC > Earth System Science Interdisciplinary Center, University of Maryland

  keywords :

  EARTH SCIENCE > ATMOSPHERE > PRECIPITATION > PRECIPITATION AMOUNT

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Earth Science Keywords, Version 7.0

  license :

  No constraints on data access or use.

  metadata_link :

  gov.noaa.ncdc:XXXXX

  naming_authority :

  gov.noaa.ncdc

  platform :

  GOES (Geostationary Operational Environmental Satellite), GMS (Japan Geostationary Meteorological Satellite), METEOSAT, TIROS > Television Infrared Observation Satellite, DMSP (Defense Meteorological Satellite Program)

  processing_level :

  NASA Level 3

  product_version :

  v01r03

  project :

  GPCP > Global Precipitation Climatology Project

  publisher_email :

  jjwang@umd.edu

  publisher_name :

  NOAA National Centers for Environmental Information (NCEI)

  publisher_url :

  https://www.ncei.noaa.gov

  references :

  Huffman et al. 1997, http://dx.doi.org/10.1175/1520-0477(1997)078<0005:TGPCPG>2.0.CO;2; Adler et al. 2003, http://dx.doi.org/10.1175/1525-7541(2003)004<1147:TVGPCP>2.0.CO;2; Huffman et al. 2009, http://dx.doi.org/10.1029/2009GL040000; Adler et al. 2017, Global Precipitation Climatology Project (GPCP) Daily Analysis: Climate Algorithm Theoretical Basis Document (C-ATBD)

  sensor :

  Imager, TOVS > TIROS Operational Vertical Sounder, SSMI > Special Sensor Microwave/Imager

  source :

  /data1/GPCP_CDR/GPCP_Output/1DD//bin/199610/stfsg3.19961001.s

  spatial_resolution :

  1 degree

  standard_name_vocabulary :

  CF Standard Name Table (v41, 22 February 2017)

  summary :

  Global Precipitation Climatology Project (GPCP) Daily Version 1.3 gridded, merged satellite/gauge precipitation Climate data Record (CDR) from 1996 to present.

  time_coverage_duration :

  P1D

  time_coverage_end :

  1996-10-01T23:59:59Z

  time_coverage_start :

  1996-10-01T00:00:00Z

  title :

  Global Precipitation Climatatology Project (GPCP) Climate Data Record (CDR), Daily V1.3

In [2]:

# display the precipitation field over the entire globe, using the first daily time step [0] in the dataset (calendar date = 1996-10-01)
# set display size to be 24 x 12
ds.precip[0].plot(figsize=(24,12))

# display the precipitation field over the entire globe, using the first daily time step [0] in the dataset (calendar date = 1996-10-01) # set display size to be 24 x 12 ds.precip[0].plot(figsize=(24,12))

Out[2]:

<matplotlib.collections.QuadMesh at 0x7fc17cb0f6a0>

In [3]:

# Add the Dask Progress Bar so we can watch how long it takes to work on the data
from dask.diagnostics import ProgressBar

# Add the Dask Progress Bar so we can watch how long it takes to work on the data from dask.diagnostics import ProgressBar

In [4]:

# There are corrupted values in the dataset, so we want to create a Mask to prevent those from being visualized
# set precipitation minimum to zero mm/day and less than 1000 mm/day
mask = (ds.precip >= 0 ) & (ds.precip < 1000)

# There are corrupted values in the dataset, so we want to create a Mask to prevent those from being visualized # set precipitation minimum to zero mm/day and less than 1000 mm/day mask = (ds.precip >= 0 ) & (ds.precip < 1000)

In [5]:

# using the Progress Bar
# calculate the climatology of global precipitation over the entire record, mean precipitation over time.
with ProgressBar():
    precip_clim = ds.precip.where(mask).mean('time').compute()

# using the Progress Bar # calculate the climatology of global precipitation over the entire record, mean precipitation over time. with ProgressBar(): precip_clim = ds.precip.where(mask).mean('time').compute()

[########################################] | 100% Completed | 23.29 s

In [6]:

# plot the mean climatology of precipitation over the entire globe
precip_clim.plot(figsize=(24,12))

# plot the mean climatology of precipitation over the entire globe precip_clim.plot(figsize=(24,12))

Out[6]:

<matplotlib.collections.QuadMesh at 0x7fc17c1889d0>

About the figure¶

This figure is the global precipitation of the entire earth over the last 26 years.

You can see the intertropical convergence zone (lighter yellows and greens), and areas in the tropics which experience high rainfall

Credits¶

Thanks to Ryan Abernathey for giving his talk at SciPy2022 and his live coding session.

Thanks to the PANGEO Funders and Contributors

---

Last update: 2022-11-15