Chapter-7: Create netCDF for compound data

The CF Conventions provided recommendations on netCDF structure for two types of complex DSGs, i.e. time series of profiles and trajectory of profiles.

Time Series of Profiles

• Time series of profiles at a single station (H.5.2): There is only one stations in a dataset; for this station, multiple profiles were measured at multiple times.

• Multidimensional array representations of time series profiles (H.5.1): There are multiple stations in a dataset; for each station, one or more profiles were measured at multiple times; and there are the same number of profiles for each station, and the same number of vertical levels for every profile, whereas the time values and the vertical level values can, but don’t have to be the same. For the situation where all time series share common time values, and all profiles share common vertical level values, we can use the representation in the example H.17 of Appendix H.5.1 for a more efficient use of storage space.

• Ragged array representation of time series profiles (H.5.3): There are multiple stations, the number of profiles and vertical levels for each station varies.

Trajectory of Profiles

• Profiles along a single trajectory (H.6.2)

• Multidimensional array representation of trajectory profiles (H.6.1): There’re multiple trajectories in the file, and there’re the same number of profiles for all trajectories and the same number of vertical levels for all profiles. This representation can also be used for trajectories with different number of profiles or vertical levels, at the cost of some wasted storage space due to missing values in the arrays.

• Ragged array representation of trajectory profiles (H.6.3): The number of profiles and the vertical levels for each trajectory varies.

Note

It is not always a good practice to pack all data into one single netCDF file. If the data structure becomes too complicated, it is often a better option to separate the features e.g. by stations and put them into different netCDF files.

For example, the Global Temperature and Salinity Profile Programme (GTSPP)[7] dataset from NOAA is a ragged array representation of trajectory of profiles trajectories of profiles (H.6.3). ↓

xarray.Dataset {
dimensions:
	trajectory = 371 ;
	profile = 132501 ;
	obs = 419569 ;

variables:
	object trajectory(trajectory) ;
		trajectory:cf_role = trajectory_id ;
		trajectory:comment = Constructed from org_type_platform_cruise ;
		trajectory:ioos_category = Identifier ;
		trajectory:long_name = Trajectory ID ;
	object org(trajectory) ;
		org:comment = From the first 2 characters of stream_ident:
Code  Meaning
AD  Australian Oceanographic Data Centre
AF  Argentina Fisheries (Fisheries Research and Development National Institute (INIDEP), Mar del Plata, Argentina
AO  Atlantic Oceanographic and Meteorological Lab
AP  Asia-Pacific (International Pacific Research Center/ Asia-Pacific Data-Research Center)
BI  BIO Bedford institute of Oceanography
CF  Canadian Navy
CS  CSIRO in Australia
DA  Dalhousie University
FN  FNOC in Monterey, California
FR  Orstom, Brest
FW  Fresh Water Institute (Winnipeg)
GE  BSH, Germany
IC  ICES
II  IIP
IK  Institut fur Meereskunde, Kiel
IM  IML
IO  IOS in Pat Bay, BC
JA  Japanese Meteorologocal Agency
JF  Japan Fisheries Agency
ME  EDS
MO  Moncton
MU  Memorial University
NA  NAFC
NO  NODC (Washington)
NW  US National Weather Service
OD  Old Dominion Univ, USA
RU  Russian Federation
SA  St Andrews
SI  Scripps Institute of Oceanography
SO  Southampton Oceanographic Centre, UK
TC  TOGA Subsurface Data Centre (France)
TI  Tiberon lab US
UB  University of BC
UQ  University of Quebec at Rimouski
VL  Far Eastern Regional Hydromet. Res. Inst. of V
WH  Woods Hole

from https://www.nodc.noaa.gov/GTSPP/document/codetbls/gtsppcode.html#ref006 ;
		org:ioos_category = Identifier ;
		org:long_name = Organization ;
	object type(trajectory) ;
		type:comment = From the 3rd and 4th characters of stream_ident:
Code  Meaning
AR  Animal mounted recorder
BA  BATHY message
BF  Undulating Oceanographic Recorder (e.g. Batfish CTD)
BO  Bottle
BT  general BT data
CD  CTD down trace
CT  CTD data, up or down
CU  CTD up trace
DB  Drifting buoy
DD  Delayed mode drifting buoy data
DM  Delayed mode version from originator
DT  Digital BT
IC  Ice core
ID  Interpolated drifting buoy data
IN  Ship intake samples
MB  MBT
MC  CTD and bottle data are mixed for the station
MI  Data from a mixed set of instruments
ML  Minilog
OF  Real-time oxygen and fluorescence
PF  Profiling float
RM  Radio message
RQ  Radio message with scientific QC
SC  Sediment core
SG  Thermosalinograph data
ST  STD data
SV  Sound velocity probe
TE  TESAC message
TG  Thermograph data
TK  TRACKOB message
TO  Towed CTD
TR  Thermistor chain
XB  XBT
XC  Expendable CTD

from https://www.nodc.noaa.gov/GTSPP/document/codetbls/gtsppcode.html#ref082 ;
		type:ioos_category = Identifier ;
		type:long_name = Data Type ;
	object platform(trajectory) ;
		platform:comment = See the list of platform codes (sorted in various ways) at https://www.nodc.noaa.gov/GTSPP/document/codetbls/calllist.html ;
		platform:ioos_category = Identifier ;
		platform:long_name = GTSPP Platform Code ;
		platform:references = https://www.nodc.noaa.gov/gtspp/document/codetbls/callist.html ;
	object cruise(trajectory) ;
		cruise:comment = Radio callsign + year for real time data, or NODC reference number for delayed mode data.  See
https://www.nodc.noaa.gov/GTSPP/document/codetbls/calllist.html .
'X' indicates a missing value.
Two or more adjacent spaces in the original cruise names have been compacted to 1 space. ;
		cruise:ioos_category = Identifier ;
		cruise:long_name = Cruise_ID ;
	float64 station_id(profile) ;
		station_id:actual_range = [54396260 55104400] ;
		station_id:cf_role = profile_id ;
		station_id:comment = Identification number of the station (profile) in the GTSPP Continuously Managed Database ;
		station_id:ioos_category = Identifier ;
		station_id:long_name = Station ID Number ;
	float32 longitude(profile) ;
		longitude:_CoordinateAxisType = Lon ;
		longitude:actual_range = [-179.9    178.384] ;
		longitude:axis = X ;
		longitude:C_format = %9.4f ;
		longitude:colorBarMaximum = 180.0 ;
		longitude:colorBarMinimum = -180.0 ;
		longitude:epic_code = 502 ;
		longitude:FORTRAN_format = F9.4 ;
		longitude:ioos_category = Location ;
		longitude:long_name = Longitude ;
		longitude:standard_name = longitude ;
		longitude:units = degrees_east ;
		longitude:valid_max = 180.0 ;
		longitude:valid_min = -180.0 ;
	float32 latitude(profile) ;
		latitude:_CoordinateAxisType = Lat ;
		latitude:actual_range = [-68.69   83.583] ;
		latitude:axis = Y ;
		latitude:C_format = %8.4f ;
		latitude:colorBarMaximum = 90.0 ;
		latitude:colorBarMinimum = -90.0 ;
		latitude:epic_code = 500 ;
		latitude:FORTRAN_format = F8.4 ;
		latitude:ioos_category = Location ;
		latitude:long_name = Latitude ;
		latitude:standard_name = latitude ;
		latitude:units = degrees_north ;
		latitude:valid_max = 90.0 ;
		latitude:valid_min = -90.0 ;
	datetime64[ns] time(profile) ;
		time:_CoordinateAxisType = Time ;
		time:actual_range = [1.71322560e+09 1.71452124e+09] ;
		time:axis = T ;
		time:ioos_category = Time ;
		time:long_name = Time ;
		time:standard_name = time ;
		time:time_origin = 01-JAN-1970 00:00:00 ;
	int32 trajectoryIndex(profile) ;
		trajectoryIndex:instance_dimension = trajectory ;
		trajectoryIndex:ioos_category = Identifier ;
		trajectoryIndex:long_name = The trajectory to which this profile is associated. ;
	int32 rowSize(profile) ;
		rowSize:ioos_category = Identifier ;
		rowSize:long_name = Number of Observations for this Profile ;
		rowSize:sample_dimension = obs ;
	float32 depth(obs) ;
		depth:_CoordinateAxisType = Height ;
		depth:_CoordinateZisPositive = down ;
		depth:actual_range = [   0. 4477.] ;
		depth:axis = Z ;
		depth:C_format = %6.2f ;
		depth:colorBarMaximum = 5000.0 ;
		depth:colorBarMinimum = 0.0 ;
		depth:epic_code = 3 ;
		depth:FORTRAN_format = F6.2 ;
		depth:ioos_category = Location ;
		depth:long_name = Depth of the Observations ;
		depth:positive = down ;
		depth:standard_name = depth ;
		depth:units = m ;
	float32 temperature(obs) ;
		temperature:actual_range = [-2.  35.5] ;
		temperature:C_format = %9.4f ;
		temperature:cell_methods = time: point longitude: point latitude: point depth: point ;
		temperature:colorBarMaximum = 32.0 ;
		temperature:colorBarMinimum = 0.0 ;
		temperature:epic_code = 28 ;
		temperature:FORTRAN_format = F9.4 ;
		temperature:ioos_category = Temperature ;
		temperature:long_name = Sea Water Temperature ;
		temperature:standard_name = sea_water_temperature ;
		temperature:units = degree_C ;
	float32 salinity(obs) ;
		salinity:actual_range = [ 0.  39.2] ;
		salinity:C_format = %9.4f ;
		salinity:cell_methods = time: point longitude: point latitude: point depth: point ;
		salinity:colorBarMaximum = 37.0 ;
		salinity:colorBarMinimum = 32.0 ;
		salinity:epic_code = 41 ;
		salinity:FORTRAN_format = F9.4 ;
		salinity:ioos_category = Salinity ;
		salinity:long_name = Practical Salinity ;
		salinity:salinity_scale = PSU ;
		salinity:standard_name = sea_water_practical_salinity ;
		salinity:units = PSU ;

// global attributes:
	:acknowledgment = These data were acquired from the US NOAA National Oceanographic Data Center (NODC) on 2024-09-11 from https://www.nodc.noaa.gov/GTSPP/. ;
	:cdm_altitude_proxy = depth ;
	:cdm_data_type = TrajectoryProfile ;
	:cdm_profile_variables = station_id, longitude, latitude, time ;
	:cdm_trajectory_variables = trajectory, org, type, platform, cruise ;
	:Conventions = COARDS, WOCE, GTSPP, CF-1.10, ACDD-1.3 ;
	:creator_email = nodc.gtspp@noaa.gov ;
	:creator_name = NOAA NESDIS NODC (IN295) ;
	:creator_url = https://www.nodc.noaa.gov/GTSPP/ ;
	:crs = EPSG:4326 ;
	:defaultGraphQuery = longitude,latitude,station_id&time%3E=max(time)-7days&time%3C=max(time)&.draw=markers&.marker=10|5 ;
	:Easternmost_Easting = 178.38400268554688 ;
	:featureType = TrajectoryProfile ;
	:file_source = The GTSPP Continuously Managed Data Base ;
	:geospatial_lat_max = 83.58300018310547 ;
	:geospatial_lat_min = -68.69000244140625 ;
	:geospatial_lat_units = degrees_north ;
	:geospatial_lon_max = 178.38400268554688 ;
	:geospatial_lon_min = -179.89999389648438 ;
	:geospatial_lon_units = degrees_east ;
	:geospatial_vertical_max = 4477.0 ;
	:geospatial_vertical_min = 0.0 ;
	:geospatial_vertical_positive = down ;
	:geospatial_vertical_units = m ;
	:gtspp_ConventionVersion = GTSPP4.0 ;
	:gtspp_handbook_version = GTSPP Data User's Manual 1.0 ;
	:gtspp_program = writeGTSPPnc40.f90 ;
	:gtspp_programVersion = 1.8 ;
	:history = 2024-08-31 csun writeGTSPPnc40.f90 Version 1.8
.tgz files from ftp.nodc.noaa.gov /pub/data.nodc/gtspp/bestcopy/netcdf (https://www.nodc.noaa.gov/GTSPP/)
2024-08-31 Most recent ingest, clean, and reformat at ERD (erd.data at noaa.gov).
2024-09-11T19:21:22Z (local files)
2024-09-11T19:21:22Z http://localhost:8080/erddap/tabledap/erdGtsppBestNc.ncCF?&time>=2024-04-16&time<2024-05-01&orderBy("trajectory,station_id,time,depth") ;
	:id = erdGtsppBest ;
	:infoUrl = https://www.nodc.noaa.gov/GTSPP/ ;
	:institution = NOAA NODC ;
	:keywords = cruise, data, density, depth, Earth Science > Oceans > Ocean Temperature > Water Temperature, Earth Science > Oceans > Salinity/Density > Salinity, global, gtspp, identifier, noaa, nodc, observation, ocean, oceans, organization, profile, program, salinity, sea, sea_water_practical_salinity, sea_water_temperature, seawater, station, temperature, temperature-salinity, time, type, water ;
	:keywords_vocabulary = NODC Data Types, CF Standard Names, GCMD Science Keywords ;
	:LEXICON = NODC_GTSPP ;
	:license = These data are openly available to the public.  Please acknowledge the use of these data with:
These data were acquired from the US NOAA National Oceanographic Data Center (NODC) on 2024-09-11 from https://www.nodc.noaa.gov/GTSPP/.

The data may be used and redistributed for free but is not intended
for legal use, since it may contain inaccuracies. Neither the data
Contributor, ERD, NOAA, nor the United States Government, nor any
of their employees or contractors, makes any warranty, express or
implied, including warranties of merchantability and fitness for a
particular purpose, or assumes any legal liability for the accuracy,
completeness, or usefulness, of this information. ;
	:naming_authority = gov.noaa.nodc ;
	:Northernmost_Northing = 83.58300018310547 ;
	:project = Joint IODE/JCOMM Global Temperature-Salinity Profile Programme ;
	:references = https://www.nodc.noaa.gov/GTSPP/ ;
	:sourceUrl = (local files) ;
	:Southernmost_Northing = -68.69000244140625 ;
	:standard_name_vocabulary = CF Standard Name Table v70 ;
	:subsetVariables = trajectory, org, type, platform, cruise ;
	:summary = The Global Temperature-Salinity Profile Programme (GTSPP) develops and maintains a global ocean temperature and salinity resource with data that are both up-to-date and of the highest quality. It is a joint World Meteorological Organization (WMO) and Intergovernmental Oceanographic Commission (IOC) program.  It includes data from XBTs, CTDs, moored and drifting buoys, and PALACE floats. For information about organizations contributing data to GTSPP, see http://gosic.org/goos/GTSPP-data-flow.htm .  The U.S. National Oceanographic Data Center (NODC) maintains the GTSPP Continuously Managed Data Base and releases new 'best-copy' data once per month.

WARNING: This dataset has a *lot* of data.  If you request too much data, your request will fail.
* If you don't specify a longitude and latitude bounding box, don't request more than a month's data at a time.
* If you do specify a longitude and latitude bounding box, you can request data for a proportionally longer time period.
Requesting data for a specific station_id may be slow, but it works.

*** This ERDDAP dataset has data for the entire world for all available times (currently, up to and including the August 2024 data) but is a subset of the original NODC 'best-copy' data.  It only includes data where the quality flags indicate the data is 1=CORRECT, 2=PROBABLY GOOD, or 5=MODIFIED. It does not include some of the metadata, any of the history data, or any of the quality flag data of the original dataset. You can always get the complete, up-to-date dataset (and additional, near-real-time data) from the source: https://www.nodc.noaa.gov/GTSPP/ .  Specific differences are:
* Profiles with a position_quality_flag or a time_quality_flag other than 1|2|5 were removed.
* Rows with a depth (z) value less than -0.4 or greater than 10000 or a z_variable_quality_flag other than 1|2|5 were removed.
* Temperature values less than -4 or greater than 40 or with a temperature_quality_flag other than 1|2|5 were set to NaN.
* Salinity values less than 0 or greater than 41 or with a salinity_quality_flag other than 1|2|5 were set to NaN.
* Time values were converted from "days since 1900-01-01 00:00:00" to "seconds since 1970-01-01T00:00:00".

See the Quality Flag definitions on page 5 and "Table 2.1: Global Impossible Parameter Values" on page 61 of
https://www.nodc.noaa.gov/GTSPP/document/qcmans/GTSPP_RT_QC_Manual_20090916.pdf .
The Quality Flag definitions are also at
https://www.nodc.noaa.gov/GTSPP/document/qcmans/qcflags.htm . ;
	:testOutOfDate = now-45days ;
	:time_coverage_end = 2024-04-30T23:54:00Z ;
	:time_coverage_start = 2024-04-16T00:00:00Z ;
	:title = Global Temperature and Salinity Profile Programme (GTSPP) Data, 1985-present ;
	:Westernmost_Easting = -179.89999389648438 ;
}

Exercise: create a netCDF for time series of profiles

Now, let’s practice creating a netCDF for time series of profiles with the Newport Lab CTD Casts dataset. Each original file contains a single profile at a time, with measurements of sea water temperature, salinity, density, and fluorescence in multiple water depths. Since each profile has different number of water depth levels, we’ll compose the netCDF in a ragged array representation of time series profiles (H.5.3). You could randomly download a few original files to do this exercise; if you encounter any difficulties, feel free to have a look at the provided sample solution. The sample solution used three datasets, 010903NH05.nc, 011207NH05.nc and 061300NH2.nc.

import os
from glob import glob
import numpy as np
import pandas as pd
import xarray as xr
import cftime
from datetime import datetime

# List available datasets. Please change it to your file path.
tsP_files = sorted(glob(os.path.join(os.getcwd(), "dsg_tsProfile", "*.nc")))

# Read both files
ds1 = xr.open_dataset(tsP_files[0], decode_times=False)
ds2 = xr.open_dataset(tsP_files[1], decode_times=False)
ds3 = xr.open_dataset(tsP_files[2], decode_times=False)

# By printing the xarray dataset, we can inspect the structure of the source data.
ds1

<xarray.Dataset> Size: 5kB
Dimensions:            (row: 43)
Dimensions without coordinates: row
Data variables: (12/14)
    line               (row) |S2 86B ...
    longitude          (row) float64 344B ...
    latitude           (row) float64 344B ...
    station_code       (row) |S10 430B ...
    time               (row) float64 344B ...
    station            (row) |S4 172B ...
    ...                 ...
    temperature        (row) float64 344B ...
    salinity           (row) float64 344B ...
    density            (row) float64 344B ...
    fluorescence       (row) float64 344B ...
    project            (row) |S2 86B ...
    transect           (row) |S20 860B ...
Attributes:
    id:                    010903NH05
    observationDimension:  row

xarray.Dataset

• Dimensions:
  • row: 43
• Coordinates: (0)
• Data variables: (14)
  • line
    (row)
    |S2
    ...

    [43 values with dtype=|S2]

  • longitude
    (row)
    float64
    ...

    units :

    degrees_east

    [43 values with dtype=float64]

  • latitude
    (row)
    float64
    ...

    units :

    degrees_north

    [43 values with dtype=float64]

  • station_code
    (row)
    |S10
    ...

    [43 values with dtype=|S10]

  • time
    (row)
    float64
    ...

    units :

    seconds since 1970-01-01T00:00:00Z

    [43 values with dtype=float64]

  • station
    (row)
    |S4
    ...

    [43 values with dtype=|S4]

  • plain_station
    (row)
    |S4
    ...

    description :

    The station without the suffix.

    [43 values with dtype=|S4]

  • depth_or_pressure
    (row)
    float64
    ...

    units :

    meters

    [43 values with dtype=float64]

  • temperature
    (row)
    float64
    ...

    units :

    degree_C

    [43 values with dtype=float64]

  • salinity
    (row)
    float64
    ...

    units :

    psu

    [43 values with dtype=float64]

  • density
    (row)
    float64
    ...

    units :

    sigma

    [43 values with dtype=float64]

  • fluorescence
    (row)
    float64
    ...

    units :

    volts

    [43 values with dtype=float64]

  • project
    (row)
    |S2
    ...

    [43 values with dtype=|S2]

  • transect
    (row)
    |S20
    ...

    [43 values with dtype=|S20]

• Indexes: (0)
• Attributes: (2)

  id :

  010903NH05

  observationDimension :

  row

# By inspecting the longitudes, latitudes and time of the observations,
# we know that one source netCDF file only contains a single profile at a time for one station.
print(ds1.longitude.data)
print(ds1.latitude.data)
print(ds1.time.data)

[-124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686 -124.12999686 -124.12999686
 -124.12999686 -124.12999686 -124.12999686]
[44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887 44.65169887
 44.65169887]
[1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09 1.0420992e+09
 1.0420992e+09 1.0420992e+09 1.0420992e+09]

# Get the station id of all stations
station1 = np.unique(ds1.station.data)
station2 = np.unique(ds2.station.data)
station3 = np.unique(ds3.station.data)
station_index = np.concatenate([station1, station2, station3])
station = np.unique(station_index)
print("The name of each station is: ", station)

# Get the longitudes of all stations
lon1 = np.unique(ds1.longitude.data)
lon2 = np.unique(ds2.longitude.data)
lon3 = np.unique(ds3.longitude.data)
lon = np.unique(np.concatenate([lon1, lon2, lon3]))
print("The longitude of each station is: ", lon)

# Get the latitudes of all stations
lat1 = np.unique(ds1.latitude.data)
lat2 = np.unique(ds2.latitude.data)
lat3 = np.unique(ds3.latitude.data)
lat = np.unique(np.concatenate([lat1, lat2, lat3]))
# Because the latitudes of all stations are same,
# we need to make the length of latitude array match the number of stations.
lat = np.repeat(lat, len(station))
print("The latitude of each station is: ", lat)

The name of each station is:  [b'NH03' b'NH15']
The longitude of each station is:  [-124.41169686 -124.12999686]
The latitude of each station is:  [44.65169887 44.65169887]

# Get the IDs of all profiles
profID1 = np.unique(ds1.station_code.data)
profID2 = np.unique(ds2.station_code.data)
profID3 = np.unique(ds3.station_code.data)
profID = np.concatenate([profID1, profID2, profID3])
print("The ID of each profile is: ", profID)

# Get the number of observations for all profile
rowSize1 = ds1.sizes['row']
rowSize2 = ds2.sizes['row']
rowSize3 = ds3.sizes['row']
rowSize = [rowSize1, rowSize2, rowSize3]
print("The number of observations for each profile is: ", rowSize)

The ID of each profile is:  [b'010903NH03' b'011207NH03' b'061300NH15a']
The number of observations for each profile is:  [43, 46, 83]

# Get time of each profile
time1 = np.unique(ds1.time.data)
time2 = np.unique(ds2.time.data)
time3 = np.unique(ds3.time.data)
time = np.concatenate([time1, time2, time3])
print("The time of each profile is: ", time)

# Get the time unit
time_unit = ds1.time.units
print("The time unit is: ", time_unit)

The time of each profile is:  [1.04209920e+09 1.16863812e+09 9.60945600e+08]
The time unit is:  seconds since 1970-01-01T00:00:00Z

# Get the data variables (temperature, density, salinity, fluorenscence) from each profile,
# and merge them in order into a 1D array
temp = np.concatenate([ds1.temperature.data, ds2.temperature.data, ds3.temperature.data])
density = np.concatenate([ds1.density.data, ds2.density.data, ds3.density.data])
salinity = np.concatenate([ds1.salinity.data, ds2.salinity.data, ds3.salinity.data])
fluorescence = np.concatenate(
    [ds1.fluorescence.data, ds2.fluorescence.data, ds3.fluorescence.data]
)
print("The total number of observations for all profiles is: ", len(temp))

# Get the water depths from each profile, and merge them in order into a 1D array
depth = np.concatenate(
    [ds1.depth_or_pressure.data, ds2.depth_or_pressure.data, ds3.depth_or_pressure.data]
)

The total number of observations for all profiles is:  172

# Compose the netCDF dataset
ds = xr.Dataset(
    coords=(
        {
            "lon": (
                ["station"],
                lon,
                {"standard_name": "longitude", "units": "degrees_east"},
            ),
            "lat": (
                ["station"],
                lat,
                {"standard_name": "latitude", "units": "degrees_north"},
            ),
            "station_name": (
                ["station"],
                station,
                {"long_name": "station name", "cf_role": "timeseries_id"},
            ),
            "profile": (
                ["profile"],
                profID,
                {"long_name": "Profile ID", "cf_role": "profile_id"},
            ),
            "station_index": (
                ["profile"],
                station_index,
                {
                    "long_name": "which station this profile is for",
                    "instance_dimension": "station",
                },
            ),
            "row_size": (
                ["profile"],
                rowSize,
                {
                    "long_name": "number of observations for this profile",
                    "sample_dimension": "obs",
                },
            ),
            "time": (["profile"], time, {"standard_name": "time", "units": time_unit}),
            "depth": (
                ["obs"],
                depth,
                {"long_name": "Depth", "units": "m", "axis": "Z"},
            ),
        }
    ),
    data_vars=(
        {
            "density": (
                ["obs"],
                density,
                {"standard_name": "sea_water_density", "units": "sigma"},
            ),
            "fluorescence": (
                ["obs"],
                fluorescence,
                {"long_name": "Fluorescence", "units": "volts"},
            ),
            "salinity": (
                ["obs"],
                salinity,
                {"standard_name": "sea_water_practical_salinity", "units": "PSU"},
            ),
            "temperature": (
                ["obs"],
                temp,
                {"standard_name": "sea_water_temperature", "units": "degree_C"},
            ),
        }
    ),
    attrs={
        "featureType": "TimeSeriesProfile",
        "Conventions": "CF-1.11",
        "more attributes available at": "https://coastwatch.pfeg.noaa.gov/erddap/tabledap/erdNewportCtd.html"
    },
)

ds

<xarray.Dataset> Size: 7kB
Dimensions:        (obs: 172, station: 2, profile: 3)
Coordinates:
    lon            (station) float64 16B -124.4 -124.1
    lat            (station) float64 16B 44.65 44.65
    station_name   (station) |S4 8B b'NH03' b'NH15'
  * profile        (profile) |S11 33B b'010903NH03' b'011207NH03' b'061300NH15a'
    station_index  (profile) |S4 12B b'NH03' b'NH03' b'NH15'
    row_size       (profile) int64 24B 43 46 83
    time           (profile) float64 24B 1.042e+09 1.169e+09 9.609e+08
    depth          (obs) float64 1kB 1.0 2.0 3.0 4.0 5.0 ... 81.0 82.0 83.0 84.0
Dimensions without coordinates: obs, station
Data variables:
    density        (obs) float64 1kB 23.0 23.01 23.01 ... 26.19 26.19 26.19
    fluorescence   (obs) float64 1kB nan nan nan nan ... 0.163 0.16 0.16 0.164
    salinity       (obs) float64 1kB 30.11 30.12 30.12 ... 33.64 33.64 33.64
    temperature    (obs) float64 1kB 10.83 10.83 10.83 ... 8.078 8.079 8.077
Attributes:
    featureType:                   TimeSeriesProfile
    Conventions:                   CF-1.11
    more attributes available at:  https://coastwatch.pfeg.noaa.gov/erddap/ta...

xarray.Dataset

• Dimensions:
  • obs: 172
  • station: 2
  • profile: 3
• Coordinates: (8)
  • lon
    (station)
    float64
    -124.4 -124.1

    standard_name :

    longitude

    units :

    degrees_east

    array([-124.41169686, -124.12999686])

  • lat
    (station)
    float64
    44.65 44.65

    standard_name :

    latitude

    units :

    degrees_north

    array([44.65169887, 44.65169887])

  • station_name
    (station)
    |S4
    b'NH03' b'NH15'

    long_name :

    station name

    cf_role :

    timeseries_id

    array([b'NH03', b'NH15'], dtype='|S4')

  • profile
    (profile)
    |S11
    b'010903NH03' ... b'061300NH15a'

    long_name :

    Profile ID

    cf_role :

    profile_id

    array([b'010903NH03', b'011207NH03', b'061300NH15a'], dtype='|S11')

  • station_index
    (profile)
    |S4
    b'NH03' b'NH03' b'NH15'

    long_name :

    which station this profile is for

    instance_dimension :

    station

    array([b'NH03', b'NH03', b'NH15'], dtype='|S4')

  • row_size
    (profile)
    int64
    43 46 83

    long_name :

    number of observations for this profile

    sample_dimension :

    obs

    array([43, 46, 83])

  • time
    (profile)
    float64
    1.042e+09 1.169e+09 9.609e+08

    standard_name :

    time

    units :

    seconds since 1970-01-01T00:00:00Z

    array([1.04209920e+09, 1.16863812e+09, 9.60945600e+08])

  • depth
    (obs)
    float64
    1.0 2.0 3.0 4.0 ... 82.0 83.0 84.0

    long_name :

    Depth

    units :

    m

    axis :

    Z

    array([ 0.99999997,  1.99999995,  2.99999992,  3.9999999 ,  4.99999987,
            5.99999985,  6.99999982,  7.9999998 ,  8.99999977,  9.99999975,
           10.99999972, 11.9999997 , 12.99999967, 13.99999965, 14.99999962,
           15.9999996 , 16.99999957, 17.99999955, 18.99999952, 19.99999949,
           20.99999947, 21.99999944, 22.99999942, 23.99999939, 24.99999937,
           25.99999934, 26.99999932, 27.99999929, 28.99999927, 29.99999924,
           30.99999922, 31.99999919, 32.99999917, 33.99999914, 34.99999912,
           35.99999909, 36.99999907, 37.99999904, 38.99999901, 39.99999899,
           40.99999896, 41.99999894, 42.99999891,  0.99999997,  1.99999995,
            2.99999992,  3.9999999 ,  4.99999987,  5.99999985,  6.99999982,
            7.9999998 ,  8.99999977,  9.99999975, 10.99999972, 11.9999997 ,
           12.99999967, 13.99999965, 14.99999962, 15.9999996 , 16.99999957,
           17.99999955, 18.99999952, 19.99999949, 20.99999947, 21.99999944,
           22.99999942, 23.99999939, 24.99999937, 25.99999934, 26.99999932,
           27.99999929, 28.99999927, 29.99999924, 30.99999922, 31.99999919,
           32.99999917, 33.99999914, 34.99999912, 35.99999909, 36.99999907,
           37.99999904, 38.99999901, 39.99999899, 40.99999896, 41.99999894,
           42.99999891, 43.99999889, 44.99999886, 45.99999884,  1.99999995,
            2.99999992,  3.9999999 ,  4.99999987,  5.99999985,  6.99999982,
            7.9999998 ,  8.99999977,  9.99999975, 10.99999972, 11.9999997 ,
           12.99999967, 13.99999965, 14.99999962, 15.9999996 , 16.99999957,
           17.99999955, 18.99999952, 19.99999949, 20.99999947, 21.99999944,
           22.99999942, 23.99999939, 24.99999937, 25.99999934, 26.99999932,
           27.99999929, 28.99999927, 29.99999924, 30.99999922, 31.99999919,
           32.99999917, 33.99999914, 34.99999912, 35.99999909, 36.99999907,
           37.99999904, 38.99999901, 39.99999899, 40.99999896, 41.99999894,
           42.99999891, 43.99999889, 44.99999886, 45.99999884, 46.99999881,
           47.99999879, 48.99999876, 49.99999874, 50.99999871, 51.99999869,
           52.99999866, 53.99999864, 54.99999861, 55.99999859, 56.99999856,
           57.99999853, 58.99999851, 59.99999848, 60.99999846, 61.99999843,
           62.99999841, 63.99999838, 64.99999836, 65.99999833, 66.99999831,
           67.99999828, 68.99999826, 69.99999823, 70.99999821, 71.99999818,
           72.99999816, 73.99999813, 74.99999811, 75.99999808, 76.99999805,
           77.99999803, 78.999998  , 79.99999798, 80.99999795, 81.99999793,
           82.9999979 , 83.99999788])

• Data variables: (4)
  • density
    (obs)
    float64
    23.0 23.01 23.01 ... 26.19 26.19

    standard_name :

    sea_water_density

    units :

    sigma

    array([22.99989942, 23.00589942, 23.00879942, 23.03959942, 23.11199942,
           23.20479941, 23.24349941, 23.25499941, 23.26829941, 23.30569941,
           23.35349941, 23.42619941, 23.54959941, 23.6774994 , 23.8019994 ,
           23.8885994 , 23.9317994 , 23.9472994 , 23.95819939, 23.97419939,
           24.00399939, 24.02739939, 24.03999939, 24.05149939, 24.06279939,
           24.07159939, 24.08639939, 24.10709939, 24.13379939, 24.15269939,
           24.16649939, 24.18899939, 24.20679939, 24.22459939, 24.26109939,
           24.31109939, 24.35009938, 24.37859938, 24.39069938, 24.39469938,
           24.39759938, 24.40379938, 24.41339938, 23.08519942, 23.08499942,
           23.08529942, 23.08639942, 23.09279942, 23.12479942, 23.22849941,
           23.42989941, 23.6787994 , 23.8074994 , 23.9210994 , 24.05469939,
           24.19249939, 24.38339938, 24.50679938, 24.54629938, 24.67249938,
           24.77239937, 24.84509937, 24.91979937, 24.95259937, 24.96269937,
           24.97929937, 24.98909937, 25.00569937, 25.02589937, 25.05919937,
           25.07949937, 25.09429937, 25.12649937, 25.15689936, 25.16049936,
           25.17109936, 25.19309936, 25.20339936, 25.21499936, 25.21939936,
           25.22329936, 25.22959936, 25.23469936, 25.23599936, 25.25219936,
           25.29179936, 25.32809936, 25.33249936, 25.33269936, 23.23999941,
           23.23999941, 23.23999941, 23.23999941, 23.23999941, 23.23999941,
           23.23999941, 23.23999941, 23.23999941, 23.23999941, 23.24099941,
           23.24199941, 23.26899941, 23.39999941, 23.5649994 , 23.6649994 ,
           23.7059994 , 23.8859994 , 24.02999939, 24.03999939, 24.16399939,
           24.31199939, 24.41899938, 24.48899938, 24.55899938, 24.63499938,
           24.69199938, 24.71499938, 24.88899937, 25.07999937, 25.17899936,
           25.26199936, 25.35899936, 25.41199936, 25.45899936, 25.51899936,
           25.57499935, 25.60799935, 25.64699935, 25.67899935, 25.68099935,
           25.68299935, 25.69099935, 25.72199935, 25.75899935, 25.79899935,
           25.83699935, 25.84999935, 25.86799935, 25.87199935, 25.88099935,
           25.89499935, 25.90399935, 25.91599935, 25.94599934, 25.96699934,
           25.98999934, 26.00399934, 26.01299934, 26.02999934, 26.03799934,
           26.03999934, 26.04599934, 26.06299934, 26.11699934, 26.13199934,
           26.13699934, 26.14099934, 26.14299934, 26.14499934, 26.14599934,
           26.14599934, 26.14899934, 26.15399934, 26.15199934, 26.15699934,
           26.17099934, 26.18599934, 26.18999934, 26.19099934, 26.19399934,
           26.19399934, 26.19499934])

  • fluorescence
    (obs)
    float64
    nan nan nan nan ... 0.16 0.16 0.164

    long_name :

    Fluorescence

    units :

    volts

    array([       nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan,        nan,        nan,
                  nan,        nan,        nan, 0.1199    , 0.1186    ,
           0.1273    , 0.1376    , 0.1542    , 0.163     , 0.1599    ,
           0.1484    , 0.1363    , 0.131     , 0.133     , 0.1255    ,
           0.1152    , 0.1081    , 0.104     , 0.1011    , 0.1022    ,
           0.0984    , 0.0949    , 0.094     , 0.0896    , 0.0875    ,
           0.0912    , 0.0919    , 0.0932    , 0.094     , 0.0952    ,
           0.0947    , 0.0944    , 0.0934    , 0.09      , 0.0891    ,
           0.0874    , 0.0873    , 0.0867    , 0.0861    , 0.0854    ,
           0.0864    , 0.0891    , 0.0916    , 0.0921    , 0.0915    ,
           0.0942    , 0.101     , 0.1126    , 0.1256    , 0.44499999,
           0.44099999, 0.44699999, 0.44699999, 0.44699999, 0.44299999,
           0.44099999, 0.44599999, 0.44899999, 0.45299999, 0.45499999,
           0.45799999, 0.47499999, 0.51499999, 0.56599999, 0.61699998,
           0.63499998, 0.65299998, 0.68299998, 0.69199998, 0.68899998,
           0.70399998, 0.66899998, 0.69999998, 0.73699998, 0.76699998,
           0.78299998, 0.79899998, 0.79699998, 0.80199998, 0.77099998,
           0.71299998, 0.65099998, 0.56199999, 0.46899999, 0.41599999,
           0.30899999, 0.24199999, 0.22799999, 0.22999999, 0.22699999,
           0.22599999, 0.22699999, 0.21999999, 0.20599999, 0.21099999,
           0.192     , 0.185     , 0.186     , 0.184     , 0.179     ,
           0.176     , 0.176     , 0.171     , 0.171     , 0.171     ,
           0.165     , 0.165     , 0.162     , 0.16      , 0.159     ,
           0.164     , 0.163     , 0.16      , 0.163     , 0.171     ,
           0.18      , 0.181     , 0.176     , 0.176     , 0.179     ,
           0.177     , 0.171     , 0.171     , 0.172     , 0.171     ,
           0.167     , 0.167     , 0.167     , 0.163     , 0.16      ,
           0.16      , 0.164     ])

  • salinity
    (obs)
    float64
    30.11 30.12 30.12 ... 33.64 33.64

    standard_name :

    sea_water_practical_salinity

    units :

    PSU

    array([30.10949924, 30.11789924, 30.12189924, 30.16189924, 30.25889924,
           30.38579923, 30.44199923, 30.45609923, 30.47139923, 30.51999923,
           30.58819923, 30.70519922, 30.89089922, 31.09759921, 31.27559921,
           31.39899921, 31.45749921, 31.4795992 , 31.4949992 , 31.5164992 ,
           31.5544992 , 31.5807992 , 31.5955992 , 31.6091992 , 31.6229992 ,
           31.6347992 , 31.6556992 , 31.6832992 , 31.7230992 , 31.7496992 ,
           31.7705992 , 31.8018992 , 31.8343992 , 31.8599992 , 31.90539919,
           31.96969919, 32.02269919, 32.06099919, 32.07709919, 32.08239919,
           32.08659919, 32.09399919, 32.10549919, 29.86619925, 29.86549925,
           29.86529925, 29.86609925, 29.87409925, 29.92889924, 30.12059924,
           30.42929923, 30.78869922, 30.95519922, 31.10649921, 31.28669921,
           31.4729992 , 31.7351992 , 31.90139919, 31.95819919, 32.14239919,
           32.28509918, 32.38149918, 32.48109918, 32.52999918, 32.54379918,
           32.56479918, 32.57729918, 32.60029918, 32.62619918, 32.66619917,
           32.69169917, 32.71259917, 32.76189917, 32.80639917, 32.81619917,
           32.83299917, 32.86329917, 32.88049917, 32.89579917, 32.90089917,
           32.90529917, 32.91279917, 32.91869917, 32.92019917, 32.93919917,
           32.98509917, 33.02579917, 33.03059917, 33.03069917, 30.89399922,
           30.89399922, 30.89399922, 30.89399922, 30.89399922, 30.89399922,
           30.89399922, 30.89399922, 30.89399922, 30.89399922, 30.89299922,
           30.89099922, 30.89199922, 30.99299922, 31.13399921, 31.22199921,
           31.25599921, 31.40999921, 31.5369992 , 31.5469992 , 31.6529992 ,
           31.7819992 , 31.87699919, 31.93799919, 31.99999919, 32.06699919,
           32.11799919, 32.13799919, 32.29099918, 32.45799918, 32.54399918,
           32.61899918, 32.70699917, 32.75599917, 32.80099917, 32.85999917,
           32.91899917, 32.95699917, 33.00899917, 33.05099917, 33.05299917,
           33.05699916, 33.06699916, 33.10599916, 33.15399916, 33.20599916,
           33.25299916, 33.26899916, 33.29199916, 33.29699916, 33.30799916,
           33.32399916, 33.33499916, 33.34899916, 33.38299916, 33.40699916,
           33.43199916, 33.44699916, 33.45699915, 33.47399915, 33.48199915,
           33.48499915, 33.49099915, 33.50799915, 33.56199915, 33.57599915,
           33.58199915, 33.58499915, 33.58799915, 33.58899915, 33.58999915,
           33.59099915, 33.59199915, 33.59699915, 33.59499915, 33.59999915,
           33.61399915, 33.62899915, 33.63299915, 33.63399915, 33.63799915,
           33.63799915, 33.63899915])

  • temperature
    (obs)
    float64
    10.83 10.83 10.83 ... 8.079 8.077

    standard_name :

    sea_water_temperature

    units :

    degree_C

    array([10.82799973, 10.83169973, 10.83309973, 10.83469973, 10.85269973,
           10.88769972, 10.91659972, 10.91339972, 10.90579972, 10.90819972,
           10.93879972, 11.04539972, 11.16569972, 11.35239971, 11.42989971,
           11.48189971, 11.49499971, 11.50459971, 11.51019971, 11.51439971,
           11.51289971, 11.49659971, 11.49029971, 11.48509971, 11.48239971,
           11.48419971, 11.49239971, 11.49709971, 11.51989971, 11.53029971,
           11.54399971, 11.55419971, 11.59559971, 11.60689971, 11.60049971,
           11.60009971, 11.61229971, 11.61909971, 11.62159971, 11.62219971,
           11.62419971, 11.62209971, 11.61889971,  9.10689977,  9.10429977,
            9.10089977,  9.09829977,  9.09699977,  9.16949977,  9.46949976,
            9.72159975,  9.91929975,  9.92649975,  9.95369975,  9.99649975,
           10.04259975, 10.12509974, 10.16219974, 10.19159974, 10.29479974,
           10.36249974, 10.37729974, 10.39449974, 10.42569974, 10.42939974,
           10.42849974, 10.42829974, 10.43629974, 10.43599974, 10.42339974,
           10.42089974, 10.43029974, 10.46689974, 10.49149973, 10.51549973,
           10.52969973, 10.53939973, 10.55739973, 10.55989973, 10.55719973,
           10.55439973, 10.55169973, 10.54889973, 10.54839973, 10.54069973,
           10.51819973, 10.49219973, 10.48829974, 10.48759974, 12.87099967,
           12.86799967, 12.86899967, 12.87099967, 12.86999967, 12.86999967,
           12.86999967, 12.87099967, 12.86999967, 12.86999967, 12.86199968,
           12.84799968, 12.71199968, 12.42899969, 12.13099969, 11.9569997 ,
           11.8769997 , 11.54399971, 11.29299971, 11.28199971, 11.04299972,
           10.76899973, 10.57299973, 10.44099974, 10.30999974, 10.17299974,
           10.06599975, 10.02199975,  9.68999976,  9.30899976,  9.11199977,
            8.95399977,  8.77199978,  8.67799978,  8.60199978,  8.50999979,
            8.44299979,  8.42199979,  8.43199979,  8.44199979,  8.44199979,
            8.44199979,  8.44199979,  8.44199979,  8.44599979,  8.44899979,
            8.44599979,  8.44299979,  8.43999979,  8.44099979,  8.43699979,
            8.43099979,  8.42599979,  8.42099979,  8.39899979,  8.38099979,
            8.36199979,  8.34799979,  8.33799979,  8.31499979,  8.30399979,
            8.29999979,  8.29299979,  8.27099979,  8.19399979,  8.17199979,
            8.16299979,  8.15899979,  8.15499979,  8.15299979,  8.14999979,
            8.14899979,  8.14199979,  8.13299979,  8.13599979,  8.12799979,
            8.1069998 ,  8.0889998 ,  8.0839998 ,  8.0829998 ,  8.0779998 ,
            8.0789998 ,  8.0769998 ])

• Indexes: (1)
  • profile
    PandasIndex

    PandasIndex(Index([b'010903NH03', b'011207NH03', b'061300NH15a'], dtype='object', name='profile'))

• Attributes: (3)

  featureType :

  TimeSeriesProfile

  Conventions :

  CF-1.11

  more attributes available at :

  https://coastwatch.pfeg.noaa.gov/erddap/tabledap/erdNewportCtd.html

Note

1. In case you’re not familiar with the syntax of xr.Dataset() function, we would highly recommend you to check out the elaborations in Chapter 3.

2. For the convenience of elaboration, we have used only three profiles from two stations. However, in real use cases you probably need to handle large amount of data, then it is usually a better option to use loop functions. You can find a showcase with loop functions in Chapter 5.