CrocoDash.raw_data_access.datasets package

Submodules

CrocoDash.raw_data_access.datasets.cesm_ocean_output module

Data Access Module -> CESM ocean output

Two products, split because they have different variable/coordinate naming conventions (and so need different ForcingProduct metadata): - CESM_POP_OUTPUT: CESM-POP tseries output (CESM-HR FOSI, CESM2-LENS2) - CESM_MOM_OUTPUT: native MOM6 output (e.g. parent-run history/diag_table

output for nesting a child domain)

Both share the same lower-level file-discovery/subsetting helpers below.

class CrocoDash.raw_data_access.datasets.cesm_ocean_output.CESM_MOM_OUTPUT

Bases: ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='noleap', cesm='NO_LEAP', mom6='noleap')

depth_coord = 'z_l'

description = 'Native MOM6 output (full history/diagnostic files, or diag_table-extracted cross-section slices) for use as IC/OBC forcing - e.g. for nesting a child domain inside an outer/parent MOM6 run.'

eta_var_name = 'zos'

static get_mom6_output_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename=None, variables=['zos', 'thetao', 'so', 'uo', 'vo'], dataset_path=None, file_glob='*.nc', time_var_name='time', buffer_deg=1.5, preview=False)

static get_mom6_single_variable_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename=None, variables=['zos', 'thetao', 'so', 'uo', 'vo'], dataset_path=None, member=None, date_format: str = '%Y%m%d', regex='(\\d{6,8})-(\\d{6,8})', delimiter='.', preview=False)

link = 'https://mom6.readthedocs.io/en/main/api/generated/pages/Diagnostics.html'

logger = <Logger CESM_MOM_OUTPUT (INFO)>

product_name = 'cesm_mom_output'

time_units = 'days'

time_var_name = 'time'

tracer_lat_coord = 'yh'

tracer_lon_coord = 'xh'

tracer_var_names = {'salt': 'so', 'temp': 'thetao'}

tracer_x_coord = 'xh'

tracer_y_coord = 'yh'

u_lat_coord = 'yh'

u_lon_coord = 'xq'

u_var_name = 'uo'

u_x_coord = 'xq'

u_y_coord = 'yh'

v_lat_coord = 'yq'

v_lon_coord = 'xh'

v_var_name = 'vo'

v_x_coord = 'xh'

v_y_coord = 'yq'

class CrocoDash.raw_data_access.datasets.cesm_ocean_output.CESM_POP_OUTPUT

Bases: ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='noleap', cesm='NO_LEAP', mom6='noleap')

delimiter = '.'

depth_coord = ['z_t', 'z_t_150m']

description = 'CESM ocean output (POP2 grid) for use as IC and OBC, including CESM-HR FOSI and CESM2 Large Ensemble (LENS2)'

eta_var_name = 'SSH'

static get_cesm_single_variable_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename=None, variables=['SSH', 'TEMP', 'SALT', 'VVEL', 'UVEL'], dataset_path='/glade/campaign/collections/cmip/CMIP6/CESM-HR/FOSI_BGC/HR/g.e22.TL319_t13.G1850ECOIAF_JRA_HR.4p2z.001/ocn/proc/tseries/month_1', member=None, date_format: str = '%Y%m%d', regex='(\\d{6,8})-(\\d{6,8})', delimiter='.', lat_name='TLAT', lon_name='TLONG', preview=False)

link = 'https://gdex.ucar.edu/datasets/d267000/'

logger = <Logger CESM_POP_OUTPUT (INFO)>

marbl_var_names = {'ALK': 'ALK', 'ALK_ALT_CO2': 'ALK_ALT_CO2', 'DIC': 'DIC', 'DIC_ALT_CO2': 'DIC_ALT_CO2', 'DOC': 'DOC', 'DOCr': 'DOCr', 'DON': 'DON', 'DONr': 'DONr', 'DOP': 'DOP', 'DOPr': 'DOPr', 'Fe': 'Fe', 'Lig': 'Lig', 'NH4': 'NH4', 'NO3': 'NO3', 'O2': 'O2', 'PO4': 'PO4', 'SiO3': 'SiO3', 'coccoC': 'coccoC', 'coccoCaCO3': 'coccoCaCO3', 'coccoChl': 'coccoChl', 'coccoFe': 'coccoFe', 'coccoP': 'coccoP', 'diatC': 'diatC', 'diatChl': 'diatChl', 'diatFe': 'diatFe', 'diatP': 'diatP', 'diatSi': 'diatSi', 'diazC': 'diazC', 'diazChl': 'diazChl', 'diazFe': 'diazFe', 'diazP': 'diazP', 'mesozooC': 'mesozooC', 'microzooC': 'microzooC', 'spC': 'spC', 'spChl': 'spChl', 'spFe': 'spFe', 'spP': 'spP'}

product_name = 'cesm_pop_output'

time_units = 'days since 1850-01-01'

time_var_name = 'time'

tracer_lat_coord = 'TLAT'

tracer_lon_coord = 'TLONG'

tracer_var_names = {'salt': 'SALT', 'temp': 'TEMP'}

tracer_x_coord = 'nlon'

tracer_y_coord = 'nlat'

u_lat_coord = 'TLAT'

u_lon_coord = 'TLONG'

u_var_name = 'UVEL'

u_x_coord = 'nlon'

u_y_coord = 'nlat'

v_lat_coord = 'TLAT'

v_lon_coord = 'TLONG'

v_var_name = 'VVEL'

v_x_coord = 'nlon'

v_y_coord = 'nlat'

z_unit_conversion = 0.01

CrocoDash.raw_data_access.datasets.cesm_ocean_output.bbox_mask(ds, lat_min, lat_max, lon_min, lon_max, lat_name='lat', lon_name='lon', buffer_deg=0)

Boolean lat/lon mask for subsetting a dataset, handling both 0-360 and -180-180 longitude conventions.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.convert_cftime_to_numeric(ds, time_var_name='time', apply_month_shift=False, calendar: Calendar = Calendar(cf='noleap', cesm='NO_LEAP', mom6='noleap'))

Converts a cftime time coordinate to numeric (days since 1850-01-01, noleap) for safe NetCDF serialization; a no-op if the coordinate isn’t cftime. apply_month_shift is specific to the CESM-POP tseries convention (average- endpoint timestamp labeling) and should not be used for native MOM6 output.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.drop_extra_cftime_vars(ds)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.first_value(da_var)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.get_date_range_from_filename(path, regex)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.parse_dataset(variable_names: list[str], dataset_path: str | Path, start_date: str, end_date: str, date_format: str = '%Y%m%d', regex='(\\d{6,8})-(\\d{6,8})', space_character='.') → dict

Parses the dataset to find variable names and their corresponding file paths.

Parameters:

• variable_names (list[str]) – List of variable names to search for.

• dataset_path (str | xr.Dataset | Path) – Path to the dataset (or folder with dataset)

• space_character (str) – Character that separates words in variable names in the filenames. Default is “.”.

Returns:

A dictionary with variable names as keys and their file paths as values.

Return type:

dict

CrocoDash.raw_data_access.datasets.cesm_ocean_output.read_single_variable_tseries_data(dates, lat_min, lat_max, lon_min, lon_max, output_folder, output_filename, variables, dataset_path, member, date_format, regex, delimiter, lat_name, lon_name, preview, apply_month_shift, grid_coords=None)

Shared implementation for reading single-variable-per-file (tseries) output for a bounding box, used by both CESM_POP_OUTPUT.get_cesm_single_variable_data and CESM_MOM_OUTPUT.get_mom6_single_variable_data - the CESM tseries file convention (one variable per file series, date range in the filename) shows up for both CESM-POP and native MOM6 output.

grid_coords optionally maps a variable name to its own (lat_name, lon_name) pair, overriding lat_name/lon_name for that variable - needed for MOM6’s C-grid, where uo/vo live on different horizontal dims than the tracer grid.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.subset_dataset(variable_info: dict, output_path: str | Path, lat_min: float, lat_max: float, lon_min: float, lon_max: float, lat_name='lat', lon_name='lon', dates=None, preview: bool = False, apply_month_shift: bool = True, grid_coords: dict | None = None) → None

Subsets (and merges) the dataset based on the provided variable names and geographical bounds into the output path :param variable_info: A dictionary with variable names as keys and their file paths as values. :type variable_info: dict :param output_path: The path where the subsetted dataset will be saved. :type output_path: str | Path :param lat_min: Minimum latitude for subsetting. :type lat_min: float :param lat_max: Maximum latitude for subsetting. :type lat_max: float :param lon_min: Minimum longitude for subsetting. :type lon_min: float :param lon_max: Maximum longitude for subsetting. :type lon_max: float :param lat_name: Default latitude coordinate name, used for any variable

not listed in grid_coords. Default is “lat”.

Parameters:

• lon_name (str) – Default longitude coordinate name, used for any variable not listed in grid_coords. Default is “lon”.

• dates (tuple) – Just used for the file naming

• preview (bool) – If True, only previews the subsetting without saving. Default is False.

• apply_month_shift (bool) – Apply the CESM-POP tseries average-endpoint timestamp correction. Default True (matches CESM-POP); set False for native MOM6 tseries output.

• grid_coords (dict | None) – Optional {var_name: (lat_name, lon_name)} override - needed for MOM6’s C-grid, where uo/vo live on different horizontal dims than the tracer grid, so a single lat_name/lon_name pair can’t mask every variable correctly.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.subtract_month(dt)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.validate_dataset_path(dataset_path)

CrocoDash.raw_data_access.datasets.empty_products module

class CrocoDash.raw_data_access.datasets.empty_products.CESMInputData

Bases: BaseProduct

description = 'The CESM Input SVN repo holds all files CrocoDash exposes publicly themselves at the following repo link'

link = 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ocn/mom/croc'

logger = <Logger CESMInputData (INFO)>

product_name = 'CESM Inputdata'

class CrocoDash.raw_data_access.datasets.empty_products.TPXO

Bases: BaseProduct

description = 'TPXO (TOPEX/POSEIDON Global Tidal Ocean I think) is a public tidal model dataset'

link = 'https://www.tpxo.net/global'

logger = <Logger TPXO (INFO)>

product_name = 'tpxo'

CrocoDash.raw_data_access.datasets.gebco module

class CrocoDash.raw_data_access.datasets.gebco.GEBCO

Bases: BaseProduct

description = 'GEBCO (General Bathymetric Chart of the Ocean) is a public dataset of global ocean bathymetry'

static get_gebco_data_script(output_folder=None, output_filename=None)

static get_gebco_data_with_python(output_folder=None, output_filename=None)

link = 'https://www.gebco.net/data_and_products/gridded_bathymetry_data/#global'

logger = <Logger GEBCO (INFO)>

product_name = 'gebco'

CrocoDash.raw_data_access.datasets.glofas module

class CrocoDash.raw_data_access.datasets.glofas.GLOFAS

Bases: DatedBaseProduct

description = '\tGLOFAS (Global Flood Awareness System) is a public river discharge/runoff Product'

static get_global_data_with_python(dates, output_folder=PosixPath('.'), output_filename='glofas_data.nc')

Downloads glofas data using cdsapi library. Note that users need to have an account with copernicus and have cdsapi installed and configured.

Parameters:

• date (str, optional) – What dates to download.

• output_folder (str, optional) – Directory where downloaded files will be saved.

static get_processed_global_glofas_script_for_cli(dates='UNUSED', output_folder=PosixPath('.'), output_filename='processed_glofas.nc')

Downloads chlor_a data from the CESM inputdata repository by generating a script users can run in their terminal. :param date: Currently unused; placeholder for future date-based filtering. :type date: str, optional :param lat_min: Currently unused; placeholder for future spatial filtering. :type lat_min: float, optional :param lat_max: Currently unused; placeholder for future spatial filtering. :type lat_max: float, optional :param lon_min: Currently unused; placeholder for future spatial filtering. :type lon_min: float, optional :param lon_max: Currently unused; placeholder for future spatial filtering. :type lon_max: float, optional :param output_folder: Directory where downloaded files will be saved. :type output_folder: str, optional :param output_filename: filename in output directory :type output_filename: str, optional

link = 'https://ewds.climate.copernicus.eu/datasets/cems-glofas-historical?tab=download'

logger = <Logger GLOFAS (INFO)>

product_name = 'glofas'

CrocoDash.raw_data_access.datasets.glorys module

Data Access Module -> Glorys

class CrocoDash.raw_data_access.datasets.glorys.GLORYS

Bases: ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='standard', cesm='GREGORIAN', mom6='gregorian')

depth_coord = 'depth'

description = 'GLORYS (Global Ocean Physics Reanalysis) is a public dataset provided through the copernicus marine service.'

eta_var_name = 'zos'

static get_glorys_data_from_cds_api(dates, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=None, output_filename=None, variables=['zos', 'uo', 'vo', 'so', 'thetao'])

Using the copernucismarine api, query GLORYS data (any dates)

static get_glorys_data_from_rda(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename='raw_glorys.nc', variables=['time', 'latitude', 'longitude', 'depth', 'zos', 'uo', 'vo', 'so', 'thetao'], buf=1.0) → Dataset

Gather GLORYS Data on Derecho Computers from the campaign storage and return the dataset sliced to the llc and urc coordinates at the specific dates

static get_glorys_data_script_for_cli(dates: tuple, lat_min, lat_max, lon_min, lon_max, output_folder, output_filename, variables=None, name=None) → None

Script to run the GLORYS data query for the CLI

link = 'https://data.marine.copernicus.eu/product/GLOBAL_MULTIYEAR_PHY_001_030/description'

logger = <Logger GLORYS (INFO)>

product_name = 'glorys'

time_units = 'days'

time_var_name = 'time'

tracer_lat_coord = 'latitude'

tracer_lon_coord = 'longitude'

tracer_var_names = {'salt': 'so', 'temp': 'thetao'}

tracer_x_coord = 'longitude'

tracer_y_coord = 'latitude'

u_lat_coord = 'latitude'

u_lon_coord = 'longitude'

u_var_name = 'uo'

u_x_coord = 'longitude'

u_y_coord = 'latitude'

v_lat_coord = 'latitude'

v_lon_coord = 'longitude'

v_var_name = 'vo'

v_x_coord = 'longitude'

v_y_coord = 'latitude'

CrocoDash.raw_data_access.datasets.seawifs module

class CrocoDash.raw_data_access.datasets.seawifs.SeaWIFS

Bases: BaseProduct

description = 'SEAWIFS is a Chlorophyll Dataset for MOM6'

static get_global_seawifs_script_for_cli(output_folder=None, output_filename='UNUSED', username='')

Downloads chlor_a data using NASA OceanData API with authentication.

Parameters:

• username (str) – NASA Earthdata username (password will be prompted at runtime).

• date (str, optional) – Currently unused; placeholder for future date-based filtering.

• lat_min (float, optional) – Currently unused; placeholder for future spatial filtering.

• lat_max (float, optional) – Currently unused; placeholder for future spatial filtering.

• lon_min (float, optional) – Currently unused; placeholder for future spatial filtering.

• lon_max (float, optional) – Currently unused; placeholder for future spatial filtering.

• output_folder (str, optional) – Directory where downloaded files will be saved.

static get_processed_global_seawifs_script_for_cli(output_folder=PosixPath('.'), output_filename='processed_seawifs.nc')

Downloads chlor_a data from the CESM inputdata repository by generating a script users can run in their terminal. :param date: Currently unused; placeholder for future date-based filtering. :type date: str, optional :param lat_min: Currently unused; placeholder for future spatial filtering. :type lat_min: float, optional :param lat_max: Currently unused; placeholder for future spatial filtering. :type lat_max: float, optional :param lon_min: Currently unused; placeholder for future spatial filtering. :type lon_min: float, optional :param lon_max: Currently unused; placeholder for future spatial filtering. :type lon_max: float, optional :param output_folder: Directory where downloaded files will be saved. :type output_folder: str, optional :param output_filename: filename in output directory :type output_filename: str, optional

link = 'https://oceandata.sci.gsfc.nasa.gov/getfile/SEASTAR_SEAWIFS_GAC.19980201_20100228.L3m.MC.CHL.chlor_a.9km.nc'

logger = <Logger SeaWIFS (INFO)>

product_name = 'seawifs'

CrocoDash.raw_data_access.datasets.srtm module

class CrocoDash.raw_data_access.datasets.srtm.SRTM

Bases: BaseProduct

description = 'SRTM15+ is a global 15 arc-second resolution bathymetry/topography dataset compiled from SRTM land topography and satellite-derived ocean bathymetry.'

static get_srtm_data_script(output_folder=None, output_filename=None)

static get_srtm_data_with_python(output_folder=None, output_filename=None)

link = 'https://topex.ucsd.edu/WWW_html/srtm15_plus.html'

logger = <Logger SRTM (INFO)>

product_name = 'srtm'

CrocoDash.raw_data_access.datasets.utils module

CrocoDash.raw_data_access.datasets.utils.convert_lons_to_180_range(*lons)

CrocoDash.raw_data_access.datasets.utils.make_dates_end_inclusive(dates)

Return (start, end) as “%Y-%m-%d %H:%M:%S” strings, with the end pushed to the last second of its day.

APIs that treat end_datetime as a literal cutoff (rather than a whole day) will otherwise silently drop everything after midnight on the last requested day.

CrocoDash.raw_data_access.datasets.utils.write_bash_curl_script(url, script_name, output_folder, output_filename)

Module contents

CrocoDash.raw_data_access.datasets.load_all_datasets()

Dynamically import all modules in this package so that their _init_subclass hook run and populate the registry.