CrocoDash.raw_data_access.datasets package

Submodules

CrocoDash.raw_data_access.datasets.cesm_ocean_output module

Data Access Module -> CESM ocean output

Two products, split because they have different variable/coordinate naming conventions (and so need different ForcingProduct metadata): - CESM_POP_OUTPUT: CESM-POP tseries output (CESM-HR FOSI, CESM2-LENS2) - CESM_MOM_OUTPUT: native MOM6 output (e.g. parent-run history/diag_table

output for nesting a child domain)

Both share the same lower-level file-discovery/subsetting helpers below.

class CrocoDash.raw_data_access.datasets.cesm_ocean_output.CESM_MOM_OUTPUT

Bases: ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='noleap', cesm='NO_LEAP', mom6='noleap')

cesm_calendar = 'NO_LEAP'

cf_calendar = 'noleap'

depth_coord = 'z_l'

description = 'Native MOM6 output (full history/diagnostic files, or diag_table-extracted cross-section slices) for use as IC/OBC forcing - e.g. for nesting a child domain inside an outer/parent MOM6 run.'

eta_var_name = 'zos'

static get_mom6_output_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename=None, variables=['zos', 'thetao', 'so', 'uo', 'vo'], dataset_path=None, file_glob='*.nc', time_var_name='time', buffer_deg=1.5, preview=False)

static get_mom6_single_variable_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename=None, variables=['zos', 'thetao', 'so', 'uo', 'vo'], dataset_path=None, member=None, date_format: str = '%Y%m%d', regex='(\\d{6,8})-(\\d{6,8})', delimiter='.', preview=False)

link = 'https://mom6.readthedocs.io/en/main/api/generated/pages/Diagnostics.html'

logger = <Logger CESM_MOM_OUTPUT (INFO)>

mom6_calendar = 'noleap'

product_name = 'cesm_mom_output'

time_units = 'days'

time_var_name = 'time'

tracer_lat_coord = 'yh'

tracer_lon_coord = 'xh'

tracer_var_names = {'salt': 'so', 'temp': 'thetao'}

tracer_x_coord = 'xh'

tracer_y_coord = 'yh'

u_lat_coord = 'yh'

u_lon_coord = 'xq'

u_var_name = 'uo'

u_x_coord = 'xq'

u_y_coord = 'yh'

v_lat_coord = 'yq'

v_lon_coord = 'xh'

v_var_name = 'vo'

v_x_coord = 'xh'

v_y_coord = 'yq'

class CrocoDash.raw_data_access.datasets.cesm_ocean_output.CESM_POP_OUTPUT

Bases: ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='noleap', cesm='NO_LEAP', mom6='noleap')

cesm_calendar = 'NO_LEAP'

cf_calendar = 'noleap'

delimiter = '.'

depth_coord = ['z_t', 'z_t_150m']

description = 'CESM ocean output (POP2 grid) for use as IC and OBC, including CESM-HR FOSI and CESM2 Large Ensemble (LENS2)'

eta_var_name = 'SSH'

static get_cesm_single_variable_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename=None, variables=['SSH', 'TEMP', 'SALT', 'VVEL', 'UVEL'], dataset_path='/glade/campaign/collections/cmip/CMIP6/CESM-HR/FOSI_BGC/HR/g.e22.TL319_t13.G1850ECOIAF_JRA_HR.4p2z.001/ocn/proc/tseries/month_1', member=None, date_format: str = '%Y%m%d', regex='(\\d{6,8})-(\\d{6,8})', delimiter='.', lat_name='TLAT', lon_name='TLONG', preview=False)

link = 'https://gdex.ucar.edu/datasets/d267000/'

logger = <Logger CESM_POP_OUTPUT (INFO)>

marbl_var_names = {'ALK': 'ALK', 'ALK_ALT_CO2': 'ALK_ALT_CO2', 'DIC': 'DIC', 'DIC_ALT_CO2': 'DIC_ALT_CO2', 'DOC': 'DOC', 'DOCr': 'DOCr', 'DON': 'DON', 'DONr': 'DONr', 'DOP': 'DOP', 'DOPr': 'DOPr', 'Fe': 'Fe', 'Lig': 'Lig', 'NH4': 'NH4', 'NO3': 'NO3', 'O2': 'O2', 'PO4': 'PO4', 'SiO3': 'SiO3', 'coccoC': 'coccoC', 'coccoCaCO3': 'coccoCaCO3', 'coccoChl': 'coccoChl', 'coccoFe': 'coccoFe', 'coccoP': 'coccoP', 'diatC': 'diatC', 'diatChl': 'diatChl', 'diatFe': 'diatFe', 'diatP': 'diatP', 'diatSi': 'diatSi', 'diazC': 'diazC', 'diazChl': 'diazChl', 'diazFe': 'diazFe', 'diazP': 'diazP', 'mesozooC': 'mesozooC', 'microzooC': 'microzooC', 'spC': 'spC', 'spChl': 'spChl', 'spFe': 'spFe', 'spP': 'spP'}

mom6_calendar = 'noleap'

product_name = 'cesm_pop_output'

time_units = 'days since 1850-01-01'

time_var_name = 'time'

tracer_lat_coord = 'TLAT'

tracer_lon_coord = 'TLONG'

tracer_var_names = {'salt': 'SALT', 'temp': 'TEMP'}

tracer_x_coord = 'nlon'

tracer_y_coord = 'nlat'

u_lat_coord = 'TLAT'

u_lon_coord = 'TLONG'

u_var_name = 'UVEL'

u_x_coord = 'nlon'

u_y_coord = 'nlat'

v_lat_coord = 'TLAT'

v_lon_coord = 'TLONG'

v_var_name = 'VVEL'

v_x_coord = 'nlon'

v_y_coord = 'nlat'

z_unit_conversion = 0.01

CrocoDash.raw_data_access.datasets.cesm_ocean_output.bbox_mask(ds, lat_min, lat_max, lon_min, lon_max, lat_name='lat', lon_name='lon', buffer_deg=0)

Boolean lat/lon mask for subsetting a dataset, handling both 0-360 and -180-180 longitude conventions.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.convert_cftime_to_numeric(ds, time_var_name='time', apply_month_shift=False)

Converts a cftime time coordinate to numeric (days since 1850-01-01, noleap) for safe NetCDF serialization; a no-op if the coordinate isn’t cftime. apply_month_shift is specific to the CESM-POP tseries convention (average- endpoint timestamp labeling) and should not be used for native MOM6 output.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.drop_extra_cftime_vars(ds)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.first_value(da_var)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.get_date_range_from_filename(path, regex)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.parse_dataset(variable_names: list[str], dataset_path: str | Path, start_date: str, end_date: str, date_format: str = '%Y%m%d', regex='(\\d{6,8})-(\\d{6,8})', space_character='.') → dict

Parses the dataset to find variable names and their corresponding file paths.

Parameters:

• variable_names (list[str]) – List of variable names to search for.

• dataset_path (str | xr.Dataset | Path) – Path to the dataset (or folder with dataset)

• space_character (str) – Character that separates words in variable names in the filenames. Default is “.”.

Returns:

A dictionary with variable names as keys and their file paths as values.

Return type:

dict

CrocoDash.raw_data_access.datasets.cesm_ocean_output.read_single_variable_tseries_data(dates, lat_min, lat_max, lon_min, lon_max, output_folder, output_filename, variables, dataset_path, member, date_format, regex, delimiter, lat_name, lon_name, preview, apply_month_shift, grid_coords=None)

Shared implementation for reading single-variable-per-file (tseries) output for a bounding box, used by both CESM_POP_OUTPUT.get_cesm_single_variable_data and CESM_MOM_OUTPUT.get_mom6_single_variable_data - the CESM tseries file convention (one variable per file series, date range in the filename) shows up for both CESM-POP and native MOM6 output.

grid_coords optionally maps a variable name to its own (lat_name, lon_name) pair, overriding lat_name/lon_name for that variable - needed for MOM6’s C-grid, where uo/vo live on different horizontal dims than the tracer grid.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.subset_dataset(variable_info: dict, output_path: str | Path, lat_min: float, lat_max: float, lon_min: float, lon_max: float, lat_name='lat', lon_name='lon', dates=None, preview: bool = False, apply_month_shift: bool = True, grid_coords: dict | None = None) → None

Subsets (and merges) the dataset based on the provided variable names and geographical bounds into the output path :param variable_info: A dictionary with variable names as keys and their file paths as values. :type variable_info: dict :param output_path: The path where the subsetted dataset will be saved. :type output_path: str | Path :param lat_min: Minimum latitude for subsetting. :type lat_min: float :param lat_max: Maximum latitude for subsetting. :type lat_max: float :param lon_min: Minimum longitude for subsetting. :type lon_min: float :param lon_max: Maximum longitude for subsetting. :type lon_max: float :param lat_name: Default latitude coordinate name, used for any variable

not listed in grid_coords. Default is “lat”.

Parameters:

• lon_name (str) – Default longitude coordinate name, used for any variable not listed in grid_coords. Default is “lon”.

• dates (tuple) – Just used for the file naming

• preview (bool) – If True, only previews the subsetting without saving. Default is False.

• apply_month_shift (bool) – Apply the CESM-POP tseries average-endpoint timestamp correction. Default True (matches CESM-POP); set False for native MOM6 tseries output.

• grid_coords (dict | None) – Optional {var_name: (lat_name, lon_name)} override - needed for MOM6’s C-grid, where uo/vo live on different horizontal dims than the tracer grid, so a single lat_name/lon_name pair can’t mask every variable correctly.

CrocoDash.raw_data_access.datasets.cesm_ocean_output.subtract_month(dt)

CrocoDash.raw_data_access.datasets.cesm_ocean_output.validate_dataset_path(dataset_path)

CrocoDash.raw_data_access.datasets.empty_products module

class CrocoDash.raw_data_access.datasets.empty_products.CESMInputData

Bases: BaseProduct

description = 'The CESM Input SVN repo holds all files CrocoDash exposes publicly themselves at the following repo link'

link = 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ocn/mom/croc'

logger = <Logger CESMInputData (INFO)>

product_name = 'CESM Inputdata'

class CrocoDash.raw_data_access.datasets.empty_products.TPXO

Bases: BaseProduct

description = 'TPXO (TOPEX/POSEIDON Global Tidal Ocean I think) is a public tidal model dataset'

link = 'https://www.tpxo.net/global'

logger = <Logger TPXO (INFO)>

product_name = 'tpxo'

CrocoDash.raw_data_access.datasets.gebco module

class CrocoDash.raw_data_access.datasets.gebco.GEBCO

Bases: BaseProduct

description = 'GEBCO (General Bathymetric Chart of the Ocean) is a public dataset of global ocean bathymetry'

static get_gebco_data_script(output_folder=None, output_filename=None)

static get_gebco_data_with_python(output_folder=None, output_filename=None)

link = 'https://www.gebco.net/data_and_products/gridded_bathymetry_data/#global'

logger = <Logger GEBCO (INFO)>

product_name = 'gebco'

CrocoDash.raw_data_access.datasets.glofas module

class CrocoDash.raw_data_access.datasets.glofas.GLOFAS

Bases: DatedBaseProduct

description = '\tGLOFAS (Global Flood Awareness System) is a public river discharge/runoff Product'

static get_global_data_with_python(dates, output_folder=PosixPath('.'), output_filename='glofas_data.nc')

Downloads glofas data using cdsapi library. Note that users need to have an account with copernicus and have cdsapi installed and configured.

Parameters:

• date (str, optional) – What dates to download.

• output_folder (str, optional) – Directory where downloaded files will be saved.

static get_processed_global_glofas_script_for_cli(dates='UNUSED', output_folder=PosixPath('.'), output_filename='processed_glofas.nc')

Downloads chlor_a data from the CESM inputdata repository by generating a script users can run in their terminal. :param date: Currently unused; placeholder for future date-based filtering. :type date: str, optional :param lat_min: Currently unused; placeholder for future spatial filtering. :type lat_min: float, optional :param lat_max: Currently unused; placeholder for future spatial filtering. :type lat_max: float, optional :param lon_min: Currently unused; placeholder for future spatial filtering. :type lon_min: float, optional :param lon_max: Currently unused; placeholder for future spatial filtering. :type lon_max: float, optional :param output_folder: Directory where downloaded files will be saved. :type output_folder: str, optional :param output_filename: filename in output directory :type output_filename: str, optional

link = 'https://ewds.climate.copernicus.eu/datasets/cems-glofas-historical?tab=download'

logger = <Logger GLOFAS (INFO)>

product_name = 'glofas'

CrocoDash.raw_data_access.datasets.glorys module

Data Access Module -> Glorys

class CrocoDash.raw_data_access.datasets.glorys.GLORYS

Bases: MOM6ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='standard', cesm='GREGORIAN', mom6='gregorian')

cesm_calendar = 'GREGORIAN'

cf_calendar = 'standard'

depth_coord = 'depth'

description = 'GLORYS (Global Ocean Physics Reanalysis) is a public dataset provided through the copernicus marine service.'

eta_var_name = 'zos'

static get_glorys_data_from_cds_api(dates, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=None, output_filename=None, variables=['zos', 'uo', 'vo', 'so', 'thetao'])

Using the copernucismarine api, query GLORYS data (any dates)

static get_glorys_data_from_rda(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename='raw_glorys.nc', variables=['time', 'latitude', 'longitude', 'depth', 'zos', 'uo', 'vo', 'so', 'thetao'], buf=1.0) → Dataset

Gather GLORYS Data on Derecho Computers from the campaign storage and return the dataset sliced to the llc and urc coordinates at the specific dates

static get_glorys_data_script_for_cli(dates: tuple, lat_min, lat_max, lon_min, lon_max, output_folder, output_filename, variables=None, name=None) → None

Script to run the GLORYS data query for the CLI

link = 'https://data.marine.copernicus.eu/product/GLOBAL_MULTIYEAR_PHY_001_030/description'

logger = <Logger GLORYS (INFO)>

mom6_calendar = 'gregorian'

product_name = 'glorys'

time_units = 'days'

time_var_name = 'time'

tracer_lat_coord = 'latitude'

tracer_lon_coord = 'longitude'

tracer_var_names = {'salt': 'so', 'temp': 'thetao'}

tracer_x_coord = 'longitude'

tracer_y_coord = 'latitude'

u_lat_coord = 'latitude'

u_lon_coord = 'longitude'

u_var_name = 'uo'

u_x_coord = 'longitude'

u_y_coord = 'latitude'

v_lat_coord = 'latitude'

v_lon_coord = 'longitude'

v_var_name = 'vo'

v_x_coord = 'longitude'

v_y_coord = 'latitude'

CrocoDash.raw_data_access.datasets.reference module

Data Access Module -> Reference (fast, deterministic synthetic forcing)

REFERENCE_OCEAN (MOM6) generates plausible-looking forcing data purely in memory via numpy, with no network access, credentials, or campaign-storage dependency. Each access method is a pure function of its own (dates, lat/lon bbox) arguments: same inputs always produce the same output, so this is safe to use in tests (real assertions, not just “did it not crash”) and in demo notebooks that should look identical on every run.

This is not meant to be physically accurate – just close enough in shape (a warm-at-equator thermocline) that the real MOM6 regrid pipeline has something sensible to chew on, fast.

class CrocoDash.raw_data_access.datasets.reference.REFERENCE_OCEAN

Bases: MOM6ForcingProduct

boundary_fill_method = 'regional_mom6'

calendar = Calendar(cf='standard', cesm='GREGORIAN', mom6='gregorian')

cesm_calendar = 'GREGORIAN'

cf_calendar = 'standard'

depth_coord = 'depth'

description = 'Fast, deterministic synthetic ocean IC/OBC data (temperature, salinity, SSH, currents) for testing and demos -- no network or campaign-storage access required.'

eta_var_name = 'ssh'

static get_reference_ocean_data(dates: list, lat_min, lat_max, lon_min, lon_max, name=None, output_folder=PosixPath('.'), output_filename='reference_ocean.nc', variables=None, resolution_deg=0.5)

link = 'n/a'

logger = <Logger REFERENCE_OCEAN (INFO)>

marbl_var_names = {'ALK': 'ALK', 'ALK_ALT_CO2': 'ALK_ALT_CO2', 'DIC': 'DIC', 'DIC_ALT_CO2': 'DIC_ALT_CO2', 'DOC': 'DOC', 'DOCr': 'DOCr', 'DON': 'DON', 'DONr': 'DONr', 'DOP': 'DOP', 'DOPr': 'DOPr', 'Fe': 'Fe', 'Lig': 'Lig', 'NH4': 'NH4', 'NO3': 'NO3', 'O2': 'O2', 'PO4': 'PO4', 'SiO3': 'SiO3', 'coccoC': 'coccoC', 'coccoCaCO3': 'coccoCaCO3', 'coccoChl': 'coccoChl', 'coccoFe': 'coccoFe', 'coccoP': 'coccoP', 'diatC': 'diatC', 'diatChl': 'diatChl', 'diatFe': 'diatFe', 'diatP': 'diatP', 'diatSi': 'diatSi', 'diazC': 'diazC', 'diazChl': 'diazChl', 'diazFe': 'diazFe', 'diazP': 'diazP', 'mesozooC': 'mesozooC', 'microzooC': 'microzooC', 'spC': 'spC', 'spChl': 'spChl', 'spFe': 'spFe', 'spP': 'spP'}

mom6_calendar = 'gregorian'

product_name = 'reference_ocean'

time_units = 'days'

time_var_name = 'time'

tracer_lat_coord = 'latitude'

tracer_lon_coord = 'longitude'

tracer_var_names = {'salt': 'salt', 'temp': 'temp'}

tracer_x_coord = 'longitude'

tracer_y_coord = 'latitude'

u_lat_coord = 'latitude'

u_lon_coord = 'longitude'

u_var_name = 'u'

u_x_coord = 'longitude'

u_y_coord = 'latitude'

v_lat_coord = 'latitude'

v_lon_coord = 'longitude'

v_var_name = 'v'

v_x_coord = 'longitude'

v_y_coord = 'latitude'

CrocoDash.raw_data_access.datasets.seawifs module

class CrocoDash.raw_data_access.datasets.seawifs.SeaWIFS

Bases: BaseProduct

description = 'SEAWIFS is a Chlorophyll Dataset for MOM6'

static get_global_seawifs_script_for_cli(output_folder=None, output_filename='UNUSED', username='')

Downloads chlor_a data using NASA OceanData API with authentication.

Parameters:

• username (str) – NASA Earthdata username (password will be prompted at runtime).

• date (str, optional) – Currently unused; placeholder for future date-based filtering.

• lat_min (float, optional) – Currently unused; placeholder for future spatial filtering.

• lat_max (float, optional) – Currently unused; placeholder for future spatial filtering.

• lon_min (float, optional) – Currently unused; placeholder for future spatial filtering.

• lon_max (float, optional) – Currently unused; placeholder for future spatial filtering.

• output_folder (str, optional) – Directory where downloaded files will be saved.

static get_processed_global_seawifs_script_for_cli(output_folder=PosixPath('.'), output_filename='processed_seawifs.nc')

Downloads chlor_a data from the CESM inputdata repository by generating a script users can run in their terminal. :param date: Currently unused; placeholder for future date-based filtering. :type date: str, optional :param lat_min: Currently unused; placeholder for future spatial filtering. :type lat_min: float, optional :param lat_max: Currently unused; placeholder for future spatial filtering. :type lat_max: float, optional :param lon_min: Currently unused; placeholder for future spatial filtering. :type lon_min: float, optional :param lon_max: Currently unused; placeholder for future spatial filtering. :type lon_max: float, optional :param output_folder: Directory where downloaded files will be saved. :type output_folder: str, optional :param output_filename: filename in output directory :type output_filename: str, optional

link = 'https://oceandata.sci.gsfc.nasa.gov/getfile/SEASTAR_SEAWIFS_GAC.19980201_20100228.L3m.MC.CHL.chlor_a.9km.nc'

logger = <Logger SeaWIFS (INFO)>

product_name = 'seawifs'

CrocoDash.raw_data_access.datasets.srtm module

class CrocoDash.raw_data_access.datasets.srtm.SRTM

Bases: BaseProduct

description = 'SRTM15+ is a global 15 arc-second resolution bathymetry/topography dataset compiled from SRTM land topography and satellite-derived ocean bathymetry.'

static get_srtm_data_script(output_folder=None, output_filename=None)

static get_srtm_data_with_python(output_folder=None, output_filename=None)

link = 'https://topex.ucsd.edu/WWW_html/srtm15_plus.html'

logger = <Logger SRTM (INFO)>

product_name = 'srtm'

CrocoDash.raw_data_access.datasets.utils module

CrocoDash.raw_data_access.datasets.utils.convert_lons_to_180_range(*lons)

CrocoDash.raw_data_access.datasets.utils.make_dates_end_inclusive(dates)

Return (start, end) as “%Y-%m-%d %H:%M:%S” strings, with the end pushed to the last second of its day.

APIs that treat end_datetime as a literal cutoff (rather than a whole day) will otherwise silently drop everything after midnight on the last requested day.

CrocoDash.raw_data_access.datasets.utils.write_bash_curl_script(url, script_name, output_folder, output_filename)

Module contents

CrocoDash.raw_data_access.datasets.load_all_datasets()

Dynamically import all modules in this package so that their _init_subclass hook run and populate the registry.