CrocoDash.extract_forcings package

CrocoDash.extract_forcings package#

Submodules#

CrocoDash.extract_forcings.bgc module#

CrocoDash.extract_forcings.bgc.process_bgc_ic(file_path, output_path)#

Copy BGC initial condition file

Parameters: - file_path: str, path to the original BGC IC file - output_path: str, path to save the processed BGC IC file

Returns: - None

CrocoDash.extract_forcings.bgc.process_bgc_iron_forcing(nx, ny, MARBL_FESEDFLUX_FILE, MARBL_FEVENTFLUX_FILE, MARBL_FESEDFLUXRED_FILE, inputdir)#

Create dummy iron forcing files for MARBL. Parameters: - nx: int, number of grid points in x-direction - ny: int, number of grid points in y-direction - MARBL_FESEDFLUX_FILE: str, filename for sediment flux input - MARBL_FEVENTFLUX_FILE: str, filename for event flux input - inputdir: str, directory to save the generated files Returns: - None

CrocoDash.extract_forcings.bgc.process_river_nutrients(global_river_nutrients_filepath, ocn_grid, mapping_file, river_nutrients_nnsm_filepath, calendar='noleap')#

CrocoDash.extract_forcings.chlorophyll module#

CrocoDash.extract_forcings.chlorophyll.process_chl(ocn_grid, ocn_topo, inputdir, chl_processed_filepath, output_filepath, calendar='NOLEAP')#

CrocoDash.extract_forcings.driver module#

CrocoDash Forcing Extraction Driver

Orchestrates the forcing extraction workflow for a CrocoDash case. Reads config.json (written by Case.configure_forcings) and derives all paths from the case state file via case_state.read.

This module exposes run_workflow as the Python API. The CLI entry point is crocodash process (see CrocoDash.cli).

Typical Python usage:

from CrocoDash.extract_forcings.driver import run_workflow

run_workflow(config_path="~/croc_input/mycase/extract_forcings/config.json", bc=True, ic=True)
CrocoDash.extract_forcings.driver.resolve_components(args, config)#

Resolve which components to run based on CLI args and config availability.

CrocoDash.extract_forcings.driver.run_workflow(config_path, ic=False, bc=False, bgcic=False, bgcironforcing=False, tides=False, chl_=False, runoff=False, bgcrivernutrients=False, preview=False)#

Execute the forcing extraction workflow.

Parameters:

  • config_path (str or Path) – Path to the config.json written by Case.configure_forcings.

  • ic (bool) – Run initial conditions.

  • bc (bool) – Run boundary conditions.

  • bgcic (bool) – Run BGC initial conditions.

  • bgcironforcing (bool) – Run BGC iron forcing.

  • tides (bool) – Run tidal forcing.

  • chl (bool) – Run chlorophyll processing.

  • runoff (bool) – Run runoff mapping.

  • bgcrivernutrients (bool) – Run BGC river nutrients (always runs after runoff).

  • preview (bool) – Preview task graph without executing.

CrocoDash.extract_forcings.ic module#

IC (Initial Condition) forcing extraction engine for CrocoDash.

Model-agnostic: this module gets the raw t=0 snapshot, but knows nothing about how to regrid it. Callers (mom6.py, cice.py, ww3.py) supply a regrid_fn that turns the raw snapshot into that target’s own initial-condition file(s) – the GET step is shared.

CrocoDash.extract_forcings.ic.process_initial_condition(product_name: str, function_name: str, variables: list, extra_args: dict, dataset_varnames: dict, start_date: str | datetime, hgrid_path: str | Path, raw_data_dir: str | Path, output_data_dir: str | Path, regrid_fn, preview: bool = False)#

Process the initial condition (t=0) through the GET → REGRID pipeline.

Parameters:

  • product_name – The name of the data product to retrieve.

  • function_name – The function to call for retrieving data.

  • variables – Variable names to request from the download function (already resolved by the caller from its own product metadata).

  • extra_args – Extra kwargs for the download function (already resolved by the caller).

  • dataset_varnames – Opaque metadata dict forwarded to regrid_fn – this module never reads its keys itself.

  • start_date – The start date (any pandas-parseable string or datetime).

  • hgrid_path – Path to the hgrid supergrid file.

  • raw_data_dir – Directory for raw downloaded data.

  • output_data_dir – Directory for final output files.

  • regrid_fn – Target-specific regrid step, called as regrid_fn(raw_file, hgrid, start_date, output_dir, dataset_varnames) once the raw snapshot has been downloaded. Owns its own idempotency (this engine has no per-chunk state to check – IC is a single snapshot, not a date-chunked series).

  • preview – Return metadata dict without executing, default False.

CrocoDash.extract_forcings.mom6 module#

MOM6-specific OBC/IC handling for CrocoDash.

obc.py/ic.py are model-agnostic engines: they get raw data (and, for OBC, date-chunk/merge it) but know nothing about how to regrid it onto a target model’s grid. This module supplies MOM6’s own regrid step – built on regional_mom6’s segment/experiment classes and mom6_forge’s fill utilities – and the MOM6-specific piece of the GET step (turning a forcing product’s own u/v/eta/tracer var-name metadata into a download request). process_mom6_obc/process_mom6_ic are the entry points driver.py calls; they have the same external shape as the old obc.process_obc_conditions/ic.process_initial_condition did before those were generalized.

CrocoDash.extract_forcings.mom6.build_forcing_request(product_info: dict, function_args: dict = None) → tuple[list, dict]#

Build the (variables, extra_args) an access function needs from a MOM6 forcing product_info dict (u/v/eta/tracer var names).

function_args: user overrides (or access-function defaults) for the access function’s non-required arguments, as written to config.json’s forcing.function_args by configure_forcings()’s function_overrides. Merged into extra_args last so they take precedence over product_info-derived keys.

CrocoDash.extract_forcings.mom6.final_cleanliness_fill(var, x_dim, y_dim, z_dim=None)#
CrocoDash.extract_forcings.mom6.process_mom6_ic(product_name: str, function_name: str, product_information: dict, start_date, hgrid_path, vgrid_path, dataset_varnames: dict, raw_data_dir, output_data_dir, bathymetry_path, preview: bool = False, function_args: dict = None)#

Process the MOM6 initial condition (t=0) through ic.py’s GET → REGRID engine, using regional_mom6’s experiment + mom6_forge’s fill utilities as the regrid step.

Args mirror ic.process_initial_condition’s pre-generalization signature.

CrocoDash.extract_forcings.mom6.process_mom6_obc(start_date, end_date, boundary_number_conversion: dict, product_name: str, function_name: str, product_info: dict, hgrid_path, raw_dataset_path, regridded_dataset_path, output_path, get_step_days=None, regrid_step_days: int = 30, function_args: dict = None, preview: bool = False)#

Process MOM6 boundary conditions through obc.py’s GET → REGRID → MERGE engine, using regional_mom6’s Segment as the regrid step.

Args mirror obc.process_obc_conditions’s pre-generalization signature – see that function’s docstring for the shared-engine parameters, and build_forcing_request/_regrid_obc_chunk above for what’s MOM6-specific.

CrocoDash.extract_forcings.obc module#

OBC (Open Boundary Condition) forcing extraction engine for CrocoDash.

Model-agnostic: this module gets raw data and chunks/merges it, but knows nothing about how to regrid it. Callers (mom6.py, cice.py, ww3.py) supply a regrid_chunk_fn that turns one raw chunk into that target’s own per-segment output file – everything else (GET, date-chunking, idempotency, MERGE) is shared.

Three-phase pipeline per boundary:

  1. GET — download raw data, chunked by get_step (default: full range in

    one request). Chunk size is driven by data-provider constraints (API limits, download size). Each chunk is written as {boundary}_unprocessed.{start}_{end}.nc.

  2. REGRID — validate raw coverage from filenames, then open all raw files

    lazily and regrid (via the caller-supplied regrid_chunk_fn) in regrid_step-sized slices. Chunk size is driven by memory and xESMF performance. GET and REGRID chunks are fully independent.

  3. MERGE — concatenate regridded chunks into forcing_obc_segment_NNN.nc.

Each phase is idempotent: existing output files are detected and skipped, so a failed run can be safely re-started.

CrocoDash.extract_forcings.obc.process_obc_conditions(start_date, end_date, boundary_number_conversion: dict, product_name: str, function_name: str, variables: list, extra_args: dict, dataset_varnames: dict, hgrid_path, raw_dataset_path, regridded_dataset_path, output_path, regrid_chunk_fn, get_step_days=None, regrid_step_days: int = 30, preview: bool = False)#

Process boundary conditions through the GET → REGRID → MERGE pipeline.

Each phase is idempotent. Re-running after a partial failure resumes from the last completed file.

GET and REGRID chunk sizes are independent. GET defaults to the full date range in one request; REGRID defaults to 30-day slices for memory efficiency.

Parameters:

  • start_date – Forcing start date (datetime or any pandas-parseable string).

  • end_date – Forcing end date (datetime or any pandas-parseable string).

  • boundary_number_conversion – Boundary name -> target-model segment number.

  • product_name – Forcing data product name.

  • function_name – Download function name for the product.

  • variables – Variable names to request from the download function (already resolved by the caller from its own product metadata).

  • extra_args – Extra kwargs for the download function (already resolved by the caller).

  • dataset_varnames – Opaque metadata dict forwarded to regrid_chunk_fn – this module never reads its keys itself.

  • hgrid_path – Path to the hgrid supergrid file.

  • raw_dataset_path – Directory for raw downloaded data.

  • regridded_dataset_path – Directory for per-chunk regridded data.

  • output_path – Directory for final, merged output files.

  • regrid_chunk_fn – Target-specific regrid step – see _regrid_boundary.

  • get_step_days – GET chunk size in days; None = full range in one request.

  • regrid_step_days – REGRID chunk size in days.

  • preview – If True, return a dict of expected date pairs without executing any downloads or regridding.

CrocoDash.extract_forcings.runoff module#

CrocoDash.extract_forcings.runoff.generate_rof_ocn_map(rof_grid_name, rof_esmf_mesh_filepath, ocn_mesh_filepath, inputdir, grid_name, rmax, fold)#

Generate runoff to ocean mapping files if runoff is active in the compset.

CrocoDash.extract_forcings.tides module#

CrocoDash.extract_forcings.tides.process_tides(ocn_topo, inputdir, supergrid_path, vgrid_path, tidal_constituents, boundaries, tpxo_elevation_filepath, tpxo_velocity_filepath)#

CrocoDash.extract_forcings.utils module#

CrocoDash.extract_forcings.utils.check_date_continuity(boundary_file_list: dict)#

Check for overlaps or missing dates between consecutive files.

CrocoDash.extract_forcings.utils.fetch_raw_chunk(data_access_fn, dates: list, latlon: dict, output_folder: str | Path, output_filename: str, variables: list, extra_args: dict, name=None) → Path#

Download one raw data chunk, skipping if a valid output file already exists.

Shared by obc.py and ic.py — both fetch a chunk of raw data for a given date range and bounding box, and both need to be idempotent across re-runs.

CrocoDash.extract_forcings.utils.get_data_access_function(product_name: str, function_name: str)#

Load the product registry and return the raw access function for (product_name, function_name).

CrocoDash.extract_forcings.utils.is_valid_netcdf(path: Path) → bool#

Check a file’s magic bytes match a known NetCDF format (HDF5, classic, or 64-bit offset).

CrocoDash.extract_forcings.utils.parse_dataset_folder(folder: str | Path, input_dataset_regex: str, date_format: str)#

Parse a folder to find and extract dataset file information based on a regex pattern.

Parameters:

  • folder (str or Path) – Path to the folder containing the dataset files.

  • input_dataset_regex (str) – Regular expression pattern to match dataset filenames. Example: “(north|east|south|west)_unprocessed.(d{8})_(d{8}).nc”

  • date_format (str) – Date format string used to parse dates in filenames (e.g., “%Y%m%d”).

Returns:

Dictionary mapping boundaries to a list of tuples with: - Start date (datetime) - End date (datetime) - Full file path (Path)

Example: {

”north”: [(datetime(2000, 1, 1), datetime(2000, 1, 2), Path(“/path/to/north_20000101_20000102.nc”))], “east”: [(datetime(2000, 1, 3), datetime(2000, 1, 4), Path(“/path/to/east_20000103_20000104.nc”))]

}

Return type:

dict

Module contents#

Contents