Running DART as a CESM component

Welcome to the DART_interface user documentation!

Introduction

This documentation will guide you through the process of running DART within the CESM CIME framework.

The goal of the DART_interface is to allow “out-of-the-box” DA setup and execution for CESM models, within the ./case.setup, case.build, and case.submit workflow. This means that you can set up and run a data assimilation case with DART using the standard CESM workflow, without needing to manually configure DART or write custom scripts.

We will cover how to get the code, create a case, set up the case, how to change CESM options and DART namelist options, and finally run the case with data assimilation!

Please note that the science of Data Assimilation is never “out-of-the-box” particularly for coupled Earth system models. You will need to have some understanding of how DART works and how to configure it for your specific use case. This documentation is meant to guide you through the technical process of running DART within CESM, but it is not a tutorial on how to do data assimilation or how to configure DART for your specific use case. For more information on Data Assimilation with DART, visit the docs.dart.ucar.edu.

Get the code

You will need to clone the CROCODILE CESM repository and check out the branch with DART support. The branch is called full_regional_cesm_da. After checking out the branch, you will need to run git-fleximod update to download all the CESM components including DART.

git clone -b full_regional_cesm_da https://github.com/CROCODILE-CESM/CESM.git CESM_DA
cd CESM_DA/  
./bin/git-fleximod update  

Create a case

Compset choice

To use data assimilation you will need to set up a DA enabled compset.

You can see which compsets are available with DART using query__config:

cime/scripts/query_config --compsets dart

This will output something similar to:

Active component: dart
       --------------------------------------
       Compset Alias: Compset Long Name 
       --------------------------------------
   C_DA                 : 2000_DATM%NYF_SLND_MOM6_DROF%NYF_SGLC_SWAV_DART%1
   CR_JRA_DA            : 1850_DATM%JRA_SLND_SICE_MOM6%REGIONAL_SROF_SGLC_SWAV_DART%1
   CR1850MARBL_JRA_DA   : 1850_DATM%JRA_SLND_SICE_MOM6%REGIONAL%MARBL-BIO_SROF_SGLC_SWAV_DART%1
   CR_JRA_GLOFAS_DA     : 1850_DATM%JRA_SLND_SICE_MOM6%REGIONAL_DROF%GLOFAS_SGLC_SWAV_DART%1
   CR1850MARBL_JRA_GLOFAS_DA : 1850_DATM%JRA_SLND_SICE_MOM6%REGIONAL%MARBL-BIO_DROF%GLOFAS_SGLC_SWAV_DART%1
   G_DA                 : 2000_DATM%NYF_SLND_CICE_MOM6_DROF%NYF_SGLC_SWAV_DART%1
   G_JRA_DA             : 2000_DATM%JRA-1p5-2023_SLND_CICE_MOM6_DROF%JRA-1p5-2023_SGLC_SWAV_DART%1
   GR_JRA_DA            : 1850_DATM%JRA_SLND_CICE_MOM6%REGIONAL_SROF_SGLC_SWAV_DART%1
   GR1850MARBL_JRA_DA   : 1850_DATM%JRA_SLND_CICE_MOM6%REGIONAL%MARBL-BIO_SROF_SGLC_SWAV_DART%1
   GR_JRA_GLOFAS_DA     : 1850_DATM%JRA_SLND_CICE_MOM6%REGIONAL_DROF%GLOFAS_SGLC_SWAV_DART%1
   GR1850MARBL_JRA_GLOFAS_DA : 1850_DATM%JRA_SLND_CICE_MOM6%REGIONAL%MARBL-BIO_DROF%GLOFAS_SGLC_SWAV_DART%1

If you are creating a regional CESM case with CrocoDash, follow the CrocoDash instructions to create your CESM case. DART requires multiple instances of the model, which are the number of ensemble members in DART. Specify the number of ensemble members X with --ninst X along with --multi-driver for multiple instances.

If you are not using CrocoDash, you can follow the example below shows to create a case with a DA compset and 3 ensemble members using CESM’s create_newcase.

./cime/scripts/create_newcase \
                    --run-unsupported \
                    --res T62_t232 \
                    --compset G_DA \
                    --case /glade/work/$USER/da-cases/da-mom6-test.0001 \
                    --ninst 3 --multi-driver --project P86850054

--run-unsupported: This flag allows you to create a case that is not officially supported by the CESM team. This is necessary for using DART as a component because it is not yet an officially supported configuration.

--res T62_t232: This is the resolution of the model. T62 is the atmospheric resolution and t232 is the ocean resolution.

--compset G_DA: This specifies the component set to use. G_DA is a G compset with data assimilation using DART.

--case /glade/work/$USER/da-cases/da-mom6-test.0001 is the path where the case will be created. You can choose a different path if you prefer.

--ninst 3: This specifies the number of instances (ensemble members) to run.

--multi-driver: This flag indicates that the case will use multiple driver instances, which is necessary for running DART with multiple ensemble members.

--project P86850054: This is the project code for the case, which is used for accounting purposes on the computing system. You should replace this with your own project code.

Set up the case

cd into the case directory and run ./case.setup to set up the case. This will create the necessary directories and files for the case.

cd /glade/work/$USER/da-cases/da-mom6-test.0001
./case.setup

Case Options

There are a few CESM options that need to be set for the case. These are set with xmlchange commands. The options are:

./xmlchange CALENDAR=GREGORIAN 
./xmlchange DATA_ASSIMILATION_OCN=TRUE 
./xmlchange RUN_STARTDATE=2013-04-01
./preview_namelists --component esp
./case.setup --reset  

CALENDAR=GREGORIAN: required when DART is active. The build stops with an error if the calendar is anything else.

DATA_ASSIMILATION_OCN=TRUE: turns on assimilation for the ocean. Set DATA_ASSIMILATION_ATM, DATA_ASSIMILATION_LND or DATA_ASSIMILATION_ICE to TRUE for the other components. All components with DA enabled must have the same NINST (the same ensemble size). At least one of these must be TRUE when you build the case (./case.build builds filter and friends only for the components you have turned on). Once the case is built, you can set all four to FALSE to run cycles with DA turned off entirely – see Running without DA (spin-up).

DATA_ASSIMILATION_CYCLES: a CIME variable giving the number of times CIME stops the model and calls the DA script within one ./case.submit. The default is DATA_ASSIMILATION_CYCLES=1.

To run multiple DA cycles in one job submittion, set DATA_ASSIMILATION_CYCLES>1

./xmlchange DATA_ASSIMILATION_CYCLES=3

Each cycle assimilate.py is passed a cycle number that starts at 0 and counts up within that job submission only – a resubmitted job starts back at cycle == 0 for its first stop, no matter how many cycles earlier submissions already completed. Anything keyed off cycle == 0 (see Inflation on the first cycle and Ensemble perturbation on cycle 0 below) has to account for that.

RUN_STARTDATE: the start date of the run.

Note

Start date is important, as this is used to match observations to model output. Change the start date to match the observations you want to assimilate. The date format is YYYY-MM-DD.

./preview_namelists --component esp runs the DART buildnml on its own. Do this before ./case.build, because the first time buildnml runs with at least one active DA component, it sets NTASKS_ESP (the number of tasks DART runs filter on) to the largest NTASKS of the active DA components. NTASKS_ESP lives in env_mach_pes.xml, which CESM locks when you run ./case.setup. Running preview_namelists now, followed by ./case.setup --reset, gets NTASKS_ESP set and env_mach_pes.xml re-locked while there is nothing built yet.

Warning

If you skip preview_namelists, NTASKS_ESP is instead changed part way through ./case.build, when CESM generates the namelists. The changed env_mach_pes.xml no longer matches the locked copy, so ./case.submit refuses to run and tells you to run ./case.setup --reset. ./case.setup --reset sets BUILD_COMPLETE=FALSE, so you then have to build the whole case a second time.

Note

NTASKS_ESP is only ever set by buildnml while it is still at CESM’s default value of 1 – i.e. the very first time. Once it has been set, buildnml leaves it alone, even though it reruns on every ./case.submit. This means you only need the preview_namelists / case.setup --reset dance above once, right after your first case.setup. Toggling DATA_ASSIMILATION_* flags later, to change which components are assimilated or to turn DA off entirely, does not change NTASKS_ESP and does not require repeating this step.

./case.setup --reset picks up the xml changes above and re-locks env_mach_pes.xml.

You can query the options with ./xmlquery to make sure they are set correctly:

./xmlquery CALENDAR

Or see all options with

./xmlquery --listall

Or see matching options with

./xmlquery --partial DATA_ASSIM

DART is the ESP component, so its task count and root directory are the ESP variables:

./xmlquery --partial ESP

Build the case

./case.build    

Cleaning and rebuilding DART

If you change DART source code, or otherwise need to force DART be rebuilt, clean the ESP component before building again:

./case.build --clean esp
./case.build

--clean esp only removes DART’s build (the ESP component); it does not clean the build of the other CESM components so ./case.build only has to rebuild DART.

List of observation sequence files

The list of observation sequence files DART will use is in stored in

Buildconf/dart.input_data_list

This file is created by preview_namelists based on your DART_OBS_ROOT setting. See Observations for the directory layout and naming convention DART expects, and how to point DART_OBS_ROOT at your own observation archive.

DART input.nml

DART’s runtime options are in a Fortran namelist file called input.nml. You do not edit this file directly. Instead, DART’s buildnml writes one input.nml per active DA component, starting from a template of DART defaults for that component’s model and applying your changes from user_nl_dart.

The dart input.nml created by preview_namelists is in

Buildconf/dartconf/input.nml.ocn

Since in this example we have set DATA_ASSIMILATION_OCN=TRUE, we are only assimilating ocean observations and therefore the input.nml file is input.nml.ocn. If you were assimilating atmosphere observations, the input.nml file would be input.nml.atm and so on for land and sea-ice.

During the run, assimilate.py copies input.nml.{comp} to $RUNDIR/input.nml before running filter_{comp} for that component, once per component per assimilation cycle.

Note

MOM6 also reads a file called input.nml, in the same run directory. assimilate.py backs the MOM6 file up as mom_input.nml.bak before staging DART’s, and restores it when filter has finished, so the input.nml you find in $RUNDIR after a run is MOM6’s, not DART’s. Look in Buildconf/dartconf/ for DART’s.

Changing namelist options with user_nl_dart

./case.setup puts a user_nl_dart in your case directory. Add the namelist groups and variables you want to change, using normal Fortran namelist syntax:

&filter_nml
  inf_flavor = 5, 0
  inf_initial = 1.0, 1.0
  inf_sd_initial = 0.6, 0.0
/

&assim_tools_nml
  cutoff = 0.1
/

user_nl_dart applies to all active DA components.

Component specific namelist options

The components assimilate different observations, of different quantities, at different scales, so when running DA on mulitple components you will need to set some options per component rather than shared. To set an option for one component only, create a user_nl_dart_{comp} file in your case directory:

File	Component	DART model
user_nl_dart_ocn	ocean	MOM6
user_nl_dart_atm	atmosphere	CAM (CAM_DYCORE selects cam-fv or cam-se)
user_nl_dart_lnd	land	CLM
user_nl_dart_ice	sea-ice	CICE

Note

./case.setup creates user_nl_dart for you, but not the user_nl_dart_{comp} files. Create the ones you need yourself, in $CASEROOT.

Values are merged variable by variable, in this order, last one wins:

1. the DART default for that component, from param_templates/json/input_nml_{model}.json

2. user_nl_dart

3. user_nl_dart_{comp}

So a user_nl_dart_{comp} file only needs to contain the variables that differ from user_nl_dart; you can use both files together. For example, to use a 0.1 radian localization everywhere except the ocean, where you want 0.05:

# user_nl_dart
&assim_tools_nml
  cutoff = 0.1
/

# user_nl_dart_ocn
&assim_tools_nml
  cutoff = 0.05
/

input.nml.ocn is then written with cutoff = 0.05, and every other active component gets cutoff = 0.1.

Namelist options you should not set

Some options are set from the case xml variables and are overwritten by buildnml after your user_nl_dart changes are applied. Setting these in user_nl_dart has no effect:

Namelist variable	Set from
filter_nml:input_state_file_list	fixed as filter_input_list.txt, written each cycle by assimilate.py
filter_nml:output_state_file_list	fixed as filter_output_list.txt, written each cycle by assimilate.py
ensemble_manager_nml:tasks_per_node	MAX_TASKS_PER_NODE
ensemble_manager_nml:layout	fixed at 2

Warning

Do not set filter_nml:ens_size in user_nl_dart. It is set from NINST of the active DA components, but unlike the variables in the table above, a value in user_nl_dart overrides it. That does not change the number of ensemble members CESM runs, it only makes input.nml disagree with the case, and filter will fail. Change the ensemble size with --ninst at create_newcase instead.

Only variables that already exist in the component’s template can be set. A misspelled variable, or one that belongs to a DART program that is not part of this component’s build, is skipped with a warning rather than being written to input.nml:

Variable inf_sd_intial in filter_nml not found in template
Namelist filtr_nml not found in template

Checking your changes

Regenerate the namelists and look at the result:

./preview_namelists --component esp

This rewrites Buildconf/dartconf/input.nml.{comp} and prints any of the warnings above, so it is worth doing whenever you edit user_nl_dart. Namelist changes do not need a rebuild: CESM regenerates the namelists during ./case.submit.

Inflation on the first cycle

When using inf_initial_from_restart=.true. and inf_sd_initial_from_restart=.true., on the first cycle of a new case, there are no existing inflation files for filter to read.

assimilate.py turns the from_restart flags off for the first cycle and filter starts from the nameslist values (e.g. 0.6, 0.6) for inf_initial and inf_sd_initial.

Once the first inflation file exists, inf_initial_from_restart and inf_sd_initial_from_restart are set back to .true.

Warning

inf_sd_initial must be greater than zero for inflation to adapt. With inf_sd_initial = 0.0, which is the template default, the zero is written into the inflation restart and read back on every subsequent cycle, so inflation stays fixed at inf_initial for the whole run, not just the first cycle. 0.6 is a common starting value.

Ensemble perturbation on cycle 0

A cold-start multi-instance case starts every instance from an identical restart, so the ensemble has no spread until something perturbs it. Set filter_nml:perturb_from_single_instance = .true. in user_nl_dart for this case: filter then reads only the first instance’s restart and perturbs it into the full ensemble, instead of reading all NINST per-instance restarts.

assimilate.py restricts perturb_from_single_instance = .true. to cycle == 0: it is forced back to .false. on every cycle > 0, regardless of the setting in user_nl_dart. If you leave the namelist value .false. (the template default), nothing is forced on at any cycle. perturbation_amplitude controls the size of the cycle-0 perturbation and is ignored once perturb_from_single_instance is .false..

Warning

If you resubmit a job mid-experiment with perturb_from_single_instance still .true., CIME’s cycle counter restarts at 0 for the new job submission (see DATA_ASSIMILATION_CYCLES above), so cycle 0 perturbs again and discards the ensemble spread already built up by assimilation. Set it back to .false. in user_nl_dart before resubmitting an in-progress multi-submittion experiment.

Running without DA (spin-up)

You may want to run some cycles without assimilating anything – for example a spin-up period before starting DA, or to pause assimilation partway through an experiment. To do this, set all four DATA_ASSIMILATION_* flags to FALSE:

./xmlchange DATA_ASSIMILATION_OCN=FALSE

With no active DA components, buildnml skips namelist generation for DART and assimilate.py logs that it is skipping DA and returns, so ./case.submit runs the model forward without calling filter for any component. Turn one or more flags back on to resume assimilation on a later cycle; no rebuild or case.setup --reset is needed either way, since NTASKS_ESP is fixed at build time (see the note above).

Note

You must still build the case with at least one DATA_ASSIMILATION_* flag TRUE – ./case.build requires at least one active component so that the corresponding filter_{comp} executable exists. Turning all flags off only skips assimilation at run time; it does not change what was built.

Run the case

./case.submit