Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Preparing Observations for Your DART Experiment

Turn real Argo profiles from CrocoLake into DART obs_seq files scoped to your regional MOM6 domain!

A typical observation-preparation workflow consists of five main steps:

  1. Understand how DART ingests observations.

  2. Set up dartobsgen and your experiment parameters.

  3. Generate obs_seq files from CrocoLake.

  4. Inspect the result with pyDARTdiags.

  5. Trim the observations to your model domain.

This is Part 1 of the DART tutorial series: 1. Preparing Real Observations · 2. Creating Synthetic Observations · 3. Cycling DART–CESM

How this notebook feeds the rest of the series:

Tutorial 1: ──▶ obs/real/obs_seq.*.out ──▶ ready for DART
                                        │
                                        ├──▶ Tutorial 2: harvest locations for a synthetic network
                                        └──▶ Tutorial 3: assimilate observations into CESM-MOM6

Before you start

You need:

  • Access to Derecho (or an equivalent HPC system with CrocoLake staged).

  • The CrocoDash conda environment — see the CrocoDash tutorial if you have not set this up yet.

  • A DART clone — dartobsgen calls DART’s CrocoLake observation converter:

    git clone https://github.com/NCAR/DART.git

  • dartobsgen installed into your environment:

    git clone https://github.com/CROCODILE-CESM/dartobsgen.git
cd dartobsgen
pip install -e .

SECTION 1: How DART sees an observation

Step 1.1: Anatomy of an obs_seq file

DART does not read observations from netCDF, CSV, or parquet — it reads its own observation sequence (obs_seq) format. Every observation in an obs_seq file carries five pieces of information:

FieldMeaning
typeWhat was measured and by what platform, e.g. ARGO_TEMPERATURE
locationLongitude, latitude (stored in radians) and a vertical coordinate
valueThe measured value
error varianceHow much you trust the measurement (instrument + representativeness error, squared)
timeDays and seconds since the DART epoch, 1601-01-01

Here is one observation from a real file, annotated:

 OBS            1
   28.618                        <-- observation value (degrees C)
   0.0000                        <-- quality control flag
          -1           2      -1 <-- linked list bookkeeping (ignore)
obdef
loc3d
   4.858377  0.141890   10.0  3  <-- lon (rad), lat (rad), vertical, vertical coord type
kind
          10                     <-- observation type number (ARGO_TEMPERATURE)
 106740   150512                 <-- time: seconds-of-day, days since 1601-01-01
   0.04                          <-- observation error variance

The error variance is a measure of uncertainty: DART weights each observation against the ensemble spread using this number. An observation with variance 0.04 — that is, (0.2 °C)² — pulls the model harder than one with variance 1.0. For real observations the converter supplies it; in Tutorial 2 you will choose it yourself.

The forward operator is how DART compares an observation to the model: it samples the model state at the observation’s location and depth. So given an observation type and location, what does the model ‘think’ the observation should be? DART’s MOM6 interface provides the forward operator for each observation type.

Step 1.2: Assimilation windows and file naming

A cycling assimilation advances the model in fixed steps (we will use 6 hours) and assimilates, at each stop, the observations that fall inside that window. DART therefore expects one obs_seq file per window, named by the window center time:

obs_seq.2013-04-01-00000.out     # 2013-04-01 00:00, seconds-of-day = 0
obs_seq.2013-04-01-21600.out     # 2013-04-01 06:00, seconds-of-day = 21600
obs_seq.2013-04-01-43200.out     # 2013-04-01 12:00
...

The trailing number is seconds of day, zero-padded to five digits - following CESM’s yyyy-mm-dd-sssss date format. Windows are half-open, [t0, t0 + freq), so no observation ever lands in two files.

SECTION 2: Set up your experiment parameters

Step 2.1: The shared parameters cell

Every notebook in this series starts from the same parameters cell below. Set WORKDIR to a scratch directory you own — everything the series produces lives under it.

# --- CROCODILE DART tutorial series parameters (same cell in all 3 notebooks) ---
from pathlib import Path
import datetime

WORKDIR = Path("<DART_WORKDIR>")        # your scratch working directory on Derecho

START = datetime.datetime(2013, 4, 1)   # must match RUN_STARTDATE in Tutorial 3
END   = datetime.datetime(2013, 4, 4)   # 3 days -> 12 six-hour assimilation windows
FREQ  = datetime.timedelta(hours=6)

# Bounding box: the panama1 domain (lon 278-281E, lat 7-10N) padded by ~2 degrees
LAT_MIN, LAT_MAX = 5.0, 12.0
LON_MIN, LON_MAX = -84.0, -77.0

OBS_TYPES = ["ARGO_TEMPERATURE", "ARGO_SALINITY"]

REAL_OBS_DIR      = WORKDIR / "obs" / "real"       # Tutorial 1 output
SYNTHETIC_OBS_DIR = WORKDIR / "obs" / "synthetic"  # Tutorial 2 output

Step 2.2: What’s in CrocoLake?

CrocoLake is the CROCODILE observational database — quality-controlled in-situ ocean observations in a single parquet store. dartobsgen can convert the following CrocoLake observation types to DART obs_seq format:

PlatformDART observation types
Argo floatsARGO_TEMPERATURE, ARGO_SALINITY, ARGO_DOXY
Spray glidersGLIDER_TEMPERATURE, GLIDER_SALINITY
GLODAP bottle dataBOTTLE_TEMPERATURE, BOTTLE_SALINITY, nutrients, carbon system

We will use Argo temperature and salinity — the workhorse of ocean data assimilation. See the dartobsgen README for the full list.

SECTION 3: Configure and generate

Step 3.1: Configure obs generation

ObsGenConfig describes what you want: the time period, the spatial box, the observation types, and the assimilation frequency. The source object describes where the observations come from — here, CrocoLake plus your DART clone (which provides the converter).

Argument by argument:

  • start / end — the experiment period. start must match the RUN_STARTDATE you will set in Tutorial 3.

  • lat_min … lon_max — a bounding box slightly larger than the model domain. We trim precisely to the ocean mask in Section 5.

  • obs_types — which CrocoLake observations to convert.

  • assimilation_frequency — the window length; one output file per window.

  • output_dir — where the obs_seq files land.

from dartobsgen import ObsGenConfig, CrocLakeSource, generate_obs_sequences

config = ObsGenConfig(
    start=START,                      # must match RUN_STARTDATE in Tutorial 3
    end=END,
    lat_min=LAT_MIN, lat_max=LAT_MAX,
    lon_min=LON_MIN, lon_max=LON_MAX,
    obs_types=OBS_TYPES,
    assimilation_frequency=FREQ,      # one obs_seq file per 6-hour window
    output_dir=REAL_OBS_DIR,
)

source = CrocLakeSource(
    crocolake_path="<CROCOLAKE>",     # CrocoLake parquet database on Derecho
    dart_path="<DART_SRC>",           # your DART clone from "Before you start"
)

Step 3.2: Generate the obs_seq files

generate_obs_sequences walks the windows, queries CrocoLake for each one, and writes one file per window that contains observations. Empty windows are skipped — over a 3°×3° domain and 6-hour windows, expect some gaps; Argo floats surface roughly every 10 days.

REAL_OBS_DIR.mkdir(parents=True, exist_ok=True)

written_files = generate_obs_sequences(config, source, max_workers=None)

print(f"{len(written_files)} obs_seq file(s) written to {REAL_OBS_DIR.name}/")
for p in written_files:
    print("  ", Path(p).name)
Answer

Half as many files, each holding roughly twice the observations. Longer windows give the filter more data per update — but every observation in a window is compared against the same model state, so a 12-hour-old profile is treated as if it were valid now. Shorter windows respect observation timing better at the cost of more frequent (and more expensive) assimilation stops.

SECTION 4: Inspect what you made

Step 4.1: Load an obs_seq file with pyDARTdiags

pyDARTdiags reads any obs_seq file into a pandas DataFrame — one row per observation, with longitude and latitude converted to degrees.

import pydartdiags.obs_sequence.obs_sequence as obsq

obs_seq = obsq.ObsSequence(str(Path(written_files[0])))

print(f"{Path(written_files[0]).name}: {len(obs_seq.df)} observations")
print(obs_seq.df["type"].value_counts())
obs_seq.df[["type", "longitude", "latitude", "vertical", "observation", "obs_err_var"]].head()

Step 4.2: Map the observation locations

Where are the profiles, and how deep do they go?

import numpy as np
import matplotlib.pyplot as plt


def lon180(lon):
    """Convert 0-360 longitudes (DART convention) to -180..180 for plotting."""
    lon = np.asarray(lon, dtype=float)
    return np.where(lon > 180, lon - 360, lon)


fig, ax = plt.subplots(figsize=(6, 5))
sc = ax.scatter(lon180(obs_seq.df["longitude"]), obs_seq.df["latitude"],
                c=obs_seq.df["vertical"], cmap="viridis_r", s=14)
ax.set_xlim(LON_MIN, LON_MAX)
ax.set_ylim(LAT_MIN, LAT_MAX)
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
ax.set_title(f"Observation locations, first window ({len(obs_seq.df)} obs)")
plt.colorbar(sc, label="vertical (m)")
plt.show()
Answer
obs_seq.df[obs_seq.df["type"] == "ARGO_TEMPERATURE"]["vertical"].plot.hist(bins=40)

Most observations sit in the upper few hundred metres, so the filter has little direct information at depth. Deep corrections come only from the ensemble’s own correlations between surface and deep state — which is exactly why localization and inflation (Tutorial 3) matter.

SECTION 5: Trim to your model domain

Step 5.1: Why the bounding box isn’t enough

The box we queried is a rectangle padded well beyond the panama1 domain, so it captures observations that are outside the model grid entirely, or sit on model land. Assimilating an observation the model cannot represent wastes work at best and rejects obs at worst (they fail the forward operator).

The precise definition of “inside your model” is the ocean mask of your CrocoDash-generated case. dartobsgen can turn that mask into a polygon and drop every observation outside it.

from dartobsgen import trim_obs_seq, polygon_from_netcdf_mask

ocean_mask = "<CESM_CASE>/ocean_mask.nc"              # from your CrocoDash case inputdir
# No case yet? Use the staged workshop copy instead:
# ocean_mask = "<CROC_DART_OBS>/panama1_ocean_mask.nc"

poly = polygon_from_netcdf_mask(
    ocean_mask,
    mask_var="mask",
    lat_var="lat",
    lon_var="lon",
)

# Trims in place: observations outside the polygon are removed from each file.
for path in written_files:
    kept = trim_obs_seq(path, poly)
    print(f"  {Path(path).name}: {'kept' if kept else 'no obs left in domain'}")

Step 5.2: Before and after

trim_obs_seq modified the files in place, so re-reading the first window shows what survived. The obs_seq variable from Section 4 still holds the pre-trim contents.

trimmed = obsq.ObsSequence(str(Path(written_files[0])))

fig, ax = plt.subplots(figsize=(6, 5))
x, y = poly.exterior.xy
ax.plot(lon180(x), np.asarray(y), color="tab:green", lw=2, label="model domain (ocean mask)")
ax.scatter(lon180(obs_seq.df["longitude"]), obs_seq.df["latitude"],
           s=18, color="lightgray", label=f"before trim ({len(obs_seq.df)})")
ax.scatter(lon180(trimmed.df["longitude"]), trimmed.df["latitude"],
           s=18, color="tab:blue", label=f"after trim ({len(trimmed.df)})")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
ax.set_title("Observations kept for assimilation")
ax.legend()
plt.show()

Recap

Your takeaway product: obs/real/ under your WORKDIR — 12 windows of Argo temperature and salinity, trimmed to the panama domain.

Where to go from here?

Model-Observation Comparison Examples
DART Data Assimilation
Model-Observation Comparison Examples
Creating Synthetic Observations