Preparing Real Observations

Turn real Argo profiles from CrocoLake into DART obs_seq files for your regional MOM6 domain!

A typical observation-preparation workflow consists of these steps:

Set up dartobsgen
Define your experiment parameters: what observations do you need and when?
Generate obs_seq files from CrocoLake.
Inspect the result with pyDARTdiags.

This is Part 1 of the DART tutorial series:
1. Working with Real Observations · 2. Creating Synthetic Observations · 3. Cycling DART–CESM

Before you start¶

You need:

Access to Derecho (or an equivalent HPC system with CrocoLake installed).
The CrocoDash conda environment — see the CrocoDash tutorial if you have not set this up yet.

dartobsgen installed into your environment:

git clone https://github.com/CROCODILE-CESM/dartobsgen.git
cd dartobsgen
pip install -e .

A DART clone, dartobsgen calls DART’s CrocoLake observation converter:
For this tutorial you can use the DART provided on Derecho
```
/glade/work/hkershaw/croc26/DART
```
If you’re following this tutorial using a machine other than Derecho, you can download DART from GitHub.

SECTION 1: How DART sees an observation¶

Step 1.1: Anatomy of an obs_seq file¶

DART does not read observations from netCDF, CSV, or parquet. It reads its own observation sequence (obs_seq) format. Every observation in an obs_seq file has these five pieces of information:

Field	Meaning
type	What was measured and by what platform, e.g. `ARGO_TEMPERATURE`
location	Longitude, latitude (stored in radians) and a vertical coordinate
value	The measured value
error variance	How much you trust the measurement (instrument + representativeness error, squared)
time	Days and seconds since the DART epoch, 1601-01-01

Here is one observation from an obs_seq file, annotated:

 OBS            1
   28.618                        <-- observation value (e.g. Temp in degrees C)
   0.0000                        <-- quality control flag
          -1         2        -1 <-- linked list bookkeeping (ignore)
obdef
loc3d
   0.858377  0.141890   10.0  3  <-- lon (rad), lat (rad), vertical, vertical coord type
kind
          10                     <-- observation type number (ARGO_TEMPERATURE)
 3600     150512                 <-- time: seconds-of-day, days since 1601-01-01
   0.04                          <-- observation error variance

The error variance is a measure of uncertainty: DART weights each observation against the ensemble spread using this number. An observation with variance 0.04 — that is, (0.2°C)² — pulls the model harder than an observation with variance 1.0. For real observations the error variance is given by the instrument. In Tutorial 2 “synthetic obs” you will choose error variance yourself.

The forward operator is how DART compares an observation to the model: it samples the model state at the observation’s location. So given an observation type and location, what does the model ‘think’ the observation should be? DART’s MOM6 interface provides the forward operator for each observation type.

Step 1.2: Assimilation windows and file naming¶

A cycling assimilation advances the model in fixed steps and assimilates assimilates observations within a time window centered on the model stop time. In this tutorial we will use 24 hours, i.e. stopping the model once per day to assimilate observations.

DART expects one obs_seq file per window, named by the window center time. The file extension follows CESM’s yyyy-mm-dd-sssss date format. The example below shows 3 observation sequence files, and the time the file extension corresponds to.

obs_seq.2013-04-01-00000.out     # 2013-04-01 00:00:00
obs_seq.2013-04-02-00000.out     # 2013-04-02 00:00:00
obs_seq.2013-04-03-00000.out     # 2013-04-03 00:00:00

The assimilation frequency (freq) is the length of the window, and the center of the window is the analysis time, T. So the window runs from T - freq/2 to T + freq/2.

Windows are half-open, (T - freq/2, T + freq/2], so no observation ever lands in two files.

With a 24-hour frequency and START set to midnight, every window is centered on midnight (noon to noon), and holds 24 hours of observations.

SECTION 2: Set up your experiment parameters¶

Step 2.1: The shared parameters cell¶

Every notebook in this series starts from the same parameters cell below. Set DA_PROJECT_DIR to a scratch directory you own. This is where your real and synthetic observations will be stored.

# --- CROCODILE DART tutorial series parameters (same cell in all 3 notebooks) ---
from pathlib import Path
import datetime

DA_PROJECT_DIR = Path("<DART_DA_PROJECT_DIR>")        # your DA project working directory on Derecho

START = datetime.datetime(2023, 6, 15)   # must match RUN_STARTDATE in Tutorial 3
END   = datetime.datetime(2023, 6, 18)   # 3 days -> 3 one-day assimilation windows, centered on midnight
FREQ  = datetime.timedelta(hours=24)

# Bounding box:
LAT_MIN, LAT_MAX = 20.0, 25.0
LON_MIN, LON_MAX = -160.0, -155.0

OBS_TYPES = ["ARGO_TEMPERATURE", "ARGO_SALINITY"]

REAL_OBS_DIR      = DA_PROJECT_DIR / "obs" / "real" / "ocn_obs_seq"      # Tutorial 1 output
SYNTHETIC_OBS_DIR = DA_PROJECT_DIR / "obs" / "synthetic" / "ocn_obs_seq" # Tutorial 2 output

Step 2.2: What’s in CrocoLake?¶

CrocoLake is the CROCODILE observational database. It contains quality-controlled in-situ ocean observations in a single parquet store. We’ll use dartobsgen to convert the following CrocoLake observation types to DART obs_seq format:

Platform	DART observation types
Argo floats	`ARGO_TEMPERATURE`, `ARGO_SALINITY`
Spray gliders	`GLIDER_TEMPERATURE`, `GLIDER_SALINITY`
GLODAP bottle data	`BOTTLE_TEMPERATURE`, `BOTTLE_SALINITY`

We will use Argo temperature and salinity in this tutorial.

SECTION 3: Configure and generate¶

Step 3.1: Configure obs generation¶

ObsGenConfig describes what you want: the time period, the spatial box, the observation types, and the assimilation frequency. The source object describes where the observations come from: in this tutorial the source is CrocoLake plus DART (which provides the converter).

Argument by argument:

start / end - the experiment period. start must match the RUN_STARTDATE you will set in Tutorial 3.
lat_min, lat_max, lon_min, lon_max- a bounding box for the domain you are interested in.
obs_types - which CrocoLake observations to convert.
assimilation_frequency - the window length; one output file per window.
output_dir - where the obs_seq files are written to.

from dartobsgen import ObsGenConfig, CrocLakeSource, generate_obs_sequences

config = ObsGenConfig(
    start=START,                      # must match RUN_STARTDATE in Tutorial 3
    end=END,
    lat_min=LAT_MIN, lat_max=LAT_MAX,
    lon_min=LON_MIN, lon_max=LON_MAX,
    obs_types=OBS_TYPES,
    assimilation_frequency=FREQ,      # one obs_seq file per 24-hour window
    output_dir=REAL_OBS_DIR,
)

source = CrocLakeSource(
    crocolake_path="<CROCOLAKE>",     # Path to CrocoLake on Derecho from "Before you start"
    dart_path="<DART_SRC>",           # Path to DART from "Before you start"
)

Step 3.2: Generate the obs_seq files¶

generate_obs_sequences walks the windows, queries CrocoLake for each window, and writes one file per window that contains observations. Empty windows are skipped. Over a 5°×5° domain and 24-hour windows, expect some gaps in the Argo data.

REAL_OBS_DIR.mkdir(parents=True, exist_ok=True)

written_files = generate_obs_sequences(config, source, max_workers=None)

print(f"{len(written_files)} obs_seq file(s) written to {REAL_OBS_DIR.name}/")
for p in written_files:
    print("  ", Path(p).name)

Answer

Twice as many files, each holding roughly half the observations. Shorter windows respect observation timing better, but over sparse data like Argo profiles, more of those 12-hour windows come up empty. The 24-hour default gives the filter more observations per update, at the cost of treating a profile up to 12 hours old as valid at the midnight analysis time.

For ocean assimilation a 24-hour window is typical.

SECTION 4: Inspect what you made¶

Step 4.1: Load an obs_seq file with pyDARTdiags¶

pyDARTdiags reads an obs_seq file into a Pandas DataFrame - one row per observation, with longitude and latitude converted to degrees. Let’s inspect one of the observation sequence files you made.

import pydartdiags.obs_sequence.obs_sequence as obsq

obs_seq = obsq.ObsSequence(str(Path(written_files[0])))

print(f"{Path(written_files[0]).name}: {len(obs_seq.df)} observations")
print(obs_seq.df["type"].value_counts())
obs_seq.df.head()

Step 4.2: Map the observation locations¶

Where are the profiles, and how deep do they go?

import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature

def lon180(lon):
    """Convert 0-360 longitudes (DART convention) to -180..180 for plotting."""
    lon = np.asarray(lon, dtype=float)
    return np.where(lon > 180, lon - 360, lon)


fig, ax = plt.subplots(figsize=(6, 5), subplot_kw={"projection": ccrs.PlateCarree()})
sc = ax.scatter(lon180(obs_seq.df["longitude"]), obs_seq.df["latitude"],
                 transform=ccrs.PlateCarree())

ax.coastlines(resolution="10m")
ax.add_feature(cfeature.LAND, facecolor="lightgray")
ax.set_extent([LON_MIN, LON_MAX, LAT_MIN, LAT_MAX], crs=ccrs.PlateCarree())
ax.gridlines(draw_labels=True)

ax.set_title(f"Observation locations, first window ({len(obs_seq.df)} obs)")
plt.show()