Running Near-Real Time Quality Control (NRT QC)

The NRT QC module applies the automated real-time QC tests recommended for temperature and salinity profiles (Argo/CTD RTQC tests) to an input dataset and writes the original parquet enriched with:

  • one flag column per QC item β€” e.g. temp_qc_spike, qc_impossible_date β€” usable directly as training features, and

  • a final NRT flag per variable β€” temp_nrt_flag / psal_nrt_flag, the most severe flag among the variable’s applicable item columns.

Flags follow the IOC/Argo scheme: 1 (good), 3 (probably bad), 4 (bad).

Quick start

import aiqclib as aq

aq.write_config_template(file_name="/path/to/nrt_qc_config.yaml", stage="nrt_qc")
# Edit the paths, variables, and QC items in the YAML, then:
config = aq.read_config("/path/to/nrt_qc_config.yaml")
aq.run_nrt_qc(config)

The available QC items, their default thresholds, and all configuration sections are described in Configuration of Near-Real Time QC.

One configuration file per region

Some tests are region dependent (most notably the regional range test). Instead of detecting the region from coordinates, prepare one configuration file per region (e.g. Arctic, Baltic, Mediterranean): the files share the same structure and differ only in region-dependent parameters such as the regional ranges or the density inversion threshold.

Temperature-to-salinity propagation

When salinity is computed from temperature and conductivity, a temperature flagged 4 (or 3) corrupts the salinity too. Enabling the temp_to_psal item propagates the final temperature flag onto salinity with its severity, recorded in its own psal_qc_temp_to_psal column so the propagation stays traceable. Datasets with independently measured salinity simply omit the item.

Comparing against existing flags

When the input already carries NRT QC flags (e.g. temp_qc, psal_qc), give each variable its flag column in the qc_variable_sets section:

qc_variable_sets:
  - name: qc_variable_set_1
    variables:
      - name: temp
        flag: temp_qc
        pos_flag_values: [ 4, 6, 7 ]  # optional: for agreement metrics
        neg_flag_values: [ 1 ]

run_nrt_qc then writes one comparison report per variable (nrt_qc_flag_comparison_{variable}.tsv) containing a contingency table of existing vs new flag values, binary agreement metrics (accuracy, precision, recall β€” only when the pos/neg flag values are given), and a per-item breakdown showing which items drive the disagreements. Variables without a flag are skipped; omit all flags to skip the comparison step entirely.

Using QC items as training features

Every QC item is an ordinary feature class registered under a qc_-prefixed name (qc_spike, qc_global_range, …), so the flag columns can also be produced inside the Dataset Preparation pipeline by adding the names to a feature_set.