Configuration of Near-Real Time QC

The nrt_qc workflow (stage="nrt_qc") applies automated real-time QC tests to temperature and salinity profile data and writes the input parquet enriched with one flag column per QC item plus a final NRT flag per variable (see the Running Near-Real Time Quality Control (NRT QC) guide for the workflow overview).

Prepare one configuration file per region (e.g. Arctic, Baltic, Mediterranean): the files share the same structure and differ only in region-dependent parameters such as the regional ranges.

Detailed Configuration Sections

path_info_sets

Defines the location of the input data and the output directories.

• common.base_path: The root output directory.

• input.base_path: The directory containing the input parquet files.

• concat.step_folder_name: The subdirectory for the final output parquet (the intermediate flag file of the qc step and the comparison reports of the compare step default to folders named after their steps).

path_info_sets:
  - name: nrt_qc_path_1
    common:
      base_path: /path/to/data
    input:
      base_path: /path/to/input
      step_folder_name: ""
    concat:
      step_folder_name: nrt_qc

qc_variable_sets

The variables to run the QC items on. Only name is required. The optional flag names an existing NRT QC flag column in the input and enables the flag comparison report for that variable; pos_flag_values / neg_flag_values additionally enable the binary agreement metrics (same convention as the other modules).

qc_variable_sets:
  - name: qc_variable_set_1
    variables:
      - name: temp
        flag: temp_qc # Optional: existing NRT QC flag column
        pos_flag_values: [ 4, 6, 7 ] # Optional: for agreement metrics
        neg_flag_values: [ 1 ]
      - name: psal
        flag: psal_qc
        pos_flag_values: [ 4, 6, 7 ]
        neg_flag_values: [ 1 ]

qc_item_sets

The QC items to apply. An item runs only if listed; params overrides the built-in defaults per key, and the optional fail_flag (3 or 4, default 4) softens a failing test to “probably bad”.

Item	RTQC	Level	Output column(s)
impossible_date	RTQC2	profile	qc_impossible_date
impossible_location	RTQC3	profile	qc_impossible_location
global_range	RTQC6	observation	{var}_qc_global_range
regional_range	RTQC7	observation	{var}_qc_regional_range
pressure_increasing	RTQC8	observation	qc_pressure_increasing
spike	RTQC9	observation	{var}_qc_spike
gradient	RTQC11	observation	{var}_qc_gradient
digit_rollover	RTQC12	observation	{var}_qc_digit_rollover
stuck_value	RTQC13	profile	{var}_qc_stuck_value
density_inversion	RTQC14	observation	temp_qc_... and psal_qc_...
temp_to_psal	—	observation	psal_qc_temp_to_psal

qc_item_sets:
  - name: qc_item_set_1
    items:
      - name: impossible_date
      - name: impossible_location
      - name: global_range
        params: { temp: { min: -2.5, max: 40.0 },
                  psal: { min: 2.0, max: 41.0 } }
      - name: regional_range # EDIT: ranges of this file's region
        params: { temp: { min: 10.0, max: 40.0 },
                  psal: { min: 2.0, max: 40.0 } }
      - name: pressure_increasing
      - name: spike
        params: { temp: { shallow: 6.0, deep: 2.0 },
                  psal: { shallow: 0.9, deep: 0.3 },
                  depth_threshold: 500 }
      - name: gradient
        params: { temp: { shallow: 9.0, deep: 3.0 },
                  psal: { shallow: 1.5, deep: 0.5 },
                  depth_threshold: 500 }
      - name: digit_rollover
        params: { temp: 10.0, psal: 5.0 }
      - name: stuck_value
      - name: density_inversion
        params: { threshold: 0.03 }
      - name: temp_to_psal

Notes:

• regional_range has no built-in defaults — supply your region’s ranges, or the item raises an error (no silent pass).

• spike / gradient use the shallow threshold below depth_threshold (in decibars) and deep at or beyond it.

• density_inversion computes the potential density anomaly sigma-0 (UNESCO 1983) and allows inversions up to threshold (kg/m³); it flags temperature and salinity jointly.

• temp_to_psal propagates the final temperature flag onto salinity (see the Running Near-Real Time Quality Control (NRT QC) guide); omit it for independently measured salinity.

step_class_sets

The Python classes for the four NRT QC steps: reading the input (input), running the QC items (qc), aggregating the final flags and writing the output (concat), and the optional flag comparison (compare).

step_class_sets:
  - name: nrt_qc_step_set_1
    steps:
      input: InputDataSetAll
      qc: QCDataSetAll
      concat: ConcatDataSetAll
      compare: CompareFlagsAll

step_param_sets

Optional parameters per step. The input step shares the sub-step options of the other modules (column renaming, row filtering, column creation and validation); the NRT QC template disables row filtering by default.

step_param_sets:
  - name: nrt_qc_param_set_1
    steps:
      input: { sub_steps: { rename_columns: false,
                            filter_rows: false } }
      qc: { }
      concat: { }
      compare: { }

nrt_qc_sets

Ties everything together for one NRT QC run: the dataset folder, the input file, and the named sub-configurations to use.

nrt_qc_sets:
  - name: nrt_qc_0001
    dataset_folder_name: nrt_qc_0001
    input_file_name: nrt_cora_bo_4.parquet
    path_info: nrt_qc_path_1
    qc_variable_set: qc_variable_set_1
    qc_item_set: qc_item_set_1
    step_class_set: nrt_qc_step_set_1
    step_param_set: nrt_qc_param_set_1