Quality statistics — quality-collect, quality-combine¶
These commands wrap AOQuality to compute and aggregate visibility statistics across a set of MeasurementSets.
The workflow has two stages:
quality-collect— compute per-MS statistics and write them into each MS directory (in parallel, one job per MS).quality-combine— merge all per-MS statistics into a single.qsfile that can be plotted withaostats plot-gridor screened withaostats find-bad-obs/aostats find-bad-stations.
quality-collect — compute per-MS statistics in parallel¶
nenudata quality-collect L2 "202312*_NT04:SW03"
Runs aoquality collect -d DATA on every MS matching the pattern. Jobs are
distributed across nodes via the same worker pool as l1_to_l2.
# Collect on CORRECTED_DATA (e.g. after calibration) with 6 concurrent jobs
nenudata quality-collect L2_12C40S "202312*_CASA:SW03" \
-d CORRECTED_DATA -m 6
# Dry-run: print commands without executing
nenudata quality-collect L2 "202312*_NT04:SW03" --dry-run
Options¶
Option |
Default |
Description |
|---|---|---|
|
|
MS column passed to |
|
|
Max concurrent collect jobs per node |
|
false |
Print commands without executing |
|
|
Shell environment file sourced before each job |
|
|
Data-handler config file |
Node routing¶
For N1 obs_ids, each MS runs on the node assigned to that obs_id and SW.
For N2 obs_ids, MSs are round-robined across the N2 node pool in the same
order they are spread by l1_to_l2.
quality-combine — merge per-MS statistics into one file¶
nenudata quality-combine L2 "202312*_NT04:SW03" quality_l2/SW03/combined.qs
Collects the paths of all MSs matching the pattern, then runs a single
aoquality combine call:
aoquality combine quality_l2/SW03/combined.qs MS1 MS2 MS3 ...
The output path (OUT_QS) is a mandatory positional argument and must be the
full path to the desired output file. Its parent directory is created
automatically.
# Combine all SWs into separate files (one command per SW)
nenudata quality-combine L2 "202312*_NT04:SW03" quality/SW03/combined.qs
nenudata quality-combine L2 "202312*_NT04:SW04" quality/SW04/combined.qs
Options¶
Option |
Default |
Description |
|---|---|---|
|
|
Data-handler config file |
Downstream analysis¶
The .qs file produced by quality-combine can be consumed directly by the
aostats command-line tool (from the aoquality package):
# Frequency × obs / antenna × obs / LST × obs heatmaps
aostats plot-grid quality/SW03/combined.qs SNR -o plots/
# Flag observations with anomalous LST trends
aostats find-bad-obs quality/SW03/combined.qs SNR
# Flag stations with outlier statistics (CASA calibrator data)
aostats find-bad-stations quality/SW03/combined.qs SNR -o bad_stations.json
Reference¶
nenudata quality-collect¶
Run “aoquality collect” in parallel across all MSs for OBS_IDS at LEVEL.
Collect writes statistics into each individual MS (or its sidecar table). Run “quality-combine” afterwards to merge them into a combined.qs per observation.
Usage
nenudata quality-collect [OPTIONS] LEVEL OBS_IDS
Options
- -c, --config <config>¶
Data handler configuration file
- -d, --data-column <data_column>¶
MS column to pass to “aoquality collect -d”
- Default:
'DATA'
- -m, --max-concurrent <max_concurrent>¶
Maximum concurrent collect tasks per node
- Default:
1
- --dry-run¶
Print commands without running them
- --env-file <env_file>¶
Shell environment file sourced before each remote command
- Default:
'~/.bashrc'
Arguments
- LEVEL¶
Required argument
- OBS_IDS¶
Required argument
nenudata quality-combine¶
Merge per-MS aoquality tables into a single combined QS file.
Gathers all MSs matching OBS_IDS (and any :SW restriction) at LEVEL and passes them to “aoquality combine OUT_QS MS1 MS2 …”. Expects “quality-collect” to have been run first.
OUT_QS: full path to the output combined QS file.
Usage
nenudata quality-combine [OPTIONS] LEVEL OBS_IDS OUT_QS
Options
- -c, --config <config>¶
Data handler configuration file
Arguments
- LEVEL¶
Required argument
- OBS_IDS¶
Required argument
- OUT_QS¶
Required argument