Bad stations — bad-stations subgroup¶

nenudata bad-stations manages per-observation lists of bad antennas (stations). The lists are stored in a single JSON file declared in data_handler.toml and can be populated from the output of aostats find-bad-stations, propagated from calibrators to their target observations, or edited manually.

Configuration¶

Add one line to data_handler.toml before the first section header:

bad_stations_file = "bad_stations.json"

The path is relative to the working directory where nenudata is invoked. The file is created on first write; it does not need to exist beforehand.

JSON format¶

{
  "20231208_CYGA": ["MR003", "MR017"],
  "20231208_NT04": ["MR003", "MR017"],
  "20231210_NT04": [],
  "_history": [
    {
      "date": "2026-06-24",
      "action": "import",
      "sources": ["results/20231208_NT04/aoquality/bad_stations_l2a.json"],
      "note": "first nenuflow run",
      "changes": {
        "20231208_CYGA": {"added": ["MR003", "MR017"]},
        "20231208_NT04": {"added": ["MR003", "MR017"]}
      }
    },
    {
      "date": "2026-06-25",
      "action": "remove",
      "note": "MR017 was a cabling issue fixed after the run",
      "changes": {
        "20231208_CYGA": {"removed": ["MR017"]},
        "20231208_NT04": {"removed": ["MR017"]}
      }
    }
  ]
}

Keys are obs_ids. Values are sorted lists of bare antenna names (no &&* suffix). An empty list means the observation was checked and no bad station was found. A missing key means the observation has not been assessed yet.

The _history key stores an ordered audit log of every write operation (see History below). Keys starting with _ are ignored when the file is consumed by pipeline tasks.

The &&* constraint syntax expected by DP3 is formatted automatically when the file is consumed by the flagger pipeline task (see manual baseline flagging).

Typical workflow¶

1. Detect bad stations from calibrator QS files¶

aostats find-bad-stations \
    quality_l2_cal/SW03/20231208_CYGA.qs \
    quality_l2_cal/SW03/20231210_CASA.qs \
    SNR --out bad_stations_cal.json

2. Import into nenudata and propagate to targets¶

nenudata bad-stations import bad_stations_cal.json

Each calibrator obs_id found in [obs_calibration_map] has its bad-station list copied to all target observations mapped to it. A single command therefore updates both the calibrator entry and every corresponding target.

3. Review¶

nenudata bad-stations show

Displays a compact table with station numbers instead of full names:

Obs_id            N_bad  Bad stations
20231208_CYGA     2      3, 17
20231208_NT04     2      3, 17
20231210_NT04     0      (none)
20231212_NT04     -      —

A dash (—) in the Bad stations column means the observation has not been assessed (key absent from the JSON). (none) means it was assessed and no bad station was found.

4. Manual corrections¶

Always supply --note for manual edits so the reason is recorded in history:

# Add a station discovered by other means
nenudata bad-stations add "20231212*_NT04" MR005 --note "dropout seen in sol plot"

# Remove a station that turned out to be fine
nenudata bad-stations remove "20231208_NT04" MR017 --note "cabling issue fixed after run"

# Mark an observation as checked with no bad stations
nenudata bad-stations clear "20231215_NT04" --note "manually verified"

All commands accept glob patterns for OBS_IDS (e.g. "202312*_NT04"). The add command merges with the existing list; remove silently ignores antennas that are not present.

5. Review history¶

nenudata bad-stations show-history        # full log
nenudata bad-stations show-history --last 5   # last 5 entries

Commands¶

`show`¶

nenudata bad-stations show [OBS_IDS] [-c CONFIG]

Print a table of obs_ids with their bad-station counts and (abbreviated) station numbers. OBS_IDS defaults to * (all obs_ids in the data handler).

`show-history`¶

nenudata bad-stations show-history [--last N] [-c CONFIG]

Print the audit log of all changes made to the bad-stations file. Each entry shows the date, action, note, and the per-obs_id diff (added/removed antennas). Use --last N to show only the most recent N entries.

`add`¶

nenudata bad-stations add OBS_IDS ANTENNA... [--note NOTE] [-c CONFIG]

Add one or more antennas to the bad-station list for all obs_ids matching OBS_IDS. Merges with the existing list; deduplicates automatically.

Option	Default	Description
`--note` / `-n`	`""`	Reason for this change (stored in `_history`)

`remove`¶

nenudata bad-stations remove OBS_IDS ANTENNA... [--note NOTE] [-c CONFIG]

Remove one or more antennas from the bad-station list. Silently skips antennas that are not in the list.

Option	Default	Description
`--note` / `-n`	`""`	Reason for this change (stored in `_history`)

`clear`¶

nenudata bad-stations clear OBS_IDS [--note NOTE] [-c CONFIG]

Set the bad-station list to empty for all matching obs_ids. The key is kept in the JSON so the observation is recorded as “assessed, nothing flagged”.

Option	Default	Description
`--note` / `-n`	`""`	Reason for this change (stored in `_history`)

`import`¶

nenudata bad-stations import SOURCE... [--no-propagate] [--overwrite] [--note NOTE] [-c CONFIG]

Merge bad-station lists from one or more JSON files (e.g. output of aostats find-bad-stations) into bad_stations_file. SOURCE can be:

One or more explicit file paths: nenudata bad-stations import file1.json file2.json
A quoted glob pattern: nenudata bad-stations import "results/*/aoquality/bad_stations_l2a.json"

Option	Default	Description
`--no-propagate`	false	Skip automatic propagation from calibrators to their targets
`--overwrite`	false	Replace existing entries instead of merging (union)
`--note` / `-n`	`""`	Reason for this import (stored in `_history`)

By default entries are merged (union of existing and imported). With --overwrite the imported list replaces whatever was stored.

Auto-propagation — for each obs_id in any source file that appears as a value in [obs_calibration_map], the same bad-station list is written for every target obs_id that maps to it. This means importing calibrator results in a single step updates both the calibrator entry and all its targets.

When a source file contains target obs_ids directly (e.g. from a second round of detection on target data), propagation is a no-op because those obs_ids are not calibrators in the map.

History¶

Every write command (add, remove, clear, import) appends an entry to _history in the JSON file. Each entry records:

Field	Description
`date`	Date the command was run (`YYYY-MM-DD`)
`action`	Command name: `add`, `remove`, `clear`, or `import`
`note`	Free-text reason supplied via `--note` (empty string if omitted)
`sources`	(`import` only) list of source file paths that were merged
`changes`	Per-obs_id diff: `{"added": [...]}` and/or `{"removed": [...]}`

Use nenudata bad-stations show-history to inspect the log. The history is append-only; individual entries are never modified or deleted by the tool.

Using bad stations in the calpipe flagger task¶

Point baselinesflag.baselines_from_file at the JSON file. The flagger reads the JSON automatically (detected by the .json extension), looks up each input MS by obs_id substring match, and formats the antenna names as MR003&&*;MR017&&* on the DP3 command line.

[flagger]
do_baselinesflag = true
baselinesflag.baselines_from_file = 'bad_stations.json'

See calpipe — manual baseline flagging.

Reference¶

nenudata bad-stations¶

Manage per-observation bad station lists.

Usage

nenudata bad-stations [OPTIONS] COMMAND [ARGS]...

add¶

Add ANTENNAS to the bad-station list for OBS_IDS.

Usage

nenudata bad-stations add [OPTIONS] OBS_IDS ANTENNAS...

Options

-c, --config <config>¶: Data handler configuration file

-n, --note <note>¶: Reason for this change (recorded in history)

Arguments

OBS_IDS¶: Required argument

ANTENNAS¶: Required argument(s)

clear¶

Set bad-station list to empty for OBS_IDS (records “none bad”, keeps the key).

Usage

nenudata bad-stations clear [OPTIONS] OBS_IDS

Options

-c, --config <config>¶: Data handler configuration file

-n, --note <note>¶: Reason for this change (recorded in history)

Arguments

OBS_IDS¶: Required argument

flag¶

Flag bad stations from the registry via DP3 preflag, in parallel across all MSs.

Reads antenna lists from the bad-stations registry (see “bad-stations show”/ “import”) and runs DP3’s preflag step in place on every MS for OBS_IDS at LEVEL. Skips obs_ids with no entry or an empty antenna list.

Usage

nenudata bad-stations flag [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>¶: Data handler configuration file

-m, --max-concurrent <max_concurrent>¶

Maximum concurrent DP3 flag tasks per node

Default:: 1

--dry-run¶: Print commands without running them

--env-file <env_file>¶

Shell environment file sourced before each remote command

Default:: '~/.bashrc'

--only-n2¶: Restrict OBS_IDS resolution to N2 obs_ids only

Arguments

LEVEL¶: Required argument

OBS_IDS¶: Required argument

import¶

Import bad stations from one or more JSON files or a quoted glob pattern.

SOURCES: one or more paths to JSON files produced by “aostats find-bad-stations”, or a quoted glob pattern (e.g. “results/*/aoquality/bad_stations_l2a.json”).

Entries are automatically propagated calibrator -> N1 target (via obs_calibration_map) -> N2 chunk (via n2_obs_ids), so importing a calibrator’s bad stations also reaches every N2 chunk derived from its N1 targets. An entry keyed directly by an N1 obs_id still propagates to that N1’s own N2 chunks.

Usage

nenudata bad-stations import [OPTIONS] SOURCES...

Options

-c, --config <config>¶: Data handler configuration file

--no-propagate¶: Skip automatic propagation from calibrators to their N1/N2 targets

--overwrite¶: Replace existing entries instead of merging (union)

-n, --note <note>¶: Reason for this import (recorded in history)

Arguments

SOURCES¶: Required argument(s)

remove¶

Remove ANTENNAS from the bad-station list for OBS_IDS.

Usage

nenudata bad-stations remove [OPTIONS] OBS_IDS ANTENNAS...

Options

-c, --config <config>¶: Data handler configuration file

-n, --note <note>¶: Reason for this change (recorded in history)

Arguments

OBS_IDS¶: Required argument

ANTENNAS¶: Required argument(s)

show¶

Show bad stations for OBS_IDS (default: all).

Usage

nenudata bad-stations show [OPTIONS] [OBS_IDS]

Options

-c, --config <config>¶: Data handler configuration file

Arguments

OBS_IDS¶: Optional argument

show-history¶

Show the change history of the bad-stations file.

Usage

nenudata bad-stations show-history [OPTIONS]

Options

-c, --config <config>¶: Data handler configuration file

-l, --last <last>¶: Show only the last N entries (default: all)

Bad stations — bad-stations subgroup¶

Configuration¶

JSON format¶

Typical workflow¶

1. Detect bad stations from calibrator QS files¶

2. Import into nenudata and propagate to targets¶

3. Review¶

4. Manual corrections¶

5. Review history¶

Commands¶

show¶

show-history¶

add¶

remove¶

clear¶

import¶

History¶

Using bad stations in the calpipe flagger task¶

Reference¶

nenudata bad-stations¶

add¶

clear¶

flag¶

import¶

remove¶

show¶

show-history¶

See also¶

`show`¶

`show-history`¶

`add`¶

`remove`¶

`clear`¶

`import`¶