# Bad stations — bad-stations subgroup

`nenudata bad-stations` manages per-observation lists of bad antennas
(stations).  The lists are stored in a single JSON file declared in
`data_handler.toml` and can be populated from the output of
`aostats find-bad-stations`, propagated from calibrators to their target
observations, or edited manually.

## Configuration

Add one line to `data_handler.toml` before the first section header:

```toml
bad_stations_file = "bad_stations.json"
```

The path is relative to the working directory where `nenudata` is invoked.
The file is created on first write; it does not need to exist beforehand.

## JSON format

```json
{
  "20231208_CYGA": ["MR003", "MR017"],
  "20231208_NT04": ["MR003", "MR017"],
  "20231210_NT04": [],
  "_history": [
    {
      "date": "2026-06-24",
      "action": "import",
      "sources": ["results/20231208_NT04/aoquality/bad_stations_l2a.json"],
      "note": "first nenuflow run",
      "changes": {
        "20231208_CYGA": {"added": ["MR003", "MR017"]},
        "20231208_NT04": {"added": ["MR003", "MR017"]}
      }
    },
    {
      "date": "2026-06-25",
      "action": "remove",
      "note": "MR017 was a cabling issue fixed after the run",
      "changes": {
        "20231208_CYGA": {"removed": ["MR017"]},
        "20231208_NT04": {"removed": ["MR017"]}
      }
    }
  ]
}
```

Keys are obs_ids.  Values are sorted lists of bare antenna names (no `&&*`
suffix).  An empty list means the observation was checked and no bad station
was found.  A missing key means the observation has not been assessed yet.

The `_history` key stores an ordered audit log of every write operation (see
[History](#history) below).  Keys starting with `_` are ignored when the file
is consumed by pipeline tasks.

The `&&*` constraint syntax expected by DP3 is formatted automatically when
the file is consumed by the `flagger` pipeline task (see
[manual baseline flagging](#manual-baseline-flagging)).

## Typical workflow

### 1. Detect bad stations from calibrator QS files

```bash
aostats find-bad-stations \
    quality_l2_cal/SW03/20231208_CYGA.qs \
    quality_l2_cal/SW03/20231210_CASA.qs \
    SNR --out bad_stations_cal.json
```

### 2. Import into nenudata and propagate to targets

```bash
nenudata bad-stations import bad_stations_cal.json
```

Each calibrator obs_id found in `[obs_calibration_map]` has its bad-station
list copied to all target observations mapped to it.  A single command
therefore updates both the calibrator entry and every corresponding target.

### 3. Review

```bash
nenudata bad-stations show
```

Displays a compact table with station numbers instead of full names:

```
Obs_id            N_bad  Bad stations
20231208_CYGA     2      3, 17
20231208_NT04     2      3, 17
20231210_NT04     0      (none)
20231212_NT04     -      —
```

A dash (`—`) in the *Bad stations* column means the observation has not been
assessed (key absent from the JSON).  `(none)` means it was assessed and no
bad station was found.

### 4. Manual corrections

Always supply `--note` for manual edits so the reason is recorded in history:

```bash
# Add a station discovered by other means
nenudata bad-stations add "20231212*_NT04" MR005 --note "dropout seen in sol plot"

# Remove a station that turned out to be fine
nenudata bad-stations remove "20231208_NT04" MR017 --note "cabling issue fixed after run"

# Mark an observation as checked with no bad stations
nenudata bad-stations clear "20231215_NT04" --note "manually verified"
```

All commands accept glob patterns for `OBS_IDS` (e.g. `"202312*_NT04"`).
The `add` command merges with the existing list; `remove` silently ignores
antennas that are not present.

### 5. Review history

```bash
nenudata bad-stations show-history        # full log
nenudata bad-stations show-history --last 5   # last 5 entries
```

## Commands

### `show`

```
nenudata bad-stations show [OBS_IDS] [-c CONFIG]
```

Print a table of obs_ids with their bad-station counts and (abbreviated)
station numbers.  `OBS_IDS` defaults to `*` (all obs_ids in the data handler).

### `show-history`

```
nenudata bad-stations show-history [--last N] [-c CONFIG]
```

Print the audit log of all changes made to the bad-stations file.  Each entry
shows the date, action, note, and the per-obs_id diff (added/removed antennas).
Use `--last N` to show only the most recent N entries.

### `add`

```
nenudata bad-stations add OBS_IDS ANTENNA... [--note NOTE] [-c CONFIG]
```

Add one or more antennas to the bad-station list for all obs_ids matching
`OBS_IDS`.  Merges with the existing list; deduplicates automatically.

| Option | Default | Description |
|---|---|---|
| `--note` / `-n` | `""` | Reason for this change (stored in `_history`) |

### `remove`

```
nenudata bad-stations remove OBS_IDS ANTENNA... [--note NOTE] [-c CONFIG]
```

Remove one or more antennas from the bad-station list.  Silently skips
antennas that are not in the list.

| Option | Default | Description |
|---|---|---|
| `--note` / `-n` | `""` | Reason for this change (stored in `_history`) |

### `clear`

```
nenudata bad-stations clear OBS_IDS [--note NOTE] [-c CONFIG]
```

Set the bad-station list to empty for all matching obs_ids.  The key is kept
in the JSON so the observation is recorded as "assessed, nothing flagged".

| Option | Default | Description |
|---|---|---|
| `--note` / `-n` | `""` | Reason for this change (stored in `_history`) |

### `import`

```
nenudata bad-stations import SOURCE... [--no-propagate] [--overwrite] [--note NOTE] [-c CONFIG]
```

Merge bad-station lists from one or more JSON files (e.g. output of
`aostats find-bad-stations`) into `bad_stations_file`.  `SOURCE` can be:

- One or more explicit file paths: `nenudata bad-stations import file1.json file2.json`
- A quoted glob pattern: `nenudata bad-stations import "results/*/aoquality/bad_stations_l2a.json"`

| Option | Default | Description |
|---|---|---|
| `--no-propagate` | false | Skip automatic propagation from calibrators to their targets |
| `--overwrite` | false | Replace existing entries instead of merging (union) |
| `--note` / `-n` | `""` | Reason for this import (stored in `_history`) |

By default entries are merged (union of existing and imported).  With
`--overwrite` the imported list replaces whatever was stored.

**Auto-propagation** — for each obs_id in any source file that appears as a
value in `[obs_calibration_map]`, the same bad-station list is written for
every target obs_id that maps to it.  This means importing calibrator results
in a single step updates both the calibrator entry and all its targets.

When a source file contains target obs_ids directly (e.g. from a second round
of detection on target data), propagation is a no-op because those obs_ids are
not calibrators in the map.

(history)=
## History

Every write command (`add`, `remove`, `clear`, `import`) appends an entry to
`_history` in the JSON file.  Each entry records:

| Field | Description |
|---|---|
| `date` | Date the command was run (`YYYY-MM-DD`) |
| `action` | Command name: `add`, `remove`, `clear`, or `import` |
| `note` | Free-text reason supplied via `--note` (empty string if omitted) |
| `sources` | (`import` only) list of source file paths that were merged |
| `changes` | Per-obs_id diff: `{"added": [...]}` and/or `{"removed": [...]}` |

Use `nenudata bad-stations show-history` to inspect the log.  The history is
append-only; individual entries are never modified or deleted by the tool.

## Using bad stations in the calpipe flagger task

Point `baselinesflag.baselines_from_file` at the JSON file.  The flagger
reads the JSON automatically (detected by the `.json` extension), looks up
each input MS by obs_id substring match, and formats the antenna names as
`MR003&&*;MR017&&*` on the DP3 command line.

```toml
[flagger]
do_baselinesflag = true
baselinesflag.baselines_from_file = 'bad_stations.json'
```

See [calpipe — manual baseline flagging](#manual-baseline-flagging).

## Reference

```{eval-rst}
.. click:: nenucal.tools.nenudata:bad_stations
   :prog: nenudata bad-stations
   :nested: full
```

## See also

- [config](config.md) — `bad_stations_file` key
- [pipeline](pipeline.md) — calibration map setup
- aostats `find-bad-stations` — upstream bad-station detection from QS files