# Pipeline steps — l1_to_l2, l2_to_l3_lst

## l1_to_l2 — run DP3 to convert L1 sub-bands to L2

Creates L2 MeasurementSets from the L1 sub-band data using DP3.  The DP3
parset is read from the `[l1_to_l2_config]` section of the data-handler config.

For N1 observations (one MS per sub-band), all sub-bands are passed to DP3
together as a comma-separated `msin` list.  For N2 observations (long
integrations), the observation is split into time chunks and each chunk is
processed independently across nodes.

### Basic usage

```bash
nenudata l1_to_l2 L2 "20231208_NT04" -c data_handler.toml
```

### Process multiple nights in parallel (4 jobs per node)

```bash
nenudata l1_to_l2 L2 "202312*_NT04" -m 4 --env_file ~/.bashrc
```

### Re-process, overwriting existing L2

```bash
nenudata l1_to_l2 L2 "20231208_NT04" --force
```

### N2 chunk size

For long N2 observations, the default chunk size is 150 time slots.  Adjust
with `--n_slots_per_chunk`:

```bash
nenudata l1_to_l2 L2_BP "202312*_NT04":SW03 -m 4 --n_slots_per_chunk 180
```

The `tolerance` option controls when the remainder chunk is merged with the
last full chunk instead of kept as a separate (short) chunk.  A value of
`0.25` means a remainder smaller than 25 % of `n_slots_per_chunk` is merged.

### Data-handler configuration

```toml
[l1_to_l2_config.L2]
dppp_config = 'dp3/ndppp_l1_to_l2.parset'

[l1_to_l2_config.L2_BP]
dppp_config = 'dp3/ndppp_l1_to_l2_bp.parset'
```

When a calibration map is present (see `update_data_handler`), the bandpass
calibration HDF5 is injected directly on the DP3 command line via
`apply_bp.parmdb=<path>`, so no shell scripting is required in the parset.

### Options

| Option | Default | Description |
|---|---|---|
| `--force` | false | Overwrite existing L2 output |
| `--l1_level` | `L1` | Name of the input data level |
| `--max_concurrent` / `-m` | `1` | Concurrent DP3 jobs per node |
| `--dry_run` | false | Print commands without running them |
| `--env_file` | `~/.bashrc` | Shell environment file sourced before each job |
| `--n_slots_per_chunk` | `150` | Time slots per chunk (N2 observations only) |
| `--tolerance` | `0.25` | Merge remainder chunk if smaller than this fraction of chunk size |
| `--hosts` | (from config) | Override node list |

## l2_to_l3_lst — LST-bin L2 data into L3

Groups L2 MSs by LST bin and calls `mstool extract_lst` to cut the matching
time range from each nightly observation.  This produces L3 MSs aligned in
LST across nights, ready for stacking.

### Basic usage

```bash
nenudata l2_to_l3_lst L3_LST "202312*_NT04"
```

### Use a different data column

```bash
nenudata l2_to_l3_lst L3_LST "202312*_NT04" --data_column CORRECTED_DATA
```

### Re-create, overwriting existing L3

```bash
nenudata l2_to_l3_lst L3_LST "202312*_NT04" --force
```

### Data-handler configuration

LST binning must be declared for the output level:

```toml
[lst_binning.L3_LST]
start = 2.0        # LST bin start in hours
end = 4.0          # LST bin end in hours
width = 0.1        # bin width in hours
longitude = 6.57   # observatory east longitude in degrees
```

### Options

| Option | Default | Description |
|---|---|---|
| `--l2_level` | `L2` | Name of the input data level |
| `--data_column` | `DATA` | Column copied into the output `DATA` column |
| `--min_fraction` | `0.8` | Minimum coverage fraction required to write a bin |
| `--force` | false | Overwrite existing L3 output |
| `--max_concurrent` / `-m` | `1` | Concurrent jobs per node |
| `--dry_run` | false | Print commands without running them |
| `--env_file` | `~/.bashrc` | Shell environment file |
| `--hosts` | (from config) | Override node list |

## update_data_handler — map target obs_ids to calibrators

Identifies calibrator obs_ids in the data-handler config (matched by
`cal_obs_id_patterns`, defaulting to `*CYGA*` and `*CASA*`) and for each
target obs_id writes the closest-in-date calibrator into the
`[obs_calibration_map]` section.  The config file is backed up with a
datestamp before being modified.

### Basic usage

```bash
nenudata update_data_handler "*" -c data_handler.toml
nenudata update_data_handler "202312*" -c data_handler.toml --dry_run
```

### Data-handler configuration

```toml
# Template path — %OBS_ID% is replaced with the calibrator obs_id at runtime.
bp_cal_path = "/net/node101/gains/%OBS_ID%/bp_cal_%OBS_ID%.h5"

# Optional: override which patterns identify calibrator obs_ids (default shown).
cal_obs_id_patterns = ["*CYGA*", "*CASA*"]
```

The `[obs_calibration_map]` section is written (or updated) automatically:

```toml
[obs_calibration_map]
20231208_NT04 = "20231210_CASA"
20231215_NT04 = "20231218_CYGA"
```

### Options

| Option | Default | Description |
|---|---|---|
| `--dry_run` | false | Print the mapping without modifying the config |
| `--config` / `-c` | `data_handler.toml` | Path to the data-handler config file |

## make_bp_sol — assemble broadband bandpass solution

Averages per-antenna gain solutions across all spectral windows for each
obs_id and saves the result as a single broadband H5Parm file at the path
given by `bp_cal_path` in the data-handler config.

For each spectral window, all H5Parm files matching `--h5_name` inside the
MS directories at `LEVEL` are loaded and averaged with per-antenna
sigma-clipping.  The per-SW averages are then concatenated in frequency order
into one combined solution file.

### Basic usage

```bash
nenudata make_bp_sol "202312*_CYGA" L2 -c data_handler.toml
```

### With diagnostic plots

```bash
nenudata make_bp_sol "202312*_CYGA" L2 --plot_dir plots/bp/
```

Writes `plots/bp/<obs_id>/spectra.pdf` and `plots/bp/<obs_id>/delay.pdf` for
each processed observation.

### Data-handler configuration

`bp_cal_path` must be set, and `%OBS_ID%` in the template is replaced with
the calibrator obs_id at runtime:

```toml
bp_cal_path = "/net/node101/gains/%OBS_ID%/bp_cal_%OBS_ID%.h5"
```

### Options

| Option | Default | Description |
|---|---|---|
| `--h5_name` | `instrument_di_bp.h5` | H5Parm filename looked up inside each MS directory |
| `--plot_dir` / `-p` | *(none)* | Directory to write per-obs diagnostic plots |
| `--config` / `-c` | `data_handler.toml` | Data-handler config file |

## Reference

```{eval-rst}
.. click:: nenucal.tools.nenudata:l1_to_l2
   :prog: nenudata l1_to_l2
   :nested: full
```

```{eval-rst}
.. click:: nenucal.tools.nenudata:l2_to_l3_lst
   :prog: nenudata l2_to_l3_lst
   :nested: full
```

```{eval-rst}
.. click:: nenucal.tools.nenudata:update_data_handler
   :prog: nenudata update_data_handler
   :nested: full
```

```{eval-rst}
.. click:: nenucal.tools.nenudata:make_bp_sol
   :prog: nenudata make_bp_sol
   :nested: full
```

## See also

- [data](data.md) — query MS paths after pipeline steps complete
- [transfer](transfer.md) — move processed data to remote sites