Pipeline steps — l1_to_l2, l2_to_l3_lst¶

l1_to_l2 — run DP3 to convert L1 sub-bands to L2¶

Creates L2 MeasurementSets from the L1 sub-band data using DP3. The DP3 parset is read from the [l1_to_l2_config] section of the data-handler config.

For N1 observations (one MS per sub-band), all sub-bands are passed to DP3 together as a comma-separated msin list. For N2 observations (long integrations), the observation is split into time chunks and each chunk is processed independently across nodes.

Basic usage¶

nenudata l1_to_l2 L2 "20231208_NT04" -c data_handler.toml

Process multiple nights in parallel (4 jobs per node)¶

nenudata l1_to_l2 L2 "202312*_NT04" -m 4 --env_file ~/.bashrc

Re-process, overwriting existing L2¶

nenudata l1_to_l2 L2 "20231208_NT04" --force

N2 chunk size¶

For long N2 observations, the default chunk size is 150 time slots. Adjust with --n_slots_per_chunk:

nenudata l1_to_l2 L2_BP "202312*_NT04":SW03 -m 4 --n_slots_per_chunk 180

The tolerance option controls when the remainder chunk is merged with the last full chunk instead of kept as a separate (short) chunk. A value of 0.25 means a remainder smaller than 25 % of n_slots_per_chunk is merged.

Data-handler configuration¶

[l1_to_l2_config.L2]
dppp_config = 'dp3/ndppp_l1_to_l2.parset'

[l1_to_l2_config.L2_BP]
dppp_config = 'dp3/ndppp_l1_to_l2_bp.parset'

When a calibration map is present (see update_data_handler), the bandpass calibration HDF5 is injected directly on the DP3 command line via apply_bp.parmdb=<path>, so no shell scripting is required in the parset.

Options¶

Option	Default	Description
`--force`	false	Overwrite existing L2 output
`--l1_level`	`L1`	Name of the input data level
`--max_concurrent` / `-m`	`1`	Concurrent DP3 jobs per node
`--dry_run`	false	Print commands without running them
`--env_file`	`~/.bashrc`	Shell environment file sourced before each job
`--n_slots_per_chunk`	`150`	Time slots per chunk (N2 observations only)
`--tolerance`	`0.25`	Merge remainder chunk if smaller than this fraction of chunk size
`--hosts`	(from config)	Override node list

l2_to_l3_lst — LST-bin L2 data into L3¶

Groups L2 MSs by LST bin and calls mstool extract_lst to cut the matching time range from each nightly observation. This produces L3 MSs aligned in LST across nights, ready for stacking.

Basic usage¶

nenudata l2_to_l3_lst L3_LST "202312*_NT04"

Use a different data column¶

nenudata l2_to_l3_lst L3_LST "202312*_NT04" --data_column CORRECTED_DATA

Re-create, overwriting existing L3¶

nenudata l2_to_l3_lst L3_LST "202312*_NT04" --force

Data-handler configuration¶

LST binning must be declared for the output level:

[lst_binning.L3_LST]
start = 2.0        # LST bin start in hours
end = 4.0          # LST bin end in hours
width = 0.1        # bin width in hours
longitude = 6.57   # observatory east longitude in degrees

Options¶

Option	Default	Description
`--l2_level`	`L2`	Name of the input data level
`--data_column`	`DATA`	Column copied into the output `DATA` column
`--min_fraction`	`0.8`	Minimum coverage fraction required to write a bin
`--force`	false	Overwrite existing L3 output
`--max_concurrent` / `-m`	`1`	Concurrent jobs per node
`--dry_run`	false	Print commands without running them
`--env_file`	`~/.bashrc`	Shell environment file
`--hosts`	(from config)	Override node list

update_data_handler — map target obs_ids to calibrators¶

Identifies calibrator obs_ids in the data-handler config (matched by cal_obs_id_patterns, defaulting to *CYGA* and *CASA*) and for each target obs_id writes the closest-in-date calibrator into the [obs_calibration_map] section. The config file is backed up with a datestamp before being modified.

Basic usage¶

nenudata update_data_handler "*" -c data_handler.toml
nenudata update_data_handler "202312*" -c data_handler.toml --dry_run

Data-handler configuration¶

# Template path — %OBS_ID% is replaced with the calibrator obs_id at runtime.
bp_cal_path = "/net/node101/gains/%OBS_ID%/bp_cal_%OBS_ID%.h5"

# Optional: override which patterns identify calibrator obs_ids (default shown).
cal_obs_id_patterns = ["*CYGA*", "*CASA*"]

The [obs_calibration_map] section is written (or updated) automatically:

[obs_calibration_map]
20231208_NT04 = "20231210_CASA"
20231215_NT04 = "20231218_CYGA"

Options¶

Option	Default	Description
`--dry_run`	false	Print the mapping without modifying the config
`--config` / `-c`	`data_handler.toml`	Path to the data-handler config file

make_bp_sol — assemble broadband bandpass solution¶

Averages per-antenna gain solutions across all spectral windows for each obs_id and saves the result as a single broadband H5Parm file at the path given by bp_cal_path in the data-handler config.

For each spectral window, all H5Parm files matching --h5_name inside the MS directories at LEVEL are loaded and averaged with per-antenna sigma-clipping. The per-SW averages are then concatenated in frequency order into one combined solution file.

Basic usage¶

nenudata make_bp_sol "202312*_CYGA" L2 -c data_handler.toml

With diagnostic plots¶

nenudata make_bp_sol "202312*_CYGA" L2 --plot_dir plots/bp/

Writes plots/bp/<obs_id>/spectra.pdf and plots/bp/<obs_id>/delay.pdf for each processed observation.

Data-handler configuration¶

bp_cal_path must be set, and %OBS_ID% in the template is replaced with the calibrator obs_id at runtime:

bp_cal_path = "/net/node101/gains/%OBS_ID%/bp_cal_%OBS_ID%.h5"

Options¶

Option	Default	Description
`--h5_name`	`instrument_di_bp.h5`	H5Parm filename looked up inside each MS directory
`--plot_dir` / `-p`	(none)	Directory to write per-obs diagnostic plots
`--config` / `-c`	`data_handler.toml`	Data-handler config file

Reference¶

nenudata l1_to_l2¶

Create L2 data (at level LEVEL) from L1 data for the given OBS_IDS

Usage

nenudata l1_to_l2 [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>¶: Data handler configuration file

--force¶: Force overwrite data if already exists

--l1_level <l1_level>¶: L1 level name

-m, --max_concurrent <max_concurrent>¶: Maximum concurrent tasks on a node

--dry_run¶: Do not do anything

--env_file <env_file>¶: Environment file

--n_slots_per_chunk <n_slots_per_chunk>¶: Specify the chunk duration in number of slots

--tolerance <tolerance>¶: Tolerance for handling the remainder chunk as a fraction of chunk_duration

--hosts <hosts>¶: Lists of hosts where to run DP3

Arguments

LEVEL¶: Required argument

OBS_IDS¶: Required argument

nenudata l2_to_l3_lst¶

Create L3 data (at level LEVEL) from L2 data for the given OBS_IDS using LST binning

Usage

nenudata l2_to_l3_lst [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>¶: Data handler configuration file

--force¶: Force overwrite data if already exists

--l2_level <l2_level>¶: Input L2 level name

--data_column <data_column>¶: Data column to extract into output DATA column

--min_fraction <min_fraction>¶: Minimum fraction of bin duration required to create an LST bin

-m, --max_concurrent <max_concurrent>¶: Maximum concurrent tasks on a node

--dry_run¶: Do not do anything

--env_file <env_file>¶: Environment file

--hosts <hosts>¶: Lists of hosts where to run mstool

Arguments

LEVEL¶: Required argument

OBS_IDS¶: Required argument

nenudata update_data_handler¶

Discover L1 obs_ids in a date range, register them, and map calibrators.

Scans the filesystem using the L1 path from data_level_path in the config. Obs_ids between START_DATE and END_DATE (inclusive, YYYYMMDD) that are not yet in [obs_ids] are added with NODE assigned to every spectral window. Newly added target obs_ids are then mapped to their closest-in-date calibrator in [obs_calibration_map].

The config file is backed up with a datestamp before being updated.

Usage

nenudata update_data_handler [OPTIONS] START_DATE END_DATE

Options

-n, --node <node>¶: Required Node to assign to all spectral windows for each new obs_id

--l1_level <l1_level>¶: Data level name used for L1 discovery

-c, --config <config>¶: Data handler configuration file

--dry_run¶: Print planned changes without writing them

Arguments

START_DATE¶: Required argument

END_DATE¶: Required argument

nenudata make_bp_sol¶

Average gain solutions across spectral windows into a bandpass H5Parm.

For each obs_id matching OBS_IDS, reads H5_NAME from every MS at LEVEL across all spectral windows, averages the solutions per SW with per-antenna sigma-clipping, concatenates in frequency, and saves the result to the path given by bp_cal_path in the data-handler config (with %OBS_ID% substituted).

Usage

nenudata make_bp_sol [OPTIONS] OBS_IDS LEVEL

Options

-c, --config <config>¶: Data handler configuration file

--h5_name <h5_name>¶

H5Parm filename relative to each MS

Default:: 'instrument_di_bp.h5'

-p, --plot_dir <plot_dir>¶: Write diagnostic spectrum and delay plots here

Arguments

OBS_IDS¶: Required argument

LEVEL¶: Required argument

Pipeline steps — l1_to_l2, l2_to_l3_lst¶

l1_to_l2 — run DP3 to convert L1 sub-bands to L2¶

Basic usage¶

Process multiple nights in parallel (4 jobs per node)¶

Re-process, overwriting existing L2¶

N2 chunk size¶

Data-handler configuration¶

Options¶

l2_to_l3_lst — LST-bin L2 data into L3¶

Basic usage¶

Use a different data column¶

Re-create, overwriting existing L3¶

Data-handler configuration¶

Options¶

update_data_handler — map target obs_ids to calibrators¶

Basic usage¶

Data-handler configuration¶

Options¶

make_bp_sol — assemble broadband bandpass solution¶

Basic usage¶

With diagnostic plots¶

Data-handler configuration¶

Options¶

Reference¶

nenudata l1_to_l2¶

nenudata l2_to_l3_lst¶

nenudata update_data_handler¶

nenudata make_bp_sol¶

See also¶