data_handler.toml reference

data_handler.toml is the configuration file read by all nenudata commands. It describes the spectral layout, data-level paths, cluster nodes, DP3 parsets, and the registry of observations.

TOML ordering note

TOML requires top-level scalar keys (e.g. bp_cal_path, cal_obs_id_patterns) to appear before the first section header ([...]) in the file. Place them at the very top or immediately after [obs_ids]/[spectral_windows] before any subsection block.

---

Required sections

[spectral_windows]

Maps a spectral window (SW) name to the inclusive range of sub-band indices it covers.

[spectral_windows]
SW01 = [192, 250]
SW02 = [251, 312]
SW03 = [313, 371]

A composite SW can also be defined as a list of existing SW names. It is treated as a concatenation of those sub-bands and is available wherever an SW argument is accepted.

[spectral_windows]
SW01 = [192, 250]
SW02 = [251, 312]
SW_ALL = ["SW01", "SW02"]   # composite

---

[data_level_path]

Maps a data-level label (arbitrary string, e.g. L1, L2, L2_BP) to a path template. The following placeholders are expanded at runtime:

Placeholder	Replaced with
%YEAR%	4-digit year from the obs_id date prefix
%MONTH%	2-digit month from the obs_id date prefix
%OBS_ID%	The obs_id (N1 short name for N2 observations)
%N1_OBS_ID%	The full NenuFAR obs_id (long name for N2)
%NODE%	The node responsible for the SW / time chunk

If neither %OBS_ID% nor %N1_OBS_ID% appears in the template the obs_id is appended to the path automatically.

[data_level_path]
L1    = "/databf/nenufar-nri/LT01/%YEAR%/%MONTH%/%N1_OBS_ID%/L1/"
L2    = "/net/%NODE%/data/users/lofareor/nenufar/obs/L2/"
L2_BP = "/databf/nenufar-nri/LT01/%YEAR%/%MONTH%/%N1_OBS_ID%/L2_BP/"

---

[obs_ids]

Registry of N1 observations (short, SW-parallel). Each key is an obs_id; the value is a list of nodes, one entry per spectral window in the order they appear in [spectral_windows]. The same node handles both reading L1 and writing L2 for that SW.

[obs_ids]
"20231208_CASA" = ["node101", "node102", "node103"]
#                   ^ SW01      ^ SW02      ^ SW03
"20231210_NT04" = ["node104", "node105", "node106"]

This section is populated automatically by nenudata update_data_handler.

---

Optional — observation registry

[n2_obs_ids.<obs_id>]

Registry of N2 observations (long, time-parallel). Each subsection key is the short obs_id used everywhere in the pipeline; the body provides:

Key	Type	Description
n1_obs_ids	list[string]	Full NenuFAR obs_id(s) used to locate the raw L1 data on disk
nodes	string	Comma-separated node list; time chunks are round-robin distributed

[n2_obs_ids.20231208_NT04]
n1_obs_ids = ["20231208_213000_20231209_070000_NT04_COSMIC_DAWN"]
nodes      = "node101,node102,node103"

Node range notation (e.g. node[101-103]) is also accepted by nenudata wherever a node string appears.

---

Optional — calibration

The calibration keys below are top-level scalars/tables and must appear before any [section] header in the file.

bad_stations_file

Path to the JSON file managed by nenudata bad-stations. Stores per-obs_id lists of bad antennas. The file is created on first write.

bad_stations_file = "bad_stations.json"

Used by nenudata bad-stations subcommands and by the flagger calpipe task when baselinesflag.baselines_from_file points to a .json file. See bad stations for the full workflow.

---

bp_cal_path

Path template to the bandpass calibration HDF5 file. %OBS_ID% is replaced with the calibrator obs_id at runtime.

bp_cal_path = "/databf/.../bp_cal/%OBS_ID%/bp_gains_%OBS_ID%.h5"

Used by l1_to_l2 when the DP3 parset contains the %BP_CAL_FILE% marker.

---

cal_obs_id_patterns

List of glob patterns used to identify calibrator observations in [obs_ids]. Defaults to ["*CYGA*", "*CASA*"] if omitted. update_data_handler excludes matching obs_ids from [obs_calibration_map].

cal_obs_id_patterns = ["*CYGA*", "*CASA*", "*3C*"]

---

[obs_calibration_map]

Maps each science obs_id to the calibrator obs_id whose BP solutions are used during L1→L2 processing. Populated automatically by nenudata update_data_handler; can also be edited by hand.

[obs_calibration_map]
"20231208_NT04" = "20231210_CASA"
"20231212_NT04" = "20231210_CASA"

---

Optional — DP3 pipeline

[l1_to_l2_config.<level>]

One subsection per L2 data level. The only required key is dppp_config, which points to the DP3 parset file used when running nenudata l1_to_l2 <level>.

[l1_to_l2_config.L2_BP]
dppp_config = "/home/users/lofareor/parsets/dppp_l1_to_l2_bp.parset"

[l1_to_l2_config.L2_12C40S]
dppp_config = "/home/users/lofareor/parsets/dppp_l1_to_l2_cyga.parset"

If the parset contains the %BP_CAL_FILE% marker, l1_to_l2 automatically injects applybandpass.parmdb=<bp_cal_path> for the correct calibrator.

---

Optional — LST binning

[lst_binning.<level>]

Defines the LST grid for nenudata l2_to_l3_lst. One subsection per L3 output level.

Key	Type	Description
start	float	Start of the LST range (hours)
end	float	End of the LST range (hours)
width	float	Width of each LST bin (hours)
longitude	float	Geographic longitude of the array (radians)

[lst_binning.L3]
start     = 6.0
end       = 10.0
width     = 0.1
longitude = 0.037

Output MS filenames include an LST tag: SW03_T063.MS for bin centre 6.3 h.

---

Optional — remote data transfer

[remote_hosts.<name>]

One subsection per remote site. Used by nenudata transfer to pull L1 data from an archive host.

Key	Type	Description
host	string	SSH hostname
level	string	Data level present on the remote (e.g. "L1")
data_path	string	Path template; supports %YEAR%, %MONTH%, %OBS_ID%, %LEVEL%, %NODE%
nodes	string	Comma-separated list or range pattern of remote nodes
password_file	string	Path to an SSH password / key file (optional)
data_handler_path	string	Absolute path to the data-handler config on the remote host (optional); when set, push_l2 rsyncs n2_obs_ids_push.json alongside it

[remote_hosts.dawn]
host              = "dawn"
level             = "L2_BP"
data_path         = "/net/%NODE%/data/obs/%LEVEL%/%OBS_ID%/"
nodes             = "node1[01-15]"
data_handler_path = "/home/user/nenuflow/data_handler.toml"  # optional

---

Optional — N2 grouping

[n2_group]

Groups multiple N2 obs_ids under a single virtual obs_id for l2_to_l3_lst. Each key is the virtual group name; the value is the list of N2 obs_ids it contains. Every listed obs_id must already be defined in [n2_obs_ids].

[n2_group]
"2023_NT04_ALL" = ["20231208_NT04", "20231210_NT04", "20231212_NT04"]

When nenudata l2_to_l3_lst receives 2023_NT04_ALL as the obs_id it processes all constituent N2 obs_ids together.

---

Minimal example

[spectral_windows]
SW03 = [313, 371]
SW04 = [372, 430]

[data_level_path]
L1    = "/databf/nenufar-nri/LT01/%YEAR%/%MONTH%/%N1_OBS_ID%/L1/"
L2_BP = "/net/%NODE%/data/users/lofareor/nenufar/obs/L2_BP/"

[obs_ids]
"20231208_CASA" = ["node101", "node102"]
"20231210_NT04" = ["node103", "node104"]

Full example

# top-level scalars must appear before any [section] header
bp_cal_path         = "/databf/.../bp_gains/%OBS_ID%/bp_cal_%OBS_ID%.h5"
cal_obs_id_patterns = ["*CYGA*", "*CASA*"]

[spectral_windows]
SW01    = [192, 250]
SW02    = [251, 312]
SW03    = [313, 371]
SW_LOW  = ["SW01", "SW02"]   # composite

[data_level_path]
L1    = "/databf/nenufar-nri/LT01/%YEAR%/%MONTH%/%N1_OBS_ID%/L1/"
L2_BP = "/net/%NODE%/data/users/lofareor/nenufar/obs/L2_BP/"

[obs_ids]
"20231208_CASA" = ["node101", "node102", "node103"]
"20231210_NT04" = ["node101", "node102", "node103"]

[n2_obs_ids.20231208_NT04_LONG]
n1_obs_ids = ["20231208_213000_20231209_070000_NT04_COSMIC_DAWN"]
nodes      = "node101,node102,node103"

[obs_calibration_map]
"20231210_NT04" = "20231208_CASA"

[l1_to_l2_config.L2_BP]
dppp_config = "/home/users/lofareor/parsets/dppp_l1_to_l2_bp.parset"

[lst_binning.L3]
start     = 6.0
end       = 10.0
width     = 0.1
longitude = 0.037

[remote_hosts.databf]
host      = "databfnfr"
level     = "L1"
data_path = "/data/nenufar-nri/LT01/%YEAR%/%MONTH%/%OBS_ID%/L1/"
nodes     = "node101,node102"

[n2_group]
"2023_NT04_ALL" = ["20231208_NT04_LONG"]