nenudata — data management¶
nenudata manages the lifecycle of NenuFAR-CD data: registering observations,
querying paths, transferring data between sites, and running the L1→L2 and
L2→L3 pipeline steps.
All commands read a TOML data-handler configuration file (default:
data_handler.toml) that describes the cluster layout, data levels, and
spectral windows.
Naming conventions — N1 and N2¶
NenuFAR data uses two naming conventions, one on each side of the L1→L2 step.
N1 is the telescope’s native naming convention: the full observation obs_id
— start and end timestamps, field, and project — for example
20231208_213000_20231209_070000_NT04_COSMIC_DAWN. Raw L1 data is stored
under this name, so the data handler registers observations under it in
[obs_ids], with one node per spectral window:
[obs_ids]
"20231208_213000_20231209_070000_NT04_COSMIC_DAWN" = ["nancep5", "nancep5", "nancep5", "nancep5"]
# one entry per spectral window
That full name is precise but long and impractical to handle, so the processed
L2 data uses a shorter convention, N2. An N2 observation is split along
the time axis into chunks, one spectral window per MS
(SW03_T000.MS, SW03_T001.MS, …) — the frequency coverage is unchanged, only
the time axis is divided. N2 entries live under [n2_obs_ids] with a short,
practical key and carry the originating N1 name(s) so the raw L1 path can still
be resolved. The nodes field is the pool of nodes across which the time
chunks are distributed in round-robin order:
[n2_obs_ids.20231208_NT04]
n1_obs_ids = ["20231208_213000_20231209_070000_NT04_COSMIC_DAWN"]
nodes = "node101,node102,node103"
The chunk size and merge tolerance for N2 processing are controlled by
--n_slots_per_chunk and --tolerance on the l1_to_l2 command.
Note
An N1 observation can also be processed straight to L2 under its N1 name (one MS per spectral window, not split in time) — this is convenient for short calibrator scans. Target observations, however, should always go through the N2 convention so the long integration is split into manageable time chunks.