Data access — get_ms, du, remove, restore-meta, make_ms_list#

get_ms — print MS paths#

Prints a space-separated list of MS paths for the given obs_ids, data level, and spectral windows. The output is designed for shell command substitution.

Examples#

# Get all L2 MSs for a set of nights, SW03
mss=$(nenudata get_ms L2 "202312*_NT04" --sws SW03)
calpipe cal/ddecal.toml $mss

# Narrow to two specific spectral windows
mss=$(nenudata get_ms L2 "20231208_NT04" --sws SW03,SW04)

# Append a sub-path inside each MS (e.g. to point at a diagnostic sub-dir)
nenudata get_ms L2 "20231208_NT04" --post_path logs

With --post_path logs, each path becomes /net/node/obs/L2/SW03.MS/logs.

Options#

Option	Default	Description
`--sws` / `-s`	`all`	Comma-separated spectral windows, or `all`
`--post_path` / `-p`	``	Appended to each MS path as a sub-directory
`--config` / `-c`	`data_handler.toml`	Data-handler config file

du — disk usage per obs_id, level and SW#

nenudata du "2024*_NT04_COSMIC_DAWN" -c data_handler.toml
nenudata du "2024*_NT04_COSMIC_DAWN" --by-sw -l L1
nenudata du "2024*_NT04:SW34" -l L2_BP --only-n2 --csv du_2024.csv

Prints, for each obs_id, the disk usage of its data at each level (all levels of the config by default, -l to choose), with the target and a total, plus totals per level. By default the sizes are those of the directories holding each obs_id’s data.

With --by-sw, or a :SW selection in the pattern, there is one row per obs_id and SW, plus a total per SW: the sizes are the sum of the MSs of each SW, so data in no SW (logs, sub-bands outside all SWs) are left out. A composite SW (e.g. SW34) gets the MSs no plain SW has, such as the N2 time chunks SW34_T*.MS; selected on its own, it gets those of its sub-SWs.

The L1 of an N2 obs_id is that of its N1 obs_ids; totals count each directory or MS once, even when the pattern matches both an N1 obs_id and its N2 obs_id. Data not reachable from the current host (e.g. node-local paths) show -. --csv also writes the table, sizes in bytes; -j sets how many du run in parallel (default 8).

remove — delete MS files#

nenudata remove L2 "202312*_NT04"

Prompts for confirmation before deleting all MSs matching the given obs_ids and spectral windows at the specified data level. Use --sws to restrict to specific spectral windows, and --dry-run to only list what would be done.

nenudata remove L2 "20231208_NT04" --sws SW03

Before deleting an MS, remove moves the files of its directory that are not part of the MS, such as the solutions and sky models nenuflow writes there, into SW03_T000_meta/ next to it, with ms_info.json (its time range and mean altitude), and the provenance records of its columns. --no-keep-meta deletes them too. It also prints, for each level made from this one, how many of its MSs exist and are marked done by make, and warns about those made before the last write of their input column, e.g. before removing an L2 once its L3 is made. The quality-collect and bad-stations flag markers of the removed MSs are cleared: an MS retrieved again is flagged again.

restore-meta — put the kept files back#

nenudata restore-meta L2_BP "20241225_NT04"

Moves the files kept by remove back into the MSs, once they exist again (e.g. retrieved from an archive), and removes the _meta directories. It stops without moving anything if an MS is missing, covers another time range than the MS the files come from, or already has one of the files (--force to overwrite it). The provenance records of columns the MS does not have, e.g. an archived copy made before nenuflow, are dropped.

make_ms_list — write MS path files for calpipe/DP3#

Creates plain text files containing MS paths, one file per (obs_id, sw) combination. These files can be passed to calpipe or DP3 via @filename syntax.

One list file per (obs_id, sw)#

nenudata make_ms_list L2 "202312*_NT04" --target ms_lists/

Creates files like ms_lists/20231208_NT04_L2_SW03 containing all MS paths on one line (space-separated by default).

One MS per line (column format)#

nenudata make_ms_list L2 "202312*_NT04" --column

Each MS path is written on its own line. Useful for tools that read line-delimited input.

One file per individual MS#

nenudata make_ms_list L2 "202312*_NT04" --per_ms --target ms_lists/

Creates a sub-directory ms_lists/20231208_NT04_L2_SW03.d/ containing one file per MS file.

Filter by elevation#

nenudata make_ms_list L2 "202312*_NT04" --min_alt 25

Only includes MSs where the mean elevation of the target field is at least 25 degrees. Without --min_alt, the min_alt of the level applies, if set (see configuration). The altitude is read once from each MS and kept in <MS>/ms_info.json.

Reference#

nenudata get_ms#

Return a list of all MS corresponding to given OBS_IDS and SWS

Usage

nenudata get_ms [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>#: Data handler configuration file

-s, --sws <sws>#: Spectral windows

-p, --post_path <post_path>#: Add post_path to the path of the MS

Arguments

LEVEL#: Required argument

OBS_IDS#: Required argument

nenudata du#

Disk usage of OBS_IDS per data level, and per SW with –by-sw.

Sizes are those of the directories holding each obs_id’s data at each level, or with –by-sw (or OBS_IDS:SW) the sum of the MSs of each SW, which leaves out what is in no SW (logs, sub-bands outside all SWs). The L1 of an N2 obs_id is that of its N1 obs_ids. Totals count each directory or MS once, even when OBS_IDS matches both an N1 obs_id and its N2 obs_id. Data not reachable from this host (e.g. on another node) show “-“.

Usage

nenudata du [OPTIONS] OBS_IDS

Options

-l, --levels <levels>#: Comma-separated data levels (default: all levels of the config)

--by-sw#: One row per obs_id and SW (implied by a :SW selection in OBS_IDS)

-j, --jobs <jobs>#

Number of du run in parallel

Default:: 8

--csv <csv_file>#: Also write the table to this CSV file, sizes in bytes

-c, --config <config>#: Data handler configuration file

--only-n2#: Restrict OBS_IDS resolution to N2 obs_ids only

Arguments

OBS_IDS#: Required argument

nenudata remove#

Remove all MS corresponding to given OBS_IDS and SWS

Usage

nenudata remove [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>#: Data handler configuration file

-s, --sws <sws>#: Spectral windows

--keep-meta, --no-keep-meta#

Move the files of each MS directory that are not part of the MS (solutions, sky models) into <MS>_meta/ before removing it

Default:: True

--dry-run#: List what would be removed and kept

Arguments

LEVEL#: Required argument

OBS_IDS#: Required argument

nenudata restore-meta#

Move the files kept by “remove –keep-meta” back into the MSs of LEVEL (e.g. after retrieving them again), after checking that each MS covers the same time range. The provenance records of columns the MS does not have are dropped.

Usage

nenudata restore-meta [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>#: Data handler configuration file

-s, --sws <sws>#: Spectral windows

--force#: Overwrite the files already in the MSs

Arguments

LEVEL#: Required argument

OBS_IDS#: Required argument

nenudata make_ms_list#

Make MS list for all OBS_IDS at level LEVEL

Usage

nenudata make_ms_list [OPTIONS] LEVEL OBS_IDS

Options

-c, --config <config>#: Data handler configuration file

--target <target>#: Target directory

-C, --column#: Write one MS per line (column format)

--min_alt <min_alt>#: Minimum mean elevation in degrees. Only MS with mean altitude ≥ this value. [default: min_alt of LEVEL, else none]

--per_ms#: Create one ms_list file per MS instead of one per (obs_id, sw)

Arguments

LEVEL#: Required argument

OBS_IDS#: Required argument

Data access — get_ms, du, remove, restore-meta, make_ms_list#

get_ms — print MS paths#

Examples#

Options#

du — disk usage per obs_id, level and SW#

remove — delete MS files#

restore-meta — put the kept files back#

make_ms_list — write MS path files for calpipe/DP3#

One list file per (obs_id, sw)#

One MS per line (column format)#

One file per individual MS#

Filter by elevation#

Reference#

nenudata get_ms#

nenudata du#

nenudata remove#

nenudata restore-meta#

nenudata make_ms_list#

See also#