Change Log

0.15 - 2026-06-27

• [Packaging] Modernised pyproject.toml to PEP 621: [project.urls], docs/dev optional-dependency extras (so Read the Docs uses pip install -e ".[docs]" and docs/requirements.txt is dropped), classifiers, and removal of the legacy [tool.poetry.group] section

• [Docs] Documentation overhaul: rewritten landing page and README (project scope, components, ecosystem), restructured installation (external tools DP3/AOFlagger/WSClean), a conceptual “sky model” section and corrected N1/N2 naming explanation, descriptive tool-page titles, and trimmed per-flag usage examples in favour of one or two real workflows plus the options reference

• [Removed] nenudata add, nenudata add_all, and the [nodes_distribution] config section — the per-SW node layout they served (one node per spectral window) is superseded by the N2 time-chunk model (update_n2_obs_ids), which distributes 12-min chunks across a node pool. Obs_ids are registered via update_data_handler

• [Change] Relative paths in config files now resolve against the file that references them: dppp_config in the data handler resolves against the data handler’s directory, int_sky_model in a calpipe config against that config’s directory (via BaseSettings.get_path), and nenudata l1_to_l2 runs DP3 from the parset’s directory so parset-internal paths (e.g. aoflag.strategy) resolve next to it

• [New] nenudata push_l2 now writes n2_obs_ids_push.json alongside the data handler after pushing, recording the N2 obs_ids and remote node pool; if data_handler_path is set in [remote_hosts.*], the JSON is also rsynced to the remote data handler directory

• [New] nenudata import-n2 N2_JSON — registers N2 obs_ids from a JSON file (generated by push_l2) into the local data handler; idempotent, skips already-registered entries

• [New] DataHandler.get_remote_data_handler_path — reads optional data_handler_path key from [remote_hosts.*] config

0.14 - 2026-06-24

• [New] nenudata quality-collect LEVEL OBS_IDS — runs aoquality collect in parallel across all matching MSs using WorkerPool; options: -d DATA_COLUMN, -m MAX_CONCURRENT, --dry-run

• [New] nenudata quality-combine LEVEL OBS_IDS OUT_QS — gathers all MSs matching the pattern and runs aoquality combine OUT_QS MS1 MS2 ... in one call; OUT_QS is a full explicit output path

0.13 - 2026-06-24

• [New] nenudata bad-stations import now accepts multiple source files or a quoted glob pattern (e.g. "results/*/aoquality/bad_stations_l2a.json")

• [New] nenudata bad-stations show-history — display the audit log of all changes with dates, actions, notes, and per-obs diffs

• [New] _history key in bad_stations.json — every write command (add, remove, clear, import) appends an entry recording what changed and why

• [New] --note / -n option on add, remove, clear, import — free-text reason stored in the history entry

0.12 - 2026-06-24

• [New] nenudata bad-stations subgroup: show, add, remove, clear, import commands to manage per-observation bad antenna lists stored in a JSON file

• [New] DataHandler.bad_stations_file — new optional bad_stations_file key in data_handler.toml; path to the JSON file managed by nenudata bad-stations

• [New] calpipe flagger: baselines_from_file now accepts .json files (output of nenudata bad-stations or aostats find-bad-stations); &&* constraint is formatted automatically at consumption time; old text format unchanged

0.11 - 2026-06-19

• [New] modeltool deattentuate: convert apparent sky model back to intrinsic by dividing by the NenuFAR beam (inverse of modeltool attenuate)

• [New] modeltool cluster: cluster a sky model into DD calibration directions using K-means, tessellate, or Voronoi algorithms; prints a patch summary table

• [New] modeltool inspect: inspect a grouped sky model; output modes: human-readable summary, DP3-ready sourcedb.sourcenames string, per-direction solution intervals

• [New] modeltool plot: scatter plot of sky model sources at RA/Dec, one color per patch, optional flux-scaled marker sizes

• [New] skymodel.patch_summary: build a per-patch astropy Table (position, N_components, Total_I_Jy, Max_I_Jy) for any grouped sky model

• [New] skymodel.get_dp3_directions: return a DP3-ready patch list string [[‘P0’],[‘P1’],…]

• [New] skymodel.get_dp3_solints: compute per-direction solution intervals scaled by source flux at a target frequency

• [New] modeltool filter: filter a sky model by component flux, patch flux, patch name, and optionally elevation (requires –ms)

• [Changed] modeltool: extract MS reading into a private _ms_beam_info helper; attenuate/deattentuate/concatenate no longer duplicate this logic

• [Changed] modeltool inspect: –output renamed to –mode / -m to avoid clash with -o (output file) convention

• [Changed] skymodel.edit_model: observing_time is now optional; elevation checks are skipped when None

• [Fixed] modeltool set_patch_name: was loading sky model via skymodel.lsmtool.load instead of the top-level lsmtool import

• [Removed] skymodel.get_dp3_solints: removed unused use_distance parameter

0.10 - 2025-11-27

• [Fixed] soltool: restore open_sol

0.9 - 2025-10-27

• [New] Add new mstool, with command concat

• [New] nenudata make_ms_list: add option to filter MSs based on elevation

• [Fixed] datahandler get_ms_path: sort the MSs

• [New] nenudata make_ms_list: add option to make a ms_list with one MS per line

0.8 - 2025-06-12

• [New] calpipe: add baselinesflag.baselines_from_file to retrieve the list of baseline to flag from a file

• [New] nenudata l1_to_l2: add a way to modify the dp3 parset with a script

• [New] nenudata: add new retrieve_l2 command to transfer L2 data sets

• [New] nenudata: add new push_l2 to push local data to remote server

• [New] imgpipe: improve out_dir option to support $sw$ and $obs_id$

• [New] calpipe: add an option to set the input data column for aoflagger

• [New] skymodel: in edit_model, add option to remove individual cmpt below given elevation

• [Changed] nenudata l1_to_l2: improve the distribution of time slots over nodes

• [Changed] nenudata l1_to_l2: store logs in logs directory

• [Fixed] nenudata get_ms: sort the MS

• [Fixed] skymodel: in edit_model, check that there is patches left at every steps

0.7.1 - 2024-07-22

• [New] calipipe: Add –n_cpu to soltool plot call

• [New] add option –hosts to nenudata l1_to_l2

• [New] modeltool attenuate: predict beam over the full bandwidth range. Add beamsquint_frequency_mhz parameter

• [Changed] calpipe: remove quality update at the end of the flagger task

• [Fixed] calpipe: make sure solutions_per_direction string is compatible with newer DP3

0.7 - 2024-07-10

• [New] nenudata l1_to_l2 add timeslot_multiple parameter to make sure nbs of time slots in time-split MS are multiple of timeslot_multiple

• [New] datahandler: add the possibility to create composite spectral windows which is a combination of other spectra windows (like SW00 = [‘SW01’, ‘SW02’])

• [Changed] DPPP -> DP3

0.6.2 - 2024-07-09

• [Changed] update requirements

0.6.1 - 2024-07-09

• [Changed] Update astropy requirement

0.6 - 2024-07-09

• [Changed] Deprecate usage of sshpass

• [New] nenudata: Add option to retrieve only specific SWs

• [Changed] vis_flagger: add a hpass_filter option to the flag_freqs_band

• [Changed] vis_flagger: add flag_time_freq_outliers

• [Changed] vis_flagger: make flagger steps optional, flag_time_freq_outliers is not on by default

• [Changed] datahandler: allow to set multiple identical nodes in the list of 2 nodes

0.5 - 2023-04-28

• [New] Add solveralgorithm and solutions_per_direction parameter to ddecal task

• [New] Add numthreads worker parameter

• [New] Add new flagttol vis_flagger command

0.4 - 2023-02-12

• [New] nenucal subtract step: copy column when no patch to subtract

• [New] Support %YEAR%, %MONTH%, %OBS_ID% in all L1 data path

• [New] datahandler: If obs_id is already in the data_path, do not add it at the end

• [New] Add –combine_obs_ids option to imgpipe. By default obs_ids are not combine.

0.3 - 2022-12-19

• [New] Update skymodel.py to nenupy >= 2.0. Updated dependency to nenupy >=2.0

• [New] nenudata: add –env_file option to l1_to_l2 command

• [New] nanudata: add –run_on_host option to retrieve command

• [New] datahandler: add n2_nodes to get_all_hosts()

• [New] nenudata: add an option to append a postfix to the outputed MS paths

• [Fixed] skymodel: polarization parameter of the compute_beam functions was not correctly set.

• [Fixed] skymodel: fix skymodel.concatenate

0.2.1 - 2022-10-19

• [Fixed] In get_ms_freqs(), use reshape instead of squeeze()

0.2 - 2022-07-24

• [New] Implement N2 data level with MS split in time slices over the different nodes

• [Fixed] Improve solution smoothing.

• [Fixed] Fix flagtool loading wrong msutils module.

• [Fixed] Fix modeltool loading wrong skymodel module.

• [Fixed] soltool: improve opening solution file without directions.

• [Fixed] Fix frequency range flagging.

0.2 (RC1) - 2022-01-13

• [Fixed] skymodel.edit_model: If min_elevation_patch <= 0, do not check patch for elevation

• [Fixed] fix aoquality command line

0.2 (beta) - 2021-11-30

• [New] add imgpipe command line tool.

• [New] add the option skymodel.int_ateam_sky_model to set a different Ateam sky model than the default one.

• [New] add the possibility to subtract without applying a calibration table by specifying an empty cal.parmdb.

• [New] add predict task.

• [New] add the option skymodel.app_sky_model_file to set a specific main field apparent sky model.

• [Changed] improve “modeltool build” messages.

• [Changed] update Ateam sky model

0.1.11 - 2021-07-13

• [Changed] update dependencies to allow python >=3.6, astropy >=3.2, keyring>=20, and include losoto from pypi. LSMtool still not include you have to install it with pip afterwards.

0.1.10 - 2021-07-01

• [Fixed] SkyModel.get_directions() was not always returning the correct patch

0.1.9 - 2021-05-18

• [Changed] soltool smooth: use astropy convolve function to smooth the data, it should be more reliable for missing data.

0.1.8 - 2021-04-30

• [Fixed] update dependencies

0.1.7 - 2021-04-30

• [Added] nenucal.msutils has been moved from libpipe.msutils.