Really Huge Problems
Keywords: imager

Next: Polarization Handling Up: imager Previous: UV table consistency: Contents Index

#
Really Huge Problems

[rgb]1,0,0IMAGER basically assumes everything fits into memory. This is in general quite fine for NOEMA data. However, for ALMA data, if you are working with a too small computer (such as my laptop, which is otherwise fine), you may be lacking physical memory (RAM, Random Access Memory), and [rgb]1,0,0IMAGER may become really inefficient by using Virtual Memory instead.

To avoid time losses, [rgb]1,0,0IMAGER prevents reading UV data whose size exceeds the available RAM, and warns the user if it exceeds half of the RAM. To treat these cases, [rgb]1,0,0IMAGER provides instead a number of tools that can work sequentially on the data set, instead of loading it in memory all at once.

1. Working by subsets:
   the magenta`READ /RANGE` command allows to select an ensemble of channels from a UV data. If this ensemble is small enough, [rgb]1,0,0IMAGER can work. At the end the magenta`WRITE /APPEND` and magenta`WRITE /REPLACE` command will allow to put these channels at their proper places in a deconvolved data cube.
2. Working on UV data files:
   Some operations on UV data, like time averaging, separation of line from continuum emission, or self-calibration, and of course, spectral resampling, are best done using all valid channels to avoid loosing sensitivity. To allow [rgb]1,0,0IMAGER to do them even for large data files, most UV-related commands have a magenta`/FILE` option which instructs the command to work from the corresponding data file, instead of the UV buffers. This includes magenta`UV_PREVIEW /FILE`, magenta`UV_BASELINE /FILE`, magenta`UV_FILTER /FILE` and especially the magenta`UV_SPLIT /FILE` commands. Time averaging can be performed by magenta`UV_TIME /FILE`, and prior sorting by time order can be done by magenta`UV_SORT /FILE`. Spectral range extraction is possible by magenta`UV_EXTRACT /FILE`, spectral resampling by magenta`UV_COMPRESS /FILE`, magenta`UV_RESAMPLE /FILE`, and magenta`UV_HANNING /FILE`.

By using the above commands, all operations can be done in a quasi sequential way, avoiding to load in memory whole data sets.

Two commands have no equivalent using the [rgb]1,0,0IMAGER buffers, and work only on files. Their magenta`/FILE` option is used to provide an homogeneous syntax, but must be present for the command line to be valid.

1. the magenta`UV_MERGE /FILE`
   that allows to merge together an arbitrary number of UV tables, in spectral line (with spectral resampling) or continuum (with flux scaling according to a spectral index) modes.
2. the magenta`UV_SPLIT /FILE`
   that combines the capabilities of magenta`UV_BASELINE` and magenta`UV_FILTER` in a single command, since both operations require the same parameters and provide complementary informations.

The magenta`UV_SORT /FILE` command also has a different behaviour than its memory only version magenta`UV_SORT`. While the latter creates a transposed version of the UV table for internal use (it cannot be saved), the former keeps the normal organisation with the visibility axis first.

An example of use of such facilities is the magenta`@`violet`image-mosaic` script that splits the magenta`uv_map` step into chunks that fit in the computer memory available to the user.

Next: Polarization Handling Up: imager Previous: UV table consistency: Contents Index