Reading the DATA column by pieces
Keywords: mrtcal-prog-manual

Next: Positions Up: Reading the DATA column Previous: Dump cycle identification Contents

# Reading the `DATA` column by pieces

MRTCAL is designed to be able to process the subscans within an arbitrary limited buffer size (`MSET BOOKKEEPING SPACE Value`, default 512 MBytes). If the buffer size is large enough, the whole `DATA` column can be loaded at once in memory and its dumps can be treated in a row. If the buffer is not large enough, the `DATA` column has to be load by pieces, and each piece has to be processed separately from the other.

In practice, the calibration routine asks the reading routine to load the piece of `DATA` which contains at least a desired range of dumps (usually a cycle of a few dumps). At least means that the reading routine ensures that the range will be present entirely in the buffer, but more dumps will also be loaded in order to fill at best the buffer. On subsequent requests, if the desired range is already present in the buffer, nothing will be reloaded, in order to save calls to `CFITSIO` and I/O accesses.

At this stage, one should consider that `CFITSIO` is asked to read a block of rows from the `DATA` column. `CFITSIO` is not able to skip some bad rows here and there, and it would be unefficient to ask for reading the desired rows one by one. So `CFITSIO` reads a piece of the `DATA` column, including good and bad dumps. But right after, MRTCAL sets a 3D chunkset ( ) pointing to the good dumps only. Then the calling routine has just to work with the 3D chunkset, with no need to care for bad dumps.

Next: Positions Up: Reading the DATA column Previous: Dump cycle identification Contents