[alma-sw-ssr] Definition pipelines

[Frederic Gueth <gueth@iram.fr>]

Barry Clark wrote:
> 
> >
> > The subject of the Section 2 on Pipeline Requirements is referred to as the
> > "Pipeline".  This may be implemented as disparate
> > tools or programs, or as separate packages provided by different groups,
> > or as a single package, as long as it fulfills the requirements.
> >
> > The subject of Section 3 on Offline Data Reduction Requirements is referred to
> > as the "Package" or "Offline Package".  This may be implemented as disparate
> > tools or programs, or as separate packages provided by different groups but
> > integrated into a single suite, or a single package.
> >
> 
> Perhaps instead:
> 
> The "Package" or "Offline Package" is a set of tools or programs, believed
> adequate for ALMA reductions, and used by ALMA staff for reductions upon
> which the behavior of the system will be judged.  It may consist of packages
> provided by different groups, with transitions provided to integrate them
> into a single suite.  The requirements will state that the Package will be
> available for installation on the observer's own computer systems.  The
> requirements on the Package are set forth in Section 3.
> 
> A "Pipeline" is a set of operations, implemented by the underlying Package,
> which takes a concise description of the way these operations are to be
> performed and accesses ALMA data, either from the ALMA archive or from
> local files, and produces a desired data product.  (For purposes of software
> requirements, the alternate definitions as a machine or set of machines,
> or as the supervisory process that invokes these operations are less useful.)
> There are several Pipelines essential to the efficient operation of ALMA.
> 
> \bullet The Calibration Pipeline operates in quasi real time, looks at only
> calibrator observations, and produces one or more of the following data

In the current draft, the calibration pipeline does not ignore the 
observations of the astronomical source: it applies (in the sense: 
store the appropriate quantity in the relevant header) the atmospheric 
calibration to all incoming observations. 

> products (depending on the type of observation and type of calibrator), and
> places the results in a location where they can be accessed both by the
> real-time system and by other reduction proceedures:  1) an antenna pointing
> offset for all antennas. 2).  Tsys for all antennas as a function of time.
> 3.) Sideband ratios for all antennas.  4.)  Antenna based flux calibration
> (TSYSJY) from a flux calibrator.  5.)  Antenna based bandpass calibration.
> 6.) Antenna based polarization leakage terms (with the usual indeterminate
> offsets from a single observation).  7.) Antenna based IF phase differences
> (from a strongly polarized calibrator).  8.) Antenna based phase calibration
> (with noise and atmospheric rms).

The list should be left open in such a general description. For
instance, 
the focus offset or the antenna positions derived from a baseline
measurement 
are missing.

> 
> \bullet The Science Pipeline will process most science data.  It's data product
> is an image cube.  This product will in many cases be adequate to achieve the
> observer's science goals.  It may access ALMA data from several observing
> sessions and even from observations not the observer's own.  It is intended
> to produce the best image possible without the intervention of an expert
> observer.  The Science Pipeline will include a data calibration phase, that
> may, in fact, run somewhat asynchronously with the image making phase; this
> should not be confused with the Calibration Pipeline above.

To avoid confusion, I would suggest to change the "calibration pipeline"
name 
to "real-time calibration pipeline". It can also be run off-line, but it
*has*
to be available in real-time.

> 
> \bullet The Quick Look Pipeline will process data from only one observing
> session, and will comprise a subset of the operations of the Science Pipeline.
> It will be sufficiently limited in its processing to produce results in a
> time short compared to the length of a typical observing session.  It's
> data products (images) will usually be available while the session is still
> in progress.
> 

The functions of each pipeline is summarized in the following list,
in which "blocks" of operations are identified:


Real-time calibration pipeline
------------------------------
   
  - Data acquisition part     - store in all incoming observation the
current 
				calibration paramaters (Tsys, bandpass, ...)

  - Telescope calibration     - reduce array calibrations (pointing,
focus, delay,
				baselines,...)
			      - results are made available to the Sequencer 

  - Astronomical calibration  - reduce astronomical calibrations
(atmopheric 
				calibration, phase rms, flux scale, bandpass, ...)
			      - results are made available to the Dynamic Scheduler


Quick-look pipeline	
-------------------

  - Monitoring tools 	      - display the current properties of the
array an/or
			        observation 
			      - need results of real-time calibration pipeline

  - Calibration pipeline      - from temperature-calibrated visibilities
to uv tables
			        (simplified calibration)
			      - need results of real-time calibration pipeline

  - Imaging pipeline   	      - from uv tables to images (simplified
version)
			      - need results of previous calibration pipeline

  - Display tools 	      - display current observations, to allow the
operator/AoD 
				easy checks of the data quality 
			      - need results of previous calibration and/or imaging
				pipelines

Science pipeline
----------------

  - Calibration pipeline      - from temperature-calibrated visibilities
to uv tables

  - Imaging pipeline 	      - from uv tables to deconvolved images





Frederic.