Techniques for Dealing with MDI List Data
SOI TN 96-128
R. S. Bogart
revised 13 March 1996
Table of Contents
Most of the MDI data products originate in list (one-dimensional) rather
than two-dimensional images.
The formats are adopted for the sake of compression in telemetry, storage,
and processing. Most complete and binned full-disc images are cropped to
remove meaningless data from beyond the solar limb etc. The list
formats are preserved by definition through the level-0 and
level-1 processing steps (decommutation and calibration of data
products), which do not involve any remapping of the data. Managing and
analyzing these data products involves several types of procedures, which
can be grouped as follows:
Each of these topics is addressed separately below, with a discussion of
the state of the existing software tools of which I am aware. The list
is undoubtedly incomplete.
- Reconstruction of Images from list format
- Construction of Lists, i.e. simulation
(or supplementing) of the onboard DEP procedures for creating the binning,
cropping, or weighting lists
- Conversion of Images to list format data
(simulation of the onboard IP procedures)
- Formats of lists and means of tying the
correct information to the data products
- Support Functions that can be called by
special purpose programs and new modules for dealing with the MDI lists
- Visual Monitoring of the data products as they are
produced and archived
It should be noted that there are really two distinct classes of list data:
cropped data, in which only selected image pixels are included, but at the
original resolution, and binned data, in which multiple image pixels are
combined. Although the former is strictly speaking a subset of the latter,
separate tools have been developed to deal with the two classes of list
data, and it may be necessary to preserve all of them, at least until the
whole field is brought into some semblance of order.
The primary module for converting list data is imgrecon by
Rick Bogart. This module is completely based on the internal function
MDI_image_reconstruct() described under Support
Functions below, and suffers from the same limitations. The primary
limitation of concern is that it does not work for list made with
vector-weighted bins (the Structure Program Medium-L data). As long
as that limitation exists, recourse may be had to any of several other
modules when necessary: make_list_image by Katie Scott,
old_imgrecon by K. Scott & R. Bogart, and med_l_recon
by Phil Scherrer. All of these modules use the same approach and data
reconstruction technique for the vector-weighted bins.
There is also a module imgrecon_vds by P. Scherrer that is virtually
identical to imgrecon but uses a different strategy, keeping the
entire dataset in memory until the end. Its limitations are likewise
identical to those of the underlying function MDI_image_reconstruct().
make_list_image and med_l_recon expect the input data to
be short word integers, and will fail if the data are not. old_imgrecon
can deal with any input data, but will explicitly force its output data to
be shorts. imgrecon and imgrecon_vds preserve data type.
imgrecon is the only one of these modules with
a man page. When last reviewed, make_list_image did not
correctly set fill values for regions outside the list, but that may
have been subsequently corrected. Once the vector-weighted bin list capability is
added to imgrecon, all the competing modules can probably safely
be dispensed with.
depsim_limb by Rick Bogart constructs the annular pixel list used for
the limb figure observations in the Structure Program, given a disc center
location, disc radius, and desired annular thickness. Its default output
format is identical to that of the downloaded limb-mask tables from the
simulator. The only limb-mask table downloaded from MDI has a different
format, but it is probably erroneous (see the discussion under Visualization
below). The module is registered and reasonably well-tested, with a man page.
fakelimb by Rick Bogart produces simulated Structure Program limb
annulus data from an image and the appropriate mask file. It is a simple
module that only operates on one image at a time. It is not registered
(location: /home/rick/soi/LimbFigure) and there is no man page.
Formats (out of date!)
The primary reference for information on individual data products sorted
by Data Product Code (DPC) is the controlled file
/home/soi/CM/tables/dpc_table.txt. This is a flat 10-column rdb file.
In principle, no data product
with a valid DPC should ever appear before the appropriate entry has been
made in that table. I do not know what mechanisms ensure this. For each
DPC (column 0) there is an associated format file (column 5), which is
either a filename (always including the extension .cfil) or ".",
in which case the associated format file is DPC.cfil where
DPC is the 8-character DPC. There does not appear to be
any simple automatic way of determining from the DPC table if a particular
DPC refers to list-type data. Of course any DPC for which the rank is not
equal to 1 does not.
The format files reside on the directory /home/soi/CM/tables/dpc_format
(which is really a link to /mdi/mdi_egse_sw/dbase/ops/dpc_format
and so not really under CM control).
There should be (but is not) one file for each referenced format file in
the DPC table. In particular, all or most of the files referenced as "."
in the DPC table are missing. Each format file is a text file consisting
of a few lines of commentary plus a single key-value pair that gives the
name of a file called the IP_LIST. Thus the format files would appear to
be completely redundant: they provide a one-to-one mapping of IP_LIST names
to format file names. The following IP_LIST's are actually in current use:
Some of these are referenced by more than one format file, which in turn may
be referenced by multiple DPC's.
- irbin_7001 warning: non-standard data type!
- irbin_7004 warning: non-standard data type!
The IP_LIST names can be used to construct the filenames of vector list
FITS files which reside on the directory
/home/soi/CM/tables/lists (which is really a link to
/mdi/mdi_egse_sw/dbase/ops/lists and so not really under CM control).
For each IP_LIST there are several files
with names of the form pixmapN_IP_LIST.ext,
where N is an optional number giving the output image size for
reconstruction (absent in the case of interest, where we are not interested
in just sampling), IP_LIST is the name of the IP_LIST as shown above,
and ext is either ascii or fits (fits in the
case of interest). The relevant FITS files are two dimensional,
consisting of a set of image locations followed by the corresponding
set of list locations. Since the image locations in the full, unsampled
vector lists are simply the record numbers, the files are really twice as
long and take twice as long to read as necessary. There is a simple module
crop in /home/rick/src that was used to reduce the standard
crop vector list by half to the file stdcrop.fits on the same
directory. It could probably be applied to the other vector lists as well.
On the same directory as the pixmap files there is a set of files with names
of the form list_xxx.fits where xxx is a name referring
to a particular type of list, e.g. vwbin, vwbin_6001, crop, crop3,
crop_0fc0, etc. These lists appear to be identical in format to the
level-0 list dumps. They are all in fact either true dumps from the IP,
processed by non-pipeline software,
or dumps which have simply been patched (adjusted) so that the address
references internally are from offset 0.
They are not constructed by ground software.
There is a suite of functions (documented under the man page for
vector_bins) by Lyle Bacon. There is a function
to initialize a bin list structure and several functions,
VW_bin_at_pixel(), that can be used to extract list information
appropriate to particular image locations.
The functions currently reside in /home/lyle/SOI/Lists. These functions
have not been designed to work for general lists, although they could
probably be modified to do so. They work only with the structure of a
FITS file representing the Structure Program Vector-Weighted Bin List
(/scr5/Lists/list_vwbin.fits). It is apparently identical to the
file list_vwbin.fits described above. It has not been tested with the file
list_vwbin_6001.fits, which is probably the one being used currently, but
they are at least similar in structure. The list structure apparently does
not agree with the documentation. The programsyyy have not been used in any
of the modules described elsewhere.
There is a function get_vector_list() included in the source code
for module imgrecon that with a small effort (replacing one global
struct with an argument and compiling in fixed path names that ought not to
be supplied as arguments anyway) could be extracted and placed in a library
(libMDI is the reasonable one) for general use. It is strictly DPC
driven and returns an appropriate bin list.
There is also a function MDI_image_reconstruct() in the source code
for module ~rick/plot/mag_gif.c (see below) that is strictly DPC
driven and works directly with the small crop list file known to be associated
with the DPC, replacing a SDS containing list data with one containing the
appropriately reconstructed data. So far it has only been demonstrated to
work with the crop table associated with data with DPC's ????0fc0, but there
is no reason to doubt that it works with any single-pixel crop data. I would
recommend it as the basis for a general library function. This would really
obviate the need for get_vector_list(). However, the functionality of
the more general-purpose summing and weighting tables remains to be supplied.
For the rectangular-weighted bin lists this should be straightforward, it is
just a matter of supplying the appropriate axis-lengths. Generalization to
the case of vector-weighted bins and the LOI mask is not so obvious.
The registered module atlas_page by Rick Bogart & Katie Scott
produces postage-stamp PostScript images from 2-dimensional image data
only; it does not know how to deal with list data, but could be modified
to do so without a great deal of difficulty, using the same code that
already exists in for example. It is extensively tested
(reams of atlas pages have been generated with it), but there is no
man page. Its default output format is tuned to typical early ground
observing sequences, in which sets of 5 filtergrams were collected to
be processed offboard. It should be modified to scale the images to fit
on one page when they fill an hour, which is the standard high-rate data
set size ( 6*10 rather than 5*7). This is an easy fix. Another useful
fix wouldd be to have the reference time printed by the image, or at least
the time of the dataset hour/day printed at the head, and for images to be
located according to their time in the hour when appropriate rather than
by their sequence number. The program has elaborate mechanisms for scaling
that are sometimes defeated by missing data, and that should probably be
abandoned in favor of DPC-driven scaling as appropriate.
There are several modules under development by Rick Bogart to create
GIF images from full-image data. They reside on ~rick/plot.
They have been used to create the images used in the Web pages. The
same remarks regarding list data (non-)processing by atlas_page apply
to these modules. They are not documented. One special issue that needs
to be addressed in building a routine DPC-driven procedure to produce
quick-look GIF's is tying the appropriate color table to the DPC.
Currently there are simply separate programs to use for photograms,
Dopplergrams, and magnetograms to supply the appropriate scaling
information and color tables.
The module showmask by Rick Bogart on ~rick/soi/LimbFigure
builds an image of the locations of the mask pixels themselves (0's and 1's).
It works with the only example of a limb mask table in the simulator data
but fails with the only example in space data,
(prog:mdi_eof,lev:lev0,ser:ffffff74_01h). It also works with
the file /home/soi/CM/tables/lists/list_limb_afb0.fits which is
presumably identical to a valid dump of a limb crop list.
Although not specifically designed for it, it also works
successfully with the only example of a crop table in the simulator data
and fails (with the same error messages) with the only example in space data,
(prog:mdi_eof,lev:lev0,ser:ffffff73_01h). The failure to
work with space data may be attributed to either errors in the tables
themselves or in the processing of them, since the data in both tables
consist solely of the value 112 (70 hex). There is no man page for
The corresponding module showlimb on the same directory displays
the values of the limb data pixels from list data, using the appropriate
mask file; there is no procedure for tying the correct mask to the data.
The same caveats that apply to showmask apply to showlimb.
It has not been definitely established whether it works successfully with
full-disk crop data, although it should.
There are numerous IDL tools for dealing with list data that have presumably
been documented elsewhere. They are outside the scope of this note.
This is a summary of the inferred meanings of all Data Product Codes
representing list (one-dimensional) data as of 11 Jan 1996.
- full-disc crop data using crop list e2000fc0
- data using boxcar bin list e2005001
- data using vector-weighted bin list e2006001
- data using LOI bin list e2007001
- c7009001, c7c09001, 47c09001, c7c19001
- data using LOI bin list e2007001 repeatedly (from circular buffer)
- xxxx7002, xxxx7003
- data using LOI list e2007002
- c3009001, c3c09001, 43c09001
- data using LOI bin list e2007002 repeatedly (from circular buffer)
- limb figure data using limb figure crop list e200afb0
- bfffff29, cfffff25, cfffff26, cfffff2a, cfffff2b, cfffff2c,
cfffff3a, cfffff3b, cfffff3c, ffffff93
- data using unknown crop lists
- 9fffffb3, bfffff24, bfffffb3, bfffffbb, cfffff24, efffffb3,
- data using unknown limb crop lists
- 8fffffb4, 8fffffb5, bfffffb4, bfffffb5, bfffffb6, bfffffb7,
- data using unknown irregular bin lists
- data using unknown irregular bin lists repeatedly (from circular buffer)
- 8fffffb1, 8fffffb2, bfffff22, bfffff23, bfffffb1, bfffffb2,
bfffffb9, bfffffba, cfffff22, cfffff23, cfffffb1, cfffffb2, ffffff96,
- data using unknown boxcar bin lists
- 8fffffb0, bfffff21, bfffffb0, ffffff95, cfffff21, cfffffb0
- data using unknown vector-weighted bin lists
- a full-disc crop list
- a limb-figure crop list
- a vector-weighted bin list
- a boxcar bin list
- an irregular bin list
- ffffff70, e1003001, e1003002
- square root lookup table
- ffffff71, e1001001
- reciprocal lookup table
- ffffff72, e1002001
- velocity lookup table
- 200xxxxx, a000xxxx, bfffff65, cfff0000, cfff0028,
- IP Register dumps
- ffffff30, ffffff80, ffffff97, ffffff9b, ffffffb7, ffffffe0,
- unknown list format data
- ffffff79, ffffff7b
- unknown lists
Please address comments and questions to the author(s).
SOI Technical Notes Index
SOI home page