Techniques for Dealing with MDI List Data

SOI TN 96-128

R. S. Bogart

revised 13 March 1996

Table of Contents

Introduction

Most of the MDI data products originate in list (one-dimensional) rather than two-dimensional images. The formats are adopted for the sake of compression in telemetry, storage, and processing. Most complete and binned full-disc images are cropped to remove meaningless data from beyond the solar limb etc. The list formats are preserved by definition through the level-0 and level-1 processing steps (decommutation and calibration of data products), which do not involve any remapping of the data. Managing and analyzing these data products involves several types of procedures, which can be grouped as follows: Each of these topics is addressed separately below, with a discussion of the state of the existing software tools of which I am aware. The list is undoubtedly incomplete.

It should be noted that there are really two distinct classes of list data: cropped data, in which only selected image pixels are included, but at the original resolution, and binned data, in which multiple image pixels are combined. Although the former is strictly speaking a subset of the latter, separate tools have been developed to deal with the two classes of list data, and it may be necessary to preserve all of them, at least until the whole field is brought into some semblance of order.

Image Reconstruction

The primary module for converting list data is imgrecon by Rick Bogart. This module is completely based on the internal function MDI_image_reconstruct() described under Support Functions below, and suffers from the same limitations. The primary limitation of concern is that it does not work for list made with vector-weighted bins (the Structure Program Medium-L data). As long as that limitation exists, recourse may be had to any of several other modules when necessary: make_list_image by Katie Scott, old_imgrecon by K. Scott & R. Bogart, and med_l_recon by Phil Scherrer. All of these modules use the same approach and data reconstruction technique for the vector-weighted bins. There is also a module imgrecon_vds by P. Scherrer that is virtually identical to imgrecon but uses a different strategy, keeping the entire dataset in memory until the end. Its limitations are likewise identical to those of the underlying function MDI_image_reconstruct(). make_list_image and med_l_recon expect the input data to be short word integers, and will fail if the data are not. old_imgrecon can deal with any input data, but will explicitly force its output data to be shorts. imgrecon and imgrecon_vds preserve data type. imgrecon is the only one of these modules with a man page. When last reviewed, make_list_image did not correctly set fill values for regions outside the list, but that may have been subsequently corrected. Once the vector-weighted bin list capability is added to imgrecon, all the competing modules can probably safely be dispensed with.

List Construction

depsim_limb by Rick Bogart constructs the annular pixel list used for the limb figure observations in the Structure Program, given a disc center location, disc radius, and desired annular thickness. Its default output format is identical to that of the downloaded limb-mask tables from the simulator. The only limb-mask table downloaded from MDI has a different format, but it is probably erroneous (see the discussion under Visualization below). The module is registered and reasonably well-tested, with a man page.

Image Conversion

fakelimb by Rick Bogart produces simulated Structure Program limb annulus data from an image and the appropriate mask file. It is a simple module that only operates on one image at a time. It is not registered (location: /home/rick/soi/LimbFigure) and there is no man page.

Formats (out of date!)

The primary reference for information on individual data products sorted by Data Product Code (DPC) is the controlled file /home/soi/CM/tables/dpc_table.txt. This is a flat 10-column rdb file. In principle, no data product with a valid DPC should ever appear before the appropriate entry has been made in that table. I do not know what mechanisms ensure this. For each DPC (column 0) there is an associated format file (column 5), which is either a filename (always including the extension .cfil) or ".", in which case the associated format file is DPC.cfil where DPC is the 8-character DPC. There does not appear to be any simple automatic way of determining from the DPC table if a particular DPC refers to list-type data. Of course any DPC for which the rank is not equal to 1 does not.

The format files reside on the directory /home/soi/CM/tables/dpc_format (which is really a link to /mdi/mdi_egse_sw/dbase/ops/dpc_format and so not really under CM control). There should be (but is not) one file for each referenced format file in the DPC table. In particular, all or most of the files referenced as "." in the DPC table are missing. Each format file is a text file consisting of a few lines of commentary plus a single key-value pair that gives the name of a file called the IP_LIST. Thus the format files would appear to be completely redundant: they provide a one-to-one mapping of IP_LIST names to format file names. The following IP_LIST's are actually in current use:

Some of these are referenced by more than one format file, which in turn may be referenced by multiple DPC's.

The IP_LIST names can be used to construct the filenames of vector list FITS files which reside on the directory /home/soi/CM/tables/lists (which is really a link to /mdi/mdi_egse_sw/dbase/ops/lists and so not really under CM control). For each IP_LIST there are several files with names of the form pixmapN_IP_LIST.ext, where N is an optional number giving the output image size for reconstruction (absent in the case of interest, where we are not interested in just sampling), IP_LIST is the name of the IP_LIST as shown above, and ext is either ascii or fits (fits in the case of interest). The relevant FITS files are two dimensional, consisting of a set of image locations followed by the corresponding set of list locations. Since the image locations in the full, unsampled vector lists are simply the record numbers, the files are really twice as long and take twice as long to read as necessary. There is a simple module crop in /home/rick/src that was used to reduce the standard crop vector list by half to the file stdcrop.fits on the same directory. It could probably be applied to the other vector lists as well.

On the same directory as the pixmap files there is a set of files with names of the form list_xxx.fits where xxx is a name referring to a particular type of list, e.g. vwbin, vwbin_6001, crop, crop3, crop_0fc0, etc. These lists appear to be identical in format to the level-0 list dumps. They are all in fact either true dumps from the IP, processed by non-pipeline software, or dumps which have simply been patched (adjusted) so that the address references internally are from offset 0. They are not constructed by ground software.

Support Functionss (out of date!)

There is a suite of functions (documented under the man page for vector_bins) by Lyle Bacon. There is a function VW_bins_init() to initialize a bin list structure and several functions, VW_bins_init(), VW_bins(), VW_bin_box(), VW_bin_pixels(), VW_bin_pixel(), VW_bins_at_pixel(), and VW_bin_at_pixel(), that can be used to extract list information appropriate to particular image locations. The functions currently reside in /home/lyle/SOI/Lists. These functions have not been designed to work for general lists, although they could probably be modified to do so. They work only with the structure of a FITS file representing the Structure Program Vector-Weighted Bin List (/scr5/Lists/list_vwbin.fits). It is apparently identical to the file list_vwbin.fits described above. It has not been tested with the file list_vwbin_6001.fits, which is probably the one being used currently, but they are at least similar in structure. The list structure apparently does not agree with the documentation. The programsyyy have not been used in any of the modules described elsewhere.

There is a function get_vector_list() included in the source code for module imgrecon that with a small effort (replacing one global struct with an argument and compiling in fixed path names that ought not to be supplied as arguments anyway) could be extracted and placed in a library (libMDI is the reasonable one) for general use. It is strictly DPC driven and returns an appropriate bin list.

There is also a function MDI_image_reconstruct() in the source code for module ~rick/plot/mag_gif.c (see below) that is strictly DPC driven and works directly with the small crop list file known to be associated with the DPC, replacing a SDS containing list data with one containing the appropriately reconstructed data. So far it has only been demonstrated to work with the crop table associated with data with DPC's ????0fc0, but there is no reason to doubt that it works with any single-pixel crop data. I would recommend it as the basis for a general library function. This would really obviate the need for get_vector_list(). However, the functionality of the more general-purpose summing and weighting tables remains to be supplied. For the rectangular-weighted bin lists this should be straightforward, it is just a matter of supplying the appropriate axis-lengths. Generalization to the case of vector-weighted bins and the LOI mask is not so obvious.

Visualization

The registered module atlas_page by Rick Bogart & Katie Scott produces postage-stamp PostScript images from 2-dimensional image data only; it does not know how to deal with list data, but could be modified to do so without a great deal of difficulty, using the same code that already exists in for example. It is extensively tested (reams of atlas pages have been generated with it), but there is no man page. Its default output format is tuned to typical early ground observing sequences, in which sets of 5 filtergrams were collected to be processed offboard. It should be modified to scale the images to fit on one page when they fill an hour, which is the standard high-rate data set size ( 6*10 rather than 5*7). This is an easy fix. Another useful fix wouldd be to have the reference time printed by the image, or at least the time of the dataset hour/day printed at the head, and for images to be located according to their time in the hour when appropriate rather than by their sequence number. The program has elaborate mechanisms for scaling that are sometimes defeated by missing data, and that should probably be abandoned in favor of DPC-driven scaling as appropriate.

There are several modules under development by Rick Bogart to create GIF images from full-image data. They reside on ~rick/plot. They have been used to create the images used in the Web pages. The same remarks regarding list data (non-)processing by atlas_page apply to these modules. They are not documented. One special issue that needs to be addressed in building a routine DPC-driven procedure to produce quick-look GIF's is tying the appropriate color table to the DPC. Currently there are simply separate programs to use for photograms, Dopplergrams, and magnetograms to supply the appropriate scaling information and color tables.

The module showmask by Rick Bogart on ~rick/soi/LimbFigure builds an image of the locations of the mask pixels themselves (0's and 1's). It works with the only example of a limb mask table in the simulator data (prog:mdi_sim,lev:lev0,ser:ffffff74_01h[19630]), but fails with the only example in space data, (prog:mdi_eof,lev:lev0,ser:ffffff74_01h[25779]). It also works with the file /home/soi/CM/tables/lists/list_limb_afb0.fits which is presumably identical to a valid dump of a limb crop list. Although not specifically designed for it, it also works successfully with the only example of a crop table in the simulator data (prog:mdi_sim,lev:lev0,ser:ffffff73_01h[19630]), and fails (with the same error messages) with the only example in space data, (prog:mdi_eof,lev:lev0,ser:ffffff73_01h[25779]). The failure to work with space data may be attributed to either errors in the tables themselves or in the processing of them, since the data in both tables consist solely of the value 112 (70 hex). There is no man page for showmask. The corresponding module showlimb on the same directory displays the values of the limb data pixels from list data, using the appropriate mask file; there is no procedure for tying the correct mask to the data. The same caveats that apply to showmask apply to showlimb. It has not been definitely established whether it works successfully with full-disk crop data, although it should.

There are numerous IDL tools for dealing with list data that have presumably been documented elsewhere. They are outside the scope of this note.

Appendix: Data Product Codes

This is a summary of the inferred meanings of all Data Product Codes representing list (one-dimensional) data as of 11 Jan 1996.
xxxx0fc0
full-disc crop data using crop list e2000fc0
xxxx5001
data using boxcar bin list e2005001
xxxx6001
data using vector-weighted bin list e2006001
xxxx7001
data using LOI bin list e2007001
c7009001, c7c09001, 47c09001, c7c19001
data using LOI bin list e2007001 repeatedly (from circular buffer)
xxxx7002, xxxx7003
data using LOI list e2007002
c3009001, c3c09001, 43c09001
data using LOI bin list e2007002 repeatedly (from circular buffer)
xxxxafb0
limb figure data using limb figure crop list e200afb0
bfffff29, cfffff25, cfffff26, cfffff2a, cfffff2b, cfffff2c, cfffff3a, cfffff3b, cfffff3c, ffffff93
data using unknown crop lists
9fffffb3, bfffff24, bfffffb3, bfffffbb, cfffff24, efffffb3, ffffff47, ffffff94
data using unknown limb crop lists
8fffffb4, 8fffffb5, bfffffb4, bfffffb5, bfffffb6, bfffffb7, cfffffb4, cfffffb5
data using unknown irregular bin lists
cfffff2e
cfffff2f
data using unknown irregular bin lists repeatedly (from circular buffer)
8fffffb1, 8fffffb2, bfffff22, bfffff23, bfffffb1, bfffffb2, bfffffb9, bfffffba, cfffff22, cfffff23, cfffffb1, cfffffb2, ffffff96, ffffff99
data using unknown boxcar bin lists
8fffffb0, bfffff21, bfffffb0, ffffff95, cfffff21, cfffffb0
data using unknown vector-weighted bin lists
ffffff73
a full-disc crop list
ffffff74
a limb-figure crop list
ffffff75
a vector-weighted bin list
ffffff76
a boxcar bin list
ffffff7d
an irregular bin list
ffffff70, e1003001, e1003002
square root lookup table
ffffff71, e1001001
reciprocal lookup table
ffffff72, e1002001
velocity lookup table
200xxxxx, a000xxxx, bfffff65, cfff0000, cfff0028, cfff0050, cfff05b4
IP Register dumps
ffffff30, ffffff80, ffffff97, ffffff9b, ffffffb7, ffffffe0, bfffffed, cfffff98
unknown list format data
ffffff79, ffffff7b
unknown lists

Please address comments and questions to the author(s).

Return SOI Technical Notes Index - Return SOI home page