Oracle Major Frame Reconstruction Scheme (3/26/1997)

Working Definitions for Assembly of major frame reconstruction:

Database tables are defined and grouped in four sets.  Also defined are grades of major frame table entries, which will be used to record major frame reconstruction information as it is performed.

GFST:  the Good Frame Statistics Tables contain entries for the major frames which were assembled with no errors during initial processing of the Anotated Minor Frames (AMF).  These major frames have been decommutated and sent to DPS.  There is one GFST built per AMF session.

BFST:  the Bad Frame Statistics Tables contain entries for the minor frames which could not be built into major frames during initial processing of the AMF.  There is one BFST built per AMF session.  The minor frame entries in the BFSTs are in the same order as the minor frames were archived to the associated bad frame log file, so that if table entry N (N=1,2,...NMAX) is desired, it can be read starting from byte (N-1) times minor_frame_length of the bad frame log file.  NOTE:  to support the possibility of "odd" length minor frames, we will add a column for start byte and a column for length of archived frame, rather than rely on rigid adherence to the minor frame length.  Such odd length minor frames may not be any good, but they need to be accounted for so that they do not corrupt the location of other frames.

AFST:  the accumulated Frame Statistics Tables contain a running archive of the disposition of all expected major frames.  The entries in this file are chronologically ordered, and periodically old records are saved away to history files.  The archive could be driven by the grade of the major frame, so that MJF's which are missing or were partly filled would stay in this AFST indefinitely, until an analyst directed they be archived and cleared.

WFST:  At the beginning of a reconstruction session, entries for all of the grade 1 and 2 major frames which are outstanding are loaded into the Working Frame Statistics Table.  During major frame reconstruction, a set of working tables are needed to hold "scratch" information as the recipes for performing the reconstruction on a given major frame are built.  Following successful reconstruction of a major frame, the recipe used to construct the major frame is summarized into a grade 0 or grade 1 entry for that major frame, and that major frame's entry is updated in the AFST.  If there is no data available for a given major frame, the AFST entry will stay grade 2.  At the end of the reconstruction of all major frames in a message, the WFST is dropped (deleted) and a new one will be created during reconstruction of the next message.

Grade 0 major frames have been assembled with no minor frame errors, decommutated, and sent on as instrument records to DPS.  The GFST built by the LabVIEW virtual instrument during AMF file processing only contain entries for this grade of major frames.

Grade 1 major frames have been assembled with one or more minor frames filled.  The record of which minor frames were filled will be stored as an adjoining column as a character string (or several, depending on the minor frame length).  The names of up to M bad frame log files are also stored as columns (the number M in one column, followed by M columns of bad frame file names). This grade of major frame entry will first exist in the database table built during major frame reconstruction, and these entries will be stored in the AFST for later attempts at more complete reassembly.  These grade 1 entries will persist until they are converted to grade 0 entries by a successful reconstruction or until they expire (based on age) or are deleted by the analyst. 

Grade 2 major frames have not yet been assembled in any form.  These are placeholder entries which are inserted into the WFST, which is built by expanding the good frame table built during AMF processing to span all the entries that would be expected in that AMF based on the limit-checked time span of the AMF file.  

Onset of Reconstruction

This reconstruction scheme is developed as a set of rules that are called into adjudication after the GFST and BFST are generated by LabVIEW.  The input to the reconstructino are : the GFST, the BFST, and the AFST.  The output from the reconstruction is a reconstructin recipe table which is used by LabVIEW to construct any major frame reconstruction that is indicated, and a data commentary which records the sequence of actions taken by Oracle in the course of its examination of the input tables and its effects to build the reconstruction recipe table.The Rules are first applied to the GFST and based on the state of this table, some additional rules are invoked on the BFST.  The goal here is to establish a logical hierarchy that ranks these rules in an efficient manner that facilitates the reconstruction of major frames.  

We begin by looking at rules that handle the GFST as they rank higher on our hierarchical scheme.  The most frequently used rules are implemented first.  

1.  If a mode change and/or data rate change is found in the GFST, the boundaries of data spans within which the mode and data rate are constant, are identified.  We next implement a rule that will limit the total number of data spans that will be handled to an empirically derived value, limit_data_span_number, which we currently understand is on the order of 4.
1.1  If there are less than limit_data_span_number mode changes and/or data rate changes within a single GFST, the major frames are truncated in groups, and each data span is treated as a separated entity and all the rules are applied independently to each grouping, as if belonging to separate messages.  The AFST is updated only once, after all the major frames belonging to each time span have been reconstructed.
1.2  If there are more than limit_data_span_number mode changes and/or data rate changes within a single BFST, the entire GFST is flagged as "fail" and the following message is sent to LabVIEW:  Error XX: corrupted data, playback and reprocessing necessary.  

2.  If the GFST has at least two major frames reconstructed, we perform the ratio test (see Rule 4).  
2.a.  IF rule 4 return "Pass", we proceed as of Rule 2.1
2.b.  Esle if Rule 4 return "fail", we post a message to the data commentary that a high percentage of the minor frames are in the BFST and reconstruction is stopped.
2.1.  If there are duplicates (determined based on the same UMFDATETIME (unpacked major frame datetime) in different rows) in GFST, flag them until the unflagged entries are unique.
2.1.1.  If there is only one resulting unique major frames in the GFST, proceed as per Rule 3.
2.1.2.  If there is more than one resulting unique major frame in the GFST, check for gaps.  Gaps are identified based on the major frame time of GFST entries and major frame period appropriate for the mode and data rate.  The period is positive is the major frames are arranged in ascending order and negative if they are arranged in descending order.
2.1.2.1.  If there are gaps, expand GFST into GFST_EX by inserting the missing major frames times and creating a new major frame grade column (MFGRADE).  Assembled major frames are assigned a "grade 0" and inserted major frames are assigned a "grade 2".  During the expansion, the expected UMFC (unpacked major frame counter)  is also assigned, based on preceding UMFC.
2.1.2.1.a.  If there had been a mode change and/or data rate change, the newly inserted "grade 2" major frames are searched in the other data spans of the GFST.  
   2.1.2.1.a.a.  Should they be found, they are deleted from the data spans where they had been inserted.
   2.1.2.1.a.b.  Should the newly inserted "grade 2" major frames not be found in any other data spans, they stay "grade 2" and will go through reconstruction.
2.1.2.1.b.  If there had been no mode change and/or data rate change, proceed to Rule 2.1.2.1.1.
2.1.2.1.1.  If any of the missing major frames that have been identified as missing from the time span (in GFST), are present in the accumulated frames statistics table (AFST) with a "grade 0" (this is a check to see if some of these were assembled in a previous session by looking at the AFST), do not attempt reconstruction of these because this has already successfully been done.
2.1.2.1.2.  If any of the missing major frames are not present at all in the AFST, or are present with "grade 1" or "grade 2", select these and apply the ratio rule described below.
2.1.2.1.2.1.  If R (ratio of the number of minor frames in BFST divided by the number of major frames in GFST_EX) is greater than an empirically determined quantity, attempt reconstruction of the major frames.
2.1.2.1.2.2. Else if R is less than an empirically determined quantity, the database does not attempt reconstruction of the major frames, but sends the following message to LabVIEW:  Error XX:  Small R value, therefore, reconstruction not called for.  Wait for the next GFST.
2.1.2.2.  If there are no gaps, this means that all the major frames have been assembled within a time span (determined by the span of entries in the GFST).  Now we apply the ratio rule.
2.1.2.2.1.  If Rule 2.1.2.1.2.1. attempt reconstruction of the major frames.
2.1.2.2.2.  Else if Rule 2.1.2.1.2.2., send the following message to LabVIEW:  Error XX:  Small R value, therefore, reconstruction not called for.  Wait for the next GFST.
2.2.  If there are no duplicates, check for gaps.
2.2.1.  If there are gaps, fill in the missing major frames as per Rules 2.1.2.1.  Here, there is no need to invoke rules 2.1.2.1.a  because there are no duplicates.
2.2.2.  If there are no gaps, this means there are no missing major frames 

3.  If the GFST has one or less than one major frame reconstructed, the database will try to determine from looking at the minor frames, what major frames should have been received during the current pass.  to do the latter, all the Rules that are applied to the BFST are invoked to attempt to estimate the actual unpacked major frame time for the current minor frames.
After having estimated the actual unpacked major frame time for the current minor frames, a list of major frames is generated and represents which major frames should have been received during this pass.  This list of major frames is now copied into the GFST and the then subjected to reconstruction in the WFST.

4.  Ratio Rule
We define the following:
nBFST = number of BFST entries,
nGFST = number of GFST entries,
nmf = number of minor frames per major frame

The ratio test is intended to find out how large a fraction of the total number of minor frames in the input annotated minor frame (AMF) file have entries in the BFST.

The total number of minor frames in the input file is nBFS + nmf*nGFST, so the ratio R is

R= nBFST / ( nBFS + nmf*nGFST )

 
Cases:

The following pictograms represent the content of input data files.  The character strings represent examples of the cases of input data we are trying to identify, using 'G' to represent the minor frames that comprise good major frames in the input file of annotated minor frames, and 'B' to represent a set of minor frames that were not built into major frames on the first pass, and which thus have entries in the bad frame table.

Case
Contents of Annotated Minor Frame File
Can Use Ratio Test?
All Good:
there are no bad frames
GGGGGGGGGGGGGGGGGGGGGGGGGGG
YES
Normal Data:  
some frames are bad but the span of the good frames is approximately the same as the total data span. 
BBGGBGGGGBGGGGGGGGGGBGGGGGG
YES
High Percentage Bad:
some frames may be good, but the span of the good frames is likely to be considerably shorter than the the actual data span
BBBBBBBBGBBBBBBBBBBBBBGBBBBBBB
YES
One Good Major Frame
n*B... G ...m*B
NO
All Bad:
there are no good frame table entries upon which to begin the reconstruction process, so the BFST has to be used by itself for any reconstruction
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
NO


Of the above cases, the ratio test below is defined only for nGFST>=2.  If nGFST<2 then other methods need to be used.

if nGFST>=2 then
R = nBFST / ( nBFST + nmf*nGFST )
    if R <= Desired_R_value then Result = PASS
    else Result= FAIL
    endif
endif 

Note:  Desired_R_Value might be on the order of 0.05 for WIND data.


Rules that handle the Bad Frame Statistics Table (BFST).

Before any reconstruction may begin, the minor frames are looked at to determine their quality and improve it if at all possible.  Minor frames with an overall assessment  as "fail", won't be included as a data source during reconstruction.  Only minor frames with an overall assessment as "pass" will be included as data source during reconstruction.  Therefore, the first rules assess the overall state of the minor frames.

1.  If the unpacked minor frame count (UMNFC) is inconsistent with consistent preceding and following minor frame counts, this minor frame is suspect.
1.1.  Because a minor frame may be out of sequence but still be valid because of a start and stop in the dump, we must take care not to account for duplicate minor frames twice by truncating the BFST and process each truncation as a separate BFST.  If the current UMNFC is as described in Rule 1, and another minor frame is found with the same GRT within that same BFST, the BFST is truncated in no more than four subtables ( four is the maximum number of allowed start and stop in the dump).  Each group of truncated minor frames is examined as a new BFST and reconstruction of the group of major frames identified during the current pass is attempted using each of the truncated minor frame tables.
1.2.  Else if the current UMNFC is as described in Rule 1, and no minor frame is found with the same GRT within that same BFST the minor frame is tagged as "fail".

2.  If a mode change and/or data rate change is found in the BFST, boundaries of dataspans (conseqs) within which the mode and data rate are constant are identified..  We next implement a rule that will limit the number of minor frames per conseq, limit_minor_frame_number.  
2.1  If the number of minor frames per conseq is greater than or equal to limit_minor_frame_number (which we now consider to be 1), the contigs are flagged with a "pass".
2.2  Elseif the number of minor frames per conseq is less than limit_minor_frame_number, all the minor frames within those conseqs are flagged as "fail". 

3.  If the major frame counter was unpacked in this minor frame, indicating the major frame count should be valid, but the major frame count is inconsistent with the major frame count expected, therefore this minor frame is suspect and is tagged as "fail".  

4.  If the major frame time reported in this minor frame is inconsistent with the expected major frame time, and there are no missing frames reported since the last reported major frame time, this entire set of minor frames is suspect.

All other minor frames have a default "pass" tag.  More rules for flagging minor frames of questionable quality will be developed based on input from the analysts.  
A different set of rules is now applied to the minor frames in the BFST to attempt to estimate the unpacked major frame time.

The following set of rules apply, provided there is no mode change or change in the downlink rate.  In our database tables, this is represented by a change of the CDM (current data mode) from 0 to 1.

5.  If any of the conseqs in the BFST has at least one UMFDATETIME (unpacked major frame datetime), fill in all the minor frames times per the time present in these conseqs.  A conseq (or contig) is defined as a grouping of all the minor frames as the UMNFC (unpacked minor frame counter) goes from 0 to 249.

6.  If any conseq in the BFST has no UMFDATETIME, we cannot infer the UMFDATETIME to any minor frame of that conseq since we have no reference.  The UMFDATETIME may be determined through other means such as the UMFC, and the GRT.  It should be noted that other means of determining the UMFDATETIME are considered only where there is no UMFDATETIME within a conseq, otherwise there is no reason for considering alternative means of getting the UMFDATETIME.
7.  Datetime can be inferred from the UMFC alone in each conseq by performing a table lookup in the GFST for the UMFC present in the BFST among the expected UMFC of the "missing" major frames.
7.1.  Should a UMFC match be found we have to still compare the GRT of the minor frame that of the major frame to determine if that minor frame belongs to the major frame in question.  The GRT of the minor frame must be within the expected range of all the minor frames belonging to that major frame, that is it must not be greater than the minor frame downlink period times 250 + GRT of the major frame if the major frame order is descending, or it must not be less than the minor frame downlink period times 250 + GRT of the major frame if the major frame order is ascending
7.2.  Should a UMFC match be found and the GRT of the minor frame is greater than the minor frame downlink period times 250 + GRT of the major frame if the major frame order is ascending, or less than the minor frame downlink period times 250 + GRT of the major frame if the major frame order is descending, the major frame time cannot be inferred from the UMFC.

8.  If the GFST has at least one Good major frame assembled, or also for that mater, if there is at least one UMFDATETIME present in a BFST conseq (because one occurrence of an UMFDATETIME is enough for a base to be set) , the UMFDATETIME for each minor frame in the BFST can be determined from the Ground receipt time using the following base conversion function.
Datetime(GRT) = (GRT - direction * mfc * mfdp - GRTbase) * direction * (mfp/mfdp) + Datetimebase.
(see Dale's handout for more details)
* direction = +1 for forward time ordered data, and -1 for reverse time ordered data
* mfc = minor frame counter
* mfp = major frame period
* mfdp = major frame downlink period
* GRTbase = GRT of minor frame zero of a selected reference major frame.
Datetimebase = major frame datetime of that same reference major frame

9.  If the UMFDATETIME column is empty, this means that there is no unpacked major frame datetime within any minor frame conseq of the current BFST.  The UMFDATETIME may be determined by other means.
9.1.  The UMFDATETIME may be determined per Rule 8 from the GRT is a Datetimebase is available from the GFST.  
9.2. The UMFDATETIME may be determined per Rule 7 from the UMFC if one is available.
9.3.  Should rules 7 or 8 fail, the UMFDATETIME may be determined from the session previous to the current one.  (The details of this subrule are still being worked on.  I welcome any suggestions)
9.4.  If all fail and the UMFDATETIME can't be determined, a message such as this one is sent to LabVIEW: Error XX: All minor frames failed to be identified.  In this instance, no reconstruction will occur and the "missing" major frames will the written to the AFST as "grade 2" and the database will move on to process the next pass of data.

10.  If the UMFDATETIME for a minor frame in the BFST could only be determined from the GRT (this means that the UMFDATETIME  and UMFC columns were empty in the BFST) we will test for the identity of the major frame by allowing a little slop (determined by looking at real data, could be abs(diff)<= slop).
11. If the UMFDATETIME for a minor frame in the BFST was determined from several alternatives (which right now are GRT (provided the slop criteria is tested) and UMFC and UMFDATETIME), the following rules are used to resolve the conflict, should there be any.
11.1.  If there is an UMFC in a contig that we have computed a datetime for using GRT, the UMFC should match that expected for this major frame.
EUMFC = expected UMFC = (Datetime - Datetimebase) / (major frame period + UMFCbase
Where Datetimebase and UMFCbase are from an adjacent  major frame with no mode change between the two frames.
11.2.  If there is no agreement between  computed datetimes, for now we just note the conflict, flag the minor frame with a "fail", there by rejecting this major frame.  This test for conflict may be a starting point for more processes later on.

A new set of rules is now applied to the minor frames in the BFST to attempt to estimate the unpacked  major frame counter.

12.  If UMFC are consistent within contigs in the BFST, their values are filled in for the minor frame entries in each contig of the BFST that does not already have one.

13.  If UMFC are inconsistent within contigs, two rules may be invoked.
13.1.  If there is a large gap between the ground receipt time of the first occurrence of the inconsistency, we know that is the beginning of a new contig boundary, even though we may have consistently changing minor frames.  The contig boundary then may be identified by a combination  of the UMNFC and the GRT of the current minor frame.
13.2.  If there is no gap between the ground receipt time of the first occurrence of the inconsistency, Rule 3 is invoked.

Recipe generation

Before the onset of this stage the following has already occurred:
* The Major frames to be reconstructed have already been clearly identified.
* All the minor frames have been filled: this means that unacceptable minor frames have been flagged with a "fail" and will not be used, all acceptable minor frames have an UMFDATETIME, an UMFC, and an UMNFC.
The following is not a list of rules but a list of sequential steps that are implemented in an orderly way.

1.  We first select all "grade 1" and "grade 2" table entries from the AFST (accumulated statistic table) table that are within the current pass' time span.  These selected major frames are added into a newly created WFST (working frame statistic table).  There will be a switch during actual implementation that will allow this step to be skipped should the analyst decide not to use information from previous passes during reconstruction.

2.  Next, all "missing" major frame entries from GFST are selected, provided that they are not present in the AFST (accumulated statistic table) with a "grade 0", and not present in the WFST with a "grade 1".  This selection is added to the WFST.

3.  The WFST now has only grade 1 and 2 major frame entries, as it is intended as the "to-do" list.  It should be noted that when "grade 1" major frames are recorded (meaning that at least one minor frame belonging to that major has been found), the information as to which minor frames have been found and in which file they are is also recorded.
3.1.  If there are no entries in the WFST, the reconstruction process is complete.  LabVIEW is advised of this situation and the next pass of data are processed.
3.2.  If there are entries, they will now be processed one by one into reconstruction recipes.  
3.2.1.  If an entry in the current WFST is a "grade" 2 entry, only the BFST belonging to the current pass need to be examined to see if it contains any of the minor frames for the "grade 2" major frame.  Since all previous pass' BFSTs have been through this process already, they do not need to be examined at this point in the processing.   
3.2.1.1.  If the current BFST does not contain any minor frames belonging to this major frame time, the major frame grade for all such minor frames stays "grade 2" and is later logged into the AFST.  
3.2.1.2.  If the current BFST does contain minor frames belonging to this major frame time, they are selected and sorted by UMNFC.  From this selection, a reconstruction recipe is built which contains information such as:
Input File A:  
get from file A minor frame 0 at bite 256
get from file A minor frame 1 at bite 512
fill minor frame 2
fill minor frame 3
...
get from file A minor frame xxx at byte yyy
The new reconstruction recipe is returned to LabVIEW and later logged into the AFST as a "grade 1" major frame.
3.2.2. If an entry in the current WFST is a "grade 1" entry, the same sort of recipe is built, but the "grade 1" entry's list of which minor frames were filled along with the file names where each minor frame was located are also used.  So instead of a single BFST, several can be referenced and used to build the reconstruction recipe.
3.2.2.1.  For all recipes, if anymore data are be available in the current recipe than in the past, the recipe is returned to LabVIEW and the updated grade 1 or grade 0 entry is completed and later logged into the AFST.
3.2.2.2.  For all recipes, if no data are be available in the current recipe than in the past, the recipe is not returned back to LabVIEW for implementation.  

4.  The updated "grade 1", "grade 0", or not updated "grade 2" major frame entries are now placed in the AFST.  The WFST, having outlived its usefulness, is now dropped.  Information about all the major frames that were initially in "grade 0" state in the GFST is also placed in the AFST.  The latter is done to guard against an unnecessary attempt at reconstructing a major frame already completely assembled.

The database is now ready to process the next pass of data.


