NovaStar / Core Data Filing Process / Validation

Introduction
Process

Introduction

The validation step checks scaled values for validity and assigns the following data flags:

V for valid data
Q for questionable data
B for a break

Validation is run automatically during data filing (such as from nsrecdata and sendrpt) and can be run manually if database values are manually edited.

Validation is also run when nsdbrecalibrate and nsdbvalidate are run.

Validation is not run when data is edited in the legacy Administrator, but is run when data is edited in the new Administrator, as nsdbrecalibrate is run at that time.

Process

The process of validation has several different substeps, which are described in the following sections.

TODO all of these would greatly benefit from diagrams, especially showing the intervals. Maybe do a plot Administrator, put in PowerPoint and edit, or just show a simple case in PowerPoint.

Can a statement be made that a data value is considered valid unless one of the following conditions is met?

Finding Data Reports that are Close in Time

Validation looks at data reports that are close in time to determine whether a data report is valid. In some cases, data reports that have already been previously validated may have their validation status changed by a new report coming in.

To determine what reports are close in time, the following process is used:

The check interval is determined via the following steps:
1. The point for the data report to examine is checked for an interval field.
  1. If there is an interval, this is taken as the value to use in future steps.
  2. Otherwise, 86400 seconds (24 hours) is taken as the value to use in future steps. This is a historical default appropriate for legacy ALERT, where a daily regular report is common.
2. The point is checked for a change_interval field. If that value is greater than the value from the previous step, it is used in future steps. Otherwise, the value from the previous step remains.
3. The point is checked for a compress_interval field. If that value is greater than the value from the previous step, it is used in future steps. Otherwise, the value is unchanged.
4. If the data report is to be filed (true most of the time, false if -f is given to nsrecdata) and alarms should be checked (true most of the time, false if -a is given to nsrecdata, sendrpt, starpt, or nsdataimportrpt), then get all alarm triggers for the point.
  1. Each alarm trigger is checked in the following ways:
    1. If the alarm trigger type is not one of rise, fall, or report count, skip to the next trigger.
    2. Otherwise, check if the value from previous steps is less than the alarm trigger limit_interval plus the value from the first step. If so, use the limit_interval plus the value from the first step in future steps.
    3. If the alarm trigger reset_interval is null, skip to the next trigger
    4. Check if the value from previous steps is less than the alarm trigger reset_interval plus the value from the first step. If so, use the reset_interval plus the value from the first step in future steps.
5. Check if the repeat_report_interval is set in the global configuration table. If not, use 2 seconds as the repeat report interval.
6. Check if the repeat report interval is larger than the value from the earlier steps. If so, use the repeat report interval instead.
Subtract the calculated check interval from the time of the report to be validated. This is the start time.
Add the calculated check interval to the time of the report to be validated. This is the end time.
Get all data reports for the point between the start and end time, ordered in ascending order of report time (oldest to newest).

The result of this is the report list, which is used in the checks described below.

Zero Limit

For some calculations, values are checked not against zero itself, but the zero limit, which is calculated as follows:

Get the scaled digits from the point type for the point.
Divide 5 by 10 to the power of the scaled digits plus one. This is the zero limit.

For example, if the scaled digits are 0, the zero limit would be 0.5 - 5/10.

This means that values checked against the zero limit work somewhat as if they are rounded - with the example above, with 0 scaled digits, scaled values would be displayed with no decimals. Hence, anything less than 0.5, or anything that would round to 0, gets treated as if it were zero.

Bounds Checking

Bounds checks compare a scaled value against the limits defined for a point. For example, water level limits may be defined to match physical limits and precipitation limits may be defined based on a sensor's maximum accumulated value.

Check that the point for the data report exists. If not, set an indicator that this report is out of bounds and skip to the final step.
Check that the raw data for the report is a number. If not, set an indicator that this report is out of bounds and skip to the final step.
If the point class of the point of the report to be validated is status, loop through each point status.
1. If any point status has a use of fixed on with a corresponding data bit of 0, set an indicator that this report is out of bounds and skip to the final step.
2. If any point status has a use of fixed off with a corresponding data bit of 1, set an indicator that this report is out of bounds and skip to the final step.
If the point class of the point is counter, encoder, wind, or alertwind:
1. If the point class is alertwind, shift the data so that it is just wind run rather than wind run and wind direction combined.
2. If the point type cycle size (the value at which a counter or encoder rolls over) is >= the zero limit, check if the raw data is negative or >= the cycle size. If either is true, set an indicator that this report is out of bounds and skip to the final step.
Check that the scaled data for the report is a number. This protects against incorrect edits to the database itself. If not, set an indicator that this report is out of bounds and skip to the final step.
If either the point type upper limit is greater than 0 or lower limit is greater than the zero limit, if the scaled data is greater than the upper limit or less than the lower limit, set an indicator that this report is out of bounds and skip to the final step.
If the report was noted to be out of bounds in any previous step, regardless of if the raw or scaled value was out of bounds, check if the report is marked as verified (E) or maintenance (M). If not, remove any valid (V) flag on the report and add a questioned (Q) flag.

Main Process

Note that for the most part, flags are only added or removed at the end of the process. This is because the logic treats newly valid or questioned reports differently than reports that were questioned or valid before this validation process was run.

The data report is checked for the questioned (Q) or maintenance (M) flag. If it has either one, no further processing is done. All other flags are ignored at this step, but are relevant later on. (Note that if being run via nsdbvalidate, any questioned flags are removed before reaching this step.)
Run the bounds check. If the bounds check marks the report as questioned:
1. If the report does not have the verified (E) or maintenance (M) flags, remove the valid (V) flag and add the questioned (Q) flag.
2. Exit the validation process.
Set the allowed change value to the change value for the point.
Check if the allowed change value is greater than the zero limit:
1. Loop through the nearest reports before the report being validated from the retrieved report list, stopping either when there are no more retrieved reports or after 10 reports. This loops backwards through time - each report is older than the last. If out of reports and the allowed change value is less than the zero limit, set the change status to true. On each loop, do the following:
2. If the amount of time between the report selected for this loop iteration and the report after it is larger than the interval field of the point:
  1. If this is the first time through the loop, set an indicator that the report to be validated should have a break (B) flag added.
  2. For all other times through the loop, set an indicator that the older report should have a break (B) flag added.
3. If the report selected for this loop iteration is marked as maintenance (M) or is marked as questioned by the bounds check, go immediately to the next loop iteration.
4. If the point class of the point is status, which is a set of binary values encoded in the raw data:
  1. If the absolute value of the difference between the scaled values of the report to be validated and the report selected for this iteration of the loop is less than the zero limit, increment the valid report counter. If that counter is equal to the rounded value of the change allowed by the point, set an indicator that older report should have a valid (V) flag added, set the change status to true, and exit the loop.
  2. If the absolute value of the difference between the scaled values of the current and previous reports is greater than or equal to the zero limit:
    1. If the previous report has a valid (V) flag, then increment the not valid report counter. If this counter is equal to the rounded value of the change allowed by the point, exit the loop.
    2. Otherwise, if the previous report does not have a questioned (Q) flag, set an indicator that the previous report should have a valid (V) flag added, set an indicator that the previous report should be updated, and clear any indicators that the previous report should have a break flag added.
5. For all other point classes:
  1. If the allowed change value is less than the zero limit, this means that there are no limits on the allowed change, and no additional validation needs to be done in this loop. Set the change status to control future validation, and exit the loop.
  2. If the absolute value of the difference between the scaled values of the current and previous reports is less than or equal to the allowed change value (and, if the point class is counter, the change is positive), this is likely valid, pending further checks.
    1. Set an indicator that the older report should have a valid (V) flag added.
    2. Set an indicator that the older report should be updated, and remove the indicator that the older report should have a break flag added.
    3. Set the change status to true.
    4. Increment the valid report counter, and save the position of this report as the last known valid report.
    5. Mark this report to be updated, and clear any indicators to add a break flag to this report.
    6. If this is the second or later iteration of the loop and a report from a previous loop was valid, calculate the difference between the scaled value of the last known valid report and the scaled value of the report for this iteration of the loop. If this is greater than the zero limit, add the point change value to the allowed change value. Maybe say somewhere that the change in scaled values is computed for the cumulative change in values for the change interval and values can go up and down in the interval.
    7. Otherwise, calculate the difference between the scaled value of the report for this iteration of the loop and the scaled value for the report to be validated. If this is greater than the zero limit, add the point change value to the allowed change value. This means the allowed change value will scale appropriately with the number of intervals.
  3. Otherwise, since the change is larger than the allowed value, check other possible reasons the report could be valid:
    1. If the report selected for this iteration of the loop has the valid (V) flag:
      1. If this is the second or later iteration of the loop and a report from a previous loop was valid, calculate the difference between the scaled value of the last known valid report and the scaled value of the report for this iteration of the loop. If this is greater than the zero limit, add the point change value to the allowed change value.
      2. Otherwise, calculate the difference between the scaled value of the report for this iteration of the loop and the scaled value for the report to be validated. If this is greater than the zero limit, add the point change value to the allowed change value.
    2. Set an indicator that the report selected for this iteration of the loop should have the valid (V) flag removed.
    3. Increment the not valid report counter, used later to determine where breaks should be inserted.
    4. If this is not the first time through the loop and the report selected on the last report was marked to have the valid flag added:
      1. If the difference between the scaled value of the report selected the last time through the loop and the scaled value of the report selected this time through the loop is larger than the allowed change (or, if the point class is counter, the change is negative) and the report selected the last time through the loop does not have the valid flag, set an indicator that the report selected the last time through the loop should have the break (B) flag added.
  4. If the report selected for this iteration of the loop already has the valid (V) flag:
    1. If the valid report counter is greater than or equal to 3, exit the loop.
    2. Otherwise, if the not valid report counter is greater than or equal to the valid report counter plus 3, exit the loop.
Otherwise if the allowed change value is less than or equal to the zero limit, set the change status to true.
If the change for the point is greater than the zero limit:
1. If the position of the last valid report was saved (see step 4.5.2.3), the last valid report was not the last report checked in the loop, the valid report counter is greater than 0, and the not valid report counter is greater than the valid report counter:
  1. set an indicator that the last valid report should have the break (B) flag added.
  2. Iterate through the reports between the report being validated and the last valid report. If said intermediate report does not have the verified (E) or maintenance (M) flag, or not had an indicator to add a valid flag, mark the report to be updated and remove any indicators to have the break flag added.
2. If the valid report counter is greater than or equal to the not valid report counter and the not valid report counter is less than 3:
  1. Loop backwards through all reports checked in the loop in step 4.1.
    1. If the report has the valid (V) flag or has an indicator to add a valid flag, skip to the previous report.
    2. If the report has the maintenance (M) flag, skip to the previous report.
    3. If the report fails the bounds check, skip to the previous report.
    4. Set the allowed change to the change value for the point.
    5. Loop through all reports older than the report in the 6.2.1 loop:
      1. If an older report does not have a note to add a valid flag, continue to the next iteration of the loop.
      2. Set the next oldest report as the before report.
      3. Set the next newest report that has a note to add a valid flag from the one in the 6.2.1 loop as the after report.
      4. If the loop report scaled value minus the before report scaled value is larger than the allowed change (or, if the point class is counter, the difference is negative), remove any notes on the loop report saying to add the valid flag.
      5. Otherwise:
        
        Calculate the difference between the after report scaled value and the loop report scaled value.
        
        If the point class is not counter, or the point class is counter and the difference in the above step is positive, add a note to the loop report to add the valid flag.
        
        Otherwise, if the loop report already has a note to add the valid flag, remove any notes to add the valid flag to the after report.
        
        If neither of the above conditions, remove any notes to add the valid flag to the loop report.
      6. If the change calculated in step 4 above is not equal to zero, add the point change value to the allowed change.
    6. If the loop report does not have a note to add a break flag:
      1. Mark the report to be updated.
      2. If the loop report has a note to add the valid flag and it is the newest report retrieved overall, add a note to add the break flag.
If the change status is true, the change limit of the point is greater than the zero limit, the point change interval is not equal to 0, and reports were found when getting the list of nearby reports:
1. Initialize the rate to 0.
2. Set the last data to the scaled data from the report to be validated.
3. Set the last break status to whether the report to be validated has the break (B) flag.
4. Loop through the reports that are older than the report to be validated:
  1. If the report does not have the valid (V) flag and does not have a note to add the valid flag, continue to the next iteration of the loop.
  2. If the last break status was false, add to the rate the difference between the last data and the loop report scaled data.
  3. Set the last data to the loop report scaled data.
  4. Set the last break status to whether the loop report has the break (B) flag.
  5. If the loop report's time is older than the report to be validated's time minus the point check interval, exit the loop.
5. If the absolute value of the rate is larger than the absolute value of the point change limit, set the change status to false, and if the report to be validated does not have the maintenance (M) or verified (E) flag, remove the valid (V) flag and add the questioned (Q) flag.
If the change status is true and any report was processed in the 4.1 loop, loop through the reports processed in that loop:
1. If the loop report has been marked to be updated and it does not have the maintenance (M) or verified (E) flags:
  1. If the loop report has been marked to have the valid flag added:
    1. If the loop report does not have the valid flag or has been marked to have the break flag added:
      1. Remove the questioned (Q) flag from the loop report and add the valid (V) flag to the loop report.
      2. If the loop report has been marked to have the break flag added, add it here.
      3. If the report should be filed, update the report in the database.
      4. Set the flag on the loop report for it to be recalibrated.
      5. If the loop report has the break (B) flag, exit the loop.
  2. Otherwise if the loop report does not have the questioned (Q) flag:
    1. Remove the valid (V) flag from the loop report.
    2. Add the questioned (Q) flag to the loop report.
    3. If the report should be filed, update the report in the database.
If the change status is true:
1. If the report to be validated does not have the maintenance (M) or verified (E) flags:
  1. Add the valid (V) flag to the report to be validated.
  2. If 4.2.1 was hit, add the break (B) flag to the report to be validated.
Otherwise if at least one report was processed in the 4.1 loop:
1. If the report to be validated does not have the maintenance (M) or verified (E) flags:
  1. Remove the valid (V) flag from the report to be validated.
  2. Add the questioned (Q) flag to the report to be validated.