Skip to content

NovaStar Program Reference / Data Import / nsrecdata-log-buffer-refile

Overview

The nsrecdata-log-buffer-refile program refiles data that are buffered in the nsrecdata log file but have not been inserted in the database. Although this situation may be mitigated, it may always be an issue due to multiple processes attempting to insert data and the need to interleave these processes so that real-time data can be inserted as quickly as possible. Buffering may be due to:

  1. The system has a high load and the database tables may be locked for inserts, in which case the nsrecdata program may quit retrying to insert buffered data (see the nsrecdata --bufferretry and --retryinterval command parameters, which control buffering).
  2. The system may have been stopped or restarted and buffered data in the log file need to be processed.

This program searches the current nsrecdata log file and refiles reports that contain Action: buffer. See also the --filterfile command line parameter, which allows additional filtering. The program contains logic to ensure that only new buffered records are inserted rather than refiling all buffered records in the log file.

This program is typically run as a scheduled process to ensure that buffered data reports are filed. Other actions, such as increasing server processing power and memory, can be implemented to improve performance.

The following files are involved and are mentioned below without the leading path:

  • Input:
    • /var/log/novastar5/nsrecdata.log - nsrecdata log file, which contains log messages for all nsrecdata processes, restarted each day by the logrotate service
  • Intermediate:
    • /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{date} - lines from the nsrecdata log file containing Action: buffer, indicating lines that have already been processed from the date's nsrecdata.log file
    • or, /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{date}-nofile - lines from the nsrecdata log file containing Action: buffer, if --nofile is specified, used for testing and troubleshooting
    • /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{date}-{filter file name}-filtered-out - lines from the nsrecdata log file that contain both Action: buffer and one of the patterns given in the file specified by --filterfile
    • /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{date}-new - lines from the nsrecdata log file containing Action: buffer found in the current run of this script. /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{date} gets the contents of this file if the data is inserted successfully. Deleted afterwards unless the --keep option is given.
  • Buffered data to refile:
    • /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{date}-toinsert - buffered data formatted for loading. This is lines that have been added to the nsrecdata log file since the last time the buffer refiling script ran, edited to a format that nsdataimport can accept (report time,point numid,datavalue) Deleted after the script runs unless the --keep option is given.

The program only processes new buffered data reports, which minimizes data refiling and associated processing load. The general process is:

  1. Check if the nsrecdata-log-buffer-refile script is already running:
    1. Check if there are any processes with the same name, and if so, print an error message and exit.
  2. Load yesterday's data, if applicable:
    1. Check whether the nsrecdata-log-buffer-refile{suffix}-{date} file exists for today's date.
    2. If it does, skip forward to step 3.
    3. If it does not, pull all lines with Action: buffer from nsrecdata.log.1 into /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{yesterday's date}-new.
    4. Check whether /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{yesterday's date} exists.
    5. If not, skip to step 2.7.
    6. If yes, determine how many lines are in the file, and remove that many lines from the beginning of /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{yesterday's date}-new.
    7. If --filterfile was specified, run grep on /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{yesterday's date}-new using the filters in the provided filter file, and put the results in /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{yesterday's date}-{filter file name}-filtered-out.
    8. The file (-new if no --filterfile, -filtered-out if there is) is processed to get the data to file in the format report time,point numid,datavalue. This is placed into /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{yesterday's date}-toinsert.
    9. The -toinsert file is passed to nsdataimport, which is run using timeout with a timeout of 3600 seconds (1 hour).
    10. Delete the -toinsert file unless --keep was specified.
    11. If nsdataimport fails, delete the -new file. This means next time the script runs, it will not check anything from this run. Then, exit the script, skipping all future steps.
    12. If nsdataimport succeeds, move the -new file over the main file, unless --keep was specified, in which case copy it.
  3. Load today's data:
    1. Pull all lines with Action: buffer from nsrecdata.log into /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{today's date}-new.
    2. Check whether /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{today's date} exists.
    3. If not, skip to step 3.5.
    4. If yes, determine how many lines are in the file, and remove that many lines from the beginning of /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{today's date}-new.
    5. If --filterfile was specified, run grep on /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{today's date}-new using the filters in the provided filter file, and put the results in /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{today's date}-{filter file name}-filtered-out.
    6. The file (-new if no --filterfile, -filtered-out if there is) is processed to get the data to file in the format report time,point numid,datavalue. This is placed into /var/log/novastar5/nsrecdata-log-buffer-refile{suffix}-{today's date}-toinsert.
    7. The -toinsert file is passed to nsdataimport, which is run using timeout with a timeout of 3600 seconds (1 hour).
    8. Delete the -toinsert file unless --keep was specified.
    9. If nsdataimport fails, delete the -new file. This means next time the script runs, it will not check anything from this run. Then, exit the script, skipping all future steps.
    10. If nsdataimport succeeds, move the -new file over the main file, unless --keep was specified, in which case copy it.
  4. Delete old files:
    1. Unless --keep was specified, get all files that match the syntax /var/log/novastar5/nsrecdata-log-buffer-refile*.
    2. Iterate through these files and delete them until the file that matches either the file for yesterday or the file for today (with or without the suffix) is reached, then exit.

Command Line Usage

The command line syntax is as follows. Optional values are in square brackets.

nsrecdata-log-buffer-refile [options...]

nsrecdata-log-buffer-refile Command Line Parameters

Parameter                                     Description Default
--debug Print additional debug messages to stderr. Only standard messages are printed.
--filterfile=Filename Specify a filename with strings to match, useful when only processing certain source addresses or line numbers. This file is passed to grep -f to match the buffered data reports, in addition to matching Action: buffer. Each line is taken as a separate filter, and a line is returned if it matches any filter. For consistency, it is best if the file location given is absolute. By convention, the filter files should be stored in /usr/ns/cus/nsrecdata-log-buffer-refile/. All buffer lines are processed.
-h, --help Display the program usage.
--keep Do not delete temporary or old files. Files created prior to yesterday and intermediate processing files will be deleted.
--nofile Print matching log file lines to /var/log/novastar5/nsrecdata-log-buffer-refile-dailybufferdata-nofile.log but do not file the data. Use the --debug parameter to print log file locations. This can be used to test the program and if a separate data filing step will be used. File the data.
--suffix=Suffix A string to append to temporary files, so that separate processes can be configured, for example /var/log/novastar5/nsrecdsata-log-buffer-refile-suffix.... Note that the dash must be included in the string given. Recommended when using --filterfile so that files with different inclusion criteria are not compared. Temporary files use the script name only without a suffix.
-v, --version Display the program version.

Examples

The nsrecdata log file may contain lines similar to the following:

Jan 10 06:25:10 test-novastar nsrecdata[26609]: Line: 31 SA: 11009 Time: 01/10/2020 06:25:07 ID: 10000980 Data: 0 Scaled: 0.34 Action: buffer
Jan 10 06:25:11 test-novastar nsrecdata[26609]: Line: 31 SA: 11010 Time: 01/10/2020 06:25:08 ID: 10000981 Data: 0 Scaled: 0.34 Action: buffer

Action: buffer indicates buffered data that could not be inserted.

To do a dry run of the refiling, run the following command:

nsrecdata-log-buffer-refile --nofile

To file a subset of data that matches a given set of filters, run the following command:

nsrecdata-log-buffer-refile --filterfile=/usr/ns/cus/nsrecdata-log-buffer-refile/nsrecdata-log-buffer-refile-filter-file.txt

The filter file in the above example indicates text that must be present in the line to be refiled. To filter for only lines where the first source address is either 11009 or 11010, the filter file could contain the following:

SA: 11009
SA: 11010

To refile all the buffered data without filtering, run the following command:

nsrecdata-log-buffer-refile

Scheduling

This program is typically configured as a NovaStar scheduled process to ensure that buffered data are loaded as quickly as possible.

Because the intent is to load real-time data, the program should be run on an interval that matches the ALERT2 TDMA frame length or transmit frequency for other data, such as Iridium satellite data.

The scheduled process command line can be similar to the examples shown in the previous section. If additional output is desired for troubleshooting, create a script like the following, for example named: /usr/ns/cus/nsrecdata-log-buffer-refile/run-nsrecdata-log-buffer-refile-filter-file.sh.

#!/bin/sh
#
# Run the nsrecdata-log-buffer-refile program and redirect stdout and stderr to
# a log file that can be reviewed for troubleshooting.

# Get the location of this script.
scriptFolder=$(cd $(dirname "$0") && pwd)
scriptName=$(basename $0)

# Use a log file for troubleshooting:
# - use a location in /var/log since /usr/ns/cus is often backed up and want to keep small
logfile="/var/log/novastar5/cron-nsrecdata-log-buffer-refile.log"
now=$(date)

# Run the program:
# - redirect stdout and stderr to the log file
# - any output redirection in the script itself will go to the desired files
echo "Running ${scriptFolder}/${scriptName} at ${now}" > ${logfile}
nsrecdata-log-buffer-refile >> ${logfile} 2>&1

Then run the above script from the scheduler.

NovaStar Administrator Interface

See the Administrator scheduler.

Troubleshooting

Use the --debug and --nofile parameters to troubleshoot and review the temporary files used to process the buffered log data.

Review the log file created when running the scheduled process, if a log file is created.

See Also