NovaStar Program Reference / Data Collection / nsrecdata

Overview
Command Line Usage
Data Format Specifications
Examples
Scheduling
NovaStar Administrator Interface
Troubleshooting
See Also

Overview

The nsrecdata program receives data reports from various sources and formats and files data reports in the database. This program is one of the more complex programs in the system. The following are technical considerations related to program functionality:

Multiple nsrecdata processes can be run, each with a different main data source option, source number (see -S), and other options, to ingest data streams.
nsrecdata can be run as a service that continuously checks for data on a serial port or IP address (see main options: portnamelabel, hostname:port, and :port), or can load data from a file and then exit (see main option: filename). Continuous data collection is configured through the NovaStar Administrator System Configuration table.
Each nsrecdata process handles a specific data format (see -G and other format/parsing/decoding options), with legacy ALERT being the default.
Data formats are impacted by communication systems. For example, data transmitted over Iridium satellite can be decoded by a data vendor and provided to NovaStar as custom text format (see -G #,#,#) or ALERT2 data can be embedded in Iridium data packets and decoded by nsrecdata (see -G iridium-alert2b).
Data formats can use text (ASCII) formats that are processed line by line, or binary formats that are processed as an encoded character stream, with appropriate error handling.
Data connections can push or pull data, consistent with the functionality of data collection hardware and network design (see -c).
Alternate programs can be used for data ingest if the functionality of nsrecdata is not appropriate, using a continuous service or scheduled program.

In many systems, redundant data feeds are configured so that the risk of data loss due to problems is reduced. Much of the complexity of nsrecdata is due to handling redundant data feeds, especially legacy ALERT data, which relies on the timestamp at the NovaStar base station to set the time on the data. Consequently, different travel times can result in different time stamps at the base station for the same observation. For ALERT2 protocol, timing issues from redundant sources can result if the clocks for communications hardware drift from actual. The nsrecdata program handles these issues and performs calculations to file data reports in the database, perform quality control, and analyze alarms for notification. A log file is also written to so that if the database connection is unavailable it is possible to reload data from the log file.

Other programs are available that can be run using a schedule to import data that do not need the continuous data collection features of nsrecdata.

Command Line Usage

The command line syntax for the main configurations is:

nsrecdata ...
nsrecdata portnamelabel [options...]
nsrecdata filename      [options...]
nsrecdata hostname:port [options...]
nsrecdata :port         [options...]

General rules for command line parameters (options) are:

Options can be specified with or without space, for example -Galert2 or -G alert2.
Single-character options cannot be chained. For example -d and -k cannot be specified as -dk.
The option is case-specific; however, the value is generally case-independent. For example, -GALERT2B and -Galert2b are equivalent. When in doubt, use the values in the documentation below.

nsrecdata Command Line Parameters

Parameter	Description	Default
	Main data source options
`portnamelabel`	Data collection port or device file name. `nsrecdata` will run until the process is stopped. Use in the following cases: Decode ALERT data stream from serial port attached to an ALERT decoder. Decode ALERT2 data stream from serial port attached to an ALERT2 decoder.	`/dev/rtdd1`
`filename`	File from which to read data, used for data load from a file. `nsrecdata` will exit after the file is processed. Use `-` (single dash) to read from standard input. Set the input file format with the `-G` parameter.
`hostname:port`	Connect to or listen (use `-c0`) on the network-resolvable hostname (or IP) address on the specified port number. `nsrecdata` will run until the process is stopped. See also `-c`.
`:port`	Listen on the specified network port number. `nsrecdata` will run until the process is stopped.
	Connection/network options
`-B baudrate`	Data collection port baud rate. See the Administrator *System / Serial Port List* page.	Read `serial_port.baud_rate` database value, or `300` if no data.
`-b`	Use the baud rate set in the *Serial Port List*.
`-c [0\|1\|2]`	Network connect type: `0` - listen (data are pushed to NovaStar) `1` - connect (data are pulled by NovaStar) `2` - transient, also referred to as "call-in" (used for cell modem communications where data are pulled by NovaStar and remote device drops the connection)	`1`
`-HR string`	Heartbeat string received, used to keep network connection alive. If a decoded data string matches the string, the string specified by `-HT` will be transmitted.
`-HT string`	Heartbeat string transmitted, used to keep network connection alive. See `-HR`.
`-HW #[smhd]`	Wait between heartbeat transmits for specified number of seconds, minutes, hours, or days.	`600s` (10 minutes)
`-k`	Enable network connection keep alive feature. Need to explain.	Keep alive is not active.
`-r #[smhd]`	Data collection down and baud rate reset timeout for specified number of seconds, minutes, hours, or days.	`1h`
`-W #[smhd]`	Wait to connect for specified number of seconds, minutes, hours, or days. Need to explain why.	`10s`
`-Z #[smhd]`	Wait before reconnect for specified number of seconds, minutes, hours, or days. Need to explain why.	`5s`
	Data format/parsing/decoding options (see table below)
`-EGG`	EG&G gages with even parity. See the Data Format Specifications table below.
`-EIF`	Enhanced IFLOWS data format. See the Data Format Specifications table below.
`-ENH`	Enhanced ALERT data format. See the Data Format Specifications table below.
`-F filename`	Enables filtered ID checking. Need to document file format and functionality.
`-G format`	Specify the format of input data, which is decoded to determine data to file in the NovaStat database: `#,#,#` - input field order for date-time(1), point(2), data(3). This is referred to as a "custom" format in NovaStar Administrator and will result in line-by-line data processing. Use this parameter for text-based data feeds. Data records should end in carriage return and/or newline. Data columns can be separated by space, tab, or comma. `alert2` or `ALERT2` - ALERT2 ASCII API parsing, equivalent to `-G alert2a` (`-G alert2a` should be used to avoid confusion) `alert2a` or `ALERT2A` - ALERT2 ASCII API parsing (equivalent to `alert2`) `alert2b` or `ALERT2B` - ALERT2 Binary API parsing `alert2bn` or `ALERT2BN` - ALERT2 Binary API parsing with translation to N,C,A message types `HL5096` - HydroLynx 5096 serial data parsing `iridium-alert2b` (or uppercase or mixed case) - Iridium packet surrounding ALERT2 binary Proposed `iridium-custom` (or uppercase or mixed case) - Iridium packet surrounding custom data format (use `-G #,#,#` to specify column order if different from the default) See the Data Format Specifications table below.	For serial and IP data feeds, the default is ALERT ASCII (there is no `-G alert` since that is the default). For custom format, the default input file format if `-G` is not specified is: `MM/DD/YYYY-HH:MM:SS ID: XXXX Data: XXXX` where `ID` is the point ID (need to document how other than point ID can be matched). The default allows labels in data lines for readability.
`-R filename`	Rule file name to verify ID numbers. See Rules File Format section below.
`-s #[smhd]`	Round time of reports received to the specified number of seconds, minutes, hours, or days.	No rounding.
	Data filing/notification options
`-a alarmTrigger`	Alarm trigger to execute when data collection is down for interval specified by `-r`. Use `-a 0` or `-a` to disable alarm checking when filing data, such as when restoring data from a file.	No alarm trigger for data collection down. Do check alarm triggers.
`-f`	Do not file data received, useful for testing data parsing without filing.	File data.
`-g`	File ALERT2 data from all message types: P,N,C,A not just N. Need to link to more information on ALERT2 message types.
`-P program`	Execute the specified program after a data value is filed. The program is run as command `nsh program pointId rawValue&` where: `nsh` - is the `nsh` program, used to ensure that the NovaStar environment is configured for the program `program` - is the name of the program (or full path) to run `point` is the point identifier `rawdata` - is the raw data value See also the `-w` parameter. Need to provide example.	No program executed after filing a value.
`-S #[,#]`	Set source number for data report flag. Unique integer numbers should be used for each `nsrecdata` data feed to allow troubleshooting data collection and processing. For ALERT2 the first source number is for self report protocol and the second source number is for concentrator protocol. Need to explain how shows up in data records.	`1`
`-w #[smhd]`	Wait time after data value is filed before `-P` program is executed. Specified the number of seconds, minutes, hours, or days.	`1s`
`-y #[smhd]`	Wait after data (parsing?) before filing for specified number of seconds, minutes, hours, or days.	`0s`
`-Y #[smhd]`	Wait before refiling if database locked for specified number of seconds, minutes, hours, or days. Need to explain why.	`1s`
`-z #secs`	Set interval that prevents filing of repeat reports. OBSOLETE! Use repeat report interval in the NovaStar configuration table.	`repeat_report_interval`
	Logging options
`-d [#]`	Set the logging level: `0` - logging disabled `1` - critical error messages only `2` - normal operation messages `3` - verbose output `4` - debugging output If a level is not specified, the current logging level will be incremented by 1.	`0`
`-l #`	Log file: `0` - `nssyslog` `1` - `nsrecdata.log` `2` - `nsautointer.log` Need to explain loading data from log for ALERT and ALERT2.	`1`
`-q`	Quiet - no messages. Same as `-v`.
`-v`	Do NOT display status messages for incoming data, applied when the option is detected. Same as `-q`.	Log messages for incoming data.
	Testing options
`--dumpconfig=outputFile`	Write the configuration to a file, useful for testing program setup. The output file can be: a valid filename `stdout` `stderr` The configuration is output just prior to data filing logic. If the input parameters result in an error opening the input data stream, exit will be postponed until the configuration is output.

Data Format Specifications

The following table summarizes data format specifications that are handled by this command. These formats are indicated by the -G and other command parameters.

Data Format Specifications

Data Format Specification	Description	Example Command Parameter
ALERT	The legacy ALERT specification defines data packets that adhere to the ALERT protocol. This specification has limitations such as relying on the base station for timestamp and data quality issues, which have been addressed by the ALERT2 specification. See the Data Formats / ALERT documentation.	No `-G` option is necessary since legacy ALERT is the default.
ALERT2	The ALERT2 specification defines data packets that adhere to the ALERT2 protocol. This specification overcomes limitations of the ALERT specification, in particular measurement time is included in data and data quality are improved due to use of Time Division Multiple Access (TDMA). See the Data Formats / ALERT2 documentation.	`-G alert2a` `-G alert2b` `-G alert2bn`
EG&G	EG&G gages (need to add documentation).	`-EGG`
EIF	Enhanced IFLOWS format (need to add documentation).	`-EIF`
ENH	Enhanced ALERT data format (need to add documentation).	`-ENH`
Iridium	Data from Iridium Satellite Communications via data integration vendors. See the Data Formats / Iridium documentation. Currently the default iridium specification is that provided by Kore with data payload using the ALERT2 binary format. Other vendor data specifications can be enabled as needed.	`-G iridium-alert2b` `-G iridium-custom` with `-G #,#,#` Proposed.
Others	Other formats will be listed here as documentation is implemented. Other data collection programs are also available to parse and file data from files.

Rules File Format

The following is the format used with rules file (-R parameter).

# Comment (blank lines are also skipped)
ID LowLimit HighLimit TimeLimit IDVerify,* IDChange

The following table describes the columns. Columns can be separated by space or tab. Need to complete this documentation.

Rules File Columns

Column	Description
`ID`	Point identifier for data point to file.
`LowLimit`
`HighLimit`
`TimeLimit`
`IDVerify,*`
`IDChange`

Examples

The following are examples of common use. Need more examples.

Example - Listen for ALERT2 Binary Data Reports on a Port

The following command configures data collection to listen for ALERT2 binary data reports, with the following options. This configuration is similar to that in many production systems.

nsrecdata 0.0.0.0:1234 -Galert2b -c0 -d1 -S1

0.0.0.0:1234 - IP address and port number to listen on
-Galert2b - indicate the data format being parsed
-c0 - indicates that the source will push data to nsrecdata
-S sets the source line to 1
-d1 sets the logging level to 1 (critical messages only)

Example - Load ALERT2 Binary Data Reports from a File

The following command processes a file named filename.bin that contains raw ALERT2 packets in binary form, for example to test parsing, with the following options:

nsrecdata filename.bin -Galert2b -S1 -d1

filename.bin - file to process
-S - sets the source line to 1
-d1 - sets the debug level to 1 (critical messages only)

Example - Listen for Iridium ALERT2 Binary Data Reports on a Port

This is a proposed configuration for San Bernardino.

The Iridium satellite and satellite modem hardware provide a way to transmit station data to NovaStar. This configuration uses the -G iridium-alert2b option to indicate that Iridium data packets contain ALERT2 binary data payload. Different Iridium data integration vendors may use different specifications for the data. The default that has been implemented follows the specification from Kore. Additional options will be implemented if other data vendors and specifications need to be supported.

The following command configures data collection.

nsrecdata 0.0.0.0:1234 -Giridium-alert2b -c0 -d1 -S1

0.0.0.0:1234 - IP address and port number to listen on
-Giridium-alert2b - indicate Iridium data packets surrounding ALERT2 binary data packets
-c0 - indicates that the source will push data to nsrecdata
-S sets the source line to 1
-d1 sets the logging level to 1 (critical messages only)

To configure this approach in the Administrator, use a Format of iridium-alert2b (new configuration choice).

Example - Listen for Iridium Custom Data Reports on a Port

This is a proposed possible configuration for Larimer County which is using Campbell data loggers and Iridium. Legacy ALERT format could be used but has the drawback of no timestamp and still requires additional nsrecdata coding to decode the packets.

The Iridium satellite and satellite modem hardware provide a way to transmit station data to NovaStar.

One option is that the Iridium data vendor strips the Iridium wrapper and pushes data to NovaStar similar to other custom data. For example, Micro Specialties provides data that has been communicated over Iridium but is pushed in simple format. In this case the -G #,#,# option can be used without any indication that Iridium was used.

Another option is to package data packets as a custom format such as text lines separated by a delimiter and embed within Iridium packet. Different Iridium data integration vendors may use different specifications for the data. For example, this approach can be taken with Kore data vendor.

The following command configures data collection.

nsrecdata 0.0.0.0:1234 -Giridium-custom -G3,1,2 -c0 -d1 -S1

0.0.0.0:1234 - IP address and port number to listen on
-Giridium-custom - indicate Iridium data packets surrounding custom data packets
-G3,1,2 - specify the column order for data to be filed (this option is unnecessary if the order matches the default)
-c0 - indicates that the source will push data to nsrecdata
-S sets the source line to 1
-d1 sets the logging level to 1 (critical messages only)

To configure this approach in the Administrator, use a Format of iridium-custom (new configuration choice). Administrator will need to be updated to show the column selections and format the command to use 2 -G options. The resulting options saved to the database will need to be parsed and properly displayed.

Scheduling

nsrecdata program processes are typically configured to run as a continuous process (service) for each main data collection data feed, using the NovaStar Administrator System / Configuration List.

The program can also be run from the command line as needed, for example to load data from a file.

NovaStar Administrator Interface

The NovaStar Administrator is used to configure an nsrecdata command line so that the system can run.

System / Configuration List

The NovaStar Administrator System / Configuration List page allows nsrecdata data collection processes to be configured by defining configuration property names starting with alert_port, for example as shown in the following image. Adding a new configuration for an nsrecdata process appends a sequential number, for example: alert_port2. Although the name alert_port is used by convention due to historical reasons, nsrecdata can be configured to ingest custom data that does not adhere to ALERT or ALERT2 protocol specification.

NovaStar Configuration List (see full-size image)

Configure ALERT2 Binary Feed on Serial Port

The nsrecdata command line for a continuous data collection process is configured in the Administrator using the Edit button on the right side of the table record to edit. Input fields are displayed for each command line parameter, with parameters (options) listed based on the Value and main controlling Options. For example, the following shows the interface when configuring an ALERT2 binary data feed (-G alert2b) from a serial port (/dev/rtdd1 corresponds to the serial port, which is command line portnamelabel). Note that some of the choices such as Format do not exactly match the spelling of command line documentation; however the command line options formed by the Administrator and saved to the Options for configuration list do agree with nsrecdata. Note also that an additional Options field is provided to enter options that are not specifically handled by the Administrator, such as new options.

NovaStar ALERT2 Binary Configuration (see full-size image)

Configure Custom Data Feed on Internet Port

The following shows the interface when configuring a custom text data feed from an internet port, which uses Format of CUSTOM to configure the -G #,#,# option. The -c option is used to indicate the whether nsrecdata pulls or pushes data through the connection. For example, this configuration can be used to connect to a cell modem or satellite data feed that pushes data to NovaStar using a simple column-based text data format.