Specifications > Interrogation > File Specification

M5 File

Email Comments

 

The M5 File format is the newest file format and was implemented for production data submission in September 2021. It is a JSON file format with header (called Interrogation File) and detail (called Interrogation Field Data) sections. The most current schema is 2022-04. Refer to www.json.org for information about JSON (JavaScript Object Notation) and to json-schema.org for information about schema used to validate JSON files. This free online validator can be used to manually validate M5 files against the schema. JSON is a standard data interchange format between systems and is human readable.

 

M5 and I5 software can be used create files of this type.

 

File Structure

The file contents are composed of two tables with a one-to-many relationship. The Interrogation File table has one or more Interrogation Field Data associated with it.

 

 

Interrogation File Fields

The fields of the Interrogation File provide a global context to the related field data records, such as the location (Site Code) of where the events occurred and how to process the data in PTAGIS (Data Role).

 

Field

Description

Domain

Schema

Identifier and location of the M5 schema that this file should be validated against.

URL identifying the location of the schema.json file

File Name

A unique file name associated with the dataset.

SIT-YYYY-DOY-R-SEQ.json

 

SIT: 3-character site code

YYYY: 4-digit year of the File Created date

DOY: Day of Year of the File Created date

R: Single character representing the Data Role (see table below for values)

SEQ: Left-padded 3-digit integer representing the sequence number

Site Code

Three-character site code registered with PTAGIS representing the interrogation site where the data was collected.

Valid interrogation site code registered with PTAGIS

File Created

Date and time denoting the start time of the monitoring period covered in this file.

RFC 3339/ISO8601 date format 

File Closed

Date and time denoting the end time of the monitoring period covered in this file.

RFC 3339/ISO8601 date format

Data Role

Designates the type of data within this file and where it will be validated, stored and/or loaded into the PTAGIS database.

Valid Data Role

System Description

Describes the system that created this file, including any version information.

Free text up to 100 characters.

 

 

Data Role

Data role describes the purpose of the data file and how PTAGIS should process it.

 

Validate: schema validation is performed to verify the file format

Load: contents of the a valid file will be processed into the database and available for reporting

Stored: physical file will be stored in the PTAGIS file archive in perpetuity

 

Role Value

Description

Validate

Load

Store

Primary (P)

Primary or production dataset available via PTAGIS.

X

X

X

Secondary (S)

Backup or secondary data from a redundant system.

X

 

X

Patch (X)

Patch data to supplement a production interrogation period.

X

X

X

Test Submission (T)

For testing submission only. Any data inside of file will not be processed.

 

 

X

NOTE: The Data Role is associated as a component of the file name in abbreviated form (e.g. P) and again within the Interrogation File structure in explicit form (e.g. Primary).

 

 

Interrogation Field Data

The fields of the Interrogation Field Data describe the detail of an observation of a PIT tag or other message from a transceiver or other equipment.

 

 

Field

Description

Domain

Field Data Type*^#

Defines the type of data message and related properties.

Valid Field Data Type

Timestamp*^#

The local data and time the message was recorded.

RFC 3339/ISO8601 date format 

Field Data*^#

A PIT tag observation, transceiver diagnostic, environmental or system metadata message recorded from a transceiver, data logger, PLC or other device. The message is recorded as it is transmitted and not parsed in any way.

Free text up to 500 characters.

Field Source*^#

Describes the field device or other type of data source for the Field Data.

Valid Field Source Type

System ID*^#

A unique identifier registered with PTAGIS that identifies a master controller, transceiver unit or probe.

Two-character hexadecimal ID

Subsystem ID

A unique identifier registered with PTAGIS that identifies an antenna or similar subsystem. This field should be specified if using the Tag Field Data Type and the data originates from a multiplexing transceiver where the transceiver ID is different from the antenna IDs.

Two-character hexadecimal ID

Tag Code^#

A real-time tag code to be processed as an observation record. This field can be used in place of Field Data if a PIT tag code has already been parsed from the raw transceiver message. If this field exists, the data in Field Data will be ignored

Valid hexadecimal PIT tag code

Signal Level

FDX-B tag detection signal level measurement in mV. This field can be used in addition to the Tag Code field when the PIT tag code and signal level have already been parsed from the raw transceiver message.

Number

Sbyc Action Type#

The type of SbyC diversion taken with with the detection of this tag code. Only present if the Field Data Type = SbycTag.

Valid Action Type

Sbyc Action Code#

The SbyC action code of the tag detected.Only present if the Field Data Type = SbycTag.

Integer

Sbyc Gate Action#

The SbyC gate action that was triggered with the detection of this tag code. Only present if the Field Data Type = SbycTag.

Valid Gate Action

Sbyc Processing Latency

The latency in milliseconds to complete the SbyC action from when the tag is decoded.Only present if the Field Data Type = SbycTag.

Number

* Required if Field Data Type = Raw

^ Required if Field Data Type = Tag

# Required if Field Data Type = SbycTag

 

 

 

 Field Data Type

 

Type

Description

Raw

Designates the message in Field Data is raw unparsed data.

Tag

Designates that the PIT tag code has already been parsed and is located in the Tag Code field. Data in Field Data will be ignored.

SbycTag

Designates the PIT tag code was parsed for SbyC operations and is located in the Tag Code field. SbyC fields will also be present.

Event

Records an event from a source other than a transceiver, such as a sample rate or facility gate status at a juvenile bypass facility. Available in schema 2022-04 and later.

 

If creating an M5 file from parsed tag detection records, the Field Data Type should be set to 'Tag' indicating a real-time tag code or to 'SbycTag' indicating the tag was parsed for the purpose of conducting Separation by Code. Additional SbyC metadata will be included in SbycTag records. Most of the properties of Interrogation Field Data are optional (nullable) except where the schema defines required properties based upon the value of Field Data Type. The following  examples are valid data and required fields are highlighted based upon the value of Field Data Type:

 

{

  "$schema": "https://ptagis.org/content/schemas/interrogation/2022-04/schema.json",

  "fileName": "B2J-2023-091-P-011.json",

  "siteCode": "B2J",

  "fileCreated": "2023-04-01T10:00:00.1082182-08:00",

  "fileClosed": "2023-04-01T11:00:00.0978084-08:00",

  "dataRole": "Primary",

  "systemDescription": "M5 Monitor Service 2.0.2",

  "interrogationFieldData": [

    {​​​​​​​​​​​​​

      "fieldDataType": "Raw",

      "timestamp": "2022-09-01T10:00:01-08:00",

      "fieldData": "TAG: 08 2020 - 08 - 13T03: 26:16.944 3E7.0000001D08 511 mV",

      "fieldSource": "Transceiver",

      "systemID": "08",

      "subsystemID": "08"

    }​​​​​​​​​​​​​,

    {​​​​​​​​​​​​​

      "fieldDataType": "Tag",

      "timestamp": "2022-09-01T10:00:01-08:00",

      "fieldData": "TAG: 08 2020 - 08 - 13T03: 26:16.944 3E7.0000001D08 511 mV",

      "fieldSource": "Transceiver",

      "systemID": "08",

      "subsystemID": "08",

      "tagCode": "3E7.0000001D08",

      "signalLevel": 511

    }​​​​​​​​​​​​​,

    {​​​​​​​​​​​​​

      "fieldDataType": "SbycTag",

      "timestamp": "2022-09-01T10:00:01-08:00",

      "fieldData": "TAG: 08 2020 - 08 - 13T03: 26:16.944 3E7.0000001D08 511 mV",

      "fieldSource": "Transceiver",

      "systemID": "08",

      "subsystemID": "08",

      "tagCode": "3E7.0000001D08",

      "signalLevel": 511,

      "sbycActionType": "DefaultAll",

      "sbycActionCode": 100,

      "sbycGateActionType": "NoOp",

      "sbycProcessingLatency": 100

    }​​​​​​​​​​​​​

    {

      "fieldDataType": "Event",

      "timestamp": "2023-04-01T10:00:00.1105875-08:00",

      "fieldData": "Sample Gate Open Status: Started",

      "fieldSource": "Plc"

    },

  ]

}​​​​​​​​​​​​​

 

Because the schema only validates which properties are required based on Field Data Type value, the following example is also valid and the server would process this as raw data and ignore the value in Tag Code. Also note the required System ID field has a value of "GG" which the schema returns as valid even though it's an invalid transceiver identifier; this type of validation is handled when the file is processed for loading into the database.

{​​​​​​​​​​​​​

      "fieldDataType": "Raw",

      "timestamp": "2020-01-01T05:00:02-07:00",

      "fieldData": "TAG: 08 2020-08-13T03: 26:16.944 3E7.0000001D08 511 mV",

      "fieldSource": "Transceiver",

      "systemID": "GG",

      "subsystemID": "08",

      "tagCode": "3E7.0000001D08",

      "signalLevel": 511

    }​​​​​​​​​​​​​

 

 

 Field Source Type

 

Type

Description

Transceiver

Designates the message in Field Data originated from a transceiver and should be processed by PTAGIS. For example, a real-time tag detection or diagnostic report.

Plc

A message from a Programmable Logic Controller (PLC) typically used at mainstem hydro facilities. Messages of this type include metadata such as facility gate disposition or temperature/humidity level monitoring.

Probe

Output from a sensor probe to measure water depth, temperature or other environmental metadata at an interrogation site.

System

A system related message to indicate an operational behavior or error condition. Typically, these messages are associated with a real-time software system such as M5.

Extraneous

Typically identified by a formatting utility (e.g. I5) as extraneous and PTAGIS will ignore during processing. For example, the listing of available commands from a transceiver that was included in a memory download.

 

SbyC Action Type

 

Type

Description

None

No action was triggered.

DefaultIgnore

Default action triggered because tag code is a timer or test tag.

DefaultNotFound

Action code associated with the tag code was not found in lookup table, so the default action was triggered.

DefaultAll

Divert by default, all tag codes trigger the same action.

Override

The override gate action was triggered based on the action code assigned to the tag code.

DefaultSkip

The default action was triggered because the override protocol instructed a skip (e.g. daily or seasonal limit has been reached or the protocol time period has been exceeded).

DefaultNA

The default gate action was triggered because the action code assigned to the tag code is not applicable at this diversion gate.

Ignore

No gate action was triggered because the tag is a timer tag.

Duplicate

No gate action was triggered because the tag code is a duplicate.

StandBy

No gate action was triggered because the SbyC service is in a stand-by state.

Failed

Indicates a divert request failed to trigger a PLC bit.

 

SbyC Gate Action Type

 

Type

Description

NoOp

Indicates the gate should take no action. This is often used to override a gate that opens by default for any PIT tag.

Open

Indicates the gate should open.

Close

Indicates the gate should close.

Left

Indicates the gate should open to the left side.

Right

Indicates the gate should open to the right side.

 

 

 

 

 

 


Version: 1.07

Published: 9/6/2023