The PTAGIS database consists of relational and dimensional database systems that extract, transform, load (ETL) and collate MRR and interrogation data submitted via field data collection systems. Data contributors submit MRR and interrogation data in formatted text files (raw data files), which the automated ETL processes validate against the file and data specifications. All submitted files are checked against file format and data validation rules before being loaded into the database. Data contributors are notified of the processing result via automated email notifications and can also view the status from within the software used to submit files and through the online query system.  All raw files are archived after ETL processes are complete and the data contained within those files is made available in near real time to query via online tools.

 

MRR and interrogation files have slightly different ETL processes, but each process follows these general steps:

 

1.Check file format against the appropriate file specification

2.Check values against the current data specification

3.Check values against existing records and configuration settings in the database

 

If all validation checks are passed, all records from a file will be loaded into the database. If any validation checks fail, none of the records from the file are loaded into the database and the data contributor is alerted.

Version: 1.07

Published: 9/6/2023