EVENT LOG FOR SITE CFF The first line of each entry contains a date/time stamp & who is reporting. The most recent entry is listed first. Date Time Who ---------- ----- -------- 12/31/2012 08:17 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 12/31/2012 08:17 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 12/31/2012 08:17 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 12/31/2012 08:17 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 12/31/2012 08:14 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 12/31/2012 08:14 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 12/31/2012 08:14 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF from Network 2 12/31/2012 08:14 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 12/17/2012 09:03 Alan YKFP personnel report that a power outage occurred from ~15:10 PST 12-15- 2012 to ~12:50 PST 12-16-2012. Examination of .tmp files on both PCs yields no interrogation records. No further data exist on either PC to fill the data gap. 12/16/2012 12:53 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 12/16/2012 12:53 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 12/16/2012 12:52 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 12/16/2012 12:52 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 12/15/2012 15:17 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 12/15/2012 15:17 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 12/15/2012 15:16 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF from Network 2 12/15/2012 15:16 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/29/2012 07:56 Scott_Li It appears that the facility is once again without power. Jerod and Mike both called this morning asking if we detected a power outage. As of yet , we have not received any e-mails from the facility indicating the power was lost. Jerod will call when status of the facility is confirmed. 11/27/2012 08:52 Alan A power outage occurred on 11-19-2012 from ~0334 PST to 1328 PST. No data exist on either PC to fill the remaining data gap. 11/27/2012 08:44 Alan A power outage occurred from ~2200 PST 10-19-2012 to 0637 PST 10-22-2012. No data exist on either PC to fill the remaining data gap. 11/21/2012 07:08 scottl@b 01:NoSync,01:RdrToMaster,02:NoSync,02:RdrToMaster,02:VTTFailed: Alarms caused during maintenance on the transceivers 11/20/2012 14:27 Troy PLC programming has concluded. Several AC Power ON/OFF messages were generated during the programming process. Please ignore. Persistent Email logic was added in that the PLC will try to email until it is successful. Also, redundant email via Network 2 was added so that we have the best chance of catching a power failure. Outlook rules will have to be modified to accommodate Network 2 messages. 11/20/2012 14:24 Scott_Li Reverted PC2 back to Minimon ver 1.7.0. the reasoning for this was that this site will be physically inaccessable in the near future and software upgrades via the Sat. connection are a gamble. Unchecked the M4 site monitoring service using the new tool provided by the Portland team. Transceivers are well tuned and no problems to report. YIN personell report that the generators are still having issues with the fuel pressure and there will more than likely be future power ouitages. Contractor is still working out the kinks and still have the responsibility for the operation of the facility. 11/20/2012 14:19 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 14:19 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 14:17 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 14:16 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 14:12 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 14:12 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF from Network 2 11/20/2012 14:12 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 14:11 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/20/2012 14:10 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/20/2012 14:04 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 14:04 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 13:59 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 13:59 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/20/2012 13:58 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 13:58 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 13:56 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:56 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:43 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 13:43 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:43 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 13:43 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:42 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:42 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:41 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:41 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:40 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:40 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:37 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF from Network 2 11/20/2012 13:37 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/20/2012 13:23 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:23 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 13:22 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:21 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:20 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:19 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:19 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:18 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:17 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:16 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:15 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:14 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:13 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:12 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:12 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 13:12 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 13:12 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:12 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:05 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/20/2012 13:05 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 13:04 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF from Network 2 11/20/2012 13:04 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/20/2012 13:00 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 13:00 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 13:00 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 13:00 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 12:54 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/20/2012 12:54 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 12:54 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 12:54 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/20/2012 12:47 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 12:47 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 12:45 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 12:45 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/20/2012 12:41 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON from Network 2 11/20/2012 12:41 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 12:41 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 12:41 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 12:38 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/20/2012 12:38 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF from Network 2. 11/20/2012 12:04 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON from Network 2 11/20/2012 11:49 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF from network 2 11/20/2012 11:49 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 11:16 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/20/2012 11:10 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 11:09 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 11:09 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 11:02 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/20/2012 10:33 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/20/2012 10:33 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/19/2012 14:26 troy@psm The PLC reports AC Power on CKT1 is ON 11/19/2012 13:30 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/18/2012 08:23 Alan Examination of both PCs shows no data other than approximately 45 minutes of diagnostics and time stamps. No further data remain to fill the gap on 11-17-2012. 11/17/2012 15:07 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/17/2012 07:20 Alan A power outage was reported this morning at 03:12. Xymon shows both networks as unavailable since around the same time. I will attempt to contact the YKFP for more information. More info will be posted when it is available. 11/17/2012 03:12 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/17/2012 03:12 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/13/2012 12:47 Scott_Li Replaced PC2 due to suspected failing power supply. Reloaded M4 Ver. 1.4.6 and configured all settings. All test uploads, devices and PCA session was verified operational. Retrieved system events for John T review. M4 platform came back online at 12:15:24. Files prior to this were generated during the acid test to ensure proper operation. Possibly the last trip up here before the place is snowed in. 11/06/2012 08:27 Alan File CFF12310.D9 was created from CFF12310.D2.TMP to fill a gap in data from yesterday's maintenance. 11/06/2012 07:23 Scott_Li The latest file gap was caused during maintenance activities, closing open windows caused the PC to blue screen and crash dump. No reason at all that this should have happen. 11/05/2012 13:55 scottl Traveled to site to investigate why PC2 suddenly shut itself down on Nov.1 at 1802 hrs. Pulled all system events for that day and all other events logged by the PC O/S. Reviewed the logs from Minimon and no indication of any related shutdown activity other than initaited by user CFF_PC2. Checked all power settings in BIOS and in O/S, Verified all updating services were turned off, disabled the JAVA servies updater (Known to have issues but not related to system shutdowns). The system performed an automated controlled shutdown for some reason that at this time cannot be explained. Will forward all system events to Chris M. for review. Tested all PLC messaging that could be done, ran SOLA UPS completely dead and then reutrned power, both units came back online along with associated devices without any problems or issues. Killed main breaker feeding the entire PIT tag room, re-energiszed with no ill effects. Checked and auto tuned transcievers, no roblems to report there. Upgraded PC2 with M4 Ver. 1.4.5. Exported and copied config files to thumb drive. 11/05/2012 13:22 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/05/2012 12:53 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/05/2012 12:53 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/05/2012 12:39 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/05/2012 12:39 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/05/2012 12:36 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/05/2012 12:36 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/05/2012 12:09 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/05/2012 12:09 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/05/2012 12:07 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/05/2012 12:07 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/01/2012 11:46 Troy Please disregard the last AC Power event(s). There was not a power outage, the input to the PLC was forced to test the email to text relay function with a more descriptive message. 11/01/2012 11:42 troy@psm The PLC reports AC Power on PNL2-CKT3 is ON 11/01/2012 11:42 troy@psm The PLC reports AC Power on PNL2-CKT1 is ON 11/01/2012 11:39 troy@psm The PLC reports AC Power on PNL2-CKT3 is OFF 11/01/2012 11:38 troy@psm The PLC reports AC Power on PNL2-CKT1 is OFF 11/01/2012 10:34 Troy Please disregard the last AC Power event. There was not a power outage, the input to the PLC was forced to test the email to text relay function. 11/01/2012 10:20 troy@psm The PLC reports AC Power on CKT2 is ON 11/01/2012 10:16 troy@psm The PLC reports AC Power on CKT2 is OFF 11/01/2012 09:11 Troy A final test of the AC Power ON/OFF email was conducted this morning between 10:04 and 10:08. These are not valid AC Power ON/OFF message. From this point forward unless noted in a pit event these message should be considered valid. 11/01/2012 09:08 troy@psm The PLC reports AC Power on CKT2 is ON 11/01/2012 09:07 troy@psm The PLC reports AC Power on CKT1 is ON 11/01/2012 09:04 troy@psm The PLC reports AC Power on CKT2 is OFF 11/01/2012 09:03 troy@psm The PLC reports AC Power on CKT1 is OFF 10/31/2012 15:41 troy@psm The PLC reports AC Power on CKT1 is ON 10/31/2012 15:40 troy@psm The PLC reports AC Power on CKT2 is ON 10/31/2012 15:40 Troy A series of PLC programming changes were conducted today. All AC Power ON/OFF emails should be ignored between 12:00 and 5:00. 10/31/2012 15:36 troy@psm The PLC reports AC Power on CKT1 is OFF 10/31/2012 15:36 troy@psm The PLC reports AC Power on CKT2 is ON 10/31/2012 15:36 troy@psm The PLC reports AC Power on CKT1 is ON 10/31/2012 15:32 troy@psm The PLC reports AC Power on CKT2 is OFF 10/31/2012 15:32 troy@psm The PLC reports AC Power on CKT1 is OFF 10/31/2012 15:31 troy@psm The PLC reports AC Power on CKT2 is ON 10/31/2012 15:31 troy@psm The PLC reports AC Power on CKT1 is ON 10/31/2012 14:48 troy@psm The PLC reports AC Power on CKT2 is OFF 10/31/2012 14:48 troy@psm The PLC reports AC Power on CKT1 is OFF 10/31/2012 14:48 troy@psm The PLC reports AC Power on CKT2 is ON 10/31/2012 14:47 troy@psm The PLC reports AC Power on CKT1 is ON 10/31/2012 13:31 troy@psm The PLC reports AC Power on CKT2 is OFF 10/31/2012 12:51 troy@psm The PLC reports AC Power on CKT1 is OFF 10/31/2012 12:49 troy@psm The PLC reports AC Power on CKT2 is ON 10/31/2012 12:49 troy@psm The PLC reports AC Power on CKT1 is ON 10/31/2012 12:45 troy@psm The PLC reports AC Power on CKT1 is ON 10/31/2012 12:43 troy@psm The PLC reports AC Power on CKT2 is ON 10/31/2012 12:42 troy@psm The PLC reports AC Power on CKT1 is OFF 10/31/2012 12:40 troy@psm The PLC reports AC Power on CKT2 is OFF 10/31/2012 12:29 troy@psm The PLC reports AC Power on CKT1 is ON 10/30/2012 06:49 Alan A power outage occurred on 10-29-2012 from 0211 to 0713 PST. Both PCs remained on UPS power until ~0315 PST. No data exist on either PC to fill the remaining data gap. 10/23/2012 07:21 Troy Fixed the navigation error in the HMI program so that PSMFC could access the page that toggles power to its sonicwall and satellite modem via the network 2. There doesn't appear to be enough bandwidth to get any information from the PLC. Change the communication timeout in the HMI from 1000mS to 0ms and this did not fix the problem. Tried several times to toggle power to the Sonicwall and Sat Modem. This may or may not have occurred,no information is coming back from the PLC. Either way the Modem and/or Sonicwall are still down for Network 1. 10/01/2012 10:12 Alan Two power outages occurred, one on 9/21/2012 from 0129 to 1255 PST and the other on 9/24/2012 from 0000 to 0811 PST. Files CFF12265.A2 and B2 were submitted to help fill the gaps in data. No further data exist on eithe PC to fill the gaps in data for either power outage. 09/26/2012 07:51 Alan Examination of both PCs shows no data that could fill the data gap on 9/24/2012. 09/24/2012 07:07 Scott_Li Per Jerod at YIN, the generators are down with no explanation. Some work was performed last week on some of the controls but no details are available. Per Jerod, this is WCW issue and they should be the ones to correct the problems. Jerod to call when more is known. 09/21/2012 13:22 Don_Warf YN reports that they have an electrician working at the site to resolve a power loss issue. More to come. 09/20/2012 06:49 Scott_Li Upgraded both data collection PC's to Minimon ver. 1.70. Repaired cat 5 jack in the Panduit on the 116.2 connection for PC2. Ran a new Cat 5 cable from the network cabinet to the PIT tag room for use on PSMFC .115 network. YIN will use this connection for access to the PSMFC network. Function tested the cycling of circuits , #7 and 8. YIN will eventually utilize these circuits to toggle power on their servers in the basement. Labeled and documented power control schema as much as possible. Scott Livingston Sr. Engineering Technician PSMFC-PTAGIS 7103 W. Clearwater Ave. Suite H. Kennewick, Wa. 99336 Office: 509.735.2773 x.2 Mobile: 509.539.0894 Scottl@psmfc.org 08/30/2012 06:50 alan@bay 01:VTTFailed: Noted 08/17/2012 13:59 Scott_Li Travel to site and found that PC2 had Auto Updates turned on. After updating, the PC rebooted and got hung up in some boot menu. Turned off updates on both primary and backup PC. No other problems to discovered. 08/14/2012 12:22 scottl Rewired external light on fuel bldg bypassing the emergency lighting hot leg. Emergency lighting, internal and external lights are still fully functional. Interference caused by the emergency lighting charge circuit appears to have gone away when duplicating the internal lights ON and exterior light ON scenerio. Will monitor over time but looks as if this might have done the trick. Also, cycled power on both Sat. Modems to restore internet connectivity. It is unclear as to why both network connections were lost. Speculate that the Hughs NOC had something to do with it. 08/13/2012 07:40 troy@psm The PLC reports AC Power on CKT2 is ON 08/13/2012 07:40 troy@psm The PLC reports AC Power on CKT1 is ON 08/13/2012 07:36 troy@psm The PLC reports AC Power on CKT2 is OFF 08/13/2012 07:36 troy@psm The PLC reports AC Power on CKT1 is OFF 08/09/2012 12:52 scottl Changed out the POE network switch with a standard unmannaged switch. Installed 3 individual 28 VAC power supplies for the cameras. All POE devices have now been converted to external power source. Noise levels on the transceivers are now at 9 to 10 percent and stable. 08/02/2012 06:16 Rclark@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: Site noise issues. 08/01/2012 10:53 troy@psm The PLC reports AC Power on CKT2 is ON 08/01/2012 10:53 troy@psm The PLC reports AC Power on CKT1 is ON 08/01/2012 10:51 troy@psm The PLC reports AC Power on CKT2 is OFF 08/01/2012 10:51 troy@psm The PLC reports AC Power on CKT1 is OFF 08/01/2012 09:50 troy@psm The PLC reports AC Power on CKT2 is ON 08/01/2012 09:30 troy@psm The PLC reports AC Power on CKT2 is OFF 08/01/2012 09:30 troy@psm The PLC reports AC Power on CKT1 is OFF 07/31/2012 06:07 Rclark@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: Was addressed on 7/30/2012. 07/30/2012 06:08 Rclark@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: This will be addressed during the next site visit. 07/29/2012 06:40 darren@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: The noise levels have increased since yesterday afternoon. This will be investigated on Monday. 07/28/2012 06:30 darren@b 01:VTTFailed,02:VTTFailed: These alarms will be addressed during the next site visit. 07/27/2012 14:16 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 14:16 troy@psm The PLC reports AC Power on CKT1 is ON 07/27/2012 14:14 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 14:14 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 13:41 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 13:41 troy@psm The PLC reports AC Power on CKT1 is ON 07/27/2012 13:39 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 13:39 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 13:21 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 13:17 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 13:16 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:58 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 12:49 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:48 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 12:46 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 12:40 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 12:39 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:35 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 12:35 troy@psm The PLC reports AC Power on CKT1 is ON 07/27/2012 12:33 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:33 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 12:32 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 12:32 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:31 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:31 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 12:25 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 12:25 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 09:40 troy@psm The PLC reports AC Power on CKT2 is ON 07/27/2012 09:39 troy@psm The PLC reports AC Power on CKT1 is ON 07/27/2012 09:38 troy@psm The PLC reports AC Power on CKT2 is OFF 07/27/2012 09:37 troy@psm The PLC reports AC Power on CKT1 is OFF 07/27/2012 06:53 darren@b 01:VTTFailed,02:VTTFailed: Alarms will be investigated during the next site visit. 07/26/2012 05:59 darren@b 01:VTTFailed,02:VTTFailed: These alarms will be addressed during the next site visit. 07/25/2012 06:51 darren@b 01:VTTFailed: Alarm noted. 07/25/2012 06:12 darren@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: These alarms will be addressed during the next site visit. 07/24/2012 06:25 darren@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: Alarm noted on all three systems. PSMFC is currently working on the noise issues at this site. 07/23/2012 07:01 alan@bay 01:VTTFailed,02:VTTFailed,03:VTTFailed: Noted 07/23/2012 06:58 alan@bay 01:VTTFailed,02:VTTFailed: Noted 07/22/2012 08:04 alan@bay 01:VTTFailed,02:VTTFailed,03:VTTFailed: Noted 07/21/2012 07:06 alan@bay 01:VTTFailed,02:VTTFailed: Noted 07/18/2012 10:00 Alan File CFF12199.D1 was created from a tmp file to fill a gap in data. No further problems to report. 07/18/2012 06:59 darren@b 01:NoSync,01:RdrToMaster: Alarm generated by PSMFC perosnnel. 07/17/2012 11:13 Troy A few of the Control Relays were toggled to ensure devices were still plugged into the proper outlet. This resulted in a brief data gap in PC1 which will have to be patched by PC2. Temp file and potential patch files are on a thumb drive and have been sent in via email to Kennewick staff. 07/17/2012 09:30 Troy A new PLC program was loaded to handle email messages when AC power is ON or OFF. There is a 2 min timer that starts when AC power is ON to allow the network to come up and get the email out. AC power OFF sends an email immediatley. No actual power outages occured today so all emails concerning AC Power on/off shoudl be ignored for the 10:00am to 10:30am period today. 07/17/2012 09:26 troy@psm The PLC reports AC Power on CKT2 is ON 07/17/2012 09:26 troy@psm The PLC reports AC Power on CKT1 is ON 07/17/2012 09:24 troy@psm The PLC reports AC Power on CKT2 is OFF 07/17/2012 09:23 troy@psm The PLC reports AC Power on CKT1 is OFF 07/17/2012 09:16 troy@psm The PLC reports AC Power on CKT1 is ON 07/17/2012 09:15 troy@psm The PLC reports AC Power on CKT2 is ON 07/17/2012 07:45 Alan Both PCs crashed and recovered. File CFF12198.E1 was created from a tmp file and fills the gap in data. No further problems to report. 07/13/2012 09:26 Rclark@b 02:VTTFailed: Noted. 07/13/2012 07:10 Alan The site has not submitted files since CFF12194.C1 (7/12/2012 0900 PST), most likely due to the poor internet connection. At this time there are no zip files in the temp folder. PC1 and PC2 have been verified as operating normally. PSMFC will be onsite next week and will investigate then. 07/12/2012 09:16 Alan Examination of files on PC1 and PC2 show no data other than xcvr diagnostics during the file gaps on 7/5 & 7/9 2012. Due to the poor remote connectivity to this site, no attempt will be made to upload the diagnostics. 07/12/2012 06:19 scottl@b 01:NoSync,01:RdrToMaster,02:NoSync,02:RdrToMaster,03:MemFull,03:VTTFailed: Alarms were generated during maintenance. 07/11/2012 13:37 troy@psm The PLC reports AC Power is OFF 07/11/2012 13:33 troy@psm The PLC reports AC Power is OFF 07/11/2012 13:29 troy@psm The PLC reports AC Power is OFF 07/11/2012 13:07 scottl Swapped out all ups's with tripp lites. Noise levels are still at 30 percent, hit rates are at 70 percent. Will continure to track down noise issues as they arrise. 07/11/2012 12:38 scottl@b 03:MemFull: Cleared memory and set not to store TT in the buffer. 07/11/2012 12:31 troy@psm The PLC reports AC Power is OFF 07/11/2012 12:31 troy@psm The PLC reports AC Power is OFF 07/11/2012 12:10 troy@psm The PLC reports AC Power is OFF 07/11/2012 11:53 troy@psm The PLC reports AC Power is OFF 07/11/2012 09:59 Troy testing of the email functionality was performed today. All AC POWER OFF emails, event logs and texts should be ignored. 07/11/2012 09:58 troy@psm The PLC reports AC Power is OFF 07/11/2012 09:50 troy@psm AC Power Off Test 07/11/2012 09:50 troy@psm AC Power Off Test 07/11/2012 06:40 Troy@bay 01:VTTFailed,02:VTTFailed,03:MemFull,03:VTTFailed: These will be addressed on the next site visit. 07/10/2012 07:38 Troy@bay 01:VTTFailed,02:VTTFailed,03:MemFull,03:VTTFailed: PSMFC is in the process of correcting the issues that cause these alarms. 07/08/2012 09:07 scottl@b 01:VTTFailed,02:VTTFailed,03:MemFull,03:VTTFailed: Will check on these alarms next week 07/05/2012 06:16 scottl@b 01:VTTFailed,02:VTTFailed,03:MemFull,03:VTTFailed: Noted, Mem full messages will be suppressed 07/02/2012 06:24 darren@b 03:MemFull: Alarm will be cleared tomorrow during a site visit. 07/01/2012 07:11 Troy@bay 03:MemFull: Noted 06/30/2012 06:40 Troy@bay 03:MemFull: Noted 06/29/2012 06:28 Rclark@b 03:MemFull: Noted. 06/28/2012 06:30 scottl@b 01:VTTFailed,02:VTTFailed,03:MemFull,03:VTTFailed: Noted 06/27/2012 07:10 Troy@bay 03:MemFull: Noted 06/26/2012 07:14 Troy@bay 03:MemFull: This will be addressed on the next site visit. 06/25/2012 07:20 Troy@bay 03:MemFull: This will be addressed on the next site visit. 06/24/2012 09:18 Troy@bay 03:MemFull: This will be addressed on the next site visit. 06/23/2012 07:45 Troy@bay 03:MemFull,03:MemLow: This will be addressed on the next site visit. 06/22/2012 08:04 Troy@bay 03:MemLow: This alarm will be addressed on the next site visit. 06/21/2012 11:33 Troy_Hum Test email Troy Humphrey Field Systems Engineer PSMFC-PTAGIS 509-735-2773 ext.5 509-539-0963 cell 06/21/2012 07:09 Troy@bay 01:VTTFailed,03:MemLow: Noted. 06/20/2012 07:40 Troy@bay 01:VTTFailed: Noted 06/19/2012 06:47 alan@bay 01:VTTFailed: Noted 06/18/2012 06:06 scottl@b 01:VTTFailed: Noted 06/17/2012 05:34 darren@b 01:VTTFailed: Alarm will be addressed during the next site visit. PSMFC is working on a solution to some noise issues at this facility. 06/16/2012 05:19 darren@b 01:VTTFailed: Alarm has been noted and will be addressed during the next site visit. 06/15/2012 22:22 darren@b 01:VTTFailed,02:VTTFailed,03:VTTFailed: Problems are being addressed by PSMFC to alleviate the issues with these systems. 06/13/2012 06:11 darren@b 01:VTTFailed: Alarm noted. 06/12/2012 12:18 Don_Warf test 06/08/2012 14:16 Scott_Li This is only a test. 06/07/2012 14:08 Troy Modified the PLC program remotely (very slow process) so that the "counter" for CKT1 is incremented every time power is lost. I then modified the Data Station Plus in the Kennewick data room so that it compares the HMI Count to the plc count and sends an email when they are different. I could not get the texting to work like it does on other HMI's. Basically I'm finding that a bad email address will keep the entire program from running properly. I created a rule in my outlook that forwards an AC power related event to everyone's phone. I turned off the program that constantly monitors the health of the network as this has proven to Scott and I to be a nuisance and very un- reliable. 06/07/2012 06:27 Rclark@b 01:VTTFailed: Noted. 06/06/2012 06:31 Rclark@b 01:VTTFailed: This will be addressed during the next site visit. 06/05/2012 06:27 Rclark@b 01:VTTFailed: Noted. 06/04/2012 07:42 Alan File CFF12149.C9 contains data to cover the gap caused by the power outage on 5-28-2012. No further data exist on either PC to fill this gap. 06/03/2012 07:17 alan@bay 01:VTTFailed: Noted 06/02/2012 07:47 alan@bay 01:VTTFailed,02:VTTFailed: Noted 06/01/2012 18:29 Don_Warf Found the left side Sola UPS in the PLC panel powered off. Power was restored by holding in the UPS "On" button until power came back on. This powered up the PLC, SonicWall and satellite modem. Comms are now back up on both networks. The UPS will be replaced or reviewed for proper operation. 05/29/2012 09:11 Don_Warf Per Kevin at Premier Electric: The generator set failed at Castile Falls yesterday morning during a scheduled duty cycle switch over. Personnel are in-route to the site. More info to follow. 05/28/2012 06:23 rclark@b 01:VTTFailed: Noted. 05/26/2012 06:49 rclark@b 01:VTTFailed: Noted. 05/23/2012 06:40 Rclark@b 01:VTTFailed: Noted. 05/22/2012 06:34 Alan In order to re-establish automated uploads, MiniMon was exited briefly on PC1 to delete a zipfile. There will be a small file gap at ~6:25PST 5-22- 2012. No activity was observed on PC2 during PC1's downtime. Files CFF12142.H1 to CFF12143.C1 were manually pushed to PTAGIS. 05/22/2012 06:11 Rclark@b 01:VTTFailed: Noted. 05/21/2012 06:01 Rclark@b 01:VTTFailed: This will be addressed during the next site visit. 05/18/2012 07:20 alan@bay 01:VTTFailed: Noted 05/17/2012 06:36 Troy@bay 01:VTTFailed: Noted. 05/13/2012 06:29 rclark@b 01:VTTFailed: This will be addressed during the next site visit. 05/12/2012 07:10 rclark@b 01:VTTFailed,02:VTTFailed: This will be addressed during the next site visit. 05/10/2012 07:30 Troy@bay 01:VTTFailed,02:VTTFailed: Noted 05/09/2012 06:01 Rclark@b 01:VTTFailed: This will be addressed when access to the site is available. 05/07/2012 06:31 Rclark@b 01:VTTFailed: This will be addressed as soon as access to the site becomes possible. 05/06/2012 06:50 Troy@bay 01:VTTFailed,02:VTTFailed: These alarms will be addressed on the next site visit. 05/05/2012 07:49 Troy@bay 01:VTTFailed: Noted 05/04/2012 07:58 alan@bay 01:VTTFailed: Noted 05/03/2012 07:10 alan@bay 01:VTTFailed: Noted 05/02/2012 07:19 Troy@bay 01:VTTFailed,02:VTTFailed: These alarms will be addressed once access to the site is available. 04/29/2012 09:45 alan@bay 01:VTTFailed: Noted 04/28/2012 08:27 alan@bay 01:VTTFailed: Noted 04/26/2012 07:04 Troy@bay 01:VTTFailed: Noted 04/24/2012 08:43 alan@bay 01:VTTFailed: These systems will be investigated when the facility is accessible. 04/21/2012 07:31 darren@b 01:VTTFailed,02:VTTFailed: these systems will be investigated when the facility is accessible. 04/16/2012 07:38 alan@bay 01:VTTFailed,02:VTTFailed: Noted 04/13/2012 06:24 scottl@b 01:VTTFailed,02:VTTFailed: Noted 04/11/2012 06:11 Rclark@b 01:VTTFailed,02:VTTFailed: This will be addressed during the next site visit. 04/08/2012 06:39 rclark@b 01:VTTFailed: This will be checked during the next site visit. 04/07/2012 06:31 rclark@b 01:VTTFailed,03:VTTFailed: This will be addressed during the next site visit. 04/06/2012 06:41 alan@bay 01:VTTFailed: Noted 04/05/2012 06:07 Rclark@b 01:VTTFailed,02:VTTFailed: Will be addressed during the next site visit. 04/03/2012 06:10 Rclark@b 01:VTTFailed,02:VTTFailed: These alarms will be addressed during the next site visit. 04/02/2012 06:20 Rclark@b 01:VTTFailed: This will be addressed during the next site visit. 04/01/2012 08:33 Troy@bay 01:VTTFailed: Noted 03/31/2012 07:29 Troy@bay 01:VTTFailed,02:VTTFailed: When this site becomes accessible PSMFC will address these issues. 03/29/2012 09:05 scottl Addendum to the previous entry, called Mike Babcock (Data Manager) for the YIN and left message to informed him that the site had lost communication. 03/29/2012 08:33 scottl Communications to the site was lost at 0400 hrs and restored at 0850 hrs on 3/29/2012. It is unclear at this time why the outage occurred. 03/29/2012 07:57 Troy@bay 01:VTTFailed: Noted 03/28/2012 07:14 alan@bay 01:VTTFailed,02:VTTFailed: These alarms will be addressed when the site becomes accessible. 03/27/2012 07:27 Troy@bay 01:VTTFailed: These alarms will be addressed when the site becomes accessible. 03/26/2012 07:09 Troy@bay 01:VTTFailed,02:VTTFailed: These alarms will be addressed when this site is accessible. 03/25/2012 06:45 alan@bay 01:VTTFailed,02:VTTFailed: PSMFC will investigate these alarms when the facility becomes accessible. 03/24/2012 06:37 alan@bay 01:VTTFailed: PSMFC will investigate these alarms when the facility becomes accessible. 03/23/2012 06:45 alan@bay 01:VTTFailed,02:VTTFailed: PSMFC will investigate these alarms when the facility becomes accessible. 03/22/2012 06:46 alan@bay 01:VTTFailed: PSMFC will investigate this alarm when the facility becomes accessible. 03/21/2012 06:31 alan@bay 01:VTTFailed: PSMFC will investigate this alarm when the facility becomes accessible. 03/20/2012 06:37 alan@bay 01:VTTFailed: PSMFC will investigate this alarm when the facility becomes accessible. 03/19/2012 10:12 alan@bay 01:VTTFailed: PSMFC will investigate this alarm when the facility becomes accessible. 03/19/2012 07:35 alan@bay 01:VTTFailed: PSMFC will investigate this alarm when the facility becomes accessible. 03/18/2012 06:49 darren@b 01:VTTFailed: PSMFC will investigate this alarm when the facility becomes accessible. 03/17/2012 07:17 darren@b 01:VTTFailed: PSMFC is aware of the issues with this system and will address them when we can get access to the facility. 03/15/2012 06:51 Troy@bay 01:VTTFailed: Noted 03/14/2012 06:43 darren@b 01:VTTFailed,02:VTTFailed: These alarms will be addressed when PSMFC can gain access to this facility. The transceivers are running with enough efficiency to detect and decode PIT tags. 03/13/2012 06:21 darren@b 01:VTTFailed: Alarms will be addressed as soon as PSMFC can get to this facility. 03/09/2012 07:22 alan@bay 01:VTTFailed: This will be addressed during the next site visit. 03/05/2012 07:36 Rclark@b 01:VTTFailed,02:VTTFailed: This will be addressed during the next site visit. 03/04/2012 08:03 rclark@b 01:VTTFailed: This will be addressed during the next site visit. 03/01/2012 08:39 Rclark@b 01:VTTFailed: This will be addressed during the next site visit 02/28/2012 07:40 Rclark@b 01:VTTFailed: This will be checked during the next site visit. 02/27/2012 07:05 Rclark@b 01:VTTFailed: This will be investigated as soon as the weather allows. 02/25/2012 09:06 Troy@bay 01:VTTFailed: Noted 02/24/2012 07:32 alan@bay 01:VTTFailed,02:VTTFailed: Noted 02/23/2012 07:36 Troy@bay 01:VTTFailed: Noted 02/22/2012 07:52 Troy@bay 01:VTTFailed: Noted 02/21/2012 07:15 Troy@bay 01:VTTFailed: Noted 02/20/2012 09:00 Darren This is an amendment to the previous entry. This site is not currently under construction. 02/20/2012 08:22 darren@b 01:VTTFailed: Alarm will be addressed during the next site visit. This site is still currently under construction. 02/19/2012 07:43 darren@b 01:VTTFailed: Alarm will be addressed during the next site visit. 02/17/2012 07:11 scottl@b 01:VTTFailed,02:VTTFailed: Noted 02/15/2012 07:36 darren@b 01:VTTFailed: Alarm noted. Reader efficiency is still very high. The site is not reachable due to snow. PSMFC will be on site when we can reach the facility. 02/14/2012 07:31 darren@b 01:VTTFailed: Alarm will be addressed during the next site visit. 02/11/2012 10:52 alan@bay 01:VTTFailed: Noted 02/10/2012 07:52 alan@bay 01:VTTFailed: Noted 02/09/2012 08:11 alan@bay 01:VTTFailed: Noted 02/08/2012 07:50 alan@bay 01:VTTFailed: PSMFC is aware of the slight noise on this system and will address all issues when we can reach the facility. 02/07/2012 08:20 alan@bay 01:VTTFailed: PSMFC is aware of the slight noise on this system and will address all issues when we can reach the facility. 02/06/2012 07:29 alan@bay 01:VTTFailed,03:VTTFailed: PSMFC is aware of the slight noise on this system and will address all issues when we can reach the facility. 02/03/2012 07:42 Troy@bay 01:VTTFailed: Noted 02/01/2012 07:29 darren@b 01:VTTFailed: PSMFC is aware of the slight noise on this system and will address all issues when we can reach the facility. 01/31/2012 07:35 darren@b 01:VTTFailed: PSMFC is aware of this alarm. We are unable to visit the site but feel the system is able to detect PITtags with high efficiencies. 01/29/2012 08:45 rclark@b 01:VTTFailed: Noted 01/28/2012 08:19 rclark@b 01:VTTFailed: Noted 01/27/2012 07:35 Rclark@b 01:VTTFailed: Noted 01/26/2012 06:34 rclark@b 01:VTTFailed: Noted. 01/25/2012 07:17 Rclark@b 01:VTTFailed: Noted 01/24/2012 07:14 Rclark@b 01:VTTFailed: Noted. 01/22/2012 08:00 Troy@bay 01:VTTFailed: The TT appear to be firing now. 01/21/2012 10:31 Troy@bay 01:VTTFailed: Noted 01/19/2012 07:08 Troy@bay 01:VTTFailed: Noted 01/18/2012 08:02 Troy@bay 01:VTTFailed: Noted. 01/17/2012 07:16 Troy@bay 01:VTTFailed: Noted 01/16/2012 07:29 alan@bay 01:VTTFailed: Noted 01/15/2012 08:33 alan@bay 01:VTTFailed: Noted 01/15/2012 08:33 alan@bay 01:VTTFailed: Noted 01/14/2012 06:48 alan@bay 01:VTTFailed: Noted 01/07/2012 09:20 darren@b 01:VTTFailed: Alarm noted. PSMFC is aware of the noise situation at this site and is addressing issues as the construction continues. 01/04/2012 07:49 darren@b 01:VTTFailed: Alarm noted. 01/03/2012 07:57 darren@b 01:VTTFailed: PSMFC is aware of the noise situation of these systems. 01/01/2012 00:00 PTOC_Bot Annual log file initiation for CFF #### End of Event Log for CFF ###