EVENT LOG FOR SITE WEA Most recent activity to be listed first. First line of each entry lists date, time and who is reporting. Subsequent lines of each entry aligned under Time column. Do not use tabs, only spaces. Date Time Who What ---------- ----- -------- ----------------------------------------------- 12/11/2003 17:44 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise: Excessive noise levels occurred during the open/close of sub panel #3 breaker from 06:15-10:07. This is the first major noise event since the 11/12/03. Coils 05-08 where in Stby during the noise event. 12/11/03 Coils 05-08 have been inspected and re-tuned while the ladder is dewatered. The coils will remain on during the testing of the new gantry crane motors. The noise reports will be used to determine if the noise levels are affecting both ladders. Unknown when testing will be over. Coils 01-04 were also tuned. 12/02/2003 17:29 acarson@ When arriving on site the alarm from the UPS supplying power to PC1 was on and PC1 was powered down. PC2 was collecting data, which makes me believe that the UPS to PC1 did not reset itself after AC power was turned back on. After pushing the on/off button on the UPS it then supplied power to PC1. The UPS was tested by temporarily disconnecting the UPS from AC and no problems were found. Any gaps in data will be filled Dec 4th. The right ladder will be de-watered Dec 3rd. Coils 05-08 have been placed in standby after the ladder is completely de-watered. The transceivers are placed in standby remotely while maintenance personnel are working in the area. Mike Bruno mentioned all sub panels are back online. Coils 01-04 are still reporting the same low noise levels since 11/12/03 at 12:45:00. They will also be testing the newly installed frequency driven motor on the gantry cranes on Dec 3rd. More later……… 11/20/2003 15:11 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,05:ExcessiveNoise,06:ExcessiveNoise,08 :ExcessiveNoise: Alarms caused during analog board (NPO) replacement and normal maintenance on site. 11/20/2003 15:11 acarson@ 01:LowExcCurrent,05:LowExcCurrent,05:TestTagFailure,06:LowExcCurrent,06:Te stTagFailure,07:LowExcCurrent,07:TestTagFailure,08:LowExcCurrent: Alarms caused during analog board (NPO) replacement and normal maintenance on site. 11/20/2003 15:07 acarson@ Analog boards on coils 05-08 were replaced w/ NPO boards. All transceivers were re-tuned and no problems found. On 11/12/03 around 12:40:00 noise levels on the Left ladder have decreased on coils 01-04. Coil 04 which has had noise issues since the initial install had the biggest change from 8% noise averages to 0-1%. I have graphed the noise reports for the entire day and forwarded it to Mike Bruno. The control room log does not show any activity during the sharp decrease in noise levels. Mike indicated that all sub panels are being replaced and power is being switch around as they move from one panel to the next. The panel that is down now is coming back on line Nov. 23rd. I will continue to monitor the noise levels on the Left ladder until the noise levels return to normal. File generation was changed on PC1 & PC2 to every 3hrs. due to recent power outages that lasted longer than the UPS's. Power to the modems nest need to be placed on a UPS so data can get to the computer platform when power is lost. I will send this request to Shane Bickford and Mike Bruno. As of 11/20/03 14:16:00 all files marked for Upload/archive have been successfully uploaded. 11/17/2003 11:57 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise,05 :ExcessiveNoise,06:ExcessiveNoise,07:ExcessiveNoise,08:ExcessiveNoise: Excessive noise levels will be looked at during next site visit Nov 20th. Per Dave Marvin request, PC1's file transfer settings was changed from automated to manual. I will be uploading data manually each day until settings are return to normal. 11/17/2003 10:39 Dave I spoke with Anthony Carson, BioMark, this morning regarding data collection at PRD, RIS, and WEL and our PSMFC firewall issues. We decided that, to ensure the integrity and security of the data files at those sites, Anthony will disable automated file uploads in Minimon (by switching to the 'archive only' mode) until the firewall issue has been resolved. Data files will be sent manually. At Anthony's request, I've disabled file-checking for PRA, RIA, and WEA in etc/sites until further notice. 11/10/2003 09:49 acarson@ Power to the computer room was shut down while replacing breakers. Data was lost from WEA03312.C (11/08/03, 18:00) - WEA03313.B (11/09/03, 04:04) on both pc's. 10/21/2003 10:09 acarson@ 03:ExcessiveNoise,04:ExcessiveNoise,06:ExcessiveNoise: Excessive noise level on 2003/10/19 17:04:13 was seen on all coils except 07 & 08. Excessive noise level on 2003/10/15 12:01:59 was seen on only coils 05 & 06. Cause is unknown. 09/23/2003 16:09 acarson@ 03:ExcessiveNoise,04:ExcessiveNoise: Excessive noise levels due to cleaning of fish viewing windows 09/19/2003 13:26 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise,05 :ExcessiveNoise,06:ExcessiveNoise,07:ExcessiveNoise,08:ExcessiveNoise: Alarms caused during de-watering of fish ladder to clean fish veiwing window on both ladders. 08/22/2003 11:31 acarson@ 07:LowExcCurrent: Low current alarm caused during cleaning fish viewing windows. In addition, Gap in data starting on WEA03230.A was caused during a power outage that lasted about an hour. PC1 was not responding to PCA. Wells personnel re-booted PC1 and missing data will be recovered on Monday. 08/22/2003 11:17 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise,05 :ExcessiveNoise,06:ExcessiveNoise,07:ExcessiveNoise,08:ExcessiveNoise: Excessive noise alarms look to be from de-watering both laddders for cleaning the fish viewing windows. I will contact DCPUD personnel on Monday to confirm. 08/18/2003 14:01 Darren_C The loading problem for WEA has been passed to Anthony Carson of Biomark and he will take care of the situation. I was able to PCA into PC2 and it is up and running minimon and collecting data, I was not able to PCA into PC1 however. The situation with PRA is that they computers have been removed from the PUD's network until the computers can be cleaned of a virus. Anthony is in the process of correcting the problem. 08/12/2003 09:25 acarson@ 07:LowExcCurrent: Alarms caused when ladder was de-watered to clean the fish viewing windows. 08/11/2003 12:52 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise,05 :ExcessiveNoise,06:ExcessiveNoise,07:ExcessiveNoise,08:ExcessiveNoise: On 7/24/03 both the right and left ladders were de-watered for fish viewing window cleaning, 8:31:27 - 9:27:28 (R) and 9:56:44 - 10:25:44 (L). On 8/7/03 both right and left ladders were de-watered for fish viewing window cleaning, 8:41:03 - 9:35:41 (R) and 10:08:53 - 10:46:53 (L) causing the excessive noise level alarms. 07/24/2003 08:55 Brett_T. Site visit to WEA on 7/22/03. Coils 01 to 06 had drifted slightly. It tweaked them a little and brought the noise levels down to the 6-8% range. Coils 07 and 08 required no adjustments. Overall, everything looked very good. 04/25/2003 08:53 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise,05 :ExcessiveNoise,06:ExcessiveNoise,07:ExcessiveNoise,08:ExcessiveNoise: Alarms caused during normal cleaning of fish viewing windows. The West ladder was de-watered from 8:30:20 - 9:34:20 and the East from 10:08:14 - 10:48:14. 04/25/2003 08:52 acarson@ 01:LowExcCurrent,03:LowExcCurrent: Alarms caused during normal cleaning of fish viewing windows. The West ladder was de-watered from 8:30:20 - 9:34:20 and the East from 10:08:14 - 10:48:14. 04/21/2003 10:11 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise,05 :ExcessiveNoise,06:ExcessiveNoise,08:ExcessiveNoise: Wells Dam lost power for 15 min. on 04/15/03 around 8:00a.m. causing alarms on all coils. 04/21/2003 10:03 acarson@ 03:BadOscFreq: Noise reports indicate increased noise levels on all coils 01-04 during same time. I will continue to monitor this problem. 04/11/2003 11:36 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise: Alarms caused during last site visit. 04/11/2003 11:09 acarson@ 01:LowExcCurrent,02:LowExcCurrent,02:TestTagFailure,03:BadOscFreq,03:LowEx cCurrent,03:TestTagFailure,04:LowExcCurrent,04:TestTagFailure: Alarms caused during site visit. 04/09/2003 18:39 acarson@ Changed clocks on PC1- 17:22:30 to 17:32:30 and PC2- 17:22:30 to 17:31:50 during file WEA03099.E. Also Archived all files in PC2. 04/09/2003 18:21 acarson@ 04:ExcessiveNoise: Replaced analog boards in all East side transceivers, coils 01-04. I noticed a noise increase in almost all coils. I will monitor to determine if the increased noise levels are "true" and are having an effect on detection efficiency. There was little difference in how well the stick tag and test tag read while tuning/calibrating transceivers with original or new NPO analog boards. 04/09/2003 10:33 acarson@ 04:ExcessiveNoise: Excessive noise levels due to switching main breaker during 18:00-18:30 time frame. Still monitoring. 03/23/2003 17:54 acarson@ 04:ExcessiveNoise: Excessive noise levels are being shown on all coils 01- 04 during this noise spike. After raising the noise alarm threshold, only coil 04 has sent an alarm. I will continue to monitor the alarms during this time frame. 03/16/2003 15:49 acarson@ 01:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise: Noise spikes that occur daily between 17:30-18:00 are triggering the excessive noise alarms on almost all East ladder antennae 01-04. Those coils on the East side that do not trigger the alarms still show noise spikes. There are days when the noise spikes do not set off any of the coils excessive noise alarms but still show an increase in noise level during that time frame. This is suspected to be from the breakers on units 7 & 8. The breakers are usually used during this time of the day for the increased power consumption when people get home from work. The noise spikes are brief and do not effect the average noise levels on all coils. Because there may not be a solution to fix this problem, the noise threshold alarms will be raised from 25 to 35 on coils 01-04. I will increase the threshold every week until the noise alarms for this particular event are decreased. 03/07/2003 14:47 Dave I updated the ptagis3.test_tags table to associate each (previously- missing) coil_id value with each virtual timer tag at this site. This essentially enables the validation routines for the Timer Tag reports. 03/06/2003 10:15 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise: At 17:41:27 on 3/4/03 all East side coils indicated excessive noise levels for approximately 10 minutes. Then the next day at about the same time it happened again, 17:45:42. I contacted Mike Prey in the control room who told me that Unit #7 breaker was closed on 3/4/03 at 17:49:42 and Unit #8 was closed on 3/5/03 at 17:45:02. The breaker event does occur almost once a day so I will continue to monitor to see if the noise only occurs during Units 7 & 8. 03/06/2003 08:18 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise,04:ExcessiveNoise: Excessive noise levels on all East bank coils occurring at same time the last two days. I will contact DCPUD personel to help find the noise source. 02/26/2003 18:14 acarson@ 02/26/03 Reset all transceivers, tuned coils 01-04 and set noise alarm thresholds to 25% on all coils. The following are the in/out of service dates for both ladders this season. West Ladder: Out of Service, 0713 12-03-02. In Service, 1621 01-22-03. East Ladder: Out of Service, 0717 01-27-03. In Service, 1435 02-20-03. 02/23/2003 16:47 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise,03:ExcessiveNoise: The East ladder has re-watered and coils 01-04 will need to be tuned on next site visit, scheduled for 02/26/03. I will PCA into PC2 and adjust noise alarm thresholds until then. 02/17/2003 16:57 acarson@ 02:ExcessiveNoise,07:ExcessiveNoise: Alarms caused during re-tuning and calibrating of all transceivers. 02/17/2003 16:55 acarson@ 01:LowExcCurrent,02:LowExcCurrent,04:BadExcFreq,04:BadOscFreq,04:LowExcCur rent,07:LowExcCurrent: Alarms caused during re-tuning and calibrating of all transceivers. 02/13/2003 16:28 acarson@ 2/13/02 All 2002 data for PC1 & PC2 has moved to folder WEA2002Data under Minimon for each PC. Inserted pipe insulation foam around Coils 01 & 02 exciter cables to test the effects of vibration to the cables after water- up. All other exciter cables will need the sweeping 90' PVC pipes removed before the foam will fit. 02/13/2003 09:27 acarson@ 01:ExcessiveNoise,02:ExcessiveNoise: Biomark personnel on site re-tuning transceivers 02/12/03 and 02/13/03. 02/12/2003 18:36 acarson@ 01:ExcessiveNoise,03:ExcessiveNoise,06:ExcessiveNoise,07:ExcessiveNoise: Excessive noise levels due to Coils 05-08 being placed in "Scan" mode once the West ladder was watered-up. 2/12/03 8:00 Coils 05-08 re-tuned. Coils 01-04 re-tuned to a de-watered ladder and inspected for damage. East ladder is scheduled to water-up 2/27/03. 02/06/2003 18:39 Dave Redundant WEA data files were uploaded to PTAGIS a total of three times today. The first set was uploaded at ~9:35 am today. The second set was uploaded at ~12:30 pm. We were successful in removing the second set prior to processing the interrogation data. However, I did not catch a third, re-redundant, set of files that was uploaded at 5:56 pm, and included in the processing queue for the 6:15 pm load. At 6:20 pm, I pcA'd into WEA- PC1, and verified that the data directory is now empty, and all the files have been archived. For kicks, I also pcA'd into WEA-PC2, and verified that machine is NOT uploading data files. However, its data directory contains 1787 files dating well back into 2002; this results in VERY slow response time when a user (like me) attempts to view the upload status (or the contents of that directory). I suggest that the files in the WEA-PC2 Minimon data directory be archived as soon as practical, and this directory be archived on a regular basis (at least monthly) to prevent the accumulation of more than ~125 data files. 02/06/2003 17:51 acarson@ 2/3/03 Contacted Jeff Bunzter to have him confirm the communications between the computers and the Wells Dam router. Scott L. spoke with Ben Carter of DCPUD E. Wenatchee and confirmed the connection from E. Wenatchee to PSMFC. Both confirm communications is operational but still no direct connection from PC1 & PC2 to PSMFC. 2/4/03 Left message with Dan Stolp, DCPUD Wells Dam, about the problem. 2/5/03 Dan will check the connection from Wells Dam to E. Wenatchee. 2/6/03 Contacted by Jeff Bentzler that the router in E. Wenatchee was altered and caused the break in communications. Router was re-configured and I was able to PCA into PC1 and manually send the data files. 02/06/2003 17:24 acarson@ 01:LowExcCurrent,02:LowExcCurrent: Alarms caused by the East ladder being de-watered. Coils 01-04 have been placed in "stby" until inspected and ladder water-up. 02/03/2003 14:17 Scott_Li Per the contractual agreement with Biomark, I contacted Anthony Carson and advised him on how to approach and diagnose the problem with the interrogation computers with regards to not being able to send data files to PTAGIS. Anthony C. will try and resolve the issue through the appropriate channels within the DCPUD and post the results to the PTAGIS eventlog. 02/03/2003 08:25 Scott_Li Called Jeff Bunztler at the project and aked if he could reboot PC1 and check status of PC2. Of course requested that he try and change the current data file prior to rebooting in order to preserve the data residing in the temp file. Once the computer is back online, I will pull the log files to try and determine why the PC was in a coma state. More soon.. 01/31/2003 14:55 acarson@ Last data received was 1/29/03 at 06:32. I was not able to PCA into PC1 or PC2. Contacted control room and had them send diagnostic test data on both PC1&2 which resulted in an Error: connecting to FTTP session #26002- (1101), Host not found. I will get a hold of Jeff on Monday to find the source of the network problems. As of now both PC1 & PC2 are collecting data. PC1 has around 10 files still to be downloaded from the data folder. The control room personnel could not give any reason for the failure. 01/13/2003 08:38 Scott_Li Recieved the message that the site was had not submitted data files. Called the project and asked Jeff B. to reboot PC1. I also requested that before he does that to try and change the current data file in order to preserve the data . Will follow up to ensure the reboot was successful. 01/01/2003 12:20 Dave Initialized new event log for 2003. 2002 event log has been archived. ####End of Event Log for WEA###