EVENT LOG FOR SITE 3_LMJ Most recent activity to be listed first. First line of each entry lists date, time & who is reporting. Subsequent lines of each entry aligned under Time column. Date Time Who What -------- ----- -------- ----------------------------------------------- 10/30/98 15:04 scottl The Sun file server was shut Down on Tuesday Oct. 27th. due to Hard drive failure. Data Collection was not interupted as the collection pc's where still running. Replaced the file server and reconfigured. Tested and verified proper operation with Todd K. Assistance. Copied all data files from pc1 that have not been uploaded since the failure and transfered those files via FTP to Sockeye for processing. This facility will be de-watered on Sunday, November 1, 98' for the season per Bill Spurgeon. 10/22/98 10:59 dlwarf LMJ-10, B-DIVERSION: Annunciator warnings acknowledged 10/14/98 11:44 scottl Removed remote monitor (mort tunnel)from this facility. The system will be on loan to ODFW(Sue Knapp). 10/13/98 09:14 dlwarf LMJ-10, B-DIVERSION: Annunciator warnings acknowledged 09/16/98 08:41 scottl Per Dave at the Walla Walla Distict help desk, The T-1 line is down. US West is working on the problem at this time. Dave will call when the communications have been re-established. 09/10/98 12:58 scottl Installed new firmware rev for controller 00. This is the controller that has been problematic over the last few months with lockup problems. The firmware was written to do an entire system reset after ever tag code is detected. This should eliminate the lockup do to invalid SOT characters. The controller will be monitored for some time for performance. The rest of the high speed controllers at other sites will be upgraded pending the evaluation of this test. 09/10/98 12:58 scottl Installed new firmware rev for controller 00. This is the controller that has been problematic over the last few months with lockup problems. The firmware was written to do an entire system reset after ever tag code is detected. This should eliminate the lockup do to invalid SOT characters. The controller will be monitored for some time for performance. The rest of the high speed controllers at other sites will be upgraded pending the evaluation of this test. 09/08/98 08:33 dlwarf 400X controller 00 locked up on 9/4 and was reset at 9:30am on 9/5. 09/08/98 08:32 dlwarf 400X controller 00 locked up on 9/4 and was reset at 9:30am on 9/5. 08/26/98 14:15 carters LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 08/14/98 11:26 carters LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 08/14/98 11:25 carters LMJ-3, A-DIVERSION: Annunciator warnings acknowledged 08/14/98 11:25 carters LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 08/13/98 09:04 scottl Per Ken Fone ( assistant biologist)The scheduled power outage for this facility was canceled until further notice. 08/10/98 08:14 scottl Assistant biologist reported that there will be a power outage at the facility on Aug. 13th after 1000. The duration of the outage was not known. 07/25/98 10:31 dlwarf High Speed controller for down stream of separator monitors locked up. Had smolt monitoring person power cycle both units. Telnetted to site to verify operation. 07/07/98 11:42 scottl The high speed controller locked up again on 7/5/98. Called Bill spurgeon on 7/6 and had himn power cycle the high speed controller. This corrected the problem and data collection continued. We will be installing a power line monitor at this facility in the near future to help diagnose the problem. Should be in the next week or so. 07/02/98 13:37 dlwarf LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 06/16/98 13:58 carters Big Brother flagged LMJ with disk utilization above 90%. Removed 1997 data sets from pc1/logfiles/archive. --Carter 06/15/98 08:10 scottl E-mail wrote by brad peterson. We had another 400X controller lock-up Thursday at Lower Monumental. Scott ran quit a few tests for me and it appeared the controller was functioning properly but it was sending garbage data and multimon was rightfully ignorin this data (no start of text character). I met Scott at Lo. Mo. Friday afternoon and clipped a processor monitor to t ailing controller, enabling us to monitor the execution of the firmware. Th reader cards were passing the proper data to the CPU card which was formatti messages properly and passing them on to the communications card which appea to be working OK. That is, it went though the proper sequences for acceptin data for transmission but, what came out or the RS-232 port was garbage. Consistent garbage but, garbage never the less. It appears the comms card initialization parameters are getting corrupted, a which, the ports are longer set up for 115k,N,8,1 operation, resulting in a of communication between the controller and the A and B computers. Once the comms card is re-initialized, by cycling the power and executing the initialization function, the controller returns to normal operation. The problem is most likely due to dirty power but, we're not sure. In any c there are a number of fixes we can implement that will prevent this from happening in the future. 06/12/98 15:27 scottl replaced reader card for coil 54 on diversion exit. tested good 06/10/98 15:00 davem SMP personnel report that on June 10 at 1500 the PIT tag override of the sample timer was initiated. 06/07/98 09:37 carters LMJ-4, DIVERSION EXIT: Annunciator warnings acknowledged 06/02/98 00:00 davem SMP personnel report that fish collected on June 1 and 2 were bypassed on June 2 rather than barged due to high river flows and GBT concerns. 05/28/98 12:53 scottl Used telnet session to determine status on gate controllers and generalize coil efficencies while data base is being corrected. 05/27/98 14:29 dlwarf Visited for a general maintenance check. 400X controller 00 has run with no problems since power cycle. Ran 10 sticks on the separator with these results: 00 read 10 02 read 10 04 read 10 06 read 10 08 read 10 0A read 10 0C read 10 0E read 9 05/27/98 09:27 dlwarf LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 05/27/98 09:27 dlwarf LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 05/25/98 21:28 dlwarf Telnetted to site. Controller 00 still running after yesterdays power cycle. 05/24/98 21:10 dlwarf Telnetted to site at 9:00pm. Controller 00 still running after power cycle. 05/24/98 13:14 dlwarf Found 00 400x controller locked up. Verified that the Rocket port was still alive then power cycled the controller to bring back on line. 05/22/98 09:04 carters LMJ-10, B-DIVERSION: Annunciator warnings acknowledged 05/21/98 14:40 dlwarf Visited for a general maintenance check. Ran 10 sticks on the 8 separator coils with these results: 00 read 10 02 read 10 04 read 9 06 read 10 08 read 10 0A read 10 0C read 10 0E read 8 All sticks were successfully diverted 05/20/98 08:22 dlwarf LMJ-10, B-DIVERSION: Annunciator warnings acknowledged 05/20/98 08:18 dlwarf LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 05/20/98 08:17 dlwarf LMJ-3, A-DIVERSION: Annunciator warnings acknowledged 05/18/98 08:40 dlwarf LMJ-4, DIVERSION EXIT: Annunciator warnings acknowledged 05/18/98 08:38 dlwarf LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 05/15/98 11:00 carters LMJ-10, B-DIVERSION: Annunciator warnings acknowledged Look at 05:30:20 below... LOWER MONUMENTAL DAM JUVENILE (LMJ) Monitor Nbr: 2 -- A-SEPARATOR GATE 1.44 tag/min Coil: 00 Mean (M) = 93.90% Std Dev (S) +--------------------+-------------+----------------------------------- | End Time | Tags | Coil Efficiency + Of | Per +------+---------------------------- | Detection Interval | Minute | % |+- M-3S Mean(|) +--------------------+-------------+------+| 15-may-1998 05:59:03 0.32 |* 66.67 B=========================== 15-may-1998 05:49:43 1.50 |** 100.00 |--------------------------- 15-may-1998 05:36:23 15.38 |*** 80.00 |--------------O============ 15-may-1998 05:34:47 5.00 |*** 95.00 |--------------------------- 15-may-1998 05:30:43 52.17 |**** 95.00 |--------------------------- 15-may-1998 05:30:201200.00 |***** 65.00 <-O========================= 15-may-1998 05:30:19 600.00 |***** 45.00 <-O========================= 15-may-1998 05:30:16 1.71 |** 90.00 |-------------------------O= 15-may-1998 05:18:14 1.38 |** 90.00 |-------------------------O= 15-may-1998 05:03:37 1.72 |** 95.00 |--------------------------- 05/15/98 10:58 carters LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 05/14/98 09:18 carters LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 05/14/98 09:17 carters LMJ-3, A-DIVERSION: Annunciator warnings acknowledged 05/14/98 09:16 carters LMJ-4, DIVERSION EXIT: Annunciator warnings acknowledged 05/13/98 09:52 carters LMJ-4, DIVERSION EXIT: Annunciator warnings acknowledged 05/13/98 09:51 carters LMJ-10, B-DIVERSION: Annunciator warnings acknowledged 05/13/98 09:51 carters LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 05/13/98 09:50 carters LMJ-3, A-DIVERSION: Annunciator warnings acknowledged 05/13/98 09:49 carters LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 05/12/98 13:13 scottl Replaced exciter on A-diversion coils 20,22 due to dead exciter. Retuned coils 24,26 A-diversion. Coil 20 detected 8 of 10 coil 22 detected 9 of 10 coil 24 detected 10 of 10 coil 26 detected 10 of 10 The up stream coils orientation was poor. Current on exciter is max 1.0 Amp. No other problems to report. 05/11/98 09:26 carters LMJ-5, A-RACEWAY: Annunciator warnings acknowledged 05/09/98 11:33 scottl Gate yoke mount was physicaly broken at approx. 2200 5/8/98. Arrived at site at 1100 hrs and talked to site bio. Maint. personnel is going to weld yoke mount back onto slide gate. All fish are being sent to the raceway. Gate should be repipared by this afternoon. Will follow up 5/10 for status. 05/07/98 19:40 carters LMJ-4, DIVERSION EXIT: Annunciator warnings acknowledged 05/07/98 19:40 carters LMJ-9, B-SEPARATOR GATE: Annunciator warnings acknowledged 05/07/98 19:39 carters LMJ-3, A-DIVERSION: Annunciator warnings acknowledged 05/07/98 19:39 carters LMJ-2, A-SEPARATOR GATE: Annunciator warnings acknowledged 05/05/98 13:04 dlwarf Visited for a general maintenance check. Found a dead exciter for coils 24 and 26 on A diversion. Replaced exciter. Found coil 48 down on the subsample. Replaced exciter. Ran 10 sticks on the separator monitors with these results: Coil 00 read 10 02 read 10 04 read 10 06 read 10 08 read 10 0A read 10 0C read 8 0E read 9 04/29/98 15:00 scottl Changed delay before operate time in PLC from 22 to 26 on the A-side diversion gate. The upstream coils where slightly early, This should compensate for this condition. Changed delay before operate timing parameters in PLC for the B-side diversion gate from 22 to 26 for the same conditon as above. Changed Exciter on the sample room monitor due to low efficiency on coil 4A. Stick tested A-raceway monitor and all coils detected 100 percent of 10 stick tags thrown. Adjusted soft close valve on the a-side slide gate due to eliminate slamming. Did the same for the B-side also. No other problems found during general maintenence check. 04/29/98 09:30 dougc LMJ-7, SAMPLE ROOM: Old warnings cleared per request from Don Warf. 04/29/98 09:29 dougc LMJ-4, DIVERSION EXIT: Old warnings cleared per request from Don Warf. 04/29/98 09:29 dougc LMJ-10, B-DIVERSION: Old warnings cleared per request from Don Warf. 04/29/98 09:28 dougc LMJ-6, A-EXIT: Old warnings cleared per request from Don Warf. 04/29/98 09:27 dougc LMJ-3, A-DIVERSION: Old warnings cleared per request from Don Warf. 04/29/98 09:26 dougc LMJ-2, A-SEPARATOR GATE: Old warnings cleared per request from Don Warf. 04/24/98 11:33 dlwarf Found loose power connector on coil 00 and 02. Ran 35 sticks and read 1000n both coils. 04/21/98 11:16 scottl found A-separator gate valve stuck. Purged all the oil out of line and cycled gate several times.Turned oiler flow to a minimum. Stick test ran 100 percent of 20 tags on all 4 coils and diverted all 20. Replaced exciter on a-diversion coils 20,22. Coils 24,26 need tuned for optimum performance. Will tune on the next visit. Informed separator operator to cycle gates twicw per shift to excersise gates. 04/16/98 08:44 scottl Repaired connection on exciter for coil 1b-B-raceway. Coil detected 10 of stick tags. Timer tag is now functioning also. Stick tested A-raceway, all coils detected 10 of 10 stick tags. General maintenance check found no other problems. 04/14/98 15:15 scottl Pre-season testing and tuning for this facility has been completed. Changed timeouts and retrys in pc-nfs mount to 3 and 3. This should eliminate the errors abort,retry,ignore errors from the network card. 04/12/98 12:30 davem SMP personnel report the "PIT tag system" was out of service on April 12 from 1230 to 1540 to replace the plastic air lines with copper tubing. The PIT tag system was also out on April 13 from 1445 to 1500 and 1815 to 1840 to replace a water trap on the air supply line. Power was out at the fish facility from 2305 on April 9 to 0330 on April 10 due to a ground fault on the fish facility transformer. 04/08/98 12:44 scottl Found the backup machine (pc2) at this facility locked-up with error message failure reading drive d (ABORT,RETRY,IGNORE.) When abort command was issued the comm buffers dumped and saw error messages pertaining to com1 overflows. Rebooted machine to continue data collection. This problem is being investigated for the root cause 04/08/98 11:01 carters Stick tests: All coils a-separator 20 of 20. b-separator 20 of 20 on coils 08, 0a, 0c; 19 of 22 on 0e. 04/02/98 08:33 carters Bill Spurgeon reports that facility began collection of fish on Wed. April 1, 1998 for transport. PIT interrogation equipment is operational and has been collecting timer tag information. 03/31/98 12:19 carters Bill Spurgeon reports that Lower Monumental will begin passing fish tomorrow 4/1/98. 03/31/98 12:02 carters IP packets now being sent. No action was taken by PSMFC. Was there a power outage in the PITTAG room? 03/30/98 14:32 carters Pete Peterson reports that no IP packets are being transmitted from the PSMFC side of the router at LoMo. Either all three machines are down, the blackbox hub is down, or the fiber to the router is down. PSMFC will investigate today or tomorrow. 03/20/98 08:02 carters LoMo is not yet watered up. We plan to water up on the 23 of March. Per Bill Spurgeon e-mail, 3/19/98. PIT interrogation equipment is operating and is collecting timer tag information now.