EVENT LOG FOR SITE GRA Most recent activity to be listed first. First line of each entry lists date, time & who is reporting. Subsequent lines of each entry aligned under Time column. Do not use tabs, only spaces. Date Time Who What ---------- ----- -------- ----------------------------------------------- 12/13/2001 11:34 Dave Ken McIntyre informed me that the adult trap at GRA was dewatered and shut down for the season by 1100h today (12/13/01). I've retrieved the last of the data files, closed Multimon and PTTP, and archived the remaining data files on both PC1 and PC2. The computers are still running, and the coils/transceivers are still powered-up. We are no longer polling for GRA data. 12/03/2001 10:40 Dave Well, the COE said it wouldn't happen, but it did. We lost power to GRA this morning, right after 6:00AM. Unfortunately, I was unable to pcA into GRA from home last night, so I didn't have a chance to disable PTTP. Multimon was on file 103 when it crashed today, so it reverted to 101 when power was restored. Fortunately, 102 had not been transferred before the crash, so it was still resident in the pc1data directory, and Multimon only backfilled the one file before skipping onto the (new) 104. I discovered this when I finally got into the office and pcAed into GRA at 9:30. I fixed the problem by renaming the duplicate 101 file to GRA01337.901 inside and out, transferring it to Gladstone, and moving it from pc1data to the archive directory. I then copied the archive of the original 101 file back into the pc1data directory. Now, if there are any more outages, there won't be any gaps to backfill. Auto-uploads from PTTP remain disabled until Thursday. Today's data files can be manually transferred tomorrow, Tuesday's files can be transferred Wednesday, and Wednesday's files can be transferred on Thursday, at which point PTTP auto-uploads can be enabled again. 11/29/2001 15:07 Scott_Li The PTTP client has been restarted on these computers. 11/27/2001 09:08 Scott_Li The PTTP client has been disabled until Thursday , Nov. 29 01. 11/15/2001 09:40 darren@p A4:ReaderReset,A5:InputSyncDetect,A5:InputSyncLost,B4:TestTagFailure: Alarms on A4 & A5 were caused by a random reader reset. This problem will be addressed this year. PTOC personnel and Destron-Fearing have a fix (possibly) that will be tried at MCJ first and then deployed system wide. The test tag failure will be further investigated on another file to see if it has returned. . 11/07/2001 08:52 Scott_Li Did not receive weekly stick tag test from NMFS. Called Neil at the facility about this weeks stick test result and was told that the printer has not been turned on therefore no hardcopy to fax. Asked them to please turn on the printer prior to the tag test so that we receive a report here in the Kennewick office. All indications are that the facility is running well and the diversion gates are operating. Also , will be evaluating MM ver. 8.0 to see if this version has the parallel port bug that was found in 7.04. Will report results to Sandy D. 11/01/2001 10:59 Scott_Li Traveled to the site yesterday to re-evaluate why the diversion gates were periodically locking in the open or closed state. It was previously thought that there could be some port contention between the primary and backup computers. After installing an isolation circuit on the output pins of each computer, the problem still existed. The problem is consistently repeatable. If pc2 is controlling the gates and a divert signal is sent to either of the two diversion gates at the exact time that pc1 is detected by pc2, pc2's control lines ( parallel port ) lock in the high state therefore holding the diversion gates open. To prove that the situation was at the software level rather than a hardware issue, the signal lines for both computers was disconnected from the gate relay box. Bottom Line, This issue is a Multimon Bug. I will report this bug to Gordon Anderson via Sandy Downing and maybe get this fixed in the next revision. For now, pc2 was disconnected from the gate control relay box. Its far better to take the slight chance of missing a few fish if pc1 goes down rather than fill the trap with fish. 10/29/2001 15:04 Scott_Li The facility computers have recently experienced a problem with the diversion gate controls. This problem has occurred in the past on very infrequent basis. What we know as of now is that the parallel port control lines that send the signal to a relay controlled, voltage level converter box that actuated the gate solenoids are locking in a 0 or 5v state, therefore either keeping the gates open or not opening at all. After duplicating the problem, it appears that there may be some port contention between PC1 and PC2. For instance, If PC2 is controlling the gates, for whatever reason, and PC1 comes back into operation at the exact time that a tagged fish is detected, PC2 and PC1 may both try to divert the fish therefore causing both inputs to the control relay to be at 5 volts, in turn locking the ports. Short term, we plan to put some diode isolation on the outputs of the parallel ports to eliminate port conflicts. The long term solution ,if our thinking is correct, would be to install a PLC and eliminate the relay logic. Control modifications will be done Tuesday, Oct 30. 10/22/2001 09:47 Scott_Li Ken Thomas informed us this morning that the handpass pass diversion test on the diversion gates was not working this morning but PC1 was collecting data. Could not PCA into GRA-PC1 although, the PTTP loader was successfully sending files. With no other option, Had Neil at the facility reboot PC1 . The diversion gates are now working properly but still for some reason cannot PCA into the computer. Neil also indicated that the phone line in the Pittag room was dead, will place a trouble ticket with COE personnel to have it repaired. The unfortunate part of this is that PTTP , prior to the reboot had sent files 295.101-295.102 to Ptagis. As we know , Multimon will create a .101 file when it shoud increment to .104. Talked with Dave M and we decided to take PC1 out of the loop and let PC2 control the diversion gates until the situation with PC1 can be corrected. Will travel to the site if necessary no later than tomorrow, Oct.23rd for repairs. PC1 has experienced this problem with the parallel port not sending a control pulse to the gate solenoids several times in the past year. Although several attempts have been made to isolate the cause of this infrequent , periodic coma of the parallel port on this computer, no root cause has been determined. Immediate change of plans requires a visit to GRA, PC2 was given control of the gates but was not activating them.. Will respond... 10/19/2001 11:32 Dave Responded to power outage at Lower Granite. According to COE, outage lasted from ~8:10-8:40 AM PDT. Logged into both the juvenile and adult sites a few minutes before 9:00 AM PDT. Created *.104 files on both machines. Renamed duplicate *.101 files to *.101x. After 9:00 AM PDT I retrieved the *.203-204 files from the GRJ and GRA backup PCs. I programmatically closed the truncated *.203 files, and then submitted all the *.203-204 files to PTAGIS. I moved all outstanding files from the backup machines to their respective archives, and compressed the archives on all four machines. I restarted PTTP Uploader on GRJ-PC1 and GRA-PC1. I noted that GRA PC1 and PC2 are 2:20 and 1:10 fast, respectively; GRJ PC1 and PC2 are 6:50 and 5:40 fast, respectively. Both pairs of machines are more than 30 seconds off synch with each other, and GRJ is more than 5 minutes out of synch with the world. These exceed our standards. However, this late in the season, with few tagged fish present, I did not reset the clocks and re-synch the machines. Finally, I added a shortcut to WinCmd to the Desktop on all four machines. 09/24/2001 12:05 Darren_C Jerry Harmon contacted our office today and explained that there was no diversion when passing sticks over the coils. This has happened two times in the past. the first time the switch for running the gate was turned off accidentally. The other time it was discovered that the parallel port was locked up and not sending the signal to trigger the gate. We could not get a hold of Jerry right away so we PCA'd into both computers and rebooted them. I spoke with Jerry at 11:45 and he confirmed that the sticks were diverting. I asked if the CWT switch was being turned off or the equipment he confirmed that the equipment was being turned off and not the switch. We are not sure how long this has been going on and neither is Jerry. The computers were rebooted at 10:13 am PST. They were not down for more than 20 seconds. 09/20/2001 22:49 Dave For reasons unknown, GRA lost power for ~1hr this afternoon, from 2:30-3:30 PST. All data files have been revised and transferred. 09/04/2001 08:52 Dave There was a power outage at Lower Granite Dam on Thursday, August 30. This affected both the GRJ and GRA fish facilities. SMP personnel report "There was a switch from turbine five to turbine six. Turbine six is not the default generator for the facility, causing the outage. Power was out from 7:55 AM to 10:25 AM (PST) according to the bio-tech at the separator. This was an unscheduled outage." Our PIT tag detectors and computers were down during this outage. When power was restored, the computers resumed data collection, but not automatic file transmission, per our standard protocol. Scott noticed the disruption in file uploads from both GRJ and GRA, and remotely contacted the sites, renamed and manually transmitted all queued files, and restored normal operations at both GRJ and GRA. I made some edits to three files from each site, and all files were loaded normally. 08/31/2001 09:01 Darren_C Due to the previous power outage, The PTTP client was not active and running therefore not transferring data. PCA'd into GRA-PC1, renamed the duplicate files to .901 & .902. Manually initiated the data transfer and was successful. 08/28/2001 22:20 Dave I updated Multimon with GRA01239.DB and MAP at ~ 2100h this evening. These configuration files reference Action Codes 106 and 107, for hatchery and wild subyearling chinook of interest to USFWS. Any of the ~100k detected from these two groups will be diverted into the adult trap. I had been trying to upload these configuration files remotely since yesterday afternoon; I was not able to connect to PC1 until this morning, and was unable to establish and maintain a reasonable file transfer rate until this evening. During the course of these events, I also synchronized both PC clocks within four seconds of Universal Time. 07/26/2001 09:58 Dave pcA'd into GRA to retrieve files left after yesterday's power outage. Renamed and archived redundant data on PC1. Re-initiated PTTP on PC1, and manually uploaded GRA01207.101-102. Manually uploaded GRA01206.204-209 on PC2. Reset time on PC2 to bring it within four seconds of PC1. The time for PC1 is ~4:45 fast, bumping up against the 5:00 maximum difference we'll tolerate. 07/26/2001 09:40 Darren_C A mail message was sent to me that GRA,GRJ had not sent in files since 6 a.m. I PCA'd into GRJ pc1 and realized that there had been a power outage at about10:35 a.m. and now that PTTP is not in the startup menu I needed to send files. I conversed with Dave Marvin on how he would like this done. We gave the copy files (101,102,103) and extension of 101x,102x,103x and then archived all of day 206. I then PCA'd into PC2 and sent the files 204,205,206,207,208 and 209. I also started PTTP on PC1 but I will let PTTP send in the files for day 207. I also checked the time sync between PC1 and PC2 as per Dave Marvin. 07/12/2001 13:59 Dave I update the Multimon environment with a new map file, "gra01193.map". This map version replaces the various proportional diversion commands for all CSS groups with a no-action "Monitor Only" protocol, per Jerry Harmon and Larry Basham. From this point forward, all CSS fish will be routed directly back to the ladder after detection. Per Jerry, there is no change in the disposition of UofI study fish (Action Code 66), which will continue to be diverted to the sample tank as encountered. Stick tags will also continue to be diverted to the sample tank. 07/10/2001 10:50 Scott_Li General maintenence check found no problems to report. displayed reader status on all ISO xceivers, hand passed test stick for diversion gate test. Ken Thomas faxed over weekly stick test All ISO coils detected 100 percent of the stick tags thrown. 400 west coils 10,14 detected 100 percent, coil 12 detected 7 of 8 stick tags. 400 east coils all detected 100 percent of the 8 tags thrown. 07/02/2001 13:20 Scott_Li Ken Thomas faxed over the weekly stick tag test. All coils on the east and west flumes detected 100 percent of the tags tested for both the 134.2 and 400 khz coils. 06/26/2001 19:43 Don_Warf Visited site for a general maintenance check. No problems found. 06/25/2001 08:37 Darren_C Ken Thomas faxed over the stick test for today. All coils are reporting a 100% efficiency. 06/20/2001 08:12 Darren_C Site was visited when it was noticed that coil 10 did not appear to be reading. When I arrived exciter was tested and it was determined that the loop tuner was bad. It was replaced and the coil was tested by passing sticks over the coil. 06/12/2001 08:04 Darren_C A general site maintenance check was performed. No problems were found. Sticks were thrown at GRJ on the separator with 100% efficiency. 06/11/2001 15:59 Scott_Li Ken Thomas faxed over this weeks stick tag test results. Could not read the printout that was faxed so the data was pulled from the raw file162.103 and tallied.(9) 400 khz tags was thrown on the east and west coils. All coils detected 100 percent of the tags. (8) 134.2 khz stick tags were thrown on both the east and west monitors. All coils detected 100 percent of the tags. Ken appearently did not do the manual hand pass diversion test. A live tagged fish came through the facility at the time of the test. The Stick tag efficiency report tool has been validated with the raw interrogation file. 06/07/2001 14:06 scottl general maintenence check was performed on 6/6/01. No problems to report. Hand pass diversion test was successfull. All equipment was checked for optimum performance. 06/04/2001 11:13 Scott_Li Ken thomas faxed over the weekly stick test and hand pass divert test for this week... All ISO and 400 Khz coils detected 100 percent of the tags thrown thru the monitor on the East and West flumes..Hand Pass diversion gate test was also successfull.. 06/01/2001 08:25 Scott_Li The diversion gates were inoperable from ~1430 hrs 5/29 to 1200 hrs 6/1. This was noticed by NMFS and U of I when none of the 4 tagged fish were diverted. All indications were that the gate was working, a manual trigger via PCA was successfull and tag codes detected said that they had been diverted. A complete system check from cables to software revealed nothing.. Cold booted PC1 and the gate functions were restored. NMFS will hand pass divert test the gates on a daily basis for 1 week to ensure the gates are working. 05/30/2001 08:12 Darren_C A general Maintenance check was performed no problems were found. The time was adjusted on PC2 ( it was 1:30 behind PC1). 05/24/2001 16:47 Dave Using the buffer downloads that Scott obtained, I found three ISO tag codes detected during the computer outage on 5/21-22. I created an interrogation file for these three tags (gra01141.z) and have loaded it to the PTAGIS3 database. 05/24/2001 08:43 Dave pcA'd into site and manually uploaded files created since 9:00am yesterday. John is diagnosing "problem" with PTTP-Uploader. 05/23/2001 09:17 Scott_Li Downloaded the buffers from ISO transceivers coil a4 east flume and coil b4 west flume to help fill in any gaps in the data set due to the recent power outage. Sent data to Dave M for processing.. 05/22/2001 17:11 Dave A series of power outages at Lower Granite Dam knocked the data collection computers off line, beginning at 2:45 pm (PST) on May 21. Both the primary and backup computers went down for the count at 3:00 pm. At the request of PTOC, NMFS personnel at the site restarted the computers at 8:20 am (PST) on May 22. A total of 17 hours and 33 minutes data were lost. This loss is tempered somewhat by the period of the data outage. Adult passage through the LGR ladder is generally negligible at night. During the previous 24-hour period, no PIT-tagged fish were detected at GRA between 7:30 pm on May 20 and 4:30 am on May 21 (both times PST). Both AM and PM crepuscular passage can be significant, however, and there's little doubt that one or more PIT-tagged fish passed through the interrogation system undetected yesterday afternoon and evening, and this morning. 05/15/2001 18:22 Chase Both sites were visited for weekly maintenance. No sticks were passed through the separator monitor at GRJ because of the high volume of fish. The gates were checked and no other problems were found. There was a power outage at both sites. The power was out from approximately 06:45 to 07:30. Both platforms were checked for timing and both were found to be within 30 seconds ( PC1 to PC2). I spoke with Dave Marvin and explained what had happened and we discovered that when PTTP fired back up that file .101 had been overwritten but there was no cause for alarm since the previous file had been sent before the outage. 05/08/2001 11:41 Scott_Li General maintenence check found no problems to report.. Switched the printer output from pc2 to pc1.. This was done to provide more accurate information to the NMFS guy's, Jerry Harmon and gang, in their daily monitoring of fish movement thru the facility. Also , delivered 6 dot matrix printers for backups.. Checked all transceivers for noise and current... 05/02/2001 10:37 Dave I replaced the existing SbyC database (gra01100.db) with gra01122.db. This version contains 550 400 kHz tags that replace the 600 ISO tags specific to the UofI radio-tagging study in the forebay. The action code remains the same (66), and so no changes to the map file were required. The close proximity of the ISO coils to the diversion gates in the East and West Weir monitors apparently resulted in a "loss" of ISO-tagged fish back to the ladder, rather than into the trap for tag recovery. UofI has therefore switched to the 400 kHz tags, with the hope that the greater distance between the 400 kHz coils and the diversion (trap) gates will provide sufficient time for the swing gates to activate. 05/02/2001 07:57 Darren_C A Site Maintenance Check was performed. No sticks were passed through the Separator monitors due to the high volume of fish passing through the facility. The gates were checked to ensure opening and closing. The 3-way gate was checked at the juvenile site. No problems were found. Ken Thomas mentioned to me that they were having a problem with gate activation on ISO tagged fish. They have Dual system (402/ISO) with the ISO coils furthest downstream. The distance from the first coil and the gate is not very far and the gate size is large. I told him that PSMFC PTOC would discuss the problem and try and come up with a viable solution for the next season. 05/01/2001 16:15 Scott_Li Ken Thomas faxed over weekly stick test results for the Adult facility.. All 400 and ISO coils detected 100 percent of the stick tags thrown thru the monitors on the East and West Flumes..Hand pass diversion gate test was successful. 04/23/2001 15:01 Scott_Li General maintenence check found no problems at either site. Ken Thomas faxed over weekly stick test results for GRA. All coils 400 and ISO detected 100 percent of the stick tags thrown. Hand pass diversion gate test was alsp successful. 04/18/2001 07:47 Darren_C A general site maintenance check was performed and no problems were found. Sticks were ran on monday by Jerry Harmon with 100 0.000000e+00fficiency. 04/16/2001 09:58 Scott_Li Ken Thomas faxed over weekly stick test results. All coils 400 khz and 134.2 Khz on East and West Flume detected 100 percent of the total stick tags thrown thru the monitors. Hand pass diversion test was successful 04/16/2001 09:09 Dave I updated the SbyC map and database at 10:30AM on Friday, 4/13/01. The current map and db files are GRA01100.MAP and GRA01100.DB, respectively. 04/14/2001 09:28 Don_Warf Visited site for a general maint. check and found no problems. 04/03/2001 09:48 Scott_Li Ken Thomas faxed over the weekly stick tag test results. All ISO coils East and West flumes detected 10 of 10 stick tags. All 400 coils East and West flumes detected 9 of 9 stick tags. 04/03/2001 09:41 Scott_Li The Aboutime program was removed from the data collection platforms at the listed facilities. This application was causing timing conflicts with the Multimon application with regards to slide gate triggering via the PLC. The Aboutime program also adjusted the system clocks for daylight savings time. The system clocks were returned to PST at approx. 1150am PST on 4/2/01. To prevent this occurance , the Aboutime application requires the user to manually re-adjust the win98 time timezone setting just prior to the switch from PST to PDT. Another gotchya! Also, the phone lines were un-plugged from the data computers until further notice and modem initialization settings within PCA were disengaged until further notice. 03/26/2001 10:54 Scott_Li Ken Thomas faxed over weekly stick test results; All 134.2 Khz coils on both East and West flumes detected 10 of 10 stick tags. All 400 Khz coils except for coil 12 ,detected 9 of 9 stick tags. Coil 12 is dead and will be repaired 3/27. 03/23/2001 14:27 Scott_Li Ken Thomas faxed over the weekly stick test results. All 400 khz coils for East and West flumes detected 9 of 9 stick tags. All 134.2 Khz coils for East and West flumes detected 10 of 10 stick tags. 03/12/2001 15:05 Scott_Li Ken Thomas faxed over weekly stick test results. All ISO coils on East and West flumes detected 10 of 10 stick tags All 400 Khz on East and West flumes detected 9 of 9 stick tags. 03/05/2001 11:01 Dave Just a re-interation and clarification of Scott's entry. The picket leads were dropped on March 5, 2001. All fish ascending the Lower Granite ladder are now passing through the PIT tag detectors. This will be the normal mode of operation until sometime in December. 03/05/2001 11:01 Scott_Li Entry for work done on Friday March 2. Replaced IDI controller boxes with the new style Pro-Log High speed controllers.Installed network card in utility pc and configured software. Ran stick test on both east and west flumes. All 4 ISO coils detected 10 of 10 stick tags. All 400 coils detected 9 of 9 stick tags. Facility is watered up but is not collecting any fish in trap at this time... Ken McEntire was not sure exactly when they would begin trapping fish. Visit also included various misc. tasks...Ken Thomas faxed over stick test results for Monday March 5th.. Again all 400 coils detected 9 of 9 stick tags... Will inform Ken to include the ISO system in his weekly testing.. Noticed on the report that the Stick tag code for the diversion gate test was not in the look up table. Will correct this later in the week via remote access... 02/16/2001 09:46 Scott_Li The generic site configuration files for this facility have been posted on the ftp server. During the site visit, (2) 400khz exciters were replaced associated with coils 12,14,00,10. Repaired broken signal return wire in controller for coil 12. Replaced timer tag on coil 12. Site has been completely function tested and is operational. Latest word on water-up is March 1st, but subject to change. 02/10/2001 11:20 scottl Turned off test submission switch in PTTP. Data files will now be redirected and processed as normal. The evaluation period for this site is complete. 01/23/2001 09:09 scott_li The .34 subnet addresses and subnet mask for GRA have changed. GRJ addresses have not changed. The Subnet mask for both facilities is now 255.255.255.128 rather than 255.0 This i/p scheme is a variable subnet generated by Pete Peterson, COE. Portland distict is revising their allowed address table for access through the firewall. Granite Adult computers: GRA-PC1, 137.161.34.130 GRA-PC2, 137.161.34.131 GRA Utility, 137.161.34.132 Granite JFF , Subnet mask, 255.255.255.128 The subnet addresses have not changed 01/05/2001 18:00 Dave Manually created new event log for 2001. ####End of Event Log for GRA###