Technical, Logistical, and Economic Considerations for the Development and Implementation of a Scottish Salmon Counter Network: Scottish Marine and Freshwater Science Vol 7 No 2

4.0 Operational Costs and Validation

4.1 Operation and Maintenance

In this section we discuss general cost considerations for operating and maintaining all major types of counter technologies, with more specific information below. Cost estimates were based on an extensive literature review ( Appendix 1), data from questionaires sent to current counter operators ( Appendix 2), and from IFR's 25-year experience with various counters. These costs represent typical budgets, but as each site is unique, this can vary greatly depending on the following:

Counter Structure

Structure type is one of the largest determinants of cost. Crump weirs and flat pads, for example, are low maintenance and require infrequent debris removal, whereas partial diversions and fish passes require moderate maintenance and require periodic monitoring and cleaning. Full river span picket fences require a high level of maintenance in-season. Most picket fence sites are inspected daily to several times per week for debris, holes, or other issues that may compromise operation. High water levels can limit fence access due to safety issues with wading in fast deep water and can also overturn fences. In these cases, complete in-season rebuilds are common. Alaskan floating fences are generally lower maintenance than traditional picket fences, as they are self cleaning and much less likely to be "blown out" during a high water event.

Debris Loads

The amount of debris (i.e., wood, bedload) that is transported downstream will affect the number and duration of site visits required to ensure proper counter operation. The frequency of visits will increase, and take longer, with higher debris loads and more frequent discharge fluctuations.

Fish Abundance

High fish abundance (i.e., high number of fish events) can rapidly fill data storage for some counters (e.g., resistivity), subsequently requiring more frequent downloading. This will require more site visits, especially if the counter does not have remote download capabilities.

Equipment Malfunction

Equipment malfunctions increase in-season mainanence costs. While some repairs can occur in the field, other instances require the equipment to be sent to the manufacturer, resulting in potential data gaps. This should be avoided since data gaps reduce data reliability which requires additional analyses. If consistent data collection is the highest priority, it is recommended to purchase backup equipment (high capital cost). Another option, which can reduce costs but is less reliable, is an assurance from the supplier that loan or lease of equipment is available in the event of equipment malfunction.

Power Supply

Power consumption varies among counter equipment. Mains power is most reliable and only susceptible to power outages, which can be mitigated by having a backup battery supply. Unfortunately, it is not always available and could be costly to install in remote sites. When on mains power, the site requires infrequent weekly visits if no remote communication is available, and less frequently if counter operation can be monitored remotely. It is important to keep in mind that power outages can damage equipment.

Alternative power supplies (i.e., batteries, solar) will moderately increase maintenance costs; the cost of batteries and other infrastructure (if installing a solar-powered system) can vary depending on the location. Site visits should occur weekly at minimum if using alternative power supplies to change batteries and ensure proper functioning. Temperature and cloud cover will influence the discharge rate of batteries and recharge rate of solar pannels, respectively. In some cases, battery or solar power can be more reliable than mains power when power outages are frequent. Solar and wind power generation can be paired with batteries to increase the time between battery changes and in some cases can be used to eliminate battery changes.

Site Access

Site access can cause costs to vary considerably. If a site is close to the operators workbase and easily accessed by vehicle, costs will be low. Conversely, if a site is farther away and poses accessibility challenges (i.e., no vehicle access), time and travel costs will increase accordingly. Here we provide a breakdown of operation and maintenance costs for all major types of counter technologies. Refer to Table 4.1 for a summary of this information.

Table 4.1. Maintenance and operation costs associated with optical beam, hydroacoustic, resistivity and video counters.

Type of counter	Typical application and channel type	Maintenance			Operations
		Annual costs	In-season costs	Remote downloading capabilities	Power supply	Equipment servicing	Debris load and structure
Optical beam counters	Fish pass	Installation: 2-4 ppl for 1 day Removal: 2-4 ppl for 1 day	Moderate	Supported by manufacturer and available for all models	12 V 40 Ah battery will last 8 days	Manufacturer: Out of country High shipping costs, long transport time Backup cost: High (£27 000)	Periodic removal required
	Full-river fences	Installation: 2-4 ppl for 2 days Removal: 2-4 ppl for 1 day	High	Supported by manufacturer and available for all models	12 V 40 Ah battery will last 8 days	Manufacturer: Out of country High shipping costs, long transport time Backup cost: High (£27 000)	Up to daily removal required
Hydroacoustic counters	Diversion fence	Installation: 2-4 ppl for 2 days Removal: 2-4 ppl for 1 day	High	Limited to no capability due to large data files	2.5-12.5 Ah	Manufacturer: Out of country High shipping costs, long transport time Backup cost: High (£22 000-48 000)	Up to daily removal required
	No structure	Installation: 2 ppl for 1 day Removal: 2 ppl for 1 day	Low	Limited to no capability due to large data files	2.5-12.5 Ah	Manufacturer: Out of country High shipping costs, long transport time Backup cost: High (£22 000-48 000))	Little to no removal required
Resistivity counters	Crump weir	Installation: 2 ppl for 1 day Removal: 2 ppl for 1 day	Low	Capable with third party software and hardware	24 V 40 Ah battery will last 5 days	Manufacturer: Scotland-based Low shipping costs, short transport time Backup cost: Low (£10 000-20 000)	Little to no removal required
	Flat pad sensor	Installation: 2 ppl for 2 days Removal: 2 ppl for 1 day	Low	Capable with third party software and hardware	24 V 40 Ah battery will last 5 days	Manufacturer: Scotland-based Low shipping costs, short transport time Backup cost: Low (£10 000-20 000)	Little to no removal required
	Tube sensor in fish pass	Installation: 2-4 ppl for 1 day Removal: 2-4 ppl for 1 day	Moderate	Capable with third party software and hardware	24 V 40 Ah battery will last 5 days	Manufacturer: Scotland-based Low shipping costs, short transport time Backup cost: Low (£10 000-20 000)	Periodic removal required
	Flume sensor in fish pass	Installation: 2-4 ppl for 1 day Removal: 2-4 ppl for 1 day	Moderate	Capable with third party software and hardware	24 V 40 Ah battery will last 5 days	Manufacturer: Scotland-based Low shipping costs, short transport time Backup cost: Low (£10 000-20 000)	Periodic removal required
Video counters	Fish pass	Installation: 2 ppl for 1 day Removal: 2 ppl for 1 day	Moderate	Limited to no capability due to large data files	Mains power	Replacement in-season Backup cost: Low (£5000)	Periodic removal required
	Diversion fence	Installation: 2-4 ppl for 2 days Removal: 2-4 ppl for 1 day	High	Limited to no capability due to large data files	Mains power	Replacement in-season Backup cost: Low (£5000)	Up to daily removal required

Notes. All sites were assumed to be 20 m wide, have road access for installation and removal, and have a 10 m-long diversion fence (where applicable). In-season maintenance costs are related to the debris load and structure: Low (little to no removal required), Moderate (periodic removal required) and High (up to daily removal required). Installation and removal time are in people days (ppl).

4.1.1 Optical Beam Counters

Maintenance

Vaki Riverwatcher fish counters are usually associated with fish passes or fish fences, and generally require minimal to moderate maintenance once installed.

Annual Costs

Equipment is typically installed prior to fish migration periods, and removed during periods of fish absence to avoid damage when not in use. Installation and removal normally takes 2 to 4 people one day. A typical installation scenario for Riverwatcher counters would be at a fish pass with good road access. Pre-built sensor units are installed in the fish pass using a crane truck. Depending on the location (easy access versus remote access) and complexity of the site (small fish pass versus full span fence), installation could take one to several days and will be site-specific. Counters, sensors and fences will also require maintenance and repair periodically (i.e., annually) to ensure that equipment is in good working order for the following season. Costs vary depending on the level of maintenance required and the location of the manufacturer. Vaki, for example, is located in Iceland, while Aquantic and SSE are located in Scotland. Costs of shipping equipment can be substantial (up to £300) and should not be overlooked.

Operational and In-season Cost Considerations

Vaki Riverwatcher counters typically require weekly visits to ensure proper operation, check for debris issues and to collect data and video validation data if there is no remote access. Depending on location, safety issues and access can require 1 to 2 technicians for half of a day.

In-season maintenance may be required if sensors and fence panels are damaged by debris. Risk of vandalism can also be high in more populated sites. Operators who responded to the questionnaire indicated costs ranging between £3000 to £10 000 annually to install, operate, download and review data from optical beam counters ( Appendix 2). The highest cost was associated with labour, followed by equipment and travel. Respondents identified ongoing maintenance as the greatest cost by task. Larger budgets were associated where high debris loads required regular removal. The following sections present cost and reliability considerations applicable to Vaki counters:

Remote Downloading

Vaki counters can be monitored, and have data downloaded, remotely. This reduces the need for more frequent site visits and increases the reliability of counter operation. Costs associated with remote downloading are low compared to weekly site visits despite an initial investment in communication equipment. Additional data transfer costs can vary (i.e., cell phone or hard wired internet access) but are relatively inexpensive.

Power Supply

Two power supplies are used for Vaki counters: mains and battery power. Mains power is the lower, and more reliable, cost option. Mains power eliminates the need for regular site visits, especially if coupled with remote downloading capabilities. Conversely, batteries require an upfront equipment cost and weekly site visits to change batteries. Vaki counters typically require 0.2 Ah per channel to operate and a 12 V 40 Ah battery would operate the counter for approximatley 8 days. Batteries need to be replaced every 2 to 3 years.

Equipment Malfunction

If counter or sensor failures occur, Riverwatcher counters must be returned to the manufacturer in Iceland for repair. This can incur high costs and result in considerable data gaps. Several respondents to the questionaire found the high cost of repairs and the need to ship equipment to Iceland problematic. One strategy is to purchase an after-sales service contract offered by Vaki; the cost of the service contract is dependent on distance from Iceland and what level of service is required by the operator. A basic service contract included with Riverwatcher is £850. Data gaps can be prevented by purchasing backup counter and sensor equipment (£27 000) on-site.

4.1.2 Hydroacoustic Counters

Maintenance

Multibeam (e.g., DIDSON 300, or BlueView M900-2250) and splitbeam sonar counters (e.g., DTX-Echosounder) have similar maintenance requirements. Depending on the site, hydroacoustic counters can be used with or without a structure. Multibeam sonars are generally used in conjuction with a diversion fence that forces fish to migrate through a constrained portion of channel rather than the entire streamwidth. A typical multibeam installation would include a diversion fence (e.g., purpose built picket fence across a portion of the stream width). Assuming the site had road access, we estimate it would take 2 to 4 people approximately 2 days to install. Depending on the location, road access and site characteristics (i.e., streamwidth, expected discharge and debris loads), installation could take several more days with the same number of staff. Splitbeam counters have a longer range than multibeam and may not need a diversion fence, but require a specific stream profile (i.e., a consistent angled bed profile).

Annual Costs

Hydroacoustic counters are typically installed prior to the migration period and removed when fish are not migrating. After removal, equipment should be cleaned and allowed to dry before storing in a dry location. Maintenance can be done during the lowest migration period if migrations are year round. Sonar counters may require maintenance and repair annually during the off-season which can require the user to return equipment to the manufacturer. This may be impracticable due to costs and time constraints and is at the discretion of the user. All hydroacoustic counter manufacturers are located in the United States. Diversion fences will also require annual maintenance during the off-season to ensure structures are in good working order for the next season.

Operational and In-season Cost Considerations

Operation costs for hydroacoustic counters vary depending on the requirements of the operator, but are generally higher than optical beam or resistivity counters. Depending on location, safety issues and site, operators will typically have 1 to 2 technicians on site daily for a half day to maintain equipment and collect data.

In-season maintenance requirements are generally low. Multibeam sonar counters have lenses that must be cleaned periodically depending on sediment loads. Splitbeam sonar counters may require cleaning or replacing of transducers. Other common requirements include changing the trajectory due to water-level fluctuations and debris removal from partial diversion fences, however this can be incorporated into daily site visits. Operators who responded to the questionaire indicated expenses ranging from £50 000 to £80 000 annually to install, operate, and download and review data for sonar counters ( Appendix 2). The highest cost was associated with labour, followed by equipment and travel. Respondents identified data collection and processing as the greatest cost by task. Backup equipment, in the case of equipment malfulction, can cost £22 000 for BlueView P900-2250 and £48 000 for a DIDSON 300. The following sections present cost and reliability considerations applicable to hydroacounstic counters:

Remote Downloading

Sonar counters have an external computer that communicates operational settings and stores recorded data. Data files are large (0.5-1 GB per hour) and downloading remotely is not a cost-effective method given that daily visits are required and there is little additional benefit.

Power Supply

Two power supplies are typically used for hydroacoustic counters which have high power requirements (2.5-12.5 Ah). Mains power or battery banks charged by generator, solar or micro-hydro generator are used. Mains power is more reliable and requires less maintenance by staff. Systems relying on battery require upfront setup and equipment costs, and regular site visits to ensure operation. Batteries need to be replaced every 2 to 3 years.

Debris Load

Sonar counters are typically used with diversion fences that require frequent visits by staff to remove debris. As the majority of operators are visiting the site daily to retrieve data, debris removal is not neccesarily an extra expense. High discharges and sediment load can also interfere with the effectiveness of sonar counters, but this is something that is beyond the operators' control.

4.1.3 Restivity Counters

Maintenance

Resistivity counters (Logie 2100C, Mark 12 counter) are usually associated with Crump weirs, tubes, flat pads or flume sensor structures which all require minimal maintenance once installed.

Annual Costs

Annual maintenance is contingent on the site. If a Crump weir (high initial capital cost) is installed at the start of a project, annual maintenance is quite minimal and consists up to 1 day of work for 1 to 2 people testing wiring, examining electrodes and other in-river equipment for damage. Generally this equipment is quite durable and should not require major repairs or replacement for at least 10 years if properly designed and installed.

If a tube or flat pad is used, it is typically installed just prior to the migration period of the fish being enumerated and removed during periods of fish absence to avoid damage when not in use. Installation and removal for both tubes and flatbeds should take 2 to 4 people one day. A typical install scenario for a flat pad would be in a small- to medium-sized river with vehicle access (max width 25 m and uniform depth of 1 m) where pre-built equipment would be installed and anchored to the stream bottom. Tube sensors are typically installed at the top of a fish pass using a crane truck; also requiring good road access. Depending on the location (remote access) and complexity of the site, installation could take several days longer and will be site-specific.

Operational and In-season Cost Considerations

Resistivity counters typically require weekly site visits to evaluate proper counter operation, check for debris issues, retrieve images, and download data if there is not remote download capabilitites. Depending on location, safety issues and access, site visits require 1 to 2 people for half of a day.

Wiring from the counter to the Crump weir electrodes can be damaged by debris during flood events and electrodes can also be displaced. These events can be expensive to repair (£3000), resulting in lost operational time. The cost would be much less to repair the same damage for flat pads and tube sensors because they can be removed from the river or fish pass (£1000). There is also the risk of counter failure in-season due to power outages, lightning strikes, and data overloading the counter. It is recommened that backup equipment is available or an assurance from the manufacturer that replacement or loan equipment will be made available during repairs.Operators who responded to our questionaire indicated expenses ranged from £3000 and £10 000 annually to install, operate, and download and review data from resistivity counters. The highest cost was associated with labour, followed by equipment and travel. Respondents identified counter ongoing maintenance as the greatest cost by task. The following sections present cost and reliability considerations applicable to resistivity counters:

Remote Downloading

Resistivity counters can be monitored, and have data downloaded, remotely, reducing the need for site visits and increasing counter reliability. Costs associated with remote downloading are low compared to weekly visits. There is an initial investment in communication equipment, and ongoing costs that are dependent on what communication platform is used (i.e., cell phone or hard wired internet access).

Power Supply

Two power supplies are used for resistivity counters: mains and battery. Mains power is the lower cost option and more reliable. Mains power eliminates the need for regular site visits, especially when coupled with remote downloading capability. Conversely, batteries require an upfront equipment cost and, at minimum, weekly site visits to change batteries. Resistivity counters typically require 0.3 Ah per channel to operate (i.e., a 12 V 40 Ah battery would last approximatley 2.5 days). Batteries need to be replaced every 2 to 3 years.

Equipment Malfunction

If counter or sensor failures occur, certain repairs can occur without shipping to manufacturer. Circuit boards and wiring within the counter can be replaced onsite with support from the manufacturer. Sensors and wiring can also be repaired as they are usually built by a local third party with common materials. Having backup counters (£10 000 for Logie, £20 000 for Mark 12) and sensor equipment on-site can avoid compromised data when failure occurs.

Debris Load

Resistivity counters are typically used with Crump weirs, flat pads, or in conjunction with fish passes where debris do not typically accumulate. Thus, site visits to address this issue are infrequent.

4.1.4 Video Counters

Maintenance

Video counters generally require a moderate level of maintenance compared to the other counter technologies. Depending on the site, video counters can be used with or without a structure, but are typically used in conjuction with a diversion fence or fish pass that constrains fish migration through a smaller channel area. A typical install of a purpose-built picket fence across a portion of stream width at a site with road access would take 2 to 4 people approximately 1 to 2 days. Conversely, depending on the location (remote access) and complexity of the site (i.e., streamwidth, expected discharge, debris loads, etc.), installation could take several days longer and will be site-specific.

Annual Costs

Video counters are typically installed prior to fish migration periods and removed when fish are not present. After removal, equipment must be cleaned and allowed to dry before storing in a dry location. Video counters may require periodical replacement, maintenance or repair. Generally, video equipment is inexpensive compared to other counter technologies; digital video recorders are £100-500 (can record video from up to 8 cameras at a time) and video cameras are £50-500 (per camera) depending on desired camera mounting locations (underwater versus above the channel). Repair is usually not an economical option due to the low price of equipment favoring full replacement. Diversion fences will also require periodic maintenance and repair (i.e., annually) to ensure structures are in good working order for the following season. These costs are variable and difficult to estimate.

Operational and In-season Cost Considerations

Operational costs for video counters vary depending on requirements of the operator, but are generally similar to optical beam and resistivity counters. Depending on location, safety issues and access, site visits require 1-2 technicians on site weekly for a half day to maintain equipment and retrieve data. In-season maintenance requirements are generally low. Most requirements are cleaning or replacing cameras, adjusting camera angles due to water-level fluctuations, removing debris from in front of lenses, cleaning flash boards, and cleaning and maintaining diversion fences. As with all counter technologies, there is the risk of video counter failure in-season. Backup equipment is recommended; this is a feasible option for most budgets due to the low cost of this equipment. Operators who responded to the questionnaire indicated expenses between £20 000 and £50 000 annually to install, operate, and download and review data from video counters. The highest cost was associated with labour, followed by equipment and travel. Respondents identified analysis as the greatest cost by task. This specifically refers to the cost associated with reviewing all video footage in order to count fish. The following sections present cost and reliability considerations applicable to video counters:

Remote Downloading

Video counters typically store recordings directly on digital video recorders ( DVR) with removeable hard drives. Data files are large (typically 3 to 9 GB per day) and remote downloading is not an efficent option. Remote downloading is not cost-effective and given the high storage capacity of most DVR units (up to 4 TB) is not necessary.

Power Supply

The same power supply options are available for video counters: mains and battery. Mains power is more reliable and requires less maintenance by staff. Batteries require an upfront equipment cost and require regular site visits to change batteries, at minimum, every second day. Batteries need to be replaced every 2 to 3 years.

Equipment Malfunction

If equipment failure occurs, video equipment can be replaced in-season at a low cost compared to other technologies. Therefore, backup equipment should be an integral part of the budget, minimizing the risk of data loss.

Debris Loads

Video counters are used in conjunction with two structures: diversion fences and fish passes. A diversion fence requires frequent visits by staff to ensure that debris load is not affecting the counter operation. Diversion fences require more frequent visits, every 1-2 days, and therefore increase cost of operations accordingly. High discharge and sediment load can also influence the effectiveness of video counters, but this is something that is beyond the operators' control.

4.2 Validation Costs

Validation of data is key to assessing counter performance. More specifically, validation can be used to evaluate the accuracy, precision, bias and uncertainty in abundance estimates generated by a counter. One of the major challenges to validation is the number of person hours required. We use simulations to examine the trade-off between validation effort and the uncertainty in population estimates. We then show how validation data can be used to achieve acceptable levels of uncertainty in population estimates. Finally, we provide recommendations on validation requirements given different levels of uncertainty.

Introduction

Validation is essential when assessing the quality of data produced by fish counters. Unfortunately it is uncommon, where 3 out of 7 of our survey respondents reported annual validation efforts ( Appendix 2). Validation involves quantifying the error rates in counter classification. These error rates are comprised of two important sources; sensor accuracy and species identification. Sensor accuracy is important for all counters and is calculated using type one (false positive) and type two (false negative) error rates. The accuracy of determining species identification is separate from sensor accuracy and is only required when two or more species overlap in their migration timing.

There are a wide variety of validation techniques, the gold standard being cross comparisons of count data to video observations of the number and species of fish. This is a true form of validation because it is independent of the counter, and can provide information on the true abundance and species composition of fish. Limitations of this technique include the requirement of high water clarity to view all fish and the ability to collect video at night, both of which can be challenging. Pseudo-validation refers to a method that does not represent the actual process being counted, or methods that are not independent of the counter. This includes methods such as dragging a dummy fish over the electrodes of a resistivity counter to interpret signal traces. This will trigger a signal for most counters but the dummy fish may differ, albeit only slightly, from a real fish migrating over the counter. Signal traces from Logie counters can also be used to validate counts but are a form of pseudo-validation because they are not independent of the counter and cannot detect false negatives (i.e., a fish passed through the counter but was not detected). Nonetheless, most pseudo-validation methods can provide adequate assessments of counter performance.

Other counters can also be used to validate the counter of interest. Although this is another form of pseudo-validation, it may be necessary due to river conditions that prohibit the use of video validation. For example, IFR used BlueView multibeam sonar to validate a flat pad resistivity counter in a remote location of British Columbia. The river was too wide and water clarity was too low to consider video validation. The main concern was that both counters may have missed fish, however it was assumed that the occurrence of missed events by both counters would have been extremely low. One of the major challenges is determining appropriate validation requirements. For example, increasing validation effort will decrease the uncertainty of abundance estimates but will also cost more time and money.

Here we used a series of simulation studies examining the effect of sensor accuracy and species composition to evaluate the trade-off between validation effort (i.e., cost) and uncertainty in abundance estimates. We simulated three counter scenarios: (1) a single species model with consistent counter accuracies of 70%, 80% and 90%, (2) a single species model with variable counter accuracies (50% or 80%), and (3) a two species model with both consistent and variable counter accuracies. We characterized validation effort as the number of fish validated (e.g., the number of fish observed on video) and assessed uncertainty in abundance estimates using three performance metrics; accuracy, precision, and bias.

Methods

For all simulations, we started with a known number of fish that moved up and down over the counter; this represents the true abundance of the population (up fish minus down fish). To put this into a validation context, this would be the number of fish observed on a video had video of the entire migration been reviewed. We termed this column of data video, where each up and down count was a row with a value of 1. We then specified the counter accuracy (e.g., consistent accuracy where 80% of the fish were correctly detected by the counter; variable accuracy: where 50% or 80% of the fish were correctly detected by the counter). We did not consider false positive up counts because these are typically rare for a well-sited counter and therefore would have little influence on the recommended validation effort. The counter accuracy was applied to the known, true abundance to back calculate the number of counts correctly recorded. A column named counter contained these data, where a value of 1 was given to a fish that was correctly counted by the counter and 0 if it was missed. A species ratio was applied to all records (observed and not observed by the counter) to divide the counts into two species. Under the column species, a value of 1 was assigned to the target species and 0 for other species. Table 4.2 provides an example of a typical validation dataset sampled to determine the relationship between validation effort and uncertainty.

To simulate the effects of different efforts on the uncertainty estimates derived from validation data, we randomly subsampled the validation dataset and calculated abundance estimates which were compared to the true abundance. Validation effort is the number of fish drawn from the population to estimate the counter accuracy and ranged from 50 to 1000 fish in increments of 50. Validation in the simulations is the equivalent to comparing the number of fish observed by video passing over the counter to the number of up counts observed by the counter. We did not validate more than 1000 fish due to extremely low error rates beyond this validation effort (i.e., < 3%). For each subsample (i.e., sample for each validation effort) we estimated the counter accuracy and species ratio by drawing from a binomial beta distribution using Monte Carlo simulations. We used a binomial beta distribution because the validation data were made up of 1's and 0's (binomial distribution component - 1's were given to fish observed and 0's were given when fish were not observed) and were used to estimate the proportion of counts (beta distribution component) that were correctly classified. We used the mean values and the posterior distributions of estimated counter accuracies and species ratios to calculate mean abundance estimates and levels of uncertainty.

Table 4.2. Example of a validation dataset. Date_time is a record of when a fish passed through the counter, channel is the channel it passed through, direction is the direction of fish movement (U=up, D=down), counter is whether or not the counter recorded the fish (1=recorded, 0=missed), video is whether or not each record was observed as a fish on the video (1=fish, 0=no fish), and species is whether or not the fish was the target species (1=target species, 0=other species). Note that date_time and channel were not used in simulations; these variables are presented to illustrate a typical validation dataset.

date_time	channel	direction	counter	video	species
13-08-01 19:54	2	U	1	1	1
13-08-01 19:54	2	U	1	1	1
13-08-01 20:01	3	U	1	1	0
13-08-01 20:01	3	U	1	1	1
13-08-01 20:03	1	U	0	1	1
13-08-01 20:04	2	D	1	1	1
13-08-01 20:04	2	D	1	1	1
13-08-01 20:04	3	U	1	1	1
13-08-01 20:04	3	U	1	1	1
13-08-01 20:04	2	U	1	1	0

Single Species Model - Constant Accuracy Scenario

The simplest scenario included three single species scenarios that differed in up count accuracy (70%, 80%, and 90%). We simulated only one population and used a down count accuracy of 80% for all scenarios. We determined the accuracy by subtracting the estimated values from the known abundance for each random sample for all validation sample sizes and calculated the root mean squared error. Accuracy is a measure of how close an estimate is to the true abundance (i.e., a combination of precision and bias). Precision was determined by subtracting the estimated values from the mean of all estimated values and calculating the root mean squared error. Precision is a measure of how repeatable an estimate is. Bias was calculated as the raw error (i.e., subtracting the known abundance from the estimated abundance). Bias is a measure of whether or not estimates are consistently higher or lower than the true abundance. All plots present the mean percent relative errors and 1 standard deviation.

Single Species Model - Variable Accuracy Scenario

To evaluate a more complex scenario, we allowed the accuracy to vary throughout the migration. To do this we simulated a population with a migration timing distribution, which was based on the migration characteristics of the North Esk (data provided by I. Simpson, MSS). When daily migration rates were within 25% of the maximum fish migration rate observed, the accuracy of the counter decreased from 80 to 50%. Although we used migration rate to alter counter accuracy, this scenario is representative of many other factors (e.g., changes in water depth) that may affect counter accuracies. We simulated this scenario 100 times (i.e., we created 100 populations). This allowed us to incorporate the stochasticity of variable accuracies because the population and migration characteristics changed with each simulation.

Two Species Model - Variable Accuracy Scenario

To further increase complexity, we explored a two species model with variable accuracy. This is the most likely scenario for salmon-bearing rivers in Scotland. The methods were identical to single species - variable accuracy simulations, except we altered the species proportion to reflect a two species system. We applied a known target species proportion of 0.8.

Results

All simulation scenarios showed that there are diminishing returns on increased validation effort. In other words, as more fish are validated the corresponding uncertainty of population estimates decreased at a slower rate.

Single Species Model - Constant Accuracy

Lower accuracy counters require more validation than higher accuracy counters to achieve the same percent relative error in the accuracy and precision of abundance estimates. Uncertainty for all three metrics can be high at low validation sample sizes and for lower up count counter accuracies ( Figure 4.1). Fish abundances were always slightly overestimated, as indicated by positive and low percent relative bias values (mean estimate always < 4%).

Figure 4.1. Plots showing the trade-off between validation effort (the number of fish validated) and error in abundance estimates for a single species at a constant accuracy (70% - grey, 80% - blue, 90% - dark blue). Error in abundance estimates are reported as the percent relative error in a) accuracy, b) precision and c) bias. Accuracy is a measure of how close an estimate is to the true abundance (i.e., a combination of precision and bias). Precision is a measure of how repeatable an estimate is. Bias is a measure of whether or not estimates are consistently higher or lower than the true abundance. Points indicate mean values and vertical bars represent the standard deviation.

Single Species Model - Variable Accuracy

The variable accuracy scenarios require more validation to achieve the same level of certainty as the 80% constant accuracy scenario ( Figure 4.2). Although the mean accuracy of the variable scenario is approximately 0.7, the uncertainty is much higher than the constant 0.8 accuracy.

Figure 4.2. Plots showing the trade-off between validation effort (the number of fish validated) and error in abundance estimates for a single species - variable accuracy (80% - grey) and variable accuracy (50-80% - blue). Error in abundance estimates are reported as the percent relative error in a) accuracy, b) precision, and c) bias. Accuracy is a measure of how close an estimate is to the true abundance (i.e., a combination of precision and bias). Precision is a measure of how repeatable an estimate is. Bias is a measure of whether or not estimates are consistently higher or lower than the true abundance. Points indicate mean values and vertical bars represent the standard deviation.

Two Species Model - Variable Accuracy

There is an increase in the mean percent relative error and variance for accuracy and precision when two species are validated for both scenarios ( Figure 4.3). The difference between the two scenarios is consistent with the single species versions, which suggests that adding a species to the model has the same impact on uncertainty in both cases. The percent relative bias tends to be slightly less negative for the variable scenario compared to the consistent scenario.

Figure 4.3. Plots showing the trade-off between validation effort (the number of fish validated) and error in abundance estimates for a two species - constant accuracy (80% - grey) and variable accuracy (50-80% - blue) scenario. Error in abundance estimates is reported as the percent relative error in a) accuracy, b) precision, and c) bias. Accuracy is a measure of how close an estimate is to the true abundance (i.e., a combination of precision and bias). Precision is a measure of how repeatable an estimate is. Bias is a measure of whether or not estimates are consistently higher or lower than the true abundance. Points indicate mean values and vertical bars represent 1 standard deviation.

Discussion

We used simulations to explore the trade-off between validation effort and multiple metrics of uncertainty in abundance estimates for different counter scenarios. Simulation scenarios range in complexity from a single species - constant counter accuracy model to two species with variable counter accuracy. We found diminishing returns in reductions of uncertainty with increased validation effort across all simulation scenarios. Using this information, we provide recommendations about the number of fish that should be validated given a suite of counter-population characteristics.

There was a decreasing non-linear relationship between validation effort versus both accuracy and precision. In other words, doubling validation effort will not reduce the accuracy of population estimates by a half. We also observed a similar relationship between validation effort and bias except that bias was nearly zero in all cases once 200 fish had been validated. These diminishing returns can help identify an optimal validation effort for a given set of counter conditions. The number of fish validated may be driven by cost in which case our results provide general guidelines for the expected level of uncertainty in abundance estimates. However, if a target level of uncertainty is identified, our results can determine the number of fish to be validated.

The main differences among scenarios were increases in the mean and variation of accuracy and precision as complexity increased. This could be due to the fact that variable accuracy scenarios were conducted over many populations (i.e., 100), whereas the constant counter accuracy scenario represented a single population. We ran the variable accuracy scenarios 100 times due to the high variability in accuracy estimates from one simulation to the next. For example, each simulation produced a unique run timing and migration rate (fish per day) that influenced counter accuracy; higher migration rates led to lower counter accuracy and lower migration rates resulted in higher accuracy. The lower and upper counter accuracy values were set a priori to be 50 and 80%, respectively. This additional variability was observed in the variation around the mean estimates for both variable counter accuracy scenarios ( Figure 4.2 and 4.3). This highlights that the variability in counter accuracy is as important as the mean counter accuracy.

One of the key points from meetings and site visits in Scotland is that simple simulations (i.e., single species - constant accuracy) would not capture the complexity of salmon populations in Scotland. For instance, the assumption that counter accuracy is constant during a migration is not valid for many counters. The accuracy of flat pad and Crump weir resistivity counter setups shift with water depth, whereby deeper water tends to reduce accuracy. Hydroacoustic applications are also susceptible to changes in water levels. These counters are setup to operate at a given water level and need to be adjusted when water levels change. However, there are many counters where constant accuracy models could be applied, such as a resistivity counter with tube sensors or a Vaki Riverwatcher counter. Both of these counter setups can have extremely high and stable accuracies, even at high migration rates. Also, these setups are typically used in fish passage structures or controlled waterways and need to be submerged so accuracies are not often affected by discharge. We assume that the standard deviations for most counter applications will be greater than the standard deviations we report for the constant accuracy simulations and that they are an over simplification of actual uncertainty. While we address this issue by increasing the complexity of the simulation models, it is important to note that our results are likely minimum estimates of uncertainty for a given number of fish validated. Furthermore, the presence of more than one species requires increased validation effort to achieve the same level of uncertainty. This is intuitive because the number of target fish validated is reduced (from the total number of fish) when there are other species present. For this, we assumed that species proportions were consistent throughout the migration, which is likely an invalid assumption.

4.3 Population Estimates with Uncertainty

Assessing uncertainty of population abundance estimates is crucial for effective fisheries management. This requires knowledge of the different sources of counting error, which were discussed and quantified in the previous section (Chapter 4.2 Validation Costs). Here we describe novel methods for incorporating different sources of counter error into estimates of population abundance. For instance, fish passage abundance estimates are often calculated by expanding the number of net up counts by the estimated accuracy of the counter. For example, if a counter produces 120 fish counts (110 up counts - 10 down counts = 100 net up counts) and it has been estimated that the counter detects 90% of the fish moving in both directions, then the total number of fish upstream of the counter is estimated to be 111. Although the estimated counter accuracy is high, the true accuracy of the counter is unknown and can only be determined by validating all fish passage events (Chapter 4.2 Validation Costs). Validating all fish passing over a counter, however, is seldom done and is not cost-effective.

Calculations of fish passage abundance rarely account for uncertainty in counter accuracy estimates. The goal of this analysis was to account for uncertainty in estimating fish passage numbers. We used Monte Carlo simulation methods to account for uncertainty in up and down count accuracy and the accuracy of identifying proportions of target species that pass over counter sensors. This procedure simulates the uncertainty in counter sensor accuracy and species proportion given information about counter accuracy derived from the previous validation data. The more fish passage events that are validated, the narrower the distribution of accuracies and the more certain the estimate. We illustrate this analysis using simulation data from the previous section (Chapter 4.2 Validation Costs).

We used Monte Carlo simulations to create probability distributions to account for error in counter accuracy and species identification. Validation data were composed of 1's (correctly counted fish or target species 1) and 0's (not counted or other species); therefore samples were drawn from a binomial beta distribution:

Equation 4.3.1

where, π is the proportion of records correctly counted, B is the binomial beta distribution, α and β are the alpha and beta parameters, respectively. Prior distributions can be used to inform sampling by specifying the alpha and beta parameters; otherwise both parameters are set to 1 for an uninformative prior, which is what we used for all simulations. Posterior distributions were simulated in R statistical software (R Development Core Team, 2012) using the MCbinomialbeta function in the 'MCMCpack package'(Martin et al. 2013). Each posterior was comprised of 500 samples drawn from the binomial beta distribution.

We used the posterior distributions to calculate abundance estimates and uncertainty. First, the number of up counts for the target species was calculated as:

Equation 4.3.2

where, is the posterior distribution for net up counts that are target species, is the total number of up counts for both the target species (1) and non-target species (2), is the number of target species up counts validated, is the posterior distribution of the up count accuracy, and is the proportion of counts that are the target species. The total number of down counts for species 1 are calculated using the same methods as the total species 1 up count estimates and are subtracted from the total species 1 up counts to get the net up counts for species 1.

We present the posterior distributions for counter accuracies, species proportions and abundance estimates for species 1. We also present the 2.5 and 97.5% credible intervals for the posterior distribution of abundance estimates. Figure 4.4 shows the posterior distributions for counter accuracy of simulated populations (A-B), species proportions (C-D), and abundance (E) using data from Chapter 4.2 (Validation Costs). The estimated distributions ( Figure 4.4A-D) are used to calculate the distribution of estimated fish passage numbers ( Figure 4.4E).

This method allows the calculation of common estimates of uncertainty such as 95% credible intervals and standard deviations. Another major advantage of this method is that prior information can be used to inform counter accuracy and species ratios. While the use of priors can be problematic and depend on the information being used to determine the prior (Gelman 2002, Kuhnert et al. 2010), they can be extremely valuable when specific counter accuracy information is lacking in a given year or a given site. In some systems, for example, it is difficult to validate a large number of fish moving downstream. Estimates of down count accuracy from small sample sizes can be misleading and have a large influence on population estimates when the ratio of down counts relative to up counts is high. Using prior information about how a counter setup should operate can provide more reliable estimates of counter accuracy in the absence of abundant validation data. It is important to note that the uncertainty this method incorporates may not account for all uncertainty and will always be an underestimate.

Figure 4.4. Posterior distributions of A) counter up and B) down accuracy, C) species proportions moving upstream and d) downstream, and E) estimated spawner abundance. Uniform prior distributions were used for A-D. The estimated spawner abundance represents the number of fish that passed over the counter while accounting for uncertainty in counter accuracy and the proportion of fish that were identified as the target species. The red line is the true abundance and the solid black line is the mean estimated spawner abundance. The broken black lines indicate the upper and lower bounds of the 2.5 and 97.5% credible interval.

Recommendations and Costs

One consideration not addressed in these simulations is how to sample video to ensure a particular number of fish will be validated. We suggest a stratified random sampling method whereby a fixed number of hours of video are viewed each day, which is divided into 5-10 random samples for each day. For example, an hour of video was reviewed each day; it could be divided into 6 randomly selected 10 min sections. This sampling method works well because it distributes the number of validated fish according to the run timing distribution. For example, during the peak migration rates, the largest number of fish will be validated, resulting in estimates of counter accuracy and species proportions when the largest number of fish are passing over the counter. This method works best for fish passage rates > 1 fish per hour but may not work with low passage rates < 1 fish per hour because the time it would take to observe the required number of fish may be excessive.

Our simulation study can be used to produce guidelines on validation requirements under a given set of conditions and for a desired level of uncertainty. However, associated cost estimates require specific information on the total number of fish that need to be validated (based on counter-population conditions) and the amount of time it will take to review these fish on video. We estimated validation time as follows:

Equation 4.3.3

where, is the time to validate in hours, is the known number of fish to be validated, is the abundance of the population, and is the number of hours from the beginning to the end of the migration in hours. is a constant applied to the hours of video footage which accounts for setting up video footage, measuring fish lengths (where applicable), and entering data. We estimate this constant to be approximately 1.1 based on average fish passage rates of 1-3 fish per hour.

Our estimate of validation time makes a number of assumptions that could be altered based on the characteristics of the population being validated. For example, average fish passage rate (per day) may not be applicable to all populations because some fish species, or populations, may migrate predominately at night. Video review could focus on nighttime hours and could reduce by half. should be larger for higher fish passage rates to account for the additional time required to record fish. Identifying the number of fish to validate also depends on the level of uncertainty a manager is willing to accept. For example, Table 4.3 shows minimum validation samples sizes required to achieve population estimates that are within 5 and 10% of the true abundance. If we consider a counter scenario with variable accuracy (0.5 or 0.8) and one species, the number of fish requiring validation to achieve < 5% relative error in the accuracy of abundance estimates is four times that of number required to achieve < 10% relative error. Acceptable levels of uncertainty are important considerations determining annual validation costs.

Table 4.3. The minimum sample size and validation time required to attain < 5% and < 10% relative error for accuracy, precision and bias. Validation time is calculated using Equation 4.3.3 with a migration period of 4 months, a population abundance of 2000 fish, and a video constant of 1.1.

Metric	% relative error	Counter accuracy	Number of Species	Minimum sample size (Mean and (1SD)	Minimum time (hrs) (Mean and (1SD)
a) Accuracy	5	0.7	1	150 (350)	115 (267)
	5	0.8	1	100 (200)	76 (153)
	5	0.9	1	50 (100)	38 (76)
	5	0.5, 0.8	1	200 (600)	153 (458)
	5	0.5, 0.8	2	300 (1000)	229 (764)
	10	0.7	1	50 (100)	38 (76)
	10	0.8	1	50 (100)	38 (76)
	10	0.9	1	50 (50)	38 (38)
	10	0.5, 0.8	1	50 (150)	38 (115)
	10	0.5, 0.8	2	100 (250)	76 (191)
b) Precision	5	0.7	1	150 (350)	115 (267)
	5	0.8	1	100 (200)	76 (153)
	5	0.9	1	50 (100)	38 (76)
	5	0.5, 0.8	1	200 (600)	153 (458)
	5	0.5, 0.8		300 (850)	229 (649)
	10	0.7	1	50 (100)	38 (76)
	10	0.8	1	50 (100)	38 (76)
	10	0.9	1	50 (50)	38 (38)
	10	0.5, 0.8	1	50 (150)	38 (115)
	10	0.5, 0.8	2	100 (250)	76 (191)
c) Bias	5	0.7	1	50 (250)	38 (191)
	5	0.8	1	50 (150)	38 (115)
	5	0.9	1	50 (100)	38 (76)
	5	0.5, 0.8	1	50 (350)	38 (267)
	5	0.5, 0.8	2	50 (300)	38 (229)
	10	0.7	1	50 (100)	38 (76)
	10	0.8	1	50 (50)	38 (38)
	10	0.9	1	50 (50)	38 (38)
	10	0.5, 0.8	1	50 (100)	38 (76)
	10	0.5, 0.8	2	50 (100)	38 (76)

References

Gelman, A. 2002. Prior distribution. In Encyclopedia of Environmetrics. Edited by A.H. El-Shaarawi and W.W. Piegorsh. Chichester. pp. 1634-1637.
Kuhnert, P.M., Martin, T.G., and Griffiths, S.P. 2010. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecology Letters 13(7): 900-914. doi: 10.1111/j.1461-0248.2010.01477.x.
Martin, A.D., Quinn, K.M., and Park, J.H. 2013. Package "MCMCpack." R Statistical Package: 1-162.
R Development Core Team. 2012. R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. http://www.R-project.org/.