September 26, 2005
John West
Environmental Scientist
Regional Water Quality Control Board
1515 Clay St., #1400
Oakland, CA 94612
Dear John:
It is two years since the study of effects of diverting stormwater from
the Vista Grande Canal into Lake Merced, using a riparian buffer to
remove contaminants, was initially proposed. Earlier this month
the first (draft) report on the results of this study was
released*. This report suggests a number of questions (in
addition to that of delay in public reporting) regarding the adequacy
of the study design and analyses. I hope that we can all work
together to assure that these questions are adequately considered
before additional testing goes forward.
Before discussing this report, however, let me again point out that the
question of public health and safety is not insignificant. The
report states, “(F)ull body water contact recreation is prohibited at
Lake Merced,” implying that this consideration may not be
important. However, Water Contact Recreation is a designated
beneficial use, and full body contact does occur, usually
inadvertently, when a kayak, shell or sailboat flips boaters into the
lake. Fishing, which is being encouraged at the lake, is also
considered to be contact recreation. Maintaining a safe boating
and fishing environment is then an important concern.
On to the study design and analyses:
1) The report recognizes that bacteria die-off and dilution are factors
contributing to reduction in coliform between the stormwater feedstock
and the lake. However, no attempt is made to evaluate the
contribution of these factors; we do not, then, know whether the result
would have been significantly different had stormwater simply run
through a pipe and into the lake.
2) Metals were observed in the stormwater, but not in the ground in the
riparian buffer and not in the lake. Where did they go?
Does the absence of metals in the ground indicate that there was little
if any percolation of the stormwater? What is the sustained
injection of metals into the lake, if any? It seems risky to
assume that no contamination has resulted simply because we can’t
figure out what the answers to these questions might be.
3) The report acknowledges the inadequacy of the statistical analysis
that has been conducted: “The monitoring design could address filling
some current data gaps in the project and reducing uncertainty in some
statistical analyses.” My observation is that this is the
inevitable result of not having prepared an adequate study plan in the
first place. This must be corrected by detailing future test
designs much more thoroughly. Simply suggesting that additional
testing might fill some gaps is not good enough.
4) Some description is provided regarding the flora found on the
riparian buffer, and the characteristics of the soil in that
area. However, no data is provided that would indicate the
expected or likely effectiveness of this buffer as a bio-filter, nor is
any analysis provided that would contribute to the understanding of
these effects for future researchers and practitioners.
5) There is some delay between a storm event and the delivery of
treated stormwater to the test site. No analysis is presented
indicating the degree to which the ground has become saturated during
this interval. How much water does the buffer absorb? How
much runs off directly into the lake? I am sure that these
factors can be measured only approximately; however, some effort should
be made. The absence of metals in the soil may indicate that in
fact little of the stormwater is actually being treated by the riparian
buffer.
6) Since average rainfall during diversion events was significantly
greater than during back-ground events the control group did not
accurately reflect the conditions under which the sample group was
tested. Assessment of other differences is then, unfortunately,
largely little more than conjecture. Obviously, it is not
possible to anticipate the size of storm events with full
accuracy. However, some forecast is possible prior to the
event. Rather than select the control group in predetermined
sequence it might be better to use this forecast as one selection
criterion.
7) The report acknowledges the effect of mixing on reduced coliform
counts, and estimates the impact if complete mixing in the entire lake
were achieved. However, as the report also states, complete
mixing within the time period being evaluated is extremely
unlikely. The amount of mixing likely to occur in the zone near
the test site is left unaddressed, (Might observed increase of metals
in the water provide an index of the rate of mixing?), with the result
that any health hazard in the test area is not adequately
evaluated. Further, the data indicates that E-coli concentrations
in the test area have risen to levels near acceptable limits.
(See Figure 3. Probability Plot of E. Coli Concentrations in Lake
Merced During Storm Events)
8) One would need a fuller description of the statistics applied to
adequately comment on the analysis itself. However, these
observations appear to be relevant:
a) Sample sizes seem to be very small,
using a t-test in this environment is generally acknowledged to be a
risky endeavor. For example, as the report states, the t-test
assumes Normality of the data distributions. Testing for
Normality with 3 sample points is impossible, and even 6 sample points
is very suspect. Perhaps some consideration should be given to
grouping this data.
b) The acceptance criterion has been established that results will be
considered significant if the probability of those results occurring as
a random event is less that 5%. While significance is generally
claimed, a great many of the probabilities provided are well away from
this threshold.
c) There are repeated attempts to identify a linear relationship where
it is quite evident none exists. (See especially Figure 4,
Probability Plot of Enterococcus Concentrations and Appendix H,
Surrogate Indicator Regression Plots.)
9) The t-test is appropriate for evaluating a single hypothesis.
It is not, however, appropriate to make a long list of tests to be
conducted, then to use the t-test to evaluate the significance of
each. Obviously, if one is using a 95% confidence limit one can
expect to see one test out of twenty indicating significance when the
process is completely random. It is necessary, then, to apply
some additional filter to assure that the tests indicated as
significant are truly outliers, and do not merely represent the tails
of a Student-t distribution. I see no indication that such a
filter has been applied.
10) Perhaps the question should be asked, Is this difference
important?, not Is this difference statistically significant? For
example, in Figure 3; Probability Plot of E. Coli Concentrations n Lake
Merced During Storm Events, the difference between storm events and
background events may or may not be statistically significant, but it
seems large enough to be important. Further, the claim made that
“Probability plots of total coliform, E. coli and enterococcus
concentrations (Figures 2 through 4) illustrate that bacteria
concentrations were well under single sample maximum concentration
criteria for full body contact recreation” is not the case. In
fact, in the test group E-Coli counts get dangerously close to this
limit. This suggests that if testing is to go forward, especially
with even larger diversions of stormwater into the lake, far better
monitoring will be required. It may be necessary to post the lake
when tests are conducted to assure that boaters and fishermen do not
use the lake during these periods.
As I’m sure you recall, I expressed serious reservations regarding the
test design when this program was proposed two years ago. It now
seems that those reservations were often well founded. Again,
before additional testing goes forward I hope that the issues raised
here will be adequately considered, and that a much more detailed
project plan will be prepared that improves both the analyses and the
protection of public health and safety.
Sincerely,
John Plummer
e.c.: Patrick Sweetland, Manager of Water and Wastewater Services
City of Daly City
Greg Bartow, Hydrologist, San
Francisco Public Utilities Commission
