 |
Friends
of Lake Merced 100 Santa Barbara Avenue Daly City, CA 94014 |
November 29, 2005
Dr. Robert Cooper, Professor Emeritus
School of Public Health
University of California at Berkeley
4863 Venner Road
Martinez, CA 94553
Dr. Cooper:
The North San Mateo County Sanitation District, a Division of the City of Daly City, working in cooperation with the San Francisco Public Utilities Commission, recently released a report describing the results of a two-year study of the effects of diverting water from the Vista Grande canal into Lake Merced. This water was known to be contaminated, both with coliform bacteria and various metals. The purpose of the study, as originally stated in the proposal of January 26, 2004 was:
Phase 2 of the
pilot project …involves using a vegetated area along a small section of
the southwestern shoreline of South Lake Merced as a stormwater
treatment wetland. The primary goal of this phase of the project
is to determine the extent to which water quality in Lake Merced is
impacted by the introduction of Vista Grande stormwater after passage
through the vegetated area.
We have read this report, and find it lacking in several important regards:
1) The design of the study was
inadequate to evaluate the effect of the riparian buffer as a
contaminant filter.
2) The analysis evaluated the probability that an impact would be reported when in fact none existed. The far more important issue, failing to identify a true impact, was not addressed.
3) Metals known to be in the feedstock simply disappeared; no account was made as to their destination.
4) There was no provision for public notice or ongoing monitoring in the event that a condition exceeding EPA standards for pubic health was encountered.
2) The analysis evaluated the probability that an impact would be reported when in fact none existed. The far more important issue, failing to identify a true impact, was not addressed.
3) Metals known to be in the feedstock simply disappeared; no account was made as to their destination.
4) There was no provision for public notice or ongoing monitoring in the event that a condition exceeding EPA standards for pubic health was encountered.
Patrick Sweetland, Manager of Water and Wastewater Resources for the City of Daly City, the manager responsible for this project, referenced you as the expert who has monitored this study and approved this report. Frankly, with errors so obvious I am surprised if you have not commented on these shortcomings. On the other hand, Patrick has not shared your report with us, so perhaps our differences are not so great after all.
I hope that you will bear with me as I expand on the points made above:
Inadequate study design
In as much as the purpose of this study was to evaluate the effectiveness of the riparian buffer as a tool for removing contaminants, it would be necessary to have some information about that buffer in order to accomplish this goal. In fact, no such information was collected, and as a result it is impossible to distinguish the effects of that buffer from such other factors as dilution and coliform die-off.
We indicated a number of observations that should be made. In each case the suggested analysis was “beyond the scope of the pilot study.” Here are a few examples (quotations are taken from the report and from supporting documents provided by the researchers.):
Some
level of treatment by the riparian buffer is likely (see the attached
document prepared by Michael J. Casteel, Ph.D., SFPUC Research
Microbiologist). Additional engineering analysis would be needed
to address this issue further. Such analysis was beyond the scope
of the pilot study.
It is perhaps a matter of opinion the degree to which isolating the effect of the riparian buffer is economically justified. Absent any assessment, however, the observation that “some level of treatment . . . is likely” seems hardly adequate. Would we have been willing to approve putting this contaminated water directly into Lake Merced, a body of water designated by the Public Utilities Commission as an emergency supply of potable water? I doubt it. If not, then, it needs to be more than “likely” that the riparian buffer is having a significant effect.
In any event, with a pure black-box study of this sort, looking only at input and output with no attempt to evaluate the mechanisms operating, failure to achieve replication of initial results is not uncommon. It is certainly not possible to extrapolate outside the range of the data with any confidence. Preparation for future study, with even larger amounts of stormwater diverted into the lake, is then tenuous at best.
Inappropriate statistical analysis
Measurements of the amount of E-coliform bacteria were taken for three storms with no diversion of Vista Grande canal water, six storms with diversion. Surface grab samples were taken from six predetermined sites for each storm, and the amount of E-coli bacteria measured in each sample. Three sample points were located in a row near the shore, three in a row somewhat further away from the discharge point.
In an earlier commentary I observed, less than precisely, “Scientific method dictates that the hypotheses tested should be the opposite of the desired outcome, the so-called null hypotheses. I have not yet read the actual statistical evaluation so perhaps this issue has been addressed. However, the hypotheses as stated are not fully testable.” The response was provided,
We
are not familiar with the null hypothesis being defined as the opposite
of the desired outcome. Our understanding is that a null hypothesis is
a hypothesis that is assumed to be true until statistical evidence
indicates otherwise, and is often the opposite result to what the
experimenter expects. The null hypotheses in the draft report were
fully testable.
A t-test was used to compare the results with and without storm diversion. The results proved not to be significant at the 95% confidence level. That is, the assumption (null hypothesis) was made that no impact resulted from the diversion of contaminated Vista Grande water, and that assumption would have been rejected only if the odds were less than one in twenty of its being correct. The researchers were unable to comprehend the problem with this approach.
The t-test is not an appropriate test for this problem, as it confounds the many potential sources of variation. This is clearly a multi-layered problem, test site within test row, test row within storm, storm within diversion/no diversion family, and family within the total data set. A quick look at the data demonstrates a clear storm impact. Less apparent is the effect of the other potential sources of variation.
To conduct the following analyses I have used the logs of the actual measurements. As the researchers point out, this data is often log-normally distributed, so this may be a prudent step. At the same time, with this data it makes virtually no difference to the analysis whether the original readings or their logs are used.
First I looked to see if there were any site-to-site differences. When looking at the raw data there is no significant pattern across the test sites (Attachment 1). However, this confounds storm-to-storm variance with site-to-site variance. I therefore normalized the data by storm and recalculated (Attachment 2). While removing the storm variance contribution starts to indicate some site-to-site pattern, it remains insignificant at the 90% confidence level. Also, comparing sites near the shore with those further out indicates no strong pattern; one might like to enquire as to why this is the case.
It is mildly interesting to note that when just those six storms during which diversion occurred are considered there is a significant site-to-site pattern, with Site 4 being significantly less than either Site 3 or Site 5 (Attachment 3). However, the inner/outer pattern one might expect still fails to appear. This may be an indication of significant mixing, a result not unexpected at the lake’s surface during storm events.
It seems relatiavely safe, then, to consider the six sites as random samples taken during each storm event, ignoring the site-to-site pattern for this analysis. That is, an unbalanced nested ANOVA is appropriate, with a major split between storms with and without diversion, and individual storms within each of those two categories. The question the researchers posed is this, is there a significant increase in the amount of E-coli bacteria found in the lake for storm events with diversion, after considering the variance attributed by individual storms? The answer is contained in Attachment 4.
The answer is, with 97.5% confidence, yes there is. But that is not the important question. Rather, we want to protect the customer, in this case the boater using the lake. We don’t want to say that everything is OK if in fact it is not. That is, we care about Type II error, the probability of saying there is no difference when in fact a difference exists. We care, then, about the power of the test, which is one minus Beta, the probability of a Type II error. For the diversion/no diversion factor the power is 0.62. There is, then, nearly a 40% chance of being wrong is we say that there is no difference between these two groups of data. That is well outside acceptable limits for protection of public health and safety.
It is bad enough that the researchers were unable to conduct this analysis properly. It is far more discouraging that when this problem was pointed out both to the researchers and to the managers responsible for this study, nobody recognized the nature of this problem, and the need to conduct a proper evaluation. Rather, the researchers claimed, somewhat defensively, “The null hypotheses in the draft report were fully testable.” Well, no they weren’t, but they could have come a lot closer than they did. I’m not ready to trust future testing to a group that does not include a substantially stronger statistical capability.
The disappearing metals.
In an earlier commentary I observed, “Metals were observed in the stormwater, but not in the ground in the riparian buffer and not in the lake. Where did they go? Does the absence of metals in the ground indicate that there was little if any percolation of the stormwater? What is the sustained injection of metals into the lake, if any? It seems risky to assume that no contamination has resulted simply because we can’t figure out what the answers to these questions might be.” The researchers responded:
A
number of physical, biological and chemical processes potentially
govern the fate of metals in the stormwater runoff diverted to the
riparian buffer/lake. Such potential processes include
accumulation in the riparian buffer soils (with any changes in soil
concentrations potentially masked by natural variability), removal by
biological uptake in the buffer or the lake, and adsorption to
particles in the lake system. Transformations among species of
individual metals are also likely. Characterization of the fate
of the metals would require additional monitoring data and engineering
analysis. Such monitoring and analysis were beyond the scope of
the pilot study.
And pigs might have wings.* It is also possible that the diverted stormwater simply ran down the slope and into the lake, with the metals sinking to a level below that captured with surface grab samples. Have the researchers provided a sufficient assessment of the impact on the lake of this major category of contaminants? I don’t think so. Again this reflects on the “black-box” approach that was taken. This approach has proven to be inadequate to safeguard public health at Lake Merced. Future studies should be designed to overcome this shortfall in the analysis.
Provision of public notice
It would seem that at a minimum some notice would be provided to boaters and other users of the lake when these tests are being conducted. Further, a testing protocol should be developed that provides adequate information about any contamination that does occur, and is maintained until the testing is complete and the lake has been restored to a healthy condition. Obviously this applies to E-coli bacteria that will experience relatively rapid die-off in the lake. I am not familiar with similar provisions regarding metals that reach the lake.
We have not had much assistance from the city officials responsible for maintaining water quality in Lake Merced. Jim Salerno, Environment Manager for the SF-PUC, after repeated requests for a written public notice and condition-monitoring plan, produced the following:
Here
is the revised monitoring plan for Lake Merced. The lake
will be monitored every two months for bacteria and other physical and
chemical properties generally from March through July each
year. Monitoring will be increased to monthly in August through
February to coincide with wet weather and the planned diversion of
dechlorinated drinking water to raise the lake level. The purpose
of these surveys is to characterize changing Lake conditions given
these different inputs. Bacteria analyses will be forwarded to
you and the Lake Merced Task Force when completed generally 3-4 days
after the survey. The completed survey results will be forwarded
approximately one month later when all analyses have been
completed. Feel free to contact me to discuss this plan if you
have any concerns.
Note that there is no specific reference to provisions for the stormwater diversion tests, no indication of follow-up procedures to be used if an out-of-tolerance condition is encountered, and public notice is limited to the Lake Merced Task Force and me, neither of us being the public.
The researchers attempt to be reassuring:
(S)tating
that Figure 3 indicates that E. coli concentrations in the test area
have risen to levels near acceptable limits is a misinterpretation of
this probability plot. What is really indicated is that there is
an approximate 98.5% probability that a sample from the population
characterized by the distribution in Figure 3 will have a concentration
lower than the single sample maximum criteria of 576 MPN/100mL for full
body contact recreation.
First, single sample readings of E-Coli bacteria of 520 MPN/100mL were observed on two occasions during this test. That is almost within measuring accuracy of the maximum criteria of 576 MPN/100mL; the statement that observed values were near acceptable limits is completely accurate. Further, that there is just a 1.5% chance of an over-limit condition is not sufficiently reassuring to eliminate the need for a comprehensive monitoring program. None has been produced.
Conclusion
The researchers now propose to increase the quantities of water diverted into Lake Merced. As indicated earlier, a black-box study does little to assist in estimating the effect of performance outside the range, in this case above the range, of the sample data. We have little if any idea if that 1.5% risk will double or increase ten-fold. The researchers do acknowledge this need:
In
any case, as stated earlier, we agree that any future increases in
diversion volume would require vigilant monitoring. Before any
additional diversions to the lake occur, we will carefully design and
document additional monitoring activities.
In addition to a much more detailed monitoring program, these steps should be taken before approval is given to continue:
1) Some provision is needed to assess
the effect of the riparian buffer, and to assure that we aren’t simply
pumping contaminated stormwater into Lake Merced.
2) An analytic framework is needed that will attempt to identify the destination of metals and other contaminants that were not traced in the current study.
3) A monitoring program is needed that includes follow-up testing if and when over-limit conditions are encountered.
4) More effective provisions for public notice are needed. The original proposal promised, “The boating contacts at Lake Merced will be kept informed of the monitoring results during the pilot test on an ongoing basis.” Boaters I know are not aware that this procedure was implemented.
5) Someone with a much better grounding in statistical analysis should be added to the research team. I have described one of the more serious errors made in the current analysis. There are several others, although less important still indicating a lack of adequate understanding of these techniques on the part of the researchers.
2) An analytic framework is needed that will attempt to identify the destination of metals and other contaminants that were not traced in the current study.
3) A monitoring program is needed that includes follow-up testing if and when over-limit conditions are encountered.
4) More effective provisions for public notice are needed. The original proposal promised, “The boating contacts at Lake Merced will be kept informed of the monitoring results during the pilot test on an ongoing basis.” Boaters I know are not aware that this procedure was implemented.
5) Someone with a much better grounding in statistical analysis should be added to the research team. I have described one of the more serious errors made in the current analysis. There are several others, although less important still indicating a lack of adequate understanding of these techniques on the part of the researchers.
I have no idea how serious a health risk may be posed by this activity; hopefully it is not great. However, so long as the research team lacks adequate analytic skills, and the managers responsible for this study lack an interest in following up on suggestions made by technically qualified members of the community, we cannot have confidence that anyone actually knows what risks have been encountered.
Thank you for taking the time to review these comments. I look forward to receiving your response.
Sincerely,
John Plummer
Attachments
Attachment
3: Evaluation of sampling site patterns --
Log10(E-Coli Bacteria) normalized by storm,
Diversion events only
Log10(E-Coli Bacteria) normalized by storm,
Diversion events only
***It has been brought to my
attention that some among my readers have been deprived in their youth
of a familiarity with the writings of Lewis Carroll. I will quote:
"'The time has come,' the Walrus said,
'To talk of many things:
Of shoes -- and ships -- and sealing-wax --
Of cabbages and kings --
And why the sea is boiling hot --
And whether pigs have wings.'"
I had intended in my allusion a relatively light-hearted reference to Wonderland. Lest that be misunderstood let me clarify: I find it no less wondrous that a group of professional engineers would simply lose a major class of contaminants without comment either on the seriousness of this problem, or the steps proposed for its remedy.
Perhaps
Dr. Cooper, being of the professorial persuasion, will have had a more
informative youth. I hope so.