In my last post, I discussed readings that could help improve your knowledge and analytical skills in addressing statistical data. Below is a check list of items to consider summarized from the Manual on Scientific Evidence Third Edition, Reference Guide on Epidemiology. Here is the list:
CHECKLIST OF PROBLEMS WITH THE USE OF STATISTICAL DATA AND ANALYSIS
I. What Sources of Error Might Have Produced a False Result?
A. What Statistical Methods Exist to Evaluate the Possibility of Sampling Error?
1. False positives and statistical significance,
2. False negatives,
B. What Biases May Have Contributed to an Erroneous Association?
1. Selection bias: Selection bias refers to the error in an observed association that results from the method of selection of cases and controls (in a case-control study) or exposed and unexposed individuals (in a cohort study).
2. Information bias: Information bias is a result of inaccurate information about either the disease or the exposure status of the study participants or a result of confounding. In a case-control study, potential information bias is an important consideration because the researcher depends on information from the past to determine exposure and disease and their temporal relationship.
3. Other conceptual problems:
a. Issue or hypothesis is improperly defined: Sometimes studies are limited by flawed definitions or premises.
b. Publication bias: the tendency for medical journals to prefer studies that find an effect. If negative studies are never published, the published literature will be biased.
c. Financial bias / Conflicts of Interest: the source of funding of studies have been shown to have an effect on the outcomes of such studies by researchers.
d. Observer bias: Is bias is with the “observers” of the research (i.e., the research team) rather than the participants. In other words, observer bias occurs when the observers (or researcher team) know the goals of the study or the hypotheses and allow this knowledge to influence their observations during the study. For example, if an observer knows that the researcher hypothesized that females speak in more complex sentences, they may believe they hear females speaking that way during the study even if it’s not really true.
e. Participant bias: This occurs when participants adjust their behavior to what they think the experimenters expect. This can be a significant problem in that, if participant bias occurs, then the results of an experiment may not be entirely due to the experimenters’ manipulation of the independent variable.
f. Research bias: (also called experimenter bias): Is a process where the scientists performing the research influence the results, to portray a certain outcome.
g. Sampling bias: (also called ascertainment bias) is a bias in which a sample is collected in such a way that some members of the intended population are less likely to be included than others. It results in a biased sample, a non-random sample of a population (or non-human factors) in which all individuals, or instances, were not equally likely to have been selected. If this is not accounted for, results can be erroneously attributed to the phenomenon under study rather than to the method of sampling. Examples include: Self-selection, Pre-screening, Advertising, etc.).
h. Exclusion bias: Results from exclusion of particular groups from the sample, e.g. exclusion of subjects who have recently migrated into the study area (this may occur when newcomers are not available in a register used to identify the source population). Excluding subjects who move out of the study area during follow-up is rather equivalent of dropout or non-response, a selection bias in that it rather affects the internal validity of the study.
i. Healthy user bias: when the study population is likely healthier than the general population, e.g. workers (i.e. someone in ill-health is unlikely to have a job as manual laborer).
j. Overmatching: matching for an apparent confounder that actually is a result of the exposure. The control group becomes more similar to the cases in regard to exposure than the general population.
k. Symptom-based sampling bias: The study of medical conditions begins with anecdotal reports. By nature, such reports only include those referred for diagnosis and treatment. A child not function in school is more likely to be diagnosed with dyslexia than a child who struggles but passes. A child examined for one condition is more likely to be tested for and diagnosed with other conditions, skewing comorbidity statistics. As certain diagnoses become associated with behavior problems or intellectual disability, parents try to prevent their children from being stigmatized with those diagnoses, introducing further bias. Studies carefully selected from whole populations are showing that many conditions are much more common and usually much milder than formerly believed.
C. Could a Confounding Factor Be Responsible for the Study Result? Confounding occurs when another causal factor (the co-founder) confuses the relationship between the agent of interest and outcome of interest. (e.g. Researchers must separate the relationship between gray hair and risk of death from that of old age and risk of death.) Confounding is a reality—that is, the observed association of a factor and a disease is actually the result of an association with a third, confounding factor.
1. What techniques can be used to prevent or limit confounding?
2. What techniques can be used to identify confounding factors?
3. What techniques can be used to control for confounding factors?
II. General Causation: Is an Exposure a Cause of the Disease?
A. Is There a Temporal Relationship?
B. How Strong Is the Association Between the Exposure and Disease?
C. Is There a Dose–Response Relationship?
D. Have the Results Been Replicated?
E. Is the Association Biologically Plausible (Consistent with Existing Knowledge)?
F. Have Alternative Explanations Been Considered?
G. What Is the Effect of Ceasing Exposure?
H. Does the Association Exhibit Specificity?
I. Are the Findings Consistent with Other Relevant Knowledge?
I would urge you to check this free and comprehensive source of information on scientific evidence.
More and more, figures and statistical information finds it way into litigation, both criminal and civil. At some point in your career as an attorney you will need to understand what can and cannot be accomplished in utilizing statistics. Most laypersons and attorneys are ill-equipped to handle such information. Oftentimes experts can find refuge in statistics which may or may not be truly relevant to the legal issue you are confronting. As Mark Twain (a/k/a Samuel Clemons) famously noted:
“Figures don’t lie, but liars figure.”
Another often quoted quip is:
“There are three kinds of lies: lies, damn lies and statistics.”
In litigation, you will often hear someone argue that the odds of being injured in a particular fashion are so low that a jury should not compensate them. However, there is a real risk in engaging in such post hoc analysis. How would you feel for example, if the State of Indiana came into court refusing to pay the Lotto Jackpot on your winning ticket by arguing that you could not have won it because the odds of winning are one in seven million.
Another way to point this same principle out, is the fallacy of using statistics to explain away a plaintiff’s untimely and unexpected demise:
“Your honor and ladies and gentlemen of the jury. Research has established that 90% of individuals involved in similar accidents survive. Accordingly, we must conclude that in spite of the evidence of lack of respiration, heartbeat, and brain wave activity, and in spite of the unfortunate burial of the decedent, in my expert opinion I conclude that he did not really die, and therefore the plaintiff estate cannot recover.”
Even though this sort of logic is flawed to its core, such arguments regularly find their way into our justice system… sometimes with disastrous effect. In order to spot such problems, you need to read about statistics, understand their limitations and how they can be misused. In this regard I would recommend the following reading:
1. Trial by Mathematics: Precision and Ritual in the Legal Process by Laurence Tribe, Harvard Law Review, 1971. This is an informative law review article addressing this topic. Mr. Tribe was the law clerk who assisted a California justice in writing a seminal opinion in this area. The court reversed a criminal conviction where a prosecutor improperly used statistical arguments in a robbery case involving a multi-racial couple. People v. Collins, 438 P.2d 33, 36-37 (Cal. 1968). Tribe had a math degree from Harvard in addition to his J.D. This opinion is often cited by courts as a prime example of how statistics and “scientific” evidence can be misused and down right dangerous to the pursuit of justice.
2. Naked Statistics by Charles Wheelan. The author strips away the arcane and technical details and focuses on the underlying thinking that drives statistical analysis. The author also clarifies key concepts such as inference, correlation, and regression analysis, reveals how biased or careless parties can manipulate or misrepresent data. Memorable examples of problems with statistics are discussed as well.
3. Calculated Risks: How to Know When Numbers Deceive You by Gerd Gigerenzer. This book does exactly what the title infers, it shows you in a concrete fashion how faulty thinking leads to people drawing incorrect conclusions from statistics and data. One of the problems discussed in the book is the famous dilemma presented by the Monty Hall Let’s Make a Deal Problem.
4. Math on Trial: How Numbers Get Used and Abused in the Courtroom by Leila Schneps. This book reviews the facts and outcomes of ten trials spanning from the nineteenth century to the present day, in which mathematical arguments were used, abused and disastrously misused resulting in unjust outcomes.
5. Reference Manual on Scientific Evidence prepared by the Federal Judicial Center. This is a free handbook that covers a number of areas of science that regularly appear in federal courtrooms. This manual is utilized by the federal judiciary as a reference book and covers both the law and science underlying a number of disciplines including epidemiology which is statistically based. This is must reading for any trial attorney who is going to take on an expert in a courtroom. This manual is regularly updated as well.
6. A Systematic Approach to Clinical Determinations of Causation in Symptomatic Spinal Disk Injury Following Motor Vehicle Crash Trauma by Michael D. Freeman, PhD, MPH, DC, Christopher J. Centeno, MD, and Sean S. Kohles, PhD. is an article which critically examines the misuse of data and pseudo-science to undermine claims of personal injury in motor vehicle accidents by defense “experts” and studies conducted in this area of litigation. This article provides an excellent survey and critic of the literature dealing with medical causation in motor vehicle collisions.
This list of reading should be both interesting and informative to the trial attorney confronted with the use of statistics. Just as it is helpful to “think like a lawyer”, it is equally useful to “think like a statistician”. Remember, numbers don’t lie, but liars figure…