In my last post, I discussed readings that could help improve your knowledge and analytical skills in addressing statistical data. Below is a check list of items to consider summarized from the Manual on Scientific Evidence Third Edition, Reference Guide on Epidemiology. Here is the list:

CHECKLIST OF PROBLEMS WITH THE USE OF STATISTICAL DATA AND ANALYSIS
 
I. What Sources of Error Might Have Produced a False Result?
 
A. What Statistical Methods Exist to Evaluate the Possibility of Sampling Error?
 
1. False positives and statistical significance,

2. False negatives,

3. Power,
 
B. What Biases May Have Contributed to an Erroneous Association?
 
1. Selection bias: Selection bias refers to the error in an observed association that results from the method of selection of cases and controls (in a case-control study) or exposed and unexposed individuals (in a cohort study).
 
2. Information bias: Information bias is a result of inaccurate information about either the disease or the exposure status of the study participants or a result of confounding. In a case-control study, potential information bias is an important consideration because the researcher depends on information from the past to determine exposure and disease and their temporal relationship.
 
3. Other conceptual problems:
 
a. Issue or hypothesis is improperly defined: Sometimes studies are limited by flawed definitions or premises.

b. Publication bias: the tendency for medical journals to prefer studies that find an effect. If negative studies are never published, the published literature will be biased.

c. Financial bias / Conflicts of Interest: the source of funding of studies have been shown to have an effect on the outcomes of such studies by researchers.

d. Observer bias: Is bias is with the “observers” of the research (i.e., the research team) rather than the participants. In other words, observer bias occurs when the observers (or researcher team) know the goals of the study or the hypotheses and allow this knowledge to influence their observations during the study. For example, if an observer knows that the researcher hypothesized that females speak in more complex sentences, they may believe they hear females speaking that way during the study even if it’s not really true.

e. Participant bias: This occurs when participants adjust their behavior to what they think the experimenters expect. This can be a significant problem in that, if participant bias occurs, then the results of an experiment may not be entirely due to the experimenters’ manipulation of the independent variable.

f. Research bias: (also called experimenter bias): Is a process where the scientists performing the research influence the results, to portray a certain outcome.
 
g. Sampling bias: (also called ascertainment bias) is a bias in which a sample is collected in such a way that some members of the intended population are less likely to be included than others. It results in a biased sample, a non-random sample[1] of a population (or non-human factors) in which all individuals, or instances, were not equally likely to have been selected.[2] If this is not accounted for, results can be erroneously attributed to the phenomenon under study rather than to the method of sampling. Examples include: Self-selection, Pre-screening, Advertising, etc.).
 
h. Exclusion bias: Results from exclusion of particular groups from the sample, e.g. exclusion of subjects who have recently migrated into the study area (this may occur when newcomers are not available in a register used to identify the source population). Excluding subjects who move out of the study area during follow-up is rather equivalent of dropout or non-response, a selection bias in that it rather affects the internal validity of the study.
 
i. Healthy user bias: when the study population is likely healthier than the general population, e.g. workers (i.e. someone in ill-health is unlikely to have a job as manual laborer).

j. Overmatching: matching for an apparent confounder that actually is a result of the exposure. The control group becomes more similar to the cases in regard to exposure than the general population.

k. Symptom-based sampling bias: The study of medical conditions begins with anecdotal reports. By nature, such reports only include those referred for diagnosis and treatment. A child not function in school is more likely to be diagnosed with dyslexia than a child who struggles but passes. A child examined for one condition is more likely to be tested for and diagnosed with other conditions, skewing comorbidity statistics. As certain diagnoses become associated with behavior problems or intellectual disability, parents try to prevent their children from being stigmatized with those diagnoses, introducing further bias. Studies carefully selected from whole populations are showing that many conditions are much more common and usually much milder than formerly believed.
 
C. Could a Confounding Factor Be Responsible for the Study Result? Confounding occurs when another causal factor (the co-founder) confuses the relationship between the agent of interest and outcome of interest. (e.g. Researchers must separate the relationship between gray hair and risk of death from that of old age and risk of death.) Confounding is a reality—that is, the observed association of a factor and a disease is actually the result of an association with a third, confounding factor.

1. What techniques can be used to prevent or limit confounding?
 
2. What techniques can be used to identify confounding factors?
 
3. What techniques can be used to control for confounding factors?
 
II. General Causation: Is an Exposure a Cause of the Disease?
 
A. Is There a Temporal Relationship?
 
B. How Strong Is the Association Between the Exposure and Disease?
 
C. Is There a Dose–Response Relationship?
 
D. Have the Results Been Replicated?
 
E. Is the Association Biologically Plausible (Consistent with Existing Knowledge)?
 
F. Have Alternative Explanations Been Considered?
 
G. What Is the Effect of Ceasing Exposure?
 
H. Does the Association Exhibit Specificity?
 
I. Are the Findings Consistent with Other Relevant Knowledge?
 
I would urge you to check this free and comprehensive source of information on scientific evidence.