Reading the Evidence
Every week a headline announces that some treatment "works" or "doesn't work," and every week patients walk into clinics carrying those headlines. A health professional who can read a study, spot its weak joints and put its numbers in perspective is far more useful than one who can only repeat what the latest article claimed. This chapter gives you the tools to judge research on spinal care and holistic therapies fairly: respecting what each tradition hopes to offer while asking, plainly, what the evidence actually shows.
By the end of this chapter you can
- Distinguish experimental from observational study designs and place each on the hierarchy of evidence.
- Explain how randomization, allocation concealment and blinding protect a trial from bias.
- Define confounding, selection bias, information bias and regression to the mean, and recognize each in a study description.
- Describe the placebo response and contextual effects, and explain why sham controls for manual and needling therapies are hard to design.
- Calculate and interpret absolute risk reduction, relative risk reduction and the number needed to treat.
- Interpret a confidence interval, a p value and an effect size, and separate statistical significance from the minimal clinically important difference.
- Read a forest plot and explain how a systematic review and meta-analysis differ from a narrative review.
- Explain how GRADE rates certainty of evidence and how clinical practice guidelines are built from it.
- Summarize the current evidence and safety profile for spinal manipulation, acupuncture, massage, yoga, mindfulness, herbal products and homeopathy.
Section 1Study designs and levels of evidence
Why design matters more than the conclusion
A research paper is an argument. Its conclusion is only as strong as the design that produced it, so the first question a careful reader asks is not "what did they find?" but "how did they find it?" Two studies can reach opposite conclusions about the same treatment simply because one compared patients who chose the treatment with patients who did not, while the other assigned treatment by chance. Learning to name a study's design in the first minute of reading is the single most useful habit in evidence based practice.
Evidence based practice is usually described as the integration of three things: the best available research evidence, the clinician's expertise, and the patient's values and circumstances. Research evidence is one leg of that stool, not the whole stool. A strong trial does not tell you what an individual patient wants, and clinical experience cannot by itself tell you whether improvement came from the treatment or from time. The three legs work together.
The basic split: did the investigator assign the exposure?
Every study design can be sorted with one question: did the researchers decide who received the intervention? If they did, the study is experimental. If they simply watched what people did and what happened to them, the study is observational. The distinction matters because when people choose their own treatments, the choosers usually differ from the non-choosers in ways that also affect outcomes. People who seek out a yoga class may already exercise more, sleep better and smoke less. Assignment by the investigator, and especially assignment by chance, breaks that link.
Experimental designs
The randomized controlled trial (RCT) assigns participants to two or more groups by a chance process, such as a computer generated sequence. Because chance decides group membership, both known and unknown characteristics (age, severity, motivation, genetics) tend to balance between groups, especially in larger trials. Any difference in outcome can then be attributed to the intervention with much more confidence. Variations include:
- Parallel group trials, the most common form, in which each group receives one treatment for the whole study.
- Crossover trials, in which each participant receives both treatments in sequence with a washout period between them. Crossovers suit stable chronic conditions and short acting treatments but fail when the first treatment has lasting effects.
- Cluster randomized trials, which randomize whole clinics, schools or communities rather than individuals. They are useful when treatment cannot be kept separate within a group, for example a clinic wide change in how back pain is triaged.
- Pragmatic trials, which test a treatment as it would be delivered in ordinary practice, with broad inclusion criteria and real world clinicians. They answer "does this help in practice?" The opposite, the explanatory trial, tests a treatment under tightly controlled conditions to ask "can this work, and how?"
A non-randomized controlled trial (sometimes called quasi-experimental) assigns treatment by some rule other than chance, such as alternating days or clinic location. It is better than no comparison but more vulnerable to imbalance between groups.
Observational designs
Observational studies are not second class. Many questions cannot ethically or practically be answered with a trial: you cannot randomize people to smoke, to suffer childhood adversity or to sustain a whiplash injury. Rare harms, such as a stroke after neck manipulation, are almost always studied observationally, because no trial could enroll enough people to see them.
- A cohort study follows a group of people over time, comparing those with and without an exposure, and counts who develops an outcome. A prospective cohort starts now and follows forward; a retrospective cohort uses existing records to look back at a cohort already assembled. Cohorts can measure incidence (new cases over time) and calculate relative risk.
- A case control study starts with the outcome. It gathers people who have a condition (cases) and similar people who do not (controls), then looks backward to compare their past exposures. Case control studies are efficient for rare outcomes and produce an odds ratio.
- A case crossover study is a special form in which each case serves as his or her own control, comparing exposures in a period just before the event with exposures in earlier periods. It has been used to study manipulation and vertebral artery stroke.
- A cross-sectional study measures exposure and outcome at the same moment, like a snapshot survey. It estimates prevalence but cannot show which came first.
- A case report or case series describes one or several patients. Case reports cannot establish that a treatment works, but they are valuable for raising new hypotheses and for flagging unexpected harms.
The hierarchy of evidence
Because some designs resist bias better than others, researchers arrange them in a hierarchy of evidence, often drawn as a pyramid. Near the top sit systematic reviews and meta-analyses of well conducted RCTs, then individual RCTs, then cohort studies, case control studies, case series and reports, and at the base, expert opinion, mechanistic reasoning and laboratory or animal research. The pyramid is a useful starting point, not a law. A large, careful cohort study can be more trustworthy than a tiny, poorly blinded RCT, and a meta-analysis of biased trials simply produces a precise estimate of a biased answer. Modern systems such as GRADE (discussed in Section 4) start trials higher and observational studies lower, then move each up or down based on how well it was actually done.
| Design | Starts with | Main strength | Main weakness | Typical measure |
|---|---|---|---|---|
| Randomized controlled trial | Random assignment to treatment | Balances known and unknown confounders | Costly; may not generalize; poor for rare harms | Risk difference, mean difference |
| Prospective cohort | Exposed and unexposed people | Shows time order; measures incidence | Confounding; long and expensive | Relative risk |
| Retrospective cohort | Existing records of a cohort | Fast and cheap | Depends on record quality | Relative risk |
| Case control | People with and without the outcome | Efficient for rare outcomes | Recall bias; choosing controls is hard | Odds ratio |
| Cross-sectional | A population at one time | Measures prevalence quickly | Cannot show which came first | Prevalence, prevalence ratio |
| Case report or series | One or a few patients | Generates hypotheses; signals harms | No comparison group | Description only |
Efficacy, effectiveness and mechanism
Three different questions are often blurred. Efficacy asks whether a treatment produces a benefit under ideal conditions. Effectiveness asks whether it helps under ordinary clinical conditions with ordinary patients. Mechanism asks how it works. A therapy can be effective without anyone knowing its mechanism (many drugs were used long before their receptors were discovered), and a proposed mechanism can be plausible while the therapy still fails in trials. In spinal care, many explanations of how manipulation works have changed over the past century while the clinical question, does it help this patient with this problem, has remained the one that matters most to patients.
Reading the methods section first
Experienced readers often skip straight from the title to the methods. A quick checklist works for most clinical papers: Who was studied, and how were they recruited? What was the intervention, and what was it compared with? What outcome was measured, when, and by whom? How many people dropped out? This structure is often summarized as PICO: Population, Intervention, Comparison, Outcome. Framing your own clinical questions in PICO form ("In adults with acute low back pain, does spinal manipulation compared with usual medical care reduce disability at four weeks?") also makes literature searching far more efficient.
Study design determines how much a result can be trusted. Experimental designs, especially randomized trials, guard best against confounding; observational designs answer questions trials cannot, such as rare harms and exposures no one could ethically assign. The evidence pyramid ranks designs on average, but the quality of each individual study still has to be judged.
Section 2Bias, confounding and placebo
Error comes in two kinds
Every measurement contains error. Random error scatters results unpredictably around the true value; it shrinks as samples grow and is what confidence intervals describe. Systematic error, or bias, pushes results consistently in one direction, and no amount of extra sample size will remove it. A bathroom scale that reads two pounds heavy every time is biased; weighing yourself a thousand times only makes you more confident in the wrong number. Most of the work of appraising a study is the hunt for bias.
Selection bias
Selection bias occurs when the people who end up in the comparison groups differ in ways that affect the outcome. In a trial, the main protection is proper randomization plus allocation concealment: the person enrolling a patient must not know which group the next patient will join. If a clinician can predict the next assignment, he or she might, even unconsciously, steer sicker patients away from the experimental treatment. Sealed opaque envelopes or central computer randomization prevent this. Trials with inadequate allocation concealment have been shown to exaggerate treatment effects.
Selection bias also enters observational research. Patients who volunteer for studies, who attend every follow up or who seek out a particular therapy are rarely typical. The healthy user effect describes how people who adopt one healthy behavior tend to adopt many others, making any single habit look more protective than it is.
Information (measurement) bias
Information bias arises when exposure or outcome is measured differently between groups. Recall bias is the classic example in case control studies: people who have had a stroke search their memories harder for possible causes than healthy controls do. Detection bias occurs when one group is examined more closely than another. Observer bias arises when an assessor who knows a patient's group rates outcomes more favorably. Many outcomes in spinal care, such as pain, disability and global improvement, are patient reported outcomes, which makes them especially sensitive to expectation.
Blinding
Blinding (also called masking) keeps people from knowing who received which treatment. Up to five groups can be blinded: participants, the clinicians delivering care, outcome assessors, data analysts and the people writing the conclusions. "Double blind" traditionally means participants and investigators, but the term is used loosely, so good reports state exactly who was blinded. In drug trials an identical looking sugar pill makes blinding straightforward. In manual therapy, acupuncture, exercise and surgery it is far harder: the clinician always knows what he or she is doing, and patients often can tell whether a firm thrust was delivered. When participants cannot be blinded, trials can still blind outcome assessors and analysts, and they can ask participants afterward which group they believed they were in to test whether blinding held.
Attrition and analysis
Attrition bias arises when people drop out unevenly. If patients who are not improving quit the treatment group while improving patients stay, the remaining treatment group looks better than reality. The standard protection is intention to treat analysis: every randomized participant is analyzed in the group he or she was assigned to, whether or not that person completed treatment. This preserves the balance created by randomization. A per protocol analysis, which includes only people who followed the plan, can be reported alongside it but tends to overstate benefit. Loss to follow up above about 20 percent generally raises concern.
Confounding
A confounder is a third variable that is associated with the exposure, independently affects the outcome, and is not simply a step on the causal path between them. Suppose a cohort study finds that people who receive regular massage have fewer heart attacks. Income may be a confounder: higher income makes regular massage more affordable, and higher income is independently linked to lower cardiovascular risk through diet, health care access and many other routes. The massage may have no direct effect at all.
Researchers handle confounding by design (randomization, restriction to one subgroup, matching cases and controls on age and sex) and by analysis (stratification and multivariable adjustment). Adjustment can only remove confounding from variables that were measured, and measured well. Residual confounding from unmeasured or poorly measured factors is the main reason observational findings are treated cautiously. Only randomization balances confounders that no one thought to measure.
Regression to the mean and natural history
Many spinal complaints fluctuate. People tend to seek care when symptoms are at their worst, and from a peak, the most likely direction is down. This statistical tendency for extreme measurements to be followed by more average ones is called regression to the mean. Combine it with the natural history of a condition (most acute low back pain improves substantially within several weeks whatever is done) and nearly any treatment started at a symptom peak will appear to work. This is why a control group is essential: only the difference between groups belongs to the treatment.
The placebo response and contextual effects
A placebo is an inert intervention made to resemble a real one. The placebo response is the total improvement seen in a placebo group, which includes natural history, regression to the mean and genuine psychobiological effects. The narrower placebo effect refers to that last component: real changes in brain and body triggered by the context of care. Brain imaging and pharmacology studies have shown that expectation of relief can activate endogenous opioid pathways (an effect partly reversed by the opioid blocker naloxone), engage prefrontal and brainstem pain modulating circuits, and reduce reported pain. Placebo effects are strongest for subjective, centrally processed outcomes like pain, nausea and fatigue, and weakest for objective measures such as tumor size or blood glucose.
The opposite, the nocebo effect, occurs when negative expectations produce real symptoms. Patients warned at length about side effects report more of them. In spinal care, alarming explanations ("your spine is degenerating," "your disc is out") can increase fear and disability, which is why careful language matters.
Researchers now often speak of contextual effects: the benefits that flow from the whole therapeutic encounter, including the clinician's warmth and confidence, a clear explanation, touch, the ritual of treatment, the setting, and the patient's expectations. These effects are real and are part of good care in every profession. They are, however, not specific to any one technique. A treatment's specific effect is the extra benefit beyond what the context provides, and estimating that specific effect is the purpose of a sham controlled trial.
| Source of improvement | What it is | How trials separate it |
|---|---|---|
| Natural history | The condition improving on its own course | Any concurrent control group, even no treatment |
| Regression to the mean | Extreme scores drifting back toward average | Concurrent control; repeated baseline measures |
| Contextual (placebo) effects | Benefits of expectation, attention and ritual | Sham or attention controlled comparison |
| Co-interventions | Other treatments taken at the same time | Recording and balancing other care |
| Specific effect | Benefit of the active ingredient itself | Difference between active and credible sham |
The problem of the sham
Designing a credible sham for a hands on therapy is one of the hardest problems in clinical research. A sham manipulation must feel like treatment without delivering the proposed active ingredient, yet no one fully agrees on what that ingredient is: the thrust, the cavitation, the position, the force, or the touch itself. Shams used in trials have included low force thrusts in neutral positions, inactive instrument devices and positioning without a thrust. If a sham is too inert, patients detect it and blinding fails, inflating the apparent effect of the real treatment. If a sham is too active, it may itself help, shrinking the difference. Acupuncture research faces the same dilemma with retractable non-penetrating needles and needling at "non-points," which may still stimulate skin and muscle. When a real therapy outperforms no treatment but does not outperform its sham, the honest conclusion is that much of the observed benefit appears to come from shared contextual factors rather than from the specific technique, not that patients did not improve.
Other biases worth naming
- Performance bias: groups receive different amounts of attention or co-intervention apart from the treatment being tested.
- Publication bias: studies with positive results are more likely to be published, published faster and published in more visible journals. It distorts every later review. (Section 4 explains how it is detected.)
- Selective outcome reporting: authors measure many outcomes and highlight only those that turned out favorable. Trial registration before enrollment, with the primary outcome stated in advance, guards against this.
- Allegiance and funding bias: investigators with professional, financial or philosophical investment in a therapy tend to report more favorable results. This applies equally to drug company trials, surgical trials and trials run by advocates of manual or holistic therapies.
- Spin: conclusions in the abstract that are more favorable than the results justify, such as emphasizing a secondary outcome when the primary outcome was negative.
Bias pushes results in one direction and cannot be fixed by a bigger sample. Randomization with concealed allocation, blinding wherever possible, intention to treat analysis and preregistered outcomes are the main defenses. Natural history, regression to the mean and contextual effects explain much of the improvement after any treatment, which is why a credible comparison group is indispensable.
Section 3Statistics for readers: effect size, confidence intervals, NNT
You do not need to calculate, but you do need to interpret
Most clinicians will never run a regression model, but every clinician has to read results sections. A small toolkit of ideas covers nearly everything: how big was the difference, how certain is it, and does it matter to a patient? This section works through each, using simple numbers you can check by hand.
Measures for yes or no outcomes
Many outcomes are dichotomous: a patient either recovered or did not, had a stroke or did not. Imagine a trial of 200 people with acute neck pain. In the control group of 100, 40 report being "much better" at four weeks. In the treatment group of 100, 60 report the same. The event rates are 0.40 in controls (the control event rate, CER) and 0.60 in the treated group (the experimental event rate, EER). From those two numbers come all the common measures:
- Absolute risk difference (for a good outcome often called absolute benefit increase; for a bad outcome, absolute risk reduction, ARR): EER minus CER, here 0.60 minus 0.40 = 0.20, or 20 percentage points.
- Relative risk (RR, also called risk ratio): EER divided by CER, here 0.60 / 0.40 = 1.5. Treated patients were 1.5 times as likely to be much better.
- Relative risk reduction (RRR) for a harmful outcome: (CER minus EER) / CER. If a vaccine lowers infection from 10 percent to 5 percent, the RRR is 50 percent while the ARR is only 5 percentage points.
- Number needed to treat (NNT): 1 divided by the absolute difference, here 1 / 0.20 = 5. On average, five patients must be treated for one additional patient to be much better who would not have been otherwise. NNTs are rounded up to the next whole number.
- Number needed to harm (NNH): the same calculation applied to an adverse outcome.
Relative measures sound more impressive than absolute ones and are favored in press releases. "Cuts risk in half" can mean 2 in 10,000 down to 1 in 10,000. Always look for the absolute numbers. The NNT is especially useful because it carries the baseline risk inside it: the same RRR gives a small NNT when the outcome is common and a huge NNT when it is rare.
| Measure | Formula | Worked example (CER 0.40, EER 0.60) | No effect value |
|---|---|---|---|
| Absolute difference | EER minus CER | 0.20 (20 percentage points) | 0 |
| Relative risk | EER / CER | 1.5 | 1 |
| Relative difference | (EER minus CER) / CER | 0.50 (50 percent more) | 0 |
| Number needed to treat | 1 / absolute difference | 5 | Infinite |
| Odds ratio | (EER/(1 minus EER)) / (CER/(1 minus CER)) | 1.5 / 0.667 = 2.25 | 1 |
The odds ratio (OR) compares odds rather than probabilities. Odds are the chance something happens divided by the chance it does not: a 0.60 probability equals odds of 1.5 (60 to 40). Case control studies can only calculate odds ratios, because the investigator chose how many cases and controls to include, so true risks are unknown. When an outcome is rare (below about 10 percent), the OR closely approximates the RR; when it is common, as in the example, the OR exaggerates the RR (2.25 versus 1.5).
Measures for continuous outcomes and effect size
Pain scores, disability questionnaires and range of motion are continuous outcomes. Trials report the mean difference (MD) between groups, for example 8 points on a 0 to 100 pain scale. When different trials use different scales, a meta-analysis converts results into a standardized mean difference (SMD), which expresses the difference in units of standard deviation. The best known version is Cohen's d. By widely used rules of thumb, an SMD of about 0.2 is small, 0.5 moderate and 0.8 large. These cut points are conventions, not laws; a "small" effect on mortality could be enormously important, while a "moderate" effect on a lab value might be trivial.
The question patients care about is whether a difference is big enough to feel. The minimal clinically important difference (MCID) is the smallest change in a score that patients perceive as meaningful. For 0 to 10 pain rating scales, a change of about 2 points or roughly 30 percent is commonly cited as meaningful for an individual. For the Oswestry Disability Index, values around 10 points are often quoted, and for the Roland Morris questionnaire (24 points), about 2 to 5 points. MCIDs vary with baseline severity and with the method used to derive them, so treat them as guides. Be careful too about the difference between an individual's meaningful change and a meaningful difference between group averages: a small average difference can hide the fact that some patients benefit a lot and others not at all.
The p value
A p value is the probability of seeing a result at least as extreme as the one observed if the null hypothesis (no true difference) were correct. A p value below 0.05 is conventionally called statistically significant. Three common misreadings cause trouble:
- A p value is not the probability that the null hypothesis is true, nor the probability that the result happened "by chance."
- A significant p value says nothing about the size or importance of an effect. In a trial of 5,000 people, a 1 point difference on a 100 point pain scale can be highly significant and clinically meaningless.
- A non-significant p value does not prove no effect. A small trial may simply be underpowered: too few participants to detect a real difference. "Absence of evidence is not evidence of absence."
Two error types frame the logic. A type I error (false positive, probability alpha, usually set at 0.05) is concluding a difference exists when it does not. A type II error (false negative, probability beta) is missing a real difference. Statistical power is 1 minus beta, the chance of detecting a true effect of a given size; trials are usually designed for 80 or 90 percent power. Testing many outcomes or subgroups multiplies the chance of a type I error: with 20 independent tests at the 0.05 level, about one false positive is expected by chance alone.
Confidence intervals
A 95 percent confidence interval (CI) gives a range of values for the true effect that are reasonably compatible with the data. Strictly, if the same study were repeated many times, 95 percent of the intervals calculated this way would contain the true value. In practice, read it as "the true effect is plausibly anywhere in this range." Confidence intervals are more informative than p values because they show both the size and the precision of an effect. Narrow intervals come from large samples and consistent data; wide intervals mean uncertainty.
Two reading rules cover most cases. First, if a 95 percent CI for a difference crosses zero (or for a ratio crosses 1), the result is not statistically significant at the 0.05 level. Second, compare the whole interval with the MCID. If the entire interval lies above the MCID, the effect is both real and important. If the interval is above zero but entirely below the MCID, the effect is probably real but too small to matter. If the interval spans from trivial to large, the study simply cannot tell.
Diagnostic test statistics in brief
Examination chapters introduced sensitivity (the proportion of people with a condition who test positive) and specificity (the proportion without it who test negative). Research on orthopedic and neurologic tests also reports likelihood ratios: a positive likelihood ratio above about 10 or a negative likelihood ratio below about 0.1 shifts the probability of a diagnosis substantially, while values near 1 barely change it. Reliability, the agreement between two examiners or between two occasions, is reported with the kappa statistic for categorical findings (0 is chance agreement, 1 is perfect) and the intraclass correlation coefficient for continuous measures. Studies of motion palpation for segmental restriction have generally found low inter-examiner reliability, while tests involving pain provocation tend to agree somewhat better. Recognizing that a test is unreliable is itself useful clinical knowledge.
Ask three questions of every result: how big (absolute difference, NNT, effect size), how certain (confidence interval), and does it matter (comparison with the minimal clinically important difference). A p value answers none of these by itself, and relative risks without baseline rates can mislead.
Section 4Systematic reviews and clinical guidelines
Narrative versus systematic reviews
A narrative review is an expert's summary of a topic. It can be a fine introduction, but the author chooses which studies to discuss, and an author who believes strongly in a therapy will tend, often unconsciously, to cite the studies that agree. A systematic review answers a focused question using explicit, reproducible methods that anyone could repeat:
- A protocol, ideally registered publicly before work starts, states the question (in PICO form), eligibility criteria and planned analyses.
- A comprehensive search covers several databases, trial registries and unpublished sources, so that the review is not limited to easily found, often positive, studies.
- Two reviewers independently screen studies and extract data, resolving disagreements by discussion.
- Each included study is assessed for risk of bias with a standard tool (for randomized trials, tools rate randomization, deviations from intended care, missing data, outcome measurement and selective reporting).
- Results are synthesized, statistically if the studies are similar enough, and the certainty of the evidence is rated.
Reporting standards (the PRISMA statement is the best known) require a flow diagram showing how many records were found, screened, excluded and included, so readers can see where studies were lost. The international Cochrane network produces many high quality systematic reviews in health care and publishes them with regular updates.
Meta-analysis and the forest plot
A meta-analysis is the statistical part of a systematic review: it combines results from several studies into a single pooled estimate, giving larger studies (with more precise results) more weight. Not every systematic review should include a meta-analysis; pooling studies of different patients, treatments or outcomes produces a number that answers no real question, sometimes called mixing apples and oranges.
Meta-analysis results are shown in a forest plot. Each row is one study, drawn as a square (the point estimate, with size proportional to weight) and a horizontal line (its confidence interval). A vertical line of no effect marks zero for differences or 1 for ratios. At the bottom, a diamond shows the pooled result: its center is the pooled estimate and its width is the pooled confidence interval. If the diamond does not touch the line of no effect, the pooled result is statistically significant.
Heterogeneity
Heterogeneity describes how much study results differ beyond what chance would produce. It is quantified with the I2 statistic, the percentage of variability due to real differences between studies; roughly, 25 percent is low, 50 percent moderate and 75 percent high. High heterogeneity is a signal to ask why: different patient groups, different doses or techniques, different comparison treatments or different risk of bias. A fixed effect model assumes all studies estimate one true effect; a random effects model assumes true effects vary between studies and gives a wider, more cautious interval. Reviewers explore heterogeneity with prespecified subgroup analyses and sensitivity analyses (for example, rerunning the analysis with only low risk of bias trials). A common and important finding is that the apparent benefit of a therapy shrinks when only the best trials are included.
Publication bias and small study effects
If small negative studies go unpublished, a meta-analysis of published studies will overestimate benefit. A funnel plot graphs each study's effect against its precision. Without bias, the points form a symmetrical inverted funnel: large precise studies cluster near the true effect at the top, and small imprecise studies scatter evenly on both sides at the bottom. A missing lower corner, usually on the side of no effect, suggests that small unfavorable studies are missing. Asymmetry has other causes too, including genuine differences between small and large trials, so it is a warning sign rather than proof.
Rating certainty: the GRADE approach
The GRADE system (Grading of Recommendations, Assessment, Development and Evaluation) is used by most guideline groups worldwide. It rates the certainty of a body of evidence for each outcome in four levels: high, moderate, low and very low. Evidence from randomized trials starts as high and from observational studies as low. Five factors can lower certainty: risk of bias, inconsistency (unexplained heterogeneity), indirectness (the studied patients, treatments or outcomes differ from the question), imprecision (wide confidence intervals, few events) and publication bias. Three factors can raise observational evidence: a large effect, a dose response gradient, and situations where all plausible confounding would have reduced the observed effect.
| GRADE level | Meaning in plain words | Typical wording in reviews |
|---|---|---|
| High | Further research is very unlikely to change our confidence in the estimate | "Treatment X reduces..." |
| Moderate | The true effect is probably close to the estimate, but could differ substantially | "Treatment X probably reduces..." |
| Low | The true effect may be substantially different | "Treatment X may reduce..." |
| Very low | We have very little confidence; the true effect is likely substantially different | "We are uncertain whether treatment X..." |
Clinical practice guidelines
Clinical practice guidelines are recommendations, developed systematically, to help clinicians and patients make decisions. A trustworthy guideline is built on systematic reviews, by a panel that includes several professions and patient representatives, with conflicts of interest declared and managed, using an explicit method to move from evidence to recommendation, and with plans for updating. Under GRADE, a recommendation has both a direction (for or against) and a strength (strong or conditional, also called weak). Strength depends not only on certainty of evidence but on the balance of benefits and harms, patients' values, cost and feasibility. A strong recommendation can rest on moderate evidence when harms are small and benefits clear; a conditional recommendation means that the right choice varies with individual circumstances and should be shared with the patient.
The AGREE II instrument is commonly used to appraise guideline quality across domains such as scope, stakeholder involvement, rigor, clarity, applicability and editorial independence. Readers should notice when two guidelines disagree: often they asked slightly different questions, included different evidence or weighed costs and values differently.
Systematic reviews use preplanned, reproducible methods to find and appraise all relevant studies, and meta-analyses pool their results in forest plots. Heterogeneity and publication bias must be examined, not ignored. GRADE rates how certain the evidence is, and guidelines combine that certainty with harms, values and costs to make strong or conditional recommendations.
Section 5What the evidence says about spinal manipulation and holistic therapies
Applying the tools fairly
The methods in this chapter apply to every intervention, whether a prescription drug, a surgery, an adjustment or an herbal tea. Fair appraisal means holding all of them to the same standard. The summaries below reflect the general direction of high quality systematic reviews and guidelines; individual reviews differ in detail, and new trials keep arriving, so treat these as a map, not a final verdict.
Spinal manipulation for low back pain
Spinal manipulative therapy (SMT), a high velocity, low amplitude thrust, is one of the most studied manual therapies, with dozens of randomized trials. For acute low back pain, systematic reviews have found moderate certainty evidence of modest improvements in pain and function over several weeks, on the order of 10 points on a 100 point pain scale, compared with control treatments. For chronic low back pain, large reviews pooling dozens of trials have found that SMT produces effects similar to other recommended treatments such as exercise, and better than some non-recommended comparisons, with low to moderate certainty. Against sham manipulation, differences are small and often below commonly used thresholds for clinical importance. The fair summary is that manipulation is a reasonable, guideline supported option for low back pain, roughly comparable to other recommended conservative care, rather than either a cure or a placebo.
Neck pain and headache
For neck pain, manipulation and mobilization, especially combined with exercise, show short term benefits in pain and function with low to moderate certainty, and they perform similarly to each other. For cervicogenic headache (headache referred from the neck), several trials suggest benefit. For migraine and tension type headache the evidence is more limited and inconsistent; some trials show small improvements, while sham controlled trials tend to show smaller differences. Exercise, sleep and trigger management remain central.
Conditions outside the musculoskeletal system
Historically, some chiropractic models proposed that correcting spinal subluxations could improve visceral and systemic health (Chapter 9 traces this history). Trials have tested manipulation for asthma, high blood pressure, infantile colic, ear infections, painful menstruation and other conditions. Reviews by international panels that included chiropractic researchers have concluded that there is no reliable evidence that SMT is effective for these non-musculoskeletal conditions. Much of the modern chiropractic profession accepts this finding and focuses care on musculoskeletal complaints, while some practitioners continue to hold broader traditional views. A student can respect the history of the vitalistic model and still be clear about what trials show.
Safety of manipulation
Mild, short lived side effects are common after manipulation: local soreness, stiffness, headache or tiredness, reported by a sizeable minority of patients and usually resolving within 24 to 48 hours. Serious adverse events are rare. The most discussed is cervical artery dissection (a tear in the wall of a vertebral or carotid artery) leading to stroke. Dissection can itself cause neck pain and headache before any stroke occurs, so patients with an early dissection may seek care for those symptoms. Case crossover studies found that the association between a recent chiropractic visit and vertebrobasilar stroke was similar to the association with a recent visit to a primary care physician, suggesting that much of the link reflects people with early dissection seeking care rather than manipulation causing it. A causal contribution in some cases cannot be excluded. Practical safety therefore rests on careful history and screening for red flags (sudden severe "worst ever" headache, new neck pain with neurological signs such as dizziness, double vision, slurred speech or difficulty swallowing) and on referring rather than manipulating when those are present. Lumbar manipulation has very rarely been associated with cauda equina syndrome and disc herniation.
Acupuncture
Acupuncture, from traditional East Asian medicine, inserts fine needles at defined points understood in its own framework to regulate the flow of qi along meridians. Large individual patient data meta-analyses of chronic pain (back, neck, osteoarthritis, chronic headache, shoulder pain) found that acupuncture outperformed both sham acupuncture and no acupuncture controls. The difference from no acupuncture was moderate; the difference from sham was small, though statistically reliable, suggesting that a substantial part of the benefit comes from contextual factors and that needling location may matter less than traditional theory proposes. With trained practitioners and sterile single use needles, serious adverse events (pneumothorax, infection, nerve injury) are rare; minor bruising and soreness are common.
Massage, yoga, tai chi and mindfulness
Massage shows short term benefits for low back and neck pain with low certainty, and it is very safe; caution applies to areas of suspected deep vein thrombosis, fragile skin and patients on anticoagulants receiving deep pressure. Yoga for chronic low back pain improves function and pain modestly compared with non-exercise controls, with effects similar to other forms of exercise and occasional minor strains. Tai chi has moderate to high quality evidence for reducing falls in older adults and helpful evidence for knee osteoarthritis and fibromyalgia symptoms. Mindfulness based stress reduction has been compared head to head with cognitive behavioral therapy for chronic low back pain and produced similar improvements, both better than usual care. These mind body approaches are covered more fully in Chapter 13.
Herbal products and supplements
Herbal medicine spans many traditions. Some products have reasonable trial evidence for particular uses (for example, St John's wort for mild to moderate depression), many have little or none, and natural does not mean harmless. Key safety issues are herb drug interactions (St John's wort induces liver enzymes and can reduce the effect of oral contraceptives, anticoagulants, transplant drugs and others; ginkgo and high dose fish oil may increase bleeding risk), direct toxicity (kava and some traditional products have been linked to liver injury) and product quality, since supplements are not tested before sale the way drugs are and some have been found contaminated or adulterated. Clinicians should always ask about supplements.
Homeopathy and energy therapies
Homeopathy is based on the principles that "like cures like" and that serial dilution and shaking increase potency; many remedies are diluted beyond the point where any molecule of the original substance is likely to remain. Comprehensive government sponsored reviews have found no reliable evidence that homeopathy is effective for any health condition beyond placebo. Energy therapies such as Reiki and therapeutic touch aim to channel or balance a biofield; trials are mostly small with high risk of bias, and better controlled studies have not shown effects beyond sham. These practices are usually physically harmless; their main risk is indirect, when they delay effective care for a serious condition.
| Therapy | Best supported uses | Certainty (approximate) | Main safety considerations |
|---|---|---|---|
| Spinal manipulation | Acute and chronic low back pain; neck pain; cervicogenic headache | Low to moderate | Transient soreness common; screen for dissection and cauda equina red flags |
| Acupuncture | Chronic back, neck, knee and headache pain | Moderate | Sterile single use needles; rare pneumothorax |
| Massage | Short term back and neck pain relief | Low | Avoid deep pressure over suspected clots; anticoagulant use |
| Yoga and tai chi | Chronic low back pain; falls prevention; knee osteoarthritis | Moderate | Minor strains; modify poses for osteoporosis and glaucoma |
| Mindfulness programs | Chronic pain coping, stress, anxiety | Moderate | Occasional distress in trauma survivors; trained teachers |
| Herbal products | Varies by product; a few have positive trials | Very low to moderate | Drug interactions, liver toxicity, contamination |
| Homeopathy | No condition shown beyond placebo | High certainty of no specific effect | Delay of effective care |
Spinal manipulation is a reasonable, guideline supported option for low back and neck pain with benefits similar to other recommended care, small specific effects over sham, common mild side effects and rare serious ones that careful screening helps prevent. Evidence does not support manipulation for non-musculoskeletal disease. Holistic therapies range from moderately supported (acupuncture, yoga, tai chi, mindfulness for chronic pain) to unsupported (homeopathy), and each deserves the same fair, specific appraisal.
The chapter in one breath
- Study design sets the ceiling on trustworthiness: experiments assign exposure, observational studies watch it.
- Randomized controlled trials balance known and unknown confounders; cohort, case control and cross-sectional studies answer questions trials cannot, including rare harms.
- The evidence pyramid ranks designs on average, but individual study quality must still be judged.
- Bias is systematic error; allocation concealment, blinding, intention to treat and trial registration are the main defenses.
- A confounder is linked to both exposure and outcome; only randomization controls unmeasured confounders.
- Natural history, regression to the mean and contextual effects make any treatment look effective without a control group.
- The NNT is 1 divided by the absolute risk difference; relative risks without baseline rates mislead.
- A confidence interval shows size and precision; compare it with zero for significance and with the MCID for importance.
- A p value is not the probability that the null hypothesis is true and says nothing about effect size.
- Systematic reviews use reproducible methods; forest plots show each trial and the pooled diamond; heterogeneity and publication bias need checking.
- GRADE rates certainty high, moderate, low or very low, and guidelines make strong or conditional recommendations.
- Spinal manipulation helps low back and neck pain about as much as other recommended care, is not supported for non-musculoskeletal disease, and requires red flag screening; holistic therapies range from moderately supported to unsupported.
Words worth knowing
| Term | What it means |
|---|---|
| Absolute risk reduction | The difference in event rates between control and treated groups, expressed in percentage points. |
| Allocation concealment | Keeping the upcoming group assignment hidden from those enrolling participants so they cannot influence who goes where. |
| Blinding | Keeping participants, clinicians, assessors or analysts unaware of group assignment. |
| Case control study | An observational study that starts with people who have and do not have an outcome and compares their past exposures. |
| Cohort study | An observational study that follows exposed and unexposed people over time to compare outcomes. |
| Confidence interval | A range of values for the true effect that are reasonably compatible with the study data. |
| Confounder | A third factor associated with both exposure and outcome that can create a false or distorted association. |
| Contextual effects | Benefits arising from the whole care encounter, such as expectation, attention, touch and explanation. |
| Effect size | A measure of how large a difference is, often in standard deviation units, as with Cohen's d. |
| Forest plot | A graph showing each study's estimate and confidence interval with a pooled diamond at the bottom. |
| GRADE | A widely used system for rating certainty of evidence as high, moderate, low or very low. |
| Heterogeneity | Variation in results between studies beyond what chance explains, often measured with I squared. |
| Intention to treat | Analyzing every participant in the group to which he or she was randomized, regardless of adherence. |
| Meta-analysis | Statistical pooling of results from several studies into a single weighted estimate. |
| Minimal clinically important difference | The smallest change in an outcome score that patients perceive as meaningful. |
| Nocebo effect | Real symptoms or worsening produced by negative expectations. |
| Number needed to treat | The number of patients who must be treated for one additional patient to benefit; 1 divided by the absolute risk difference. |
| Odds ratio | The ratio of the odds of an outcome in one group to the odds in another; the main measure from case control studies. |
| p value | The probability of a result at least this extreme if there were truly no difference. |
| Placebo response | All improvement seen in a placebo group, including natural history, regression to the mean and contextual effects. |
| Publication bias | The tendency for positive studies to be published more often, distorting the visible evidence. |
| Randomized controlled trial | An experiment in which participants are assigned to groups by chance and outcomes are compared. |
| Regression to the mean | The tendency of extreme measurements to be followed by values closer to average. |
| Relative risk | The risk of an outcome in one group divided by the risk in another. |
| Sham | A simulated procedure designed to resemble a real treatment without its proposed active ingredient. |
| Systematic review | A review that uses explicit, reproducible methods to find, appraise and synthesize all relevant studies on a question. |
| Type I error | Concluding that a difference exists when it does not; a false positive. |
| Type II error | Failing to detect a difference that really exists; a false negative. |