Clinical trials are intended to help us establish whether treatments are effective. Yet acupuncture research sometimes produces apparently contradictory findings. One study reports a clinically meaningful improvement, another finds little difference between acupuncture and sham treatment, while a third concludes that the available evidence is insufficient.
How can researchers investigating the same treatment reach such different conclusions?
The answer is more complicated than simply dividing studies into those that support acupuncture and those that do not.
Differences in patient selection, treatment protocols, control groups, outcome measurements and statistical methods can all influence the findings. Some apparent contradictions arise because researchers are asking fundamentally different questions.
Understanding these differences is essential for practitioners who want to interpret clinical research critically, rather than relying solely on the conclusions presented in abstracts.
The control group can change the result
One of the clearest examples comes from research into acupuncture for pain.
In a 2016 review of 139 studies, researchers investigated whether the type of control group was associated with the conclusions reached in acupuncture trials.
How Control Groups Influenced Reported Findings – Percentage of comparisons reporting positive conclusions in the 2016 review.
No Treatment – 84.3% Positive Conclusions from Non-Penetrating Sham 53.3% Positive Conclusions from Penetrating Sham 37.8% (Springer – Types of Control in Acupuncture Clinical Trials Might Affect the Conclusion of the Trials: A Review of Acupuncture on Pain Management https://link.springer.com/article/10.1016/j.jams.2016.08.001)
The findings raise an important question: are apparently conflicting results sometimes a consequence of what acupuncture is being compared against?
A no-treatment control investigates whether acupuncture offers benefits beyond receiving no additional intervention. A sham-controlled trial investigates whether the acupuncture protocol offers benefits beyond those associated with the particular sham procedure.
Those are different questions.
Furthermore, sham acupuncture is not necessarily physiologically inactive. Some techniques involve superficial skin penetration, while others use non-penetrating devices.
A 2022 BMJ consensus paper on designing high-quality acupuncture trials recognised this problem. However, its own analysis did not find that sham type consistently changed treatment effects, highlighting the uncertainty surrounding this issue. (BMJ – How to design high quality acupuncture trials—a consensus informed by evidence https://www.bmj.com/content/376/bmj-2021-067476)
The important point is that the control group must be examined before comparing the conclusions of different studies.
Are researchers delivering the same treatment?
Consider two hypothetical studies investigating acupuncture for chronic lower back pain.
The first provides six treatments over three weeks, using a fixed selection of acupuncture points.
The second provides twelve treatments over six weeks, allowing practitioners to individualise point selection and modify treatment according to the patient’s progress.
Both may be described as acupuncture trials for chronic lower back pain, but they are investigating different treatment protocols.
Treatment frequency, duration, needling techniques and individualisation may all influence outcomes.
However, it would be a mistake to assume that more individualised treatment or additional sessions necessarily produce better results. These relationships must themselves be investigated.
The 2022 BMJ consensus recommendations identify treatment adequacy, practitioner expertise and appropriate intervention reporting as important considerations when designing acupuncture trials. (BMJ – How to design high quality acupuncture trials—a consensus informed by evidence https://www.bmj.com/content/376/bmj-2021-067476)
Consequently, when two trials disagree, practitioners should examine what treatment was actually delivered before deciding whether their findings genuinely conflict.
Are the patients genuinely comparable?
Even when researchers investigate the same medical condition, the participants enrolled in different studies may have substantially different characteristics.
Chronic lower back pain provides a useful example.
One trial might recruit people experiencing relatively mild symptoms, while another includes patients with severe, longstanding pain.
The participants may differ in their previous treatment, accompanying medical conditions, medication use and baseline levels of physical function.
These differences can affect the interpretation of the results.
An individual patient data meta-analysis involving 20,827 chronic pain patients investigated whether particular patient characteristics influenced acupuncture treatment outcomes.
Of the five characteristics examined, baseline pain severity was the clearest potential moderator. Patients with more severe pain initially appeared to experience greater benefit relative to control treatment. There was no strong evidence that age or pain duration modified treatment effects. (National Library of Medicine – The effect of patient characteristics on acupuncture treatment outcomes: An Individual Patient Data Meta-Analysis of 20,827 Chronic Pain Patients in Randomized Controlled Trials https://pmc.ncbi.nlm.nih.gov/articles/PMC6450709/)
This is a useful illustration of why researchers should investigate patient characteristics rather than assume that all people with the same diagnosis will respond similarly.
What are researchers actually measuring?
Two studies can investigate the same condition and reach different conclusions simply because they measure different outcomes.
A chronic pain trial might assess pain intensity, physical function, medication consumption or quality of life.
A treatment could produce a measurable reduction in pain without a corresponding improvement in mobility.
Similarly, a statistically significant improvement does not necessarily represent a clinically meaningful change for the patient.
The 2020 Cochrane review of acupuncture for chronic non-specific lower back pain illustrates this distinction. It included 33 trials involving 8,270 participants and examined pain, back-related function and quality of life.
The review found differences in the results depending on the comparator and outcome. For example, acupuncture was more effective than no treatment for pain and function immediately after treatment, while its advantages over sham acupuncture were less certain and may not have been clinically meaningful. The certainty of evidence ranged from moderate to very low. (National Library of Medicine – Acupuncture for chronic nonspecific low back pain https://pmc.ncbi.nlm.nih.gov/articles/PMC8095030/)
This demonstrates why a headline reporting that acupuncture was effective or ineffective may omit important information about what was measured and how substantial the difference actually was.
Does practitioner experience influence outcomes?
Acupuncture is a practitioner-delivered intervention, making clinical training and experience relevant considerations when designing research.
However, the assumption that differences between practitioners explain inconsistent trial results requires evidence.
An analysis of 1,206 acupuncturists across 13 clinical trials investigated variation in treatment effects between practitioners.
Although the researchers detected statistically significant variation, its magnitude was very small and largely overlapped with what would be expected by chance. (National Library of Medicine – Do the effects of acupuncture vary between acupuncturists? Analysis of the Acupuncture Trialists’ Collaboration individual patient data meta-analysis https://pubmed.ncbi.nlm.nih.gov/33300369/)
This is an important finding because it demonstrates that a plausible explanation for inconsistent research results is not necessarily supported by the available evidence.
It also reinforces the need to distinguish between clinical assumptions and conclusions established through research.
Why study size and statistical analysis matter
A trial involving 40 participants and one involving 4,000 participants may produce different estimates of the same treatment effect simply because smaller studies are more vulnerable to random variation.
A small trial may fail to detect a genuine difference because it lacks sufficient statistical power.
Conversely, an unusually large treatment effect observed in a small study may become smaller when investigated in a larger population.
Researchers also need to consider confidence intervals, missing data, multiple outcome measurements and the risk of selective reporting.
When many different outcomes are investigated, statistically significant findings may occasionally occur by chance.
A larger sample can improve the precision of an estimate, but it cannot correct poor randomisation, inadequate allocation concealment or other systematic sources of bias.
This is why the methodological quality of a study is at least as important as its size.
When systematic reviews disagree
Systematic reviews are frequently regarded as an important source of evidence because they bring together findings from multiple studies.
Nevertheless, two reviews examining acupuncture for the same condition can reach different conclusions.
Differences may arise from their search dates, eligibility criteria, inclusion of sham-controlled versus usual-care trials, definitions of treatment success or statistical methods.
Researchers must also decide whether the included trials are sufficiently similar to combine their results.
A review that combines studies investigating substantially different interventions or patient populations may produce a pooled estimate that is difficult to interpret.
Conversely, a review applying very restrictive inclusion criteria may exclude clinically relevant research.
For practitioners, the important question is not simply which systematic review is more recent, but whether its methods and included studies are appropriate to the clinical question being considered.
What should practitioners look for when reading acupuncture research?
A practical research-reading checklist
- What condition and patient population were investigated?
- What acupuncture protocol was actually delivered?
- How many treatments did participants receive?
- What was the control intervention?
- Were the outcomes clinically meaningful?
- When were outcomes measured?
- Was the study sufficiently large and appropriately randomised?
- Were adverse events and limitations reported?
Conclusion
Conflicting clinical trial results do not necessarily mean that one research team is correct and another is wrong.
Sometimes studies investigate different patient populations, deliver different treatment protocols or measure different outcomes. Sometimes the apparent disagreement is attributable to random variation or methodological limitations. In other cases, the evidence genuinely remains uncertain.
Understanding these distinctions enables practitioners to engage more meaningfully with research, recognise where evidence is strong and identify questions requiring further investigation.
Acupuncture research should be subject to rigorous scientific scrutiny. Equally, interpreting that research requires careful consideration of precisely what was investigated and what the findings can reasonably establish.
Author: Michelle Bebbington Lic.Ac.TCM – ARA Board Member
About the author
Michelle Bebbington is an acupuncture practitioner, educator, examiner and board member of the Acupuncture Regulatory Authority (ARA).