Dr Viv Rolfe’s critically analyses research techniques and shares how limitations pose a challenge to herbal medicine research.
Non-communicable diseases like metabolic disease and mental health conditions are dramatically reducing the quality of life in young people (1). As more people globally are living in sub-optimal health, there have been attempts to define what this means, and five elements are considered important — digestive system, cardiovascular system, immune system, fatigue and mental status (2).
Readers of Herbal Reality will need no encouragement here to recognise that these are all areas where herbal remedies are beneficial and have been extensively written about on this website.
Readers will also need no introduction to the struggle that the herbal community often experiences in gaining acknowledgement that practice-based wisdom and scientific evidence exists to support the use of herbs. Mentioning the words herbal and evidence in the same sentence, or the triple threat of herbalism, holistic care or homeopathy, results in regular social media firestorms.
We need to go deeper into the debate to consider its complexity and unpick the very idea of ‘evidence’ and ‘research’ to see why herbal evidence struggles to gain traction. This article will consider difficulties within our wider research cultures and methods, and highlight the complexities faced when researching herbs. It offers advice for readers of herbal papers and researchers along the way.
Research and culture methods
There are many problems faced by medical and health research relating to research integrity and our research paradigms, and this is not an exhaustive list:
- One author identified 44% of data within an anaesthesia journal as being falsified (3)
- Papers being retracted due to scientific misconduct (4)
- The use of artificial intelligence to falsify publication material which could become a significant threat to medical decision-making (5)
- Not being able to reproduce research results — reproducibility estimated at 51–89% (6)
- An estimated half of commonly used drug treatments having unknown efficacy (7)
- Lack of external validity and generalisability of randomised controlled trials (8)
- Lack of patient involvement in defining patient-relevant outcome measures (9)
- Publication bias and tendency to publish only positive results.
And in relation to herbal medicines specifically;
- Reductionist research approaches are unsuitable for evaluating the effects of herbs (10).
In summary, research today is lacking in robustness and the finger should not just be pointed toward herbal or other complementary therapies regarding the existence of poor-quality evidence. There is much work for the research community overall to do here.
The next section focuses on systematic review (SR). In using evidence to make decisions, these research articles are placed at the top of the hierarchy pyramid. All the problems mentioned already feed into the health of these reviews, and SRs also bring their own problems.
If any reader would like a refresher on different types of evidence and an introduction to randomised controlled trials (RCT) or SRs, O’Rawe’s article on this website is excellent (10).
Systematic reviews
Systematic reviews are at the top of the evidence hierarchy. They use robust methods for retrieving and examining RCT and sometimes observational studies, and pool the data in statistical tests (meta-analyses) or provide a narrative.
It is highly problematic to place so much sway on SRs and this will be explored next.
SRs are everywhere! The number of SRs published is rising at a pace (search for SR on PubMed and look at the numbers of publications over the years). Authors may choose to conduct meta-reviews or umbrella reviews that review all the SRs on a topic. One umbrella review located 101 SRs for a CBD topic (11), and so within those SRs would be hundreds of RCTs if you think about it.
One concern is that some articles with SR in the title are not true SRs. Along with the high numbers, this makes it difficult for us to select good quality reviews. This is apparent in research that has explored the quality of SRs and rather unsurprisingly the quality is variable, although for herbal medicine research this is no different to other medical disciplines (12). SRs published with specialist organisations like Cochrane or JBI, are higher quality and should be used first to search for herbal articles.
As authoring SRs becomes popular another problem is that people are writing similar, almost identical reviews. For popular herbs like turmeric there can be multiple reviews which creates the illusion that there is more research on these topics. The similar reviews can produce different results as the authors work to different standards and include different articles. There are techniques for calculating the level of overlap that encourages readers to select the highest quality review (13). However, calculating overlap isn’t a quick step to do.
Delving more into the SR methods, they are designed to minimise bias through robust searching and using tools to describe the biases within the primary RCTs (selection, performance, detection, attrition, and publication biases). However, the proportions of reviews not reporting biases is high (14).
So SRs are problematic, and there are additional challenges when fitting herbal medicine into the reductionist SR model.
To make reviewing simpler, authors sometimes only include English language papers (bias claxon). For indigenous herbs or those connected to traditional Chinese medicine (TCM) or Ayurveda for example, language should not be restricted and local databases should be used. The exclusion of other languages can alter the results of meta-analyses (15). In a review of Andrographis paniculata for upper respiratory tract infections, 39 RCT were retrieved whereas similar reviews mentioned in this article that failed to search Chinese or Indian databases retrieved far fewer papers (16).
When searching for herbs the keywords should include the botanical name as well as any common or product names. I’d recommend using Kew’s Medicinal Plant Naming Service to identify names. For TCM or Ayurvedic herbs, the health or disease conditions should be described using authentic terms. If you are reading a SR on traditional medicines that do not use botanical or authentic names, then I would be cautious in using the results.
The highlight of any SR is the meta-analysis which is a statistical test and visual diagram created by extracting the clinical outcome data from the RCT (you can look at Hu’s paper as an example, 16). The different studies appear as items on the plot and the pooled result at the bottom. This statistical approach is problematic for herbal medicines as it factors in the variability between studies. Herbs exist in a wide range of preparations — whole herb, extracted active compound, dietary supplement — so the variability or heterogeneity is often very high. High heterogeneity can give the impression of poor quality and the conclusions can be strongly made against the clinical use of the herb all together rather than focusing on the underlying reasons. The problem stems from the primary RCTs where the details of herbs used can be insufficient.
If I was to offer an opinion based on years of authoring SRs and researching them, it would be that the majority of SRs published are not suitable for making clinical decisions. Their position at the top of the evidence hierarchy is shaky. Readers should be mindful and look for the following attributes:
- SRs published in specialist organisations like Cochrane or JBI
- SRs published in reputable journals that regularly publish them
- Look for elements that signify quality in the review — use of Cochrane risk of bias tool, registering of protocol with Prospero (or similar), use of the PRISMA reporting tool and a robust search.
Anything with SR in the title that does not refer to extensive methods or lists just one bibliographic database is inadequate for making decisions.
Weaving in complexity
Going back to our original idea that herbs can help people in sub-optimal health, we’ve already considered that herbal remedies struggle within the reductionist “pill for every ill” approach of today’s research. We could go right back to our roots and describe how using herbs can be deeply spiritual and represent an interconnectedness with nature — for healers and patients — but that is way off the radar for any clinical trial. Let’s consider other qualities of herbal medicines like the complexity of the botanical material and that herbs can impact multiple biological targets. The RCT and SR typically examine one intervention and one primary patient outcome (with secondary outcomes). So where do we start for herbs?
We need to consider the RCT which feeds the SRs with data. Authors need to clearly define the botanical materials used and in a standardised way for future comparative purposes. Is it:
- An isolated active compound or synthetic derivative
- A whole herb standardised to pharmacopoeia specifications
- Whole herb mono-preparation non-standardised
- Herbal blend with proportions of herbs listed/ extraction methods
- Dosage
- Is it in a capsule, tablet or liquid?
Providing this level of detail should be mandatory for article publication. It would allow the herbal characteristics to be compared and help with the heterogeneity problem where all herbal interventions examined in SR are placed in the same basket.
The second area that requires consideration is that herbs typically affect multiple body systems. Fortunately the ‘omic’ techniques are adept at taking a holistic view of genes, proteins etc, and can assist in our understanding of the interconnectedness of our gut microbiota and our mood, or oral health and cardiac outcomes, for example. The technology is on our side and we are able to incorporate multiple measures within a single study. One that looked at a turmeric supplement for cognition also tested for changes to the microbiota, urinary proteomics alongside surveys of participant wellbeing (17).
We can modify trial design and use a range of technologies to better understand the effects of herbs, but there are other opportunities. The RCT and SR, by narrowing their focus to reduce bias, they reduce external validity and results cannot be generalised to other conditions or groups of people. The use of pragmatic trial design is helpful as it is more patient-centred and often incorporates qualitative data for richer perspectives on health and wellbeing (18). One study included an observational component to explore complex treatments in a more real-life situation (8).
The herbal research community could focus on carrying out good quality studies that add richness through applying different technologies and designs. Alongside this, I believe we also need to reinvent the evidence hierarchy which is discussed next.
Reinventing the evidence hierarchy
We’ve discussed that placing SRs at the top of the hierarchy is problematic due to the variability of review quality. If we go back to the early imaginings of evidence-based medicine it suggested the evidence, practice and patient views were of equal importance (19). We could re-imagine these relationships and not present a hierarchy with its built-in insinuation that some research is better than others, but return to an interconnected model (Figure 1).
So what do we need to change? We need to be better at capturing the practice of herbalism. Although publishing case studies does not seem to be popular, there are many events that represent practitioners’ experiences which could be harnessed in some way? The patient voice is also absent, a criticism throughout medical research (9). This fits more with the “evidence house model” proposed by O’Rawe which includes plant chemistry, folk law and tradition (10). Methods that triangulate data from multiple sources would create stronger, more meaningful evidence relating to the complex effects of herbs. So what do we need to change? We need to be better at capturing the practice of herbalism. Although publishing case studies does not seem to be popular, there are many events that represent practitioners’ experiences which could be harnessed in some way? The patient voice is also absent, a criticism throughout medical research (9). This fits more with the “evidence house model” proposed by O’Rawe which includes plant chemistry, folk law and tradition (10). Methods that triangulate data from multiple sources would create stronger, more meaningful evidence relating to the complex effects of herbs.
Summary
So how can we grow the use of herbal remedies to benefit the growing burden of sub-optimal health?
It is a challenging time for the academic research community with declining researcher integrity and ease of use of artificial intelligence. Herbal research does not sit well within the reductionist RCT and SR approaches. Using new technologies and adapting trial design to be more real life could capture more complexity, yet still have the rigour of an RCT at its heart. By being more robust in the conduct of SR — preventing duplication, adhering to the stringent methods of specialist organisations, and taking into consideration the traditional use of the plant, we can create a stronger case for the use of herbs in society.
Organisations with a vested interest need to come together to share ideas and set the standards for researching and publishing. The herbal research community could trailblaze a more interconnected decision-making model that takes a holistic view of evidence through weaving in practice wisdom and patient perspectives.
This all feels like a dream, and it does depend on what the goal is. There are more and more people struggling with their health. Holistic evidence could be gathered for the strongest candidate herbs and conditions to raise better awareness of them. Increasing the number of good quality papers would help influence the medical and regulatory world in which we live, but that is a loftier aspiration.
References
- Salam, R.A., Khan, M.H., Meerza, S.S.A. et al (2024). An evidence gap map of interventions for noncommunicable diseases and risk factors among children and adolescents. Nature Medicine, 30, 290–301. https://doi.org/10.1038/s41591-023-02737-2
- Wang, W., Yan, Y., Guo, Z. et al (2021). Suboptimal Health Study Consortium and European Association for Predictive, Preventive and Personalised Medicine. All around suboptimal health — a joint position paper of the Suboptimal Health Study Consortium and European Association for Predictive, Preventive and Personalised Medicine. European Association for Predictive, Preventive and Personalized Medicine Journal, 12(4):403-433. doi: 10.1007/s13167-021-00253-2
- Carlisle, J. B. (2021). False individual patient data and zombie randomised controlled trials submitted to Anaesthesia. Anaesthesia, 76(4), 472-479
- Redactionwatch.com (2022). Nearing 5,000 retractions: A review of 2022. Available: https://retractionwatch.com/2022/12/27/nearing-5000-retractions-a-review-of-2022/
- Elali, F.R. & Rachid, L.N. (2023) AI-generated research paper fabrication and plagiarism in the scientific community. Patterns (N Y), 4(3):100706. doi: 10.1016/j.patter.2023.100706
- Freedman, L.P., Cockburn, I.M. & Simcoe, T.S. (2015) The Economics of Reproducibility in Preclinical Research. PLoS Biology, 13(6):e1002165. doi: 10.1371/journal.pbio.1002165
- Lee, M. S. (2019). Gaps of perception on evidence and the role of systematic reviews in evidence-based medicine. Integrative Medicine Research, 8(2), 131
- Teut, M., Walach, H. & Varanasi, R. et al. (2020). Recommendations for designing, conducting and reporting observational studies in homeopathy. Homeopathy, 109(03). 114-125.
- Heneghan, C., Goldacre, B. & Mahtani, K.R. (2017). Why clinical trial outcomes fail to translate into benefits for patients. Trials, 18. 122. https://doi.org/10.1186/s13063-017-1870-2
- O’Rawe, D. (2022). The evidence house of Herbal Medicine: A holistic approach to contemporary research. Available: https://www.herbalreality.com/herbalism/history/evidence-house-herbal-medicine-holistic-approach-to-contemporary-research/
- Solmi, M., De Toffol, M. & Kim, J. Y. et al (2023). Balancing risks and benefits of cannabis use: umbrella review of meta-analyses of randomised controlled trials and observational studies. British Medical Journal, 382 :e072348. doi:10.1136/bmj-2022-072348
- Rolfe, V. (2022). A quality assessment of a sample of herbal medicine systematic reviews using R-AMSTAR. Phytomedicine Plus, 100380.
- Pieper, D., Antoine, S. L. & Mathes, et al. (2014). Systematic review finds overlapping reviews were not mentioned in every other overview. Journal of Clinical Epidemiology, 67(4), 368-375.
- Wuytack, F., Regan, M., Biesty, L. et al. (2019). Risk of bias assessment of sequence generation: a study of 100 systematic reviews of trials. Systematic Reviews, 8, 13. https://doi.org/10.1186/s13643-018-0924-1
- Grégoire, G., Derderian, F. & Le Lorier, J. (1995). Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? Journal of Clinical Epidemiology, 48(1):159-63. doi: 10.1016/0895-4356(94)00098-b
- Hu, X.Y., Wu, R.H. & Logue, M. et al. (2017).Andrographis paniculata (Chuān Xīn Lián) for symptomatic relief of acute respiratory tract infections in adults and children: A systematic review and meta-analysis. PLoS One, 4;12(8):e0181780. doi: 10.1371/journal.pone.0181780
- Wightman, E., Khan, J. & Smith, E. (2023). Chronic supplementation of a multi-ingredient herbal supplement increases speed of cognitive task performance alongside changes in the urinary metabolism of dopamine and the gut microbiome in cognitively intact older adults experiencing subjective memory decline: a randomized, placebo controlled, parallel groups investigation. Frontiers in Nutrition, 10:1257516. doi: 10.3389/fnut.2023.1257516
- Patsopoulos, N.A. (2011). A pragmatic view on pragmatic trials. Dialogues in Clinical Neuroscience, 13(2):217-24. doi: 10.31887/DCNS.2011.13.2/npatsopoulos
- Haynes, R.B. (2002). What kind of evidence is it that Evidence-Based Medicine advocates want health care providers and consumers to pay attention to?. BMC Health Service Research, 2, 3. https://doi.org/10.1186/1472-6963-2-3
