4a. How large was the estimate of the intervention effect?

Dichotomous (binary) outcomes

Randomised trials can evaluate dichotomous outcomes (‘yes’ or ‘no’ classifications such as incidence of stroke, myocardial infarction or cancer) or continuous outcomes (eg, peoples’ weight, duration or intensity of symptoms, quality of life). For dichotomous outcomes, studies report the proportion of participants in whom events occur. Consider, for example, an RCT comparing dietary supplementation with placebo in which 20% and 25% of the dietary supplement and placebo groups, respectively, suffered a stroke within 36 months of follow-up (table 1). In this hypothetical example, the frequency of the outcome in each group is the cumulative incidence, also referred to as the risk and is defined as the proportion of study participants in each group who experience a stroke over the duration of the trial (36 months). How can we express these results?

View this table:

• View inline
• View popup

Table 1

Expressing randomised controlled trial results of a hypothetical dietary intervention

The best estimate of the effect of the intervention is typically referred to as the point estimate, and this estimate can be expressed as in absolute or relative terms. One possibility is to calculate the absolute difference in the risk (the absolute risk reduction (ARR) or risk difference (RD)) of stroke in the control group and the risk of stroke in the intervention group (table 1), in this case 0.05 (5%), which means that with using the dietary supplement as compared with a placebo 5 fewer per 100 people will suffer a stroke over 36 months. Using the ARR we can calculate the number needed to treat (NNT), the number of people who would need to receive treatment over a specified time period to avoid one stroke, in this case 20 people would need to receive treatment for 36 months to avoid one stroke (NNT=20). Another possibility is to express the impact of intervention as relative risk (RR) or OR (table 1), in this example 0.80 (80%) and 0.76 (76%), which are the most commonly presented measures of the interventional effect for dichotomous outcomes. We can also calculate the complement of the RR and OR—relative risk reduction (RRR) or relative odds reduction (ROR). In this example the RRR is 20%, meaning that of the individuals who experience the adverse outcome in the intervention group, 20% fewer (in relative terms) will experience the event if receiving the intervention.

If the investigators take the time-to-an-event into account, what is sometimes called a survival analysis, a method often used to evaluate therapies for patients with cancer, then the calculation of the frequency of the outcome incorporates time and the resulting relative measure is called the HR. Calculating the HR requires using information about the risk collected at different time intervals and it accommodates censoring of patients, common in trials of patients with advanced cancer. The HR suffers somewhat less from the risk of bias associated with missing outcome data, taking into account if the risks differ at different time points. Expressing the relative effect over time as an HR will typically yield estimates similar to an RR or OR, particularly if events are rare (eg, <20%) though they are typically further from 1.0 (null value) than the RR or OR. Occasionally when the risk of events are high, the HR will be substantially different from the RR or OR. The interpretation of HR is similar to the interpretation of RR, however the latter does not take into account the timing of an outcome occurrence.

RRR is a commonly presented measure of effect in clinical studies, and unfortunately, particularly by the media. From clinicians, patients or the general public’s point of view, the most useful option by far is the ARR or RD,6–8 ideally presented alongside the baseline (control group) absolute risk.9 By contrast, relative effects can be misleading. Consider the following hypothetical example, an RR reduction in a study is 50% and the effect size, at first glance, is considered large. However, a 50% reduced risk may mean a reduction in adverse (eg, stroke) events from 40% to 20% (a 20% ARR), or it can mean a risk reduction from 0.2% (2/1000) to 0.1% (1/1000), a 0.1% ARR (see figure 1). Different ARRs based on the same corresponding RRRs often have very different implications, for example, for a person considering changing their diet or taking an additional nutritional supplement. For full transparency, if the measure of association in an RR or OR, the ‘experimental group risk’ and the ‘control group risk’ should always be clearly presented.9 If the measure of association is an HR based on time-to-event data, there is guidance on calculating the risk in the experimental and control group.10

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Relative and absolute effects in context to baseline risk. RRR, relative risk reduction.

4b. How precise was the estimate of the intervention effect?

Measures of treatment effect observed in a study, presented in table 2 and box 2, are called point estimates (eg, MD, RR, RD) and represent the best estimate of the size of the true effect of the intervention.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

CIs in studies of different size. RR, relative risk; x-axis, different possible RR; y-axis, probability of the true relative risk reduction having indicated value; see description box 2; 95% CI calculated using the standard formula for 95% CI, which takes into account the data presented in table 1 and the width of the CI.

To describe the precision of the estimate, we refer to the range of plausible estimates around the point estimate which is called the CI—the range of values in which the true effect of the intervention likely lies.2 More specifically, if a point estimate were true and one was to repeat an identical study a multitude of times, 95% of the time the estimate would fall within the interval. Our definition of a 95% CI and the ‘true’ estimate applies if, and only if, ‘all the assumptions used to compute the intervals are correct’.15 Box 2 presents an example that illustrates the use of the CI in a trial reporting on a binary outcome, and box 3 returns to the opening clinical scenario to resolve the how large (or small) the results are based on PREDIMED.

5a. Were the study patients similar to the patient in my practice?

It is reasonable to consider that the results of a trial are sufficiently applicable to your patient if he or she meets the eligibility criteria for enrolment in the RCT. Often, however, your patient will differ from the participants in the trial (eg, your patient is older, has a more severe disease, consumes a different pre-intervention dietary pattern, is better or more poorly nourished or has additional comorbidities).

It is possible that the results of the study can be applied even if there are differences between your patient and those enrolled in the trial. The question clinicians should ask themselves is whether those differences would lead to a substantially different effect on the outcome of interest; the answer will often be no, leaving the clinicians confident in applying the results. In cases in which the answer is ‘maybe’, clinicians will be less confident in applying the results, but if benefits in the eligible population clearly outweigh the harms and burdens, then recommending nutritional intervention may still be appropriate.

An often relevant issue is a patient’s baseline nutritional status. Nutritional status can vary in its definition and includes the overall state of nourishment as well as markers of tissue nutrient concentrations. Unlike many drugs, nutrients are absorbed, metabolised and retained to some degree to meet homeostatic functions; however, individuals can vary considerably in markers of baseline status, and lower status indicators generally increase the risk of nutrient deficiency. Vitamin D is a well-studied example whereby plasma 25(OH)D3 is regarded as a marker of vitamin D status. Consider, for example, a trial in critically ill patients with vitamin D deficiency of vitamin D supplementation.16 The Correction of Vitamin D Deficiency in Critically Ill Patients (VITdAL-ICU) trial randomised and analysed 475 patients in Austria to a high dose of vitamin D3 (a bolus of 540 000 IUs followed by 5 monthly doses of 90 000 IU) or placebo. After 6 months follow-up, investigators found similar length of hospital stay, hospital mortality and 6-month mortality. However, it is reasonable to query whether effects in this trial differed by baseline vitamin D status, a known effect modifier of the response to vitamin D supplementation.17

RCTs may present results from subgroup analyses in which the effects of the intervention are tested in subsets of participants according to one or more baseline participant characteristics (potential effect modifiers such as nutrient status). Before applying the results of a subgroup analysis, users of the literature should assess its validity using published criteria that include 11 questions (eg, statistically significant test of interaction (a statistical test for the difference in the treatment effect between subgroups), consistent findings across-related outcomes).18 Based on the VITdAL-ICU trial, the criteria for the assessment of the validity of subgroup analysis are presented in table 3. While valid subgroup findings that can be confidently applied to practice are rare, the more criteria that are met the more likely the subgroup effect is real or valid. In the VITdAL-ICU trial authors reported baseline vitamin D status, and the effect of vitamin D3 on hospital mortality differed significantly (p value for interaction=0.04) between patients with severe deficiency (HR 0.56; 95% CI 0.35 to 0.90) versus in patients with less severe deficiency (HR 1.12; 95% CI 0.72 to 1.77).

Having answered ‘yes’ to 9 of 11 questions, the subgroup on severity of vitamin D deficiency is highly valid. Such examples of highly valid subgroups are hard to find, and readers should bear this in mind when interpreting subgroups reported in RCTs. To verify one’s findings for the validity of RCT subgroups, readers should continuously look to see if the subgroup findings have been replicated in emerging RCTs (ie, question 9) and particularly in systematic reviews of such RCTs using similar validity criteria.19

Unfortunately, many RCTs, despite being directly spawned by observational epidemiology comparing high versus low nutrient status indicators, fail to incorporate baseline nutrient status in their screening and enrolment criteria. Indeed, significant bodies of observational epidemiology comparing high versus low 25(OH)D and erythrocyte omega 3 status (eicosapentaenoic acid [EPA]+docosahexaenoic acid [DHA] as % total fatty acids) indicate potential risk reductions in a variety of chronic disease endpoints,20 21 though questions about the causal nature of such relationships remain. Trials that have measured baseline omega 3 status have noted that baseline levels are higher than typical in Western diets and close to protective ranges seen in the epidemiological literature,22 whereas others that have reported a lower baseline status and achieved a modest protective status on supplementation, observed protective effects only in a subgroup with low fish intake.23 Many nutrients, unfortunately, do not have a readily measurable nutrient status indicator24 and it remains challenging to assess effect modification by baseline status; self-reported dietary intakes or supplement use may aid in discriminating individuals based on habitual intake and/or status, though this too is not often readily accessible or valid.

5b. Were the study interventions likely to be reproducible for my patients in my practice?

5c. Were all outcomes of important to patients considered?

Clinicians use interventions if they provide benefits important for patients. Demonstrating that a single intervention such as a reduced salt diet modestly decreases blood pressure, or that a dietary supplement slightly improves lipid profiles in an otherwise healthy patient does not alone provide adequate justification for administering an intervention. On the other hand, dietary pattern studies that involve multiple dietary changes (eg, increased monounsaturated fat and increased fruits and vegetables high in potassium, fibre) such as a Mediterranean-style diet may have additive effects and may be more likely to impact outcomes important to patients (eg, stroke risk).1

Researchers often use surrogate (intermediate) outcomes rather than those that patients consider important. In such instances, clinicians should avoid assuming meaningful improvements in patient-important outcomes-based solely on changes in surrogate outcomes. Referring back to the vitamin D example, even if it reduces bone mineral density loss, this does not mean patients will experience a favourable outcome that is important—that is, a reduction in bone fractures. What is optimal to support the decision to implement a dietary intervention is evidence that the intervention improves outcomes that are important to patients, such as quality of life or an appreciable reduction in the risk of bone fractures, myocardial infarction, stroke or mortality.

The challenges of conducting adequately powered randomised trials of dietary interventions such as foods or food patterns that measure critically important health outcomes (eg, mortality) have led the nutrition field to rely heavily on surrogate (intermediate) outcomes. While surrogate outcomes are useful, including their ability to link individual dietary components across a dose-response range to a potential disease risk in controlled feeding trials, the reliance on them presents substantial challenges. Diseases are highly complex processes involving multiple factors across pathways for which many potential surrogates, sometimes of unknown causal significance, exist. Complicating this further, diets contain numerous components that can impact surrogate outcomes in opposing directions. This can be seen most clearly in the case of macronutrients (eg, fats, protein and carbohydrates) and atherosclerotic cardiovascular disease, where there has been a long focus on the reduction of saturated fatty acids (SFA) with any macronutrient and its ability to lower low density lipoprotein [LDL-C], a surrogate biomarker with an arguably known causal relationship with atherosclerotic disease. Systematic reviews with meta-analyses of over 80 controlled feeding trials in humans28 demonstrate that replacing SFA with any macronutrient source lowers LDL-C. However, in the case of carbohydrate replacing SFA in the diet, there is a concomitant delirious increase in triglyceride and a reduction in high density lipoprotein-cholesterol [HDL-C]. This simple example only considering one domain of cardiovascular risk factors (ie, blood lipids) readily demonstrates the challenges of predicting major disease outcomes (eg, cardiovascular mortality) from surrogate outcomes. The promiscuous pleiotropy of nutrients within biological systems readily lends themselves to influencing multiple circulating biomarkers (surrogates) of potential relevance to diseases and requires careful considerations when being used to link dietary intake, particularly in the context of dietary patterns, to disease risk.

Even when a trial reports favourable results of an intervention on one outcome of importance to patients, clinicians must also consider the effects of the intervention on other patient-important outcomes. For example, potential environmental issues aside, reducing red meat consumption may result in very small reductions in the lifetime risk of cancer,29 but may also decrease patient satisfaction with diet and/or quality of life sufficiently that even if benefits were real, informed patients would choose to continue their preferred diet.30 31 RCTs often neglect to document satisfaction with diet or the impact of dietary interventions on quality of life.26 32

As with PREDIMED, clinical trials frequently report on composite outcomes to reduce the required sample sizes and the duration of follow-up. Composite outcomes can, however, be misleading if components vary in importance (eg, death vs angina event), particularly if a favourable result is driven by the least important component.33 For instance, the St Thomas’ Atherosclerosis Regression Study (STARS) randomised men with angina referred for angiography to dietetic advice (total fat no more than 27%, saturated fat 8–10% and omega-3 and omega-6 polyunsaturated fatty acid [PUFA] at 8% daily energy intake) plus usual care versus usual care alone.34 The dietetic advice group achieved the targeted fat intakes and a large ARR compared with usual care, demonstrating 41 fewer combined cardiovascular events per 100 people followed (95% CI 18 to 66, p=0.0006). However, combined cardiovascular events, a composite, included cardiovascular deaths, fatal and non-fatal myocardial infarction, stroke, as well as coronary heart disease events (angina, angioplasty, coronary artery bypass surgery). Based on one event in the dietetic group and three events in the usual care group, the ARR was 7 fewer cardiovascular deaths per 100 people followed (95% CI 6 to 20, p=0.3084), while the ARR was 24 fewer coronary heart disease events per 100 people followed (95% CI 3 to 46, p=0.0239) based on 3 events in the dietetic group and 10 events in the usual care group, indicating that coronary events, outcomes of far less importance to patients, drove the combined cardiovascular events estimate.34 As compared with STARS, PREDIMED’s composite outcomes were fewer (three vs seven) and consisted of outcomes that were of similar patient importance, though cardiovascular death would be of more importance than stroke or infarction. Among the three outcomes, only stroke proved statistically significant (5-year absolute risk 1.7%, 1.5% and 3.0% for Mediterranean+EVOO, Mediterranean+nuts and low-fat arms, respectively; with small but likely important absolute RDs ranging from 1.3% to 1.5%).

5d. Are the likely benefits of the intervention worth the potential harms and burdens (including cost if applicable) based on my patient’s values and preferences?

Based on the above issues for consideration, if you have decided that the results of a study are applicable to your patient, the last issue to consider is the balance between the probable benefits and harms of an intervention and the associated burden (eg, dietary satisfaction, access to food) and costs. The impact of an intervention is related not only to the RR reduction for a target outcome(s) important to patients, but the ARR (ie, RD) based on the baseline (or control group) risk of the outcome(s).8 Returning to the PREDIMED trial, the Mediterranean diet supplemented with EVOO reduced the RR for major cardiovascular events by 31% (95% CI 9% to 46%), that is, in about one-third.1 This may sound impressive, but the likelihood of the intervention impacting your patient may be very small. Again, this is best understood by considering the between-group RD (ie, ARR) of an outcome (ie, major cardiovascular event), in this case 2.1% over 5 years. Expressed in different absolute terms, out of 1000 patients followed, 21 fewer will have a major cardiovascular event with EVOO as compared with control. Below we discuss two patients who might consider a Mediterranean diet supplemented with four tablespoons of EVOO per day.

In the first case, a 45-year-old woman of European origin presents with the following risk factors for CVD: treated hypertension with a current blood pressure [BP] of 130/80 mm Hg, HDL cholesterol of 35 mg/dL, a total cholesterol of 200 mg/dL and who has given up smoking a month ago. She has no signs of current heart disease and the patient tells you she is following a new smoking cessation programme. As in box 3, you use the atherosclerotic cardiovascular disease calculator to find that her risk of a major cardiovascular event (myocardial infarction, stroke or death due to CVD) within the next 10 years is 2.3%. The Mediterranean diet (with added EVOO as compared with the control ‘low-fat’ diet (on evaluation the control group consumed on average 37% fat and the diet reflected more of a Mediterranean dietary pattern when considering the control group ‘Mediterranean Diet Adherence Screener’ (MEDAS) scores) may reduce the risk of major events by 31%, thus to 1.6%, an ARR of 0.7%. It is possible that the ARR may be more pronounced when applied to an American setting wherein the patients diet reflects more of a Western/Standard American Diet higher in fat, salt and sugar. Given the relatively small decrease in the risk of cardiovascular events, the potential impact of a dietary change on the patient’s quality of life, particularly while she is trying to quit smoking and the additional cost and inconvenience of following a Mediterranean diet supplemented with a half a litre of EVOO per week, the patient might prefer smoking cessation alone.

In the second case, an overweight 59-year-old African American man presents with type II diabetes who is treated for hypertension with a current BP of 130/80 mm Hg, HDL cholesterol of 35 mg/dL and total cholesterol of 200 mg/dL. He is very motivated to take preventive measures. Using the same calculator his risk of cardiovascular event within the next 10 years is 26.1%. An RR reduction of 31% for a major cardiovascular event in such a high-risk patient generates a reduction in absolute terms of 8.1%. Even when the RCT has risk of bias issues (eg, randomisation unconcealed at some centres, stopped early) many patients might still consider following a Mediterranean dietary pattern supplemented with EVOO.

When applicable, equity issues with respect to the absolute difference in risk of an outcome between the groups is a critically important issue to consider before deciding to follow an intervention. As illustrated by the example above, assuming the same RRR for a certain intervention, the benefit for the patient is more likely if the risk of a negative outcome without intervention is higher. That is, a 26.1% baseline risk in a 59-year-old African American man shows an 8.1% ARR, while the 2.3% baseline risk in 45-year-old Hispanic woman shows a 0.7% ARR with a Mediterranean diet+EVOO intervention. Further, the 10-year cardiovascular risks will often differ in men versus women and sometimes between races; sometimes trivially, but sometimes the difference in baseline risks are substantial. For instance, for an African American woman, the baseline risk of having a major cardiovascular event is 6.0% lower (20.1%) than an African American man (26.1%) and the corresponding ARR, assuming the same RRR for the intervention, is 6.2% rather than 8.1% with a Mediterranean diet+EVOO intervention. Please note, the equity scenarios above use ‘heart disease risk calculators’ with multiple assumptions, and so the estimates are accompanied by some uncertainty.

In weighing the benefits and harms of intervention, we also need information on the adverse effects of intervention (while noting that RCTs evaluating the effects of interventions often include too few patients to detect rare but serious adverse events). Despite RCT results of the Women’s Health Initiative Calcium/Vitamin D Supplementation trial, the safety of calcium with vitamin D has been questioned for the management of osteoporosis. In a re-analysis of 16 718 women who were not using personal calcium supplementation at randomisation, the HR for an increased risk in major cardiovascular events in those randomised to calcium and vitamin D was 1.16 (95% CI 1.01 to 1.34) (non-fatal myocardial infarction or revascularisation, p=0.04), whereas in the women already using calcium supplements at randomisation, the cardiovascular risk did not change after allocation to calcium and vitamin D.35 However, it must be noted, when interpreting the validity of this apparent subgroup effect using the 11 questions in table 3 (above), the answer to about half of the questions is ‘no’ suggesting the subgroup effect is not necessarily valid. Regardless, in some instances, patients may be incurring out-of-pocket costs for nutritional interventions that have the possibility of harm, or no desirable benefit.

Finally, we need to consider and explain the trade-offs between potential benefits, harms, burden and costs (if relevant) of different options in an equitable way to our patients and help them choose the option that is most compatible with their personal values and preferences.

Clinical scenario resolution

The PREDIMED study that we identified in the opening scenario found a decreased risk of a composite outcome of cardiovascular death, stroke and myocardial infarction among participants at high risk of cardiovascular events assigned to a Mediterranean-style diet supplemented with EVOO or nuts, compared with a modestly less intense Mediterranean-style (rather than low fat) diet.1 The authors reported no relevant diet-related adverse events, though there were approximately 1% more withdrawals in the intensified Mediterranean diet groups. The trial shows a 1.7–2.1% 5-year ARR in major CVD events when olive oil or nuts (primarily walnuts) as well as when modest increases in legumes and seafood are added to a Mediterranean-style diet rather than a low-fat or Western-style diet. Before applying the results of this study, it is also important to consider the current diet of your patient and the degree of dietary change they are likely to implement. If your patient follows a Western-style diet, and they can adopt a Mediterranean-style diet, we may assume that the effects will be similar or even more pronounced, particularly for stroke, the only composite event with a statistically significant reduction. In addition, we already concluded in Part 1 that the PREDIMED trial had methodological limitations with three of six validity questions at high risk of bias, therefore one needs to be cautious in applying the results of this study to your patient.

After explaining the potential benefits relative to the burden, your patient explains their hesitations to you. The patient is very motivated and finds the Mediterranean diet to be acceptable but is concerned about the financial costs of daily olive oil or nuts. You explain that the components of a Mediterranean diet allow some flexibility in defining the pattern to follow and that the costs could be reduced by focusing on a number of foods that are relatively cheap (EVOO, walnuts, whole grain oats, inexpensive legumes and fish (catfish, mackerel, sardines), as well as fruits and vegetables such as apples, oranges and carrots). The patient can purchase most of these foods in bulk, including canned or frozen fruits, vegetables and fish (if more convenient), for as little as $75 per month. Ultimately, the patient accepts that the addition of these foods to his diet would be beneficial. The patient chooses to change his diet towards a Mediterranean pattern by regularly consuming the above foods for 3 months, at which time he will return for a follow-up visit.

Overall, in addition to assessing the potential for risk of bias in the trial,3 clinicians must consider the results, from trivial to potentially large effects on all desirable (benefits) and undesirable (harms) outcomes reported.36 Subsequently, they must assess the applicability of the study based on the unique circumstances of their patient, including their patients values and preferences.5 31 36 37 While PREDIMED has risk of bias issues, the results are reasonably compelling and may appeal to some patients.