Article Text

other Versions

Relative validity and reliability of a diet risk score (DRS) for clinical practice
  1. Emily A Johnston1,
  2. Kristina S Petersen1,
  3. Jeannette M Beasley2,
  4. Tobias Krussig3,
  5. Diane C Mitchell1,
  6. Linda V Van Horn4,
  7. Rick Weiss5 and
  8. Penny M Kris-Etherton1
  1. 1Department of Nutritional Sciences, Pennsylvania State University, University Park, Pennsylvania, USA
  2. 2Department of Medicine, NYU Langone Medical Center, New York City, New York, USA
  3. 3Penn State College of Medicine, University Park, Pennsylvania, USA
  4. 4Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
  5. 5Viocare Inc, Princeton, New Jersey, USA
  1. Correspondence to Emily A Johnston, Nutritional Sciences, Pennsylvania State University, University Park, PA 16801, USA; eajohnst{at}gmail.com

Abstract

Introduction Adherence to cardioprotective dietary patterns can reduce risk for developing cardiometabolic disease. Rates of diet assessment and counselling by physicians are low. Use of a diet screener that rapidly identifies individuals at higher risk due to suboptimal dietary choices could increase diet assessment and brief counselling in clinical care.

Methods We evaluated the relative validity and reliability of a 9-item diet risk score (DRS) based on the Healthy Eating Index (HEI)-2015, a comprehensive measure of diet quality calculated from a 160-item, validated food frequency questionnaire (FFQ). We hypothesised that DRS (0 (low risk) to 27 (high risk)) would inversely correlate with HEI-2015 score. Adults aged 35 to 75 years were recruited from a national research volunteer registry (ResearchMatch.org) and completed the DRS and FFQ in random order on one occasion. To measure reliability, participants repeated the DRS within 3 months.

Results In total, 126 adults (87% female) completed the study. Mean HEI-2015 score was 63.3 (95% CI: 61.1 to 65.4); mean DRS was 11.8 (95% CI: 10.8 to 12.8). DRS and HEI-2015 scores were inversely correlated (r=−0.6, p<0.001; R2=0.36). The DRS ranked 37% (n=47) of subjects in the same quintile, 41% (n=52) within ±1 quintile of the HEI-2015 (weighted κ: 0.28). The DRS had high reliability (n=102, ICC: 0.83). DRS mean completion time was 2 min.

Conclusions The DRS is a brief diet assessment tool, validated against a FFQ, that can reliably identify patients with reported suboptimal intake. Future studies should evaluate the effectiveness of DRS-guided diet assessment in clinical care.

Trial registration details

ClinicalTrials.gov (NCT03805373).

  • nutrition assessment
  • preventive counselling
  • nutritional treatment
  • dietary patterns
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Cardiometabolic disease (CMD) is a leading cause of mortality worldwide1 2 with poor diet quality considered responsible for nearly half of the preventable deaths.3 While diet counselling reduces these risks, physicians are not routinely trained in nutrition,4 medical office visits are time-limited and few diet assessment tools are brief but reliable for this setting.5 A tool that guides brief assessment and counselling in the clinical office visit could improve rates of dietary counselling in clinical care and help reduce diet-related CMD risk.

Less than 25% of patients receive any diet assessment or nutrition counselling from a physician.6 The American College of Cardiology/American Heart Association (ACC/AHA)7 recommend reducing intake of sugar sweetened beverages, processed meats and sodium, while emphasising vegetables, fruits, legumes, nuts, whole grains and fish especially for patients with atherosclerotic cardiovascular disease risk factors. The US Preventive Services Task Force8 recommends behavioural interventions to promote healthy lifestyle changes to prevent cardiovascular disease. There is a gap between these recommendations and the global dietary intake9 and physicians can play a key role in helping patients improve diet quality.

Diet assessment is a process of systematically collecting relevant data needed to identify patients with nutrition-related problems and their causes.10 Development of a brief diet assessment tool that meets time demands but targets vulnerable patients within an office visit could facilitate preventive dietary approaches through strategic nutrition counselling and guideline-based care.7 8 11 We developed and tested a 9-item Diet Risk Score (DRS) designed to address these objectives.

Methods

Creation and scoring of the DRS

The DRS is a 9-item questionnaire created to estimate dietary risk for CMD based on data from Micha et al,3 with additional information from studies on lifestyle and cardiovascular disease (CVD) risk.12–15 Micha et al created a comparative risk assessment model using NHANES (National Health and Nutrition Examination Survey) data from 1999 to 2002 and 2009 to 2012 and meta-analyses of previously published cohort studies to estimate diet-disease relationships.

The DRS contains one question stem, ‘For the following foods, please select the frequency that best describes how often you eat each food or group of foods in a normal week’. The dietary components contributing most to cardiometabolic risk are excess intake of sodium and sugar-sweetened beverages, and inadequate intake of fruits, vegetables, nuts and marine omega-3 fatty acids3 (table 1). Excess sodium intake is responsible for the highest risk in this model, but is difficult to accurately capture using a diet screener because it is ubiquitous in foods,16 intake varies day-to-day17 and it is difficult for the lay person to estimate;18 therefore, we included three items related to intake of the major sources of sodium in the American diet, processed and restaurant food; breads, rolls and sandwiches; salty snacks.19 A single item was allotted for each of the additional dietary components to capture habitual intake. A score of 0 (lowest) to 3 (highest) is assigned for each answer. The maximum score is 27 and the score is divided into tertiles of risk: 0 to 8: low risk; 9 to 18: moderate risk; 19 to 27: high risk.

Table 1

DRS scoring criteria

Tests of validity and reliability

An online survey was created, via REDCap, a secure online platform for building and managing surveys. In order to determine whether the DRS could accurately identify individuals with poor diet quality, the DRS was compared with the HealthyEating Index (HEI)-2015 score (online supplemental table 1) calculated from a validated Food Frequency Questionnaire (FFQ; VioScreen.com). Prior to distribution, the survey was pilot tested among nutrition graduate students and researchers.

Supplemental material

Recruitment involved ResearchMatch, a national health volunteer registry supported by the US National Institutes of Health as part of the Clinical Translational Science Award programme. ResearchMatch has a large database of volunteers who have consented to recruitment for participation in studies. We recruited adults across the USA, between the ages of 35 to 75 years, an age group at higher risk for CMD than younger age groups.20 ResearchMatch data are self-reported. Age and access to ResearchMatch were the only inclusion criteria. We did not collect information on other demographics or on CMD risk.

Eligible participants were emailed a URL link directing them to a landing page with study details. Those who consented were assigned a study ID and randomly assigned one of the two surveys based on continuous alternating order of enrolment by the Survey Research Center (SRC) at Penn State University. After completion of one survey, a short (1 min 14 s) distracting video played, and then the next survey automatically opened. Once the survey was closed, participants who completed the DRS and provided their email address were contacted to complete the DRS a second time within 3 months to measure test–retest reliability.

As an incentive, participants were entered into a gift card lottery after completing both surveys. Participants could also request their detailed diet report from VioScreen at no cost to share with their healthcare provider.

Statistical analysis

Data analysis was conducted using SAS 9.4 (SAS Institute, Cary, North Carolina). A power calculation indicated that a sample of 125 individuals would provide 80% power at an alpha level of 0.05 to detect a minimum acceptable correlation coefficient of r=0.30 between the two tests.21 Participants who reported an energy intake <500 kcal/day on the FFQ were excluded as this was deemed incomplete data. Since total energy intake was not an outcome, no other exclusions were made based on reported total energy. Spearman correlations (PROC CORR) were performed and strength of agreement between quintile rankings was measured using weighted kappa.22 23 Component kappa scores were not calculated due to the categorical nature of the DRS data. Interclass correlations were calculated to determine test–retest reliability. One-way analysis of variance (ANOVA) (PROC ANOVA) was used to determine statistical differences between DRS and HEI-2015 total scores. Age and gender were only collected in aggregate and therefore not included as covariates.

Results

Relative validation

An initial automated email from ResearchMatch was sent to 5218 individuals who met inclusion criteria. Of those, 359 received a survey invitation via email and 346 accessed the DRS, FFQ or both. In total, 128 individuals completed both surveys. Only participants who completed both the FFQ and DRS were included in analyses. One participant reporting <500 calories per day was excluded from the analysis. A sensitivity analysis that excluded subjects reporting an intake of >5000 calories (n=1) was conducted; the results were unchanged (data not reported).

Total participation time lasted approximately 45 min. Mean time to completion of the DRS alone, measured under experimental conditions and estimated through participant access by the SRC, was 2 minutes.

The average DRS of respondents was 11.8 (SD 5.5) out of a maximum of 27 (lower score represents lower risk). The average HEI score was 63.3 (SD 12.1) out of 100 (higher score represents higher diet quality) (table 2). The DRS ranked 37% (n=47) of subjects in the same quintile, and 41% (n=52) within ±1 quintile of the HEI-2015 (weighted κ: 0.28). Online supplemental table 2 shows significant correlations between the DRS and the HEI-2005, 2010 and 2015 for comparison.

Table 2

Mean HEI-2015 score by DRS category

Food group correlations

In post-hoc analyses, DRS Fruit correlated with HEI-2015 Total Fruit and Whole Fruit (r = –0.68 and –0.65, respectively, both p<0.001) (table 3). DRS Vegetable correlated moderately with HEI-2015 Total Vegetable and Green Vegetable components (r = –0.43 and –0.56, respectively, both p<0.001). DRS Sugar-sweetened beverages and HEI-2015 Added Sugars were also negatively correlated (r=−0.39, p<0.001). DRS components intended to capture sodium intake did not correlate with HEI-2015 Sodium (r=0.04 to 0.07, p>0.05). Some DRS components were combined to better map to HEI-2015 components. Separately, DRS Fish and HEI-2015 Seafood/Plant Protein were significantly correlated (r=−0.48, p<0.001) as were DRS Nuts and HEI-2015 Seafood/Plant Protein (r=−0.43, p<0.001); when combined DRS Nuts+Fish had a stronger correlation with HEI-2015 Seafood/Plant Protein component (r=−0.55, p<0.001). When combined, DRS Nuts, Fish and Processed meats correlated with HEI-2015 Fatty Acid Ratio (r=−0.27, p=0.002); the DRS components also correlated with the Fatty Acid Ratio significantly (nuts: r=−0.21, p=0.002) and without reaching statistical significance (fish: r=−0.17, p=0.05, processed meats: r=−0.17, p=0.05). When DRS Breads and Sandwiches and DRS Snacks were combined, they were correlated with HEI-2015 Refined Grains (−0.43 (p<0.001)); DRS Fast Food and DRS Breads and Sandwiches combined also correlated with HEI-2015 Refined Grains (−0.53 (p<0.001)). Similarly, when DRS Fruit, DRS Vegetables and DRS Nuts were combined, there was a correlation of −0.44 (<0.001) with potassium intake (mg) measured by the FFQ. HEI-2015 mean component scores are presented in online supplemental table 3.

Table 3

Post-hoc exploratory analysis of alignment between DRS and HEI-2015 component scores and correlations

We determined percent of participants meeting the goal criteria for fruit, vegetables and processed meats since the DRS is set up to score from 0 to 3, but only two levels are possible in these three groups; 30% of DRS participants had a 0 score (low risk) for processed meats, 62% for vegetables and 56% for fruit (table 4). The sodium intake score was based on risk for intake >2300 mg/day; 70% of participants reported intakes >2300 mg/day.

Table 4

HEI component means by DRS score

Test–retest reliability

In all, 153 participants completed the initial DRS and 102 completed the second DRS. The interclass correlation was r=0.83 reflecting high reliability of the DRS.

Discussion

The DRS moderately, inversely correlated with the total HEI-2015 score derived from a validated FFQ. The DRS ranked individuals similarly based on reported dietary intake with 78% of participants ranked within the same quintile ±1. In this sample of self-selected volunteers derived from a national research registry, the DRS though brief and completed quickly, proved valid and reliable compared with a FFQ for the dietary components contributing most to CMD risk.

In post-hoc analyses, with the exception of sodium, DRS components correlated moderately with HEI-2015 components. DRS Fast Food, DRS Breads and DRS Snacks did not correlate significantly with HEI-2015 Sodium; however, they were associated with refined grain intake when these components were combined. DRS Processed meats did not map onto an HEI-2015 component, but did correlate with HEI-2015 Saturated fat and excess intake of processed meat is an independent dietary risk factor.3 Processed meat is also high in sodium but did not correlate with HEI-2015 Sodium (r=0.08, p=0.34). The mapping of these HEI component scores onto DRS component scores may assist in improving future iterations of the DRS.

Recent evidence suggests that just two questions regarding dietary intake can facilitate a discussion of dietary choices in clinical practice, one on fruits and vegetables and the other on sugar-sweetened beverage intake.24 Indeed, these were among the strongest correlations between the DRS and HEI-2015, and inadequate or excess intake of these, respectively, contributes to CMD risk. DRS Vegetables correlated well with the HEI-2015, both for total and green vegetable intake. Similarly, DRS Fruit correlated well with both total fruit (includes juice) and whole fruit (excludes juice). A study in 109 Australian adults by Cook et al, showed the use of two short screeners was adequate to estimate population intakes of fruits and vegetables, but did not accurately or adequately assess intake compared with a 74-item FFQ in individuals,25 suggesting this may be inadequate for patient care. DRS-assessed sugar-sweetened beverage intake correlated strongly with FFQ-assessed added sugar intake. There were also significant correlations between DRS Fish and DRS Nuts and HEI Seafood/Plant Protein.

While the correlations with sodium intake were non-significant, sodium intake is associated with the greatest dietary risk3 and should be discussed in patient care. The FFQ is a commonly used diet assessment tool in validation studies, however it is well known that FFQ’s do not accurately measure sodium intake.17 The DRS does not attempt to quantify sodium intake; rather, it provides a score based on specific food group intake. While DRS Sodium did not correlate with HEI Sodium, it provides sufficient opportunity for clinically initiated discussion regarding the adverse influence of high sodium intake on blood pressure and simple changes to reduce excessive intake.

The protective effects of a high quality diet against cardiovascular disease come largely from the contribution of fruits, vegetables, legumes and nuts,26 all of which contribute to overall potassium intake, a noted shortfall nutrient in the USA.27 We combined DRS Fruit, DRS Vegetables and DRS Nuts and correlated the composite score with potassium intake measured by the FFQ and found a modest, but significant correlation in the expected direction (r=−0.44, p<0.001). Assessment of diet quality is important, as is emphasising patient-specific components missing from reported dietary intake.

In the clinical nutrition setting, Registered Dietitian Nutritionists (RDNs) assess medical history, review pertinent information that may affect nutritional status and perform in-depth diet assessments.28 They may use a FFQ, 24-hour recall, a screening tool or a combination of these, many of which require in-depth nutrition knowledge and take considerable time to complete. The Mediterranean Diet Score (MDS)29 is relatively simple to administer, provides a score based on adherence to Mediterranean Diet (higher scores reflect better adherence to the Mediterranean diet), has been associated with improved health outcomes and has been tested in the electronic health record (EHR).30 31 A dietary screening tool (DST), tested in older Caucasian, rural adults, categorised participants into healthy or less healthy diet groups using principal component analysis.32 The Rapid Eating and Activity Assessment for Participants Short Form (REAP-S) is a simple 16-item paper-based tool but lacks specific guidance for counselling and intervention.21 The National Cancer Institute’s Daily Food Checklist33 does not rely on patient memory and can be completed prior to the visit, but scoring is time intensive and actionable information is not provided. Electronic food records, including MyFitnessPal and HealthWatch360 provide feedback to the user that can be shared with a healthcare provider, but rely on the quality and consistency of daily data entry. Healthcare professionals that do not have the nutrition training that RDNs do need screening tools that do not require nutrition expertise and provide basic nutrition guidance.

Studies using reminders in the EHR to improve rates of CVD risk assessment and obesity management have not made a clinically significant impact on lifestyle risk factors.34–36 Although recommended, current rates of nutritional counselling during physician visits are inadequate to help mitigate dietary risks.6 Physicians acknowledge that nutrition counselling is part of their patient care responsibilities, but cite inadequate knowledge and lack of time as barriers.37 38 Consequently, if physicians start the conversation with patients about lifestyle change, the likelihood that patients will make lifestyle changes improves.39

Energy intake, micronutrients and certain food groups are not assessed by the DRS in favour of brevity. The DRS is based on a synthesis of meta-analyses, which provide the highest quality evidence available3 and the use of NHANES data make the findings generalisable to the US population. The strengths of this study include the use of a validated FFQ for comparison and correlation. The use of ResearchMatch allowed access to a national sample of individuals within the target age group. The DRS items are related to frequency of food intake, rather than portion sizes, which can be difficult for individuals to recall and conceptualise. A validation study by Beasley et al, showed that about half of the error between actual and recorded meal intake came from portion size estimation error.40 Additionally, the high reliability suggests that individuals following their usual diet would score similarly if given the test at different times of year. Consensus has not been reached about the threshold for acceptable reliability, but some suggest >0.4 for individual nutrients is reasonable.5 The retest reliability of the DRS (0.83) is similar to that of the REAP questionnaire (0.86),41 which like the DRS does not measure intake of individual nutrients. Reliability of the short form REAP-S has not been reported. While we did not test the DRS in a diverse population, the use of primarily food groups as opposed to specific foods, and the simple wording increases the likelihood that this tool is valid across diverse groups. Future studies should be designed to test how well the DRS performs among a more diverse population with more varied food intake, accounting for race/ethnicity, age, education level and cardiometabolic risk. Finally, the DRS was valid across HEI-2005, HEI-2010 and HEI-2015 (online supplemental table 2), suggesting it is measuring aspects of the diet that have been consistent across these measures of dietary guidance and corresponding updates to HEI scoring.

This study was the first step to validate a brief, actionable diet assessment tool for use in clinical practice. Limitations include comparison of two self-report methods. There are many sources of bias in dietary assessment of sodium intake,18 which may vary by race/ethnicity.16 Sodium intake is difficult for consumers to estimate, therefore the gold standard for assessment is 24-hour urinary sodium excretion.42 Neither the DRS nor the FFQ can objectively quantify sodium intake, thus contributing to the poor agreement. Also, the DRS does not specifically measure intakes of saturated fat, whole grains or sources of non-liquid added sugars (eg, candy, energy bars). The DRS as a screener was not created to replace a detailed diet assessment performed by a RDN. Rather it is intended to assist the physician or healthcare provider to quickly address problems with adherence to the recommended dietary pattern within the office visit. We did not collect data on subject characteristics apart from age and gender, and we report them here only to describe the sample; this precludes subgroup analyses. Our sample was largely female, with access to the internet, and therefore the results may not be generalisable. The average HEI-2015 score in the USA is 59 across age groups.43 The average score in this study was 63.3, which is higher than the national average and therefore our results may underestimate the capacity of the DRS to detect high dietary risk individuals. We correlated HEI scores that are energy adjusted (per 1000 kcal) with overall intakes (DRS), which could have had an impact on the correlation between dietary components. While we were not able to calculate the agreement of individual DRS components with HEI-2015 components, it is the overall DRS score that would determine whether or not a physician intervenes on diet, not the individual DRS components. Further investigation of modifications that could improve the alignment of the DRS with the HEI-2015 is warranted. We did not measure whether the DRS is sensitive to dietary change. A study in which the DRS is used to measure the impact of an intervention in clinical practice is recommended.

The significance of our findings lies in the simple questionnaire that can be administered without additional nutrition training to initiate a conversation about diet-related CMD risk reduction. The DRS has been adapted as a cell phone application that can guide brief diet assessment and counselling; testing the DRS in a clinical setting is planned. Importantly, the application includes referral resources to RDNs who can provide more comprehensive nutrition counselling and ongoing support. The intent is for patients with a high DRS score (>18) to be referred to a RDN for a full diet assessment, nutrition counselling and intervention. This recommended threshold is based on the modifiable risk present in an individual reporting undesirable dietary choices, however, providers could also refer at a lower threshold, as providing nutrition counselling to an individual with moderate diet quality could result in improved diet quality.

The DRS is reliable, brief, validated against an established measure and provides clinical decision support, all of which suggest it would be a useful tool in clinical practice.44 In the USA alone, there were approximately 500 million primary care visits in 201545 and if less than 25% of those included any mention of nutrition, then there were over 375 million missed opportunities for prevention. Our results demonstrate that the DRS can accurately assess diet-related CMD risk factors and can be completed within a time-limited physician visit. Empirical evidence supports the impact of improved diet quality and CMD risk reduction through nutrition counselling in clinical practice.44 This tool is a much needed resource for physicians and other non-nutrition healthcare professionals to provide relevant nutrition guidance to their patients that will improve their diet and benefit their health.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors All authors contributed substantively to this project and manuscript. EAJ: Conceptualisation, funding acquisition, methodology and original draft preparation. KSP: Methodology, data curation, supervision, reviewing and editing. JMB: Funding acquisition, methodology, supervision, reviewing and editing. TK: Methodology, writing and reviewing. DCM: Funding acquisition, validation, reviewing and editing. LVV: Funding acquisition, methodology, reviewing and editing. RW: Resources, software and reviewing. PMKE: Funding acquisition, resources, supervision, reviewing and editing.

  • Funding This work was supported by AMERICAN HEART ASSOCIATION GRANT #19PRE34450165 (Emily A. Johnston). The project described was supported by the Penn State Survey Research Center, through the Social Science Research Institute. Special thanks to Thomas Gates who created the randomisation scheme and coding for this project for all his help and support. The project described was supported by the National Center for Advancing Translational Sciences, Grant TL1 TR002016 and Grant UL1 TR002014. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Ethics approval Review and approval for all procedures was obtained from the Penn State Institutional Review Board (Study 00008321).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement Data are available upon reasonable request.