Evaluating the Effect of an Intervention on Diet

NOTE: Links for “Row” in the text below will take you to the relevant portion of the Summary Table.

Data Capture Considerations

Further research is needed to guide best practices to assess the effects of interventions on changes in diet. However, the following considerations and guidance are provided based on the current evidence:

  • It is important to consider not only the degree of [glossary term:] bias overall in data collected using one or more self-report instruments but the potential for [glossary term:] differential response bias between study groups (i.e., whether the bias in each group is similar or different). This is of concern in intervention studies because [glossary term:] exposure to the intervention itself can create differential error in reporting in the treatment group(s) relative to the control group. This is in addition to any other differences among groups (e.g., baseline body weight) that also might contribute to differential error.

    • Interventions that operate at an institutional, program, or environmental level (e.g., modifications to food environments, such as calorie labeling at restaurants) may be less prone to differential response bias than interventions at the individual level (e.g., behavioral interventions to increase fruit and vegetable consumption). Nonetheless, the possible presence of differential response bias should be carefully considered in any [glossary term:] intervention study because it has serious consequences for statistical tests of the differences in intakes between two or more groups.

  • If large differential response bias is probable, it is advisable to use objective sources of data, such as [glossary term:] biomarkers, [glossary term:] observation, or sales records, as the primary evaluation tool. However, it must be borne in mind that these objective sources of data themselves have serious limitations: In addition, it may be possible to collect other self-report data, such as markers of social desirability bias, which may help characterize individuals who may be particularly prone to response bias. Including this information in the analyses may reduce bias in the results (Learn More about Social Desirability). More research is needed to guide approaches in this area.
  • With the above provisos, we recommend the use of 24HRs over FFQs or screeners when evaluating the effect of an intervention on diet because 24HR data are less prone to overall bias (see the 24-hour Dietary Recall Profile, the Food Frequency Questionnaire Profile, and the Screeners Profile). However, research regarding intervention-related differential response bias, in particular, is lacking for all self-report dietary assessment instruments, including the 24HR. The collection of unbiased data (e.g., biomarkers) and self-report data in intervention trials are needed to spur knowledge in this area.
  • The food record is not recommended for dietary interventions because [glossary term:] reactivity may be particularly problematic, and in the context of an intervention study, may magnify differential response bias (see the Food Record Profile and Learn More about Reactivity). Thus, its utility in measuring the success of an intervention is limited. A food record, however, can be used to encourage and provide feedback to participants actively involved in a dietary change intervention.

We provide the following recommendations if you have carefully considered issues of differential response bias and have come to the conclusion that this source of bias is unlikely in the particular study (though not usually the case), and/or you have included an objective measure, such as a biomarker or observation, as the main evaluation instrument and have also collected self-report dietary data as secondary measures.

  • For intervention studies in which your aim is to assess the change in the [glossary term:] mean usual intake of a group between two points in time (Row 9) or the difference between two or more groups in such changes (Row 10), a single administration of the 24HR at each time is sufficient. Consider collecting recovery biomarkers in a subsample to aid in [glossary term:] measurement error adjustment for some nutrients at relevant time points (see Key Concepts about Measurement Error).
  • A single administration of the 24HR also is sufficient for intervention studies in which your aim is to estimate the difference in mean usual intake among two or more groups at post-intervention (Row 11). In this case, if you randomize the groups to intervention and control groups and assume that the pre-intervention mean intake does not significantly differ between the two groups, the difference between the two groups in post-intervention mean intake may be ascribed to the intervention (given the limitations described above). If 24HRs are used, consider the collection of recovery biomarker data for at least a subsample (this is called an [glossary term:] internal calibration sub-study) to aid in adjustment of measurement error, including [glossary term:] random error and bias.
  • Administration of an FFQ also may be acceptable for the intervention scenarios above, assuming it is tailored to the population of interest and captures the majority of food and beverage sources within the food supply for the dietary component of interest (see the Food Frequency Questionnaire Profile). To reduce bias, conduct an internal calibration sub-study using less-biased 24HR or recovery biomarker data from a subsample. Such [glossary term:] calibration studies should be performed on a subset of both control and intervention groups and at each relevant time point.
  • A screener also may be acceptable if the focus of your intervention is limited to one or a few specific dietary components that are concentrated in relatively few food sources (see the Screener Profile). Carefully select the screener to ensure that it is tailored to the population and captures the majority of food and beverage sources within the food supply for the dietary component of interest.

    • To reduce bias, conduct an internal calibration sub-study with a less biased instrument, such as the 24HR. As above, these calibration studies should be performed on a subset of participants from both control and intervention groups and at each relevant time point. If internal calibration is not possible, screeners that include [glossary term:] scoring algorithms (Learn More about Scoring Algorithms for Screeners) based on 24HR in an external calibration study, preferably in a similar study population, may enable estimation of less biased estimates of intake. However, this approach would not provide separate scoring algorithms for control and intervention groups, and at varying time points, and therefore it would not adjust for differential response bias, should it exist.

  • In studies in which your objective is to describe the change in some characteristic of the distribution of usual intake rather than only the mean (for example, the proportion of the group above or below some threshold) between two points in time (Row 12) or among groups (Row 13 and Row 14), we recommend conducting two administrations of the 24HR at baseline and at post-intervention in at least a subsample. As noted above, FFQ and screeners are not recommended when interest is in the distribution of intakes because analytic strategies are not yet available to address the bias. FFQs and screeners are not the ideal instruments for estimating distributions of usual intake, but if they are calibrated to a less-biased measure through an internal calibration sub-study or an external calibration study and if appropriate analytical procedures are applied, it is possible to use them for this purpose.

Data Analysis Considerations

Specific analytic strategies to account for error in dietary intake data in the context of intervention studies are limited. As noted above, pay attention to the extent to which differential response bias in intake among groups may lead to spurious relationships and/or reduced statistical [glossary term:] power for detecting actual differences.

As in the data capture section, the suggestions below are based on the premise that you have carefully considered issues of differential response bias and come to the conclusion that this source of bias is unlikely in the particular study (not usually the case), and/or you have included an objective measure, such as a [glossary term:] biomarker or observation, as the main evaluation instrument and have also collected self-report dietary data as secondary measures.

  • If you are interested in estimating and comparing [glossary term:] mean usual intake (rather than the distribution of usual intake) at multiple points in time or among different groups and have administered a single 24HR, you do not need to adjust for [glossary term:] within-person random error in intakes. If the 24HRs in each group are not distributed evenly across weekdays and weekend days (Learn More about Day-of-Week Effect) and seasons (Learn More about Season Effect), consider analytical adjustments for these [glossary term:] nuisance effects.
  • If you are interested in estimating and comparing distributions of usual intake and have administered multiple 24HRs, you will need to use statistical modeling (Learn More about Statistical Modeling) to separate the within-person variation (the major source of which is day-to-day variation in the case of 24HRs) from the [glossary term:] between-person variation and then remove within-person variation. If you have conducted an [glossary term:] internal calibration sub-study using recovery biomarkers, you can adjust for [glossary term:] measurement error, including [glossary term:] random error and [glossary term:] bias, for some nutrients (Learn More about Biomarkers). For biomarkers that display considerable [glossary term:] day-to-day variation, such as 24-hour urinary nitrogen for protein intake, repeat biomarker assessments are needed to allow estimation of distributions.
  • Various options are available to conduct this modeling (see Describing Dietary Intake). If you have a large sample, it is usually preferable to perform the modeling separately within each of the comparative groups. If your sample size is limited, you can do joint modeling across the comparative groups if the within-person and between-person variances are similar across these groups. Methods developed for estimating usual intake distributions will partition the variance so that this can be assessed.
  • In studies in which an FFQ or a screener is used to estimate and compare means across groups and 24HR or [glossary term:] recovery biomarker data are available (e.g., from an internal calibration sub-study or an [glossary term:] external calibration study), use statistical techniques to adjust regression coefficients for bias with the 24HR or biomarker data used as reference. Estimate [glossary term:] regression calibration equations for each of the comparative groups, and at relevant points in time (i.e., both pre- and post-intervention). It is not yet known, however, if regression calibration using 24HRs will adjust for differential response bias because the recalls, too, may similarly be prone to differential response bias as described above. [glossary term:] Energy adjustment of estimates obtained from an FFQ also may reduce bias.
  • In studies using FFQs or screeners without internal calibration sub-studies but with data from an external calibration study, apply the [glossary term:] scoring algorithms from the external calibration study. For example, 24HR data from the [glossary term:] National Nutrition Examination Survey (NHANES) have been used to develop scoring algorithms that enable conversion of responses on the NCI's Dietary Screener Questionnaire to estimates of intake. However, unlike the data from an internal calibration sub-study, data from an external calibration study would not allow for separate scoring algorithms for control and intervention groups, and at varying time points, and therefore it would continue to be vulnerable to uncontrolled differential response bias.

    • Energy adjustment of estimates obtained from a FFQ also may reduce bias, but energy adjustment is not possible with screeners because they do not capture total intake.

References and Resources

The following references and resources provide additional information on the topics discussed in this section.

References

Natarajan L, Pu M, Fan J, Levine RA, Patterson RE, Thomson CA, Rock CL, Pierce JP. Measurement error of dietary self-report in intervention trials. Am J Epidemiol 2010 Oct 1;172(7):819-27. [View Abstract]

Resources

National Cancer Institute. Dietary Screener Questionnaire in the NHANES 2009-2010.

National Cancer Institute. Short Dietary Assessment Instruments.

National Cancer Institute. Usual Dietary Intakes.

National Health and Nutrition Examination Survey (NHANES). NHANES Dietary Web Tutorial