Global Navigation Element.

Winter/Spring 2008 Vol. 8 No. 1

Table of Contents

©Photodisc/Getty Images Gauging the Efficiency of Federal Research

Research at federal agencies like the U.S. Environmental Protection Agency, NASA, and the National Institutes of Health underpins government efforts to regulate the safety of everything from drugs and drinking water to airplanes and roadways. Federal research also adds to our knowledge on subjects as varied as emerging viruses, nanotechnology, and dark energy.

Congress and oversight bodies such as the White House Office of Management and Budget naturally want to know whether the tax money invested in these programs -- which totals over 100 billion dollars annually -- is being used wisely and efficiently. But what's the best way to measure the efficiency of something as unpredictable as research?

Currently OMB urges EPA and other agencies to gauge the efficiency of their research based on its ultimate outcomes -- for example, whether a research program eventually leads to cleaner air or fewer deaths. But measuring efficiency this way isn't achievable or valid for most programs, says a new report from the National Research Council that recommends a new approach to making these assessments.

Ultimate outcomes happen far in the future and can't be known at the time the research is evaluated, making them inappropriate measures of efficiency, the report notes. They are also usually out of the hands of researchers. A program might discover that a certain level of an industrial chemical in drinking water is hazardous, for instance, but it cannot control whether regulators use that knowledge to limit the amount of the chemical that enters the water supply.

Agencies and OMB would find it easier to gauge efficiency accurately if they split the concept in two, the report says. Assessments of "process efficiency" should look at how well research processes are managed, and whether program managers exercise skill in using and conserving resources. This type of efficiency can be measured quantitatively -- for example, by comparing the number of grants awarded or journal articles published against benchmarks -- as well as by panels of experts.

Expert review panels also are key to assessing "investment efficiency," a second aspect that concerns whether an agency is investing in high-quality research in areas that further its mission. So-called intermediate outcomes can also be helpful, the report adds; an evaluation might consider whether a program has increased the knowledge available for making regulatory decisions, for example.

Evaluations of research should not over-emphasize efficiency, however. Efficiency should be weighed as just one factor in the overall context of a program's quality, relevance, and effectiveness, the report stresses.   -- Sara Frueh

Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency, Committee on Science, Engineering, and Public Policy, The National Academies; and Board on Environmental Studies and Toxicology, Division on Earth and Life Studies (2008, approx. 144 pp.; ISBN 0-309-11684-8; available from the National Academies Press, tel. 1-800-624-6242; $35.00 plus $4.50 shipping for single copies).

The committee was chaired by Gilbert Omenn, professor of internal medicine, human genetics, public health, and computational biology, University of Michigan, Ann Arbor. The study was funded by the U.S. Environmental Protection Agency.

Previous Table of Contents Next

Copyright 2008 by the National Academy of Sciences