The US National Science Foundation's insistence that every research project addresses 'broader impacts' leaves many researchers baffled. Corie Lok takes a looks at the system.
Research-funding agencies are forever trying to balance two opposing forces: scientists' desire to be left alone to do their research, and society's demand to see a return on its investment.
The European Commission, for example, has tried to strike that balance over the past decade by considering social effects when reviewing proposals under its various Framework programmes for research. And the Higher Education Funding Council for England announced last year that, starting in 2013, research will be assessed partly on its demonstrable benefits to the economy, society or culture.
But no agency has gone as far as the US National Science Foundation (NSF), which will not even consider a proposal unless it explicitly includes activities to demonstrate the project's 'broader impacts' on science or society at large. "The criterion was established to get scientists out of their ivory towers and connect them to society," explains Arden Bement, director of the NSF in Arlington, Virginia.
Unfortunately, good intentions are not enough to guarantee success, says Diandra Leslie-Pelecky, a physicist at the University of Texas in Dallas who is active in popular science writing and other forms of outreach.
Leslie-Pelecky remembers a pilot project she carried out in 2001, when she was at the University of Nebraska in Lincoln. In many ways, it was typical of the kinds of things that NSF-funded researchers do to fulfil their broader-impacts requirement. She took three female graduate students on weekly visits to local classrooms, where they spent 45-minutes leading nine- and ten-year-old children in practical activities designed to teach them about electricity and circuits. The visitors also talked about their lab work and careers. In addition, Leslie-Pelecky did something less typical of broader-impacts efforts: she brought along education researchers to study the effect of this interaction on the children's perception of scientists.
Those assessments were startling, she says. After three months, most of the students said that they still weren't sure who these young 'teachers' were – except that they couldn't possibly be scientists. In their minds, scientists were unfriendly, grey-haired old men in white lab coats1.
"And that's what I worry about with broader impacts," says Leslie-Pelecky. "There are a lot of people putting time and effort into [these sorts of activities] and they have no idea if they're making any difference or not."
Many NSF-funded researchers find the foundation's definition of broader impacts to be, perhaps unsurprisingly, broad, and frustratingly vague. Among the examples of activities listed in the foundation's proposal guide are: developing educational materials for elementary, high-school and undergraduate students; involving these students in the research where appropriate; creating mentoring programmes; maintaining and operating shared research infrastructure; presenting research results to non-scientific audiences such as policy-makers; establishing international, industrial or government collaborations; developing exhibits in partnership with museums; forming start-up companies; and giving presentations to the public.
Because it lacks conceptual clarity, the broader-impacts requirement often leaves researchers unsure about what to include in their proposals, and leads to inconsistencies in how reviewers evaluate applications. "Broader impacts were designed to be open, but openness confuses a lot of people," says Luis Echegoyen, the division director for NSF chemistry.
To make matters worse, the NSF has made little attempt to systematically track how its broader-impacts requirements are being met, or how much grant money is being spent in the process. Nor does it have a system in place to evaluate the effectiveness of the various projects.
These problems with the broader-impacts requirement have been confirmed over the past decade in studies from the National Academy of Public Administration and elsewhere. In March, the NSF's oversight body, the National Science Board, launched a task force to examine how broader impacts can be improved. Chaired by Alan Leshner, chief executive of the American Association for the Advancement of Science in Washington DC, the task force is not expected to make its recommendations until 2011. In the meantime, a small number of academic institutions are already exploring ways to make broader-impacts efforts work better.
After all, says Ralph Nuzzo, a chemist and materials scientist at the University of Illinois in Urbana–Champaign, most US scientists have come to accept — even if grudgingly — that it is probably a good idea to demonstrate the wider implications of their work. "People want to do the right thing," says Nuzzo. "It's just hard to know what that is."
Research-funding agencies are forever trying to balance two opposing forces: scientists' desire to be left alone to do their research, and society's demand to see a return on its investment.
The European Commission, for example, has tried to strike that balance over the past decade by considering social effects when reviewing proposals under its various Framework programmes for research. And the Higher Education Funding Council for England announced last year that, starting in 2013, research will be assessed partly on its demonstrable benefits to the economy, society or culture.
But no agency has gone as far as the US National Science Foundation (NSF), which will not even consider a proposal unless it explicitly includes activities to demonstrate the project's 'broader impacts' on science or society at large. "The criterion was established to get scientists out of their ivory towers and connect them to society," explains Arden Bement, director of the NSF in Arlington, Virginia.
Unfortunately, good intentions are not enough to guarantee success, says Diandra Leslie-Pelecky, a physicist at the University of Texas in Dallas who is active in popular science writing and other forms of outreach.
Leslie-Pelecky remembers a pilot project she carried out in 2001, when she was at the University of Nebraska in Lincoln. In many ways, it was typical of the kinds of things that NSF-funded researchers do to fulfil their broader-impacts requirement. She took three female graduate students on weekly visits to local classrooms, where they spent 45-minutes leading nine- and ten-year-old children in practical activities designed to teach them about electricity and circuits. The visitors also talked about their lab work and careers. In addition, Leslie-Pelecky did something less typical of broader-impacts efforts: she brought along education researchers to study the effect of this interaction on the children's perception of scientists.
Those assessments were startling, she says. After three months, most of the students said that they still weren't sure who these young 'teachers' were – except that they couldn't possibly be scientists. In their minds, scientists were unfriendly, grey-haired old men in white lab coats1.
"And that's what I worry about with broader impacts," says Leslie-Pelecky. "There are a lot of people putting time and effort into [these sorts of activities] and they have no idea if they're making any difference or not."
Many NSF-funded researchers find the foundation's definition of broader impacts to be, perhaps unsurprisingly, broad, and frustratingly vague. Among the examples of activities listed in the foundation's proposal guide are: developing educational materials for elementary, high-school and undergraduate students; involving these students in the research where appropriate; creating mentoring programmes; maintaining and operating shared research infrastructure; presenting research results to non-scientific audiences such as policy-makers; establishing international, industrial or government collaborations; developing exhibits in partnership with museums; forming start-up companies; and giving presentations to the public.
Because it lacks conceptual clarity, the broader-impacts requirement often leaves researchers unsure about what to include in their proposals, and leads to inconsistencies in how reviewers evaluate applications. "Broader impacts were designed to be open, but openness confuses a lot of people," says Luis Echegoyen, the division director for NSF chemistry.
To make matters worse, the NSF has made little attempt to systematically track how its broader-impacts requirements are being met, or how much grant money is being spent in the process. Nor does it have a system in place to evaluate the effectiveness of the various projects.
These problems with the broader-impacts requirement have been confirmed over the past decade in studies from the National Academy of Public Administration and elsewhere. In March, the NSF's oversight body, the National Science Board, launched a task force to examine how broader impacts can be improved. Chaired by Alan Leshner, chief executive of the American Association for the Advancement of Science in Washington DC, the task force is not expected to make its recommendations until 2011. In the meantime, a small number of academic institutions are already exploring ways to make broader-impacts efforts work better.
After all, says Ralph Nuzzo, a chemist and materials scientist at the University of Illinois in Urbana–Champaign, most US scientists have come to accept — even if grudgingly — that it is probably a good idea to demonstrate the wider implications of their work. "People want to do the right thing," says Nuzzo. "It's just hard to know what that is."
