2 The 106th Congress and the Clinton Administration should work together to commission critical, objective analyses of the existing educational development and program evaluations underway in order to see whether the expectations of the 1994 reauthorizations were met and to suggest what still remains to be done. If it turns out, as many suspect, that the Department of Education has not been able to produce a sufficient number of high-quality, long-term development projects or enough scientifically-sound program evaluations anticipated in the legislation five years earlier, then perhaps we explore After three alternative ways of restructuring our overall approach to funding development and program evaluation rather than just once again simply reaffirming our interest and commitment in this area. OERI is certainly a logical agency to sponsor and oversee high-quality, systematic development and its recent assistant secretaries have expressed support for this type of work on many occasions. So why hasn't more been accomplished? decades of frustration and mutual recriminations, is it time to acknowledge that many of the regional educational laboratories have not been producing much high-quality, systematic development? Should the regional educational laboratories be merged with the comprehensive service centers in order to provide more efficient and effective research-based technical assistance to schools while funding separately long-term, rigorous development through a competitive process in which these and 3 other institutions could compete? While a few of the R&D centers have undertaken systematic development of promising educational models and practices, should we continue to expect most of them to develop more interest in this particular area or should these centers continue to focus more on producing applied and basic research studies as well as policy analyses? Should Congress allocate more monies designated specifically for systematic research to OERI or do the legislators think that those efforts are already being funded elsewhere? And is OERI really committed to sponsoring and overseeing systematic development in the future or should we also be exploring other alternative institutional homes such as NSF or NIH for some of the proposed development work? Similarly, why haven't we had more rigorous education program evaluations at PES in the 1990's? Certainly Alan Ginsburg and his colleagues appreciate and understand what it takes to do good in-depth, program evaluations. But has PES been too pre-occupied with its other short-term assignments to fund and oversee more long-term program evaluations? How much of the relative lack of adequate program evaluations is due to the limited funding provided for these efforts by the Congress? while PES has had two recent assessments of its organizational And and management activities, is it now time for a more in-depth, critical analysis of the extent and quality of its work in program valuation? What might be done to foster more cooperation and collaboration between PES and other agencies such as OERI and NCES in the areas of systematic development and program evaluation? Part of that problem is that educational researchers have not paid sufficient attention to the use of randomized-controlled field experiments. As Thomas Cook of Northwestern University has shown in his work, the intellectual culture in many American schools of education have discouraged the use of more rigorous quantitative evaluations using randomized assignment of participants. Almost everyone agrees on the value of complementary quantitative and qualitative evaluation studies. But Cook also points out the particular need for sustained programs of research and evaluation to try to establish causal connections between specific interventions and educational improvements--rather than relying mainly on one-shot, single-site studies. Until we change the culture of indifference or even hostility to more rigorous and scientifically-sound evaluations among some educators and government officials, our calls for the development and use of randomized-controlled field trials in education probably will not succeed. Increasingly the public and educational policy makers are coming to the realization that we need more systematic development and rigorous evaluation of model education program and practices in order to have all American children learn and thrive in our schools. Yet history suggests that the present 5 arrangements and practices in OERI and PES have not produced the quantity or type of studies and assessments needed. The current reauthorizations provide us with an opportunity and a responsibility to re-examine these issues and find, if necessary, new mechanisms as well as additional funds to support more systematic development and rigorous educational program evaluations. APPENDIX 3: Material for the Record MISSING IN PRACTICE? SYSTEMATIC DEVELOPMENT AND RIGOROUS PROGRAM EVALUATION AT THE U.S. DEPARTMENT OF EDUCATION Maris A. Vinovskis Department of History, Preliminary Draft January 1999 Paper prepared for Conference on Evaluation of Educational Policies, American Academy of Arts and Sciences, Cambridge, MA, May 13-14, 1999 |