Faculty and staff members, students, prospective students, and other constituencies will be able to see how the UI’s graduate programs stack up against those of peer institutions when the National Research Council releases the results of its most recent assessment of doctoral programs, which is expected to occur in March. The NRC is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering.
Hailed as one of the most influential assessments of graduate programs at U.S. colleges and universities, the NRC assessment of research doctoral programs involves an extensive data-collection process that is based upon surveys of universities, faculty members and students. The assessments gather information about numerous quantitative variables related to doctoral programs, such as faculty members’ scholarly activities, program characteristics and research resources.
The 2009 study will comprise data from 222 institutions and more than 5,000 programs, including fields in agricultural sciences, communication, American studies, theater and kinesiology, which were not covered in the NRC’s 1995 and 1982 studies. For a field to be included in the assessment, it must have awarded at least 500 doctoral degrees from academic year 2001-2002 to AY 2005-2006 and a minimum of 25 universities must offer similar programs.
Several key methodological changes were implemented with the 2009 assessment as well. Unlike the prior assessments, which assessed departmental quality, the focus of the 2009 assessment was programmatic quality, reflecting the cross-disciplinary nature of many of today’s graduate programs, which may not exist within the confines of a single department. The 2009 study also examined a broader range of doctoral programs – 60 fields versus the 41 fields that were assessed with the last survey in 1995. Accordingly, 60 of the UI’s doctoral programs were rated in the assessment.
The 2009 assessment does not rank programs by reputation either, unlike the prior assessments. Instead, quantitative data gathered by the assessment were used to develop ratings and rankings for programs, a change that aimed to mitigate the “halo effect” that may have inflated rankings for some programs in the past because of their affiliation with prestigious institutions.
“The other problem with reputational rankings is that once you get past the first 15-20 programs, the distinctions are largely meaningless,” said Vice Provost Richard Wheeler, who is the UI’s institutional coordinator and a member of the NRC committee that oversees the assessment. “No one knew enough about all of these programs to be able to provide an informed response.”
Instead, program ratings were derived from the data generated by the faculty questionnaires that were sent to all faculty members and the supplemental ratings questionnaires completed by a random sample of faculty members. On the all-faculty survey, faculty members rated the most important program characteristics from a list of 20 characteristics.
Then, a random sample of faculty members rated a sample of graduate programs in their fields based upon data such as racial, ethnic and gender diversity of the faculty; time to degree; student support and faculty members’ research impact through publications and citations, honors, awards and grants.
Regression analysis was then used to determine which variables most closely predicted program ratings.
Students who were doctoral candidates in five fields – economics, English, chemical engineering, neuroscience and physics – were surveyed as well to gather data on factors such as their prior education, post-graduation plans and professional development.
“The people who have been driving this believe this will be the most useful NRC ranking yet, in part because of the way the rating and ranking processes have been shifted around, but more importantly because of the immense amount of information that has been gathered,” Wheeler said.
The data gathered through the supplemental faculty questionnaire “produced a lot of information about what faculty believe is the best way to understand quality in graduate programs. The report will provide an enormous amount of data connected with each program, ratings and rankings that were generated from the data, and the weights used to calculate ratings, which reflect levels of importance derived from the faculty questionnaires.”
The data are expected to be useful for college deans, unit heads and campus officials in targeting areas for program improvement and allocation of resources. The data about program characteristics are expected to be useful as well for prospective students when they are selecting graduate programs.
The Graduate College will host a series of workshops about the assessments for administrators, and faculty and staff members. The Graduate College also is developing a Web-based program called “Program Profiles” that will allow users to examine the UI’s real-time data, which will be continually updated.
“We’re hoping to launch Program Profiles at the same time as the NRC’s release of the assessment,” said Kelly Tappenden, associate dean of the Graduate College and professor of nutrition and gastrointestinal physiology in the department of food science and human nutrition. “Our plan is to work with units to understand patterns in their doctoral programs and how they compare to their peers, so areas for improvement can be targeted.”
Wheeler and Tappenden expressed appreciation to faculty and staff members at the Urbana campus for their cooperation in completing the questionnaires, and collecting and retrieving the data, which entailed finding solutions for extracting and compiling the data obtained from incompatible legacy software systems and the Banner system. The Urbana campus also exceeded the norms for the percentage of questionnaires returned by faculty members and students.
