|Editors: Ann P. Dougherty, Mountainside Publishing; Richard M. Dougherty, University of Michigan, Emeritus
Contributing Editors: William Miller, Florida Atlantic University; Barbara Fister, Gustavus Adolphus College;
Larry Hardesty, University of Nebraska at Kearney; Mignon Adams, University of the Sciences in Philadelphia;
Steven Bell, Philadelphia University; Kathleen Hoeth, Florida Gulf Coast University
|Vol. 26, No. 1||September 2005|
Meeting the Accountability Challenge
by Kathleen Hoeth
College and university administrators face philosophical and practical questions every day, but the recent trend in higher education to use assessment data for accountability has created a new challenge. Boards of Trustees, politicians, accreditation bodies, and others expect a new approach to measuring educational quality; they are demanding evidence that the institution evaluates its effectiveness in achieving its mission and then uses the assessment data for continual improvement.
The demand for data to demonstrate that educational funding
is thoughtfully deployed and produces tangible benefits permeates higher
education these days. This article is a brief for campus administrators about
(1) campus-wide evaluation tools that produce some evaluative data about
libraries, and (2) some of the library-specific assessment initiatives
librarians have mounted to generate data for library management and to meet the
The accountability trend is clear in even a cursory review of the changes that have been made over the past few years in regional accreditation requirements. Looking at the six associations’ requirements for colleges and universities and for libraries reveals an unmistakable shift away from judging quality by describing inputs and outputs, and toward an expectation of routine assessment.* Moreover, the associations clearly expect educational quality to be demonstrated by a focus on using the assessment data for continual improvement. In its Accreditation Handbook, for example, the Western Association of Schools and Colleges describes this changing context for accreditation as, “A shift toward effectiveness and performance indicators beyond inputs and resources as the organizing basis for defining and evaluating quality.”
As institutions rush to learn the language and tools of this new trend, academic libraries are an excellent place to begin looking for information, and perhaps, inspiration. Traditionally, academic library quality has been expressed in terms of inputs such as collection size and square footage. Many institutions still celebrate the library’s acquisition of its millionth volume, for instance. But the dramatic, technology-driven reinvention of libraries over the past decade has resulted in demands for new methods of assessment.
Libraries now routinely supplement traditional collections and services with digital resources and Web-based service approaches. As librarians have reinvented academic libraries, we have also worked to create methods of assessment that reflect the reality of contemporary libraries and address the shift toward assessment and accountability in higher education. The North Central Association of Colleges and Schools offers the following perspective about libraries and learning resources:
It was not that long ago that accreditation was understood to focus rather heavily on resources in the library. Accrediting teams counted staff members and the square footage allocated to the library and to book inventories. Unless libraries are used and valued by students and faculty, their impact on learning is small. In short, a library—or a learning resource center—exists to support learning and teaching. . . . Consequently, it is critical for colleges and universities to assess actual student use of equipment, materials, and media, collecting evidence that something worthwhile is happening to students because learning resources exist.
In the same spirit, rather than providing a census of its
collections, the Middle States Commission on Higher Education requires that “The
institution provides sufficient learning and information resources and services
to support the nature, scope, and level of the programs offered.” Colleges and
universities are forced to ask, “What is ‘sufficient’?” The other regional
associations have similarly broad statements, leaving librarians and
institutional effectiveness staff to figure out a new approach. Library
practitioners and scholars, therefore, have worked to create instruments and
protocols for libraries that meet demands for accountability, measure service
quality, and improve the data available to library administrators for effective
New Measures for Libraries
Rather than focusing on inputs such as collection size and staffing, the first new library measures sought to describe what libraries do. That is, librarians began to report outputs such as the number of items borrowed or the number of reference questions answered. Those measures alone, however, still fell short of addressing whether library services were sufficient. As colleges and universities created student learning outcomes over the past few years, librarians also created measures that were based on outcomes — the extent to which student and faculty contact with libraries affected them and contributed to the mission of the university.
The increasing attention to measurement and assessment in libraries during the past decade might be described as a move beyond reporting what libraries do, to capturing the ways in which student learning is affected by library services and programs.
Since libraries contribute to the academic enterprise in a
variety of ways including providing collections, creating research assignments,
serving as an intellectual locus for the campus, and teaching bibliographic
skills, there are several tools that may be used to evaluate one or more aspects
of library sufficiency or quality. Most of these tools were first discussed in
Library Issues by Stoffle and Phipps in March
Institutions that participate in the National Survey of Student Engagement (NSSE) have an opportunity to use a portion of its scores as a library assessment tool. That this survey may be used for an additional purpose may be welcome news in Institutional Research offices across the country; the level of assessment activity required on campus has risen dramatically in recent years.
“The increasing attention to measurement and assessment in libraries during the past decade might be described as a move beyond reporting what libraries do, to capturing the ways in which student learning is affected by library services and programs.”
NSSE, administered by the Indiana University Center for Postsecondary Research, surveys first-year and senior students regarding their levels of engagement in the college experience. The key idea underlying the instrument is that student learning is tied to five areas of good practices in undergraduate education and may be analyzed through student engagement. The five areas measured are:
1. Level of academic challenge,
2. Active and collaborative learning,
3. Student-faculty interaction,
4. Enriching educational experiences, and
5. Supportive campus environment.
Increasingly popular, the most recent NSSE survey was conducted earlier this year at 538 four-year institutions and includes data from more than 660,000 randomly selected first-year and senior students.
Amy Mark and Polly Boruff-Jones have argued that
bibliographic instruction, the library’s efforts to develop information literacy
competencies in students, may be assessed through an analysis of specific NSSE
results. Through case studies at the University of Mississippi and Indiana
University-Purdue University Indianapolis, they have demonstrated that NSSE
findings can be used as pre- and post-tests for assessing student progress in
attaining Information Literacy Standards as defined by the Association of
College and Research Libraries (ACRL).
Since its launch in 2000, the NSSE data has produced statistically significant
results and benchmarks for comparison groups.
SAILS: Standardized Assessment of Information Literacy Skills
If Information Literacy is a primary focus in your institution’s library, in addition to looking at NSSE, consider an instrument in development that is specifically intended to address information literacy programs. The SAILS project, coming out of Kent State University, has as its purpose, the creation of a standardized, valid, and reliable instrument for assessing information literacy programs. The instrument developed to date has 126 items that are intended to test the 5 standards, 22 performance indicators, 87 outcomes, and 138 objectives articulated in the Higher Education Information Literacy Standards adopted by ACRL.
One key difference between NSSE and SAILS is that the latter looks at information literacy among cohorts of students, rather than focusing on individuals. In practice, this means that institutions can see the specific institution average and the cross-institutional average, as well as the average by major, class standing, or other factors. What factors are the most important at your institution?
Funded by a three-year federal grant from the Institute of Museum and Library Services, the third year of the SAILS project has just been completed. The third phase included 70 participating institutions and the project team is working to analyze the results. SAILS may eventually be used to compare information literacy measures with other indicators of student achievement, such as those enumerated in NSSE, for the purpose of clarifying what role information literacy may have in student success.
Both NSSE and SAILS assess libraries in relationship to the
institution—their own institution, their students and faculty, tying them all
together. The following programs compare libraries to libraries. This is still
important, but it is not the same. Campus leadership should be aware of the
difference; do not think that just because you are already involved in LibQUAL+TM
you don’t need to look at participating in the NSSE or SAILS programs. It’s
apples and oranges.
Overall service quality in libraries is increasingly being evaluated through LibQUAL+TM, a library-specific instrument developed under the auspices of the Association of Research Libraries (ARL). While the practice of librarianship has been changing, it remains solidly based in a commitment to service. Librarians in all types of libraries work to ensure that their organizations provide high quality service to library users in support of the goals of the library’s parent institution. It would be rare indeed to discover an academic library, for example, that did not include the concept of service quality in its mission to support the teaching, learning, and research missions of the college or university in which it operates.
Librarians have sought improved methods of assessment to reflect the changes in libraries and provide pertinent information to library administrators for planning and management purposes. Consequently, library organizations such as ARL have engaged in a number of measurement projects including LibQUAL+TM, which measures expectations and perceptions of library service quality.
The project has grown from 12 participating libraries in 2000, to 199 libraries in 2005. Libraries use the LibQUAL+TM survey instrument to solicit and track users’ opinions of service quality. The survey has its origins in SERVQUAL, a service quality instrument that has been used widely in the commercial sector. Grounded in the gap theory of service quality, the instrument was developed for the for-profit sector in the 1980s.The gap theory of service quality measures three things.
1. Minimum level of service, or, the lowest level users will put up with,
2. Desired level of service, or, what they want,
3. Perceived level of service, or, how they think the agency or company is doing.
"Given the increasing investment in electronic information resources in libraries, assessing such services has become critical. A major goal of DigiQUAL is to identify best practices for digital library services."
Once the measures are collected, the gap between perceived
and desired levels of service are defined and may be used as the basis for
planning and implementing service improvements. LibQUAL+TM survey
data are analyzed and each participating library receives reports of the
aggregate response data as well as its own institutional data so that they may
compare local library performance with that of peer institutions. Librarians
report that they have used the survey data to inform practical
decision-making, identify best practices, analyze service deficits, and allocate
Digital Library Assessment Project
The success of LibQUAL+TM has spawned a related
project that is working to modify the LibQUAL+TM survey for assessing
services provided by the National Science, Math, Engineering and Technology
Education Digital Library (NSDL) program. Applying the LibQUAL+TM
protocol to NSDL has been dubbed, “DigiQUAL.” Given the increasing investment in
electronic information resources in libraries, assessing such services has
become critical. A major goal of DigiQUAL is to identify best practices for
digital library services. Eventually, this project may result in the ability to
assess service quality for digital libraries and generate data for librarians to
use in improving the retrieval of relevant information and learner preparation
for an information-rich society.
Another effort working toward determining the effectiveness of digital collections is dependent upon establishing measures. The E-Metrics project began in 2000 and was intended to explore defining and collecting data on the use of electronic resources. It became clear in the course of the E-Metrics project, however, that consistent standards for data did not exist. Without usable definitions, measurement and analysis is difficult and complex. Project COUNTER (Counting Online Usage of Networked Electronic Resources) began in 2002 to address the problem.
Project COUNTER is working to establish practices for consistently and accurately determining the usage of digital resources. Publishers have been invited to participate in the project, as they must agree to use the standards and protocols for defining and recording online usage. The current COUNTER Codes of Practice are published on the ARL website.
With standards in place, library administrators will be able
to compare the cost per use of digital resources, and that is a powerful tool in
the management of digital assets. Whether publishers accept and adhere to the
COUNTER Code remains to be seen.
MINES: Measuring the Impact of Networked Electronic Services
Knowing which digital resources are used, how they are used, and by whom becomes a very complex question in a networked environment. Based on an established method for determining the indirect costs of using digital resources for grant-funded activities, MINES differs from E-Metrics and COUNTER which quantify total usage. MINES for Libraries™ collects demographic information about those using electronic resources and the purpose for which they are using them, through a transaction-based survey. Data collected over a two-year period from 14 research libraries revealed, for example, that of the overall use of digital resources for all of the libraries 70 percent occurred outside the library and 30 percent inside the library.
Added to the ARL New Measures program in May 2003, MINES for Libraries TM was implemented in 16 Canadian institutions last year through a contract between ARL and the Ontario Council of University Libraries (OCUL).
The Time is Now
This is an exciting and challenging time for academic libraries. The rapid development of information technologies has presented myriad opportunities and challenges. New technology-based tools have spurred librarians to reconsider and redefine collections, services, the skill sets required of library staff, and the attributes of library facilities. In some cases, old planning assumptions must be turned upside down—for instance, the assumptions that all users are in a building, and that the building contains all of the library’s collections. Clearly, contemporary libraries operate with different assumptions about the ways in which they might best carry out their responsibilities.
Assessment is evolving rapidly in education; from K-12 school
report cards, to new accreditation requirements for colleges and universities.
Librarians are committed to meeting the assessment challenge. We are finding new
ways of measuring the quality of collections, services, and programs, and better
ways of demonstrating the library’s contribution to student learning. For those
libraries that have not yet moved away from merely calculating inputs and
outputs to assessing their role and impact on teaching and learning—the time is
—Kathleen Hoeth can be reached at firstname.lastname@example.org
Cook, C., Heath, F., & Thompson, B. (2003). “Zones of tolerance in perceptions of library service quality: A LibQUAL+.” Portal: Libraries and the Academy, 3(1), 113-123.
Gedeon, J., Radcliff, C. & O’Connor, L. Project SAILS: Facing the Challenges of Information Literacy Assessment, presented at the American Association for Higher Education 2004 Assessment Conference, June 14, 2004, Denver, Colorado.
Hernon, P. (2002). “Quality: New directions in the research.” The Journal of Academic Librarianship 28(4), 224-231.
Kyrillidou, M., Franklin, B. & Plum, T. Assessing the Value of Networked Electronic Services: The MINES survey. Panel Session at the ACRL Annual Conference, Minneapolis, MN, April 8, 2005. http://www.libqual.org/documents/admin/MINES%20Panel%20ACRL.ppt
Mark, A.E. & Boruff-Jones, P. D. (2003). “Information literacy and student engagement: What the National Survey of Student Engagement reveals about your campus.” College & Research Libraries, 64(6), 480-493.
OCUL MINES results for 2004-2005 http://www.arl.org/stats/mines/ocul/MINESFINAL2005.HTML.
Parasuraman, A., Berry, L. L. & Zeithaml, V.A. (1991). “Refinement and reassessment of the SERVQUAL Scale.” Journal of Retailing 67, 420-450.
For Further Information
ACRL Information Competency Standards for Higher
ARL New Measures Initiative
The Evaluation Center
Internet Resources for Higher Education
National Survey of Student Engagement
Outcomes Assessment Resources on the Web
*A review of the regional accreditation requirements reveals a clear shift away from judging quality by describing inputs and outputs, and toward an expectation of routine assessment, with a focus on using assessment data for continual improvement.
Library Issues: Briefings for Faculty and Administrators (ISSN 0734-3035) is published bimonthly beginning September 1980 by Mountainside Publishing Co., Inc., 321 S. Main St., #213, Ann Arbor, MI 48104; (734) 662-3925. Library Issues, Vol. 26, no.1 © 2005 by Mountainside Publishing Co., Inc. Subscriptions: $80/one year; $140/two years. Additional subscriptions to same address $25 each/year. Address all correspondence to Library Issues, P.O. Box 8330, Ann Arbor, MI 48107. (Fax: 734-662-4450; E-mail: apdougherty@CompuServe.com) Subscribers have permission to photocopy articles free of charge for distribution on their own campus. Library Issues is available online with a password at http://www.libraryissues.com
last modified: September 2005
Produced by Mountainside Publishing Co., Inc.