Editors: Ann P. Dougherty, Mountainside Publishing; Richard M. Dougherty, University of Michigan, Emeritus
Contributing Editors: William Miller, Florida Atlantic University; Barbara Fister, Gustavus Adolphus College;  
Larry Hardesty, University of Nebraska at Kearney;
Mignon Adams, University of the Sciences in Philadelphia;
Steven Bell
, Philadelphia University; Kathleen Hoeth, Florida Gulf Coast University
 
 
Vol. 26, No. 1 September 2005


Meeting the Accountability Challenge

by Kathleen Hoeth

College and university administrators face philosophical and practical questions every day, but the recent trend in higher education to use assessment data for accountability has created a new challenge. Boards of Trustees, politicians, accreditation bodies, and others expect a new approach to measuring educational quality; they are demanding evidence that the institution evaluates its effectiveness in achieving its mission and then uses the assessment data for continual improvement.

The demand for data to demonstrate that educational funding is thoughtfully deployed and produces tangible benefits permeates higher education these days. This article is a brief for campus administrators about (1) campus-wide evaluation tools that produce some evaluative data about libraries, and (2) some of the library-specific assessment initiatives librarians have mounted to generate data for library management and to meet the accountability challenge.
 

The Trend

The accountability trend is clear in even a cursory review of the changes that have been made over the past few years in regional accreditation requirements. Looking at the six associations’ requirements for colleges and universities and for libraries reveals an unmistakable shift away from judging quality by describing inputs and outputs, and toward an expectation of routine assessment.* Moreover, the associations clearly expect educational quality to be demonstrated by a focus on using the assessment data for continual improvement. In its Accreditation Handbook, for example, the Western Association of Schools and Colleges describes this changing context for accreditation as, “A shift toward effectiveness and performance indicators beyond inputs and resources as the organizing basis for defining and evaluating quality.”

As institutions rush to learn the language and tools of this new trend, academic libraries are an excellent place to begin looking for information, and perhaps, inspiration. Traditionally, academic library quality has been expressed in terms of inputs such as collection size and square footage. Many institutions still celebrate the library’s acquisition of its millionth volume, for instance. But the dramatic, technology-driven reinvention of libraries over the past decade has resulted in demands for new methods of assessment.

Libraries now routinely supplement traditional collections and services with digital resources and Web-based service approaches. As librarians have reinvented academic libraries, we have also worked to create methods of assessment that reflect the reality of contemporary libraries and address the shift toward assessment and accountability in higher education. The North Central Association of Colleges and Schools offers the following perspective about libraries and learning resources:

It was not that long ago that accreditation was understood to focus rather heavily on resources in the library. Accrediting teams counted staff members and the square footage allocated to the library and to book inventories. Unless libraries are used and valued by students and faculty, their impact on learning is small. In short, a library—or a learning resource center—exists to support learning and teaching. . . . Consequently, it is critical for colleges and universities to assess actual student use of equipment, materials, and media, collecting evidence that something worthwhile is happening to students because learning resources exist.

In the same spirit, rather than providing a census of its collections, the Middle States Commission on Higher Education requires that “The institution provides sufficient learning and information resources and services to support the nature, scope, and level of the programs offered.” Colleges and universities are forced to ask, “What is ‘sufficient’?” The other regional associations have similarly broad statements, leaving librarians and institutional effectiveness staff to figure out a new approach. Library practitioners and scholars, therefore, have worked to create instruments and protocols for libraries that meet demands for accountability, measure service quality, and improve the data available to library administrators for effective library management.
 

New Measures for Libraries

Rather than focusing on inputs such as collection size and staffing, the first new library measures sought to describe what libraries do. That is, librarians began to report outputs such as the number of items borrowed or the number of reference questions answered. Those measures alone, however, still fell short of addressing whether library services were sufficient. As colleges and universities created student learning outcomes over the past few years, librarians also created measures that were based on outcomes — the extent to which student and faculty contact with libraries affected them and contributed to the mission of the university.

The increasing attention to measurement and assessment in libraries during the past decade might be described as a move beyond reporting what libraries do, to capturing the ways in which student learning is affected by library services and programs.

Since libraries contribute to the academic enterprise in a variety of ways including providing collections, creating research assignments, serving as an intellectual locus for the campus, and teaching bibliographic skills, there are several tools that may be used to evaluate one or more aspects of library sufficiency or quality. Most of these tools were first discussed in Library Issues by Stoffle and Phipps in March 2003.
 

NSSE

Institutions that participate in the National Survey of Student Engagement (NSSE) have an opportunity to use a portion of its scores as a library assessment tool. That this survey may be used for an additional purpose may be welcome news in Institutional Research offices across the country; the level of assessment activity required on campus has risen dramatically in recent years.


“The increasing attention to measurement and assessment in libraries during the past decade might be described as a move beyond reporting what libraries do, to capturing the ways in which student learning is affected by library services and programs.”


NSSE, administered by the Indiana University Center for Postsecondary Research, surveys first-year and senior students regarding their levels of engagement in the college experience. The key idea underlying the instrument is that student learning is tied to five areas of good practices in undergraduate education and may be analyzed through student engagement. The five areas measured are:

    1. Level of academic challenge,

    2. Active and collaborative learning,

    3. Student-faculty interaction,

    4. Enriching educational experiences, and

    5. Supportive campus environment.

Increasingly popular, the most recent NSSE survey was conducted earlier this year at 538 four-year institutions and includes data from more than 660,000 randomly selected first-year and senior students.

Amy Mark and Polly Boruff-Jones have argued that bibliographic instruction, the library’s efforts to develop information literacy competencies in students, may be assessed through an analysis of specific NSSE results. Through case studies at the University of Mississippi and Indiana University-Purdue University Indianapolis, they have demonstrated that NSSE findings can be used as pre- and post-tests for assessing student progress in attaining Information Literacy Standards as defined by the Association of College and Research Libraries (ACRL). Since its launch in 2000, the NSSE data has produced statistically significant results and benchmarks for comparison groups.
 

SAILS: Standardized Assessment of Information Literacy Skills

If Information Literacy is a primary focus in your institution’s library, in addition to looking at NSSE, consider an instrument in development that is specifically intended to address information literacy programs. The SAILS project, coming out of Kent State University, has as its purpose, the creation of a standardized, valid, and reliable instrument for assessing information literacy programs. The instrument developed to date has 126 items that are intended to test the 5 standards, 22 performance indicators, 87 outcomes, and 138 objectives articulated in the Higher Education Information Literacy Standards adopted by ACRL.

One key difference between NSSE and SAILS is that the latter looks at information literacy among cohorts of students, rather than focusing on individuals. In practice, this means that institutions can see the specific institution average and the cross-institutional average, as well as the average by major, class standing, or other factors. What factors are the most important at your institution?

Funded by a three-year federal grant from the Institute of Museum and Library Services, the third year of the SAILS project has just been completed. The third phase included 70 participating institutions and the project team is working to analyze the results. SAILS may eventually be used to compare information literacy measures with other indicators of student achievement, such as those enumerated in NSSE, for the purpose of clarifying what role information literacy may have in student success.

Both NSSE and SAILS assess libraries in relationship to the institution—their own institution, their students and faculty, tying them all together. The following programs compare libraries to libraries. This is still important, but it is not the same. Campus leadership should be aware of the difference; do not think that just because you are already involved in LibQUAL+TM you don’t need to look at participating in the NSSE or SAILS programs. It’s apples and oranges.
 

LibQUAL+TM

Overall service quality in libraries is increasingly being evaluated through LibQUAL+TM, a library-specific instrument developed under the auspices of the Association of Research Libraries (ARL). While the practice of librarianship has been changing, it remains solidly based in a commitment to service. Librarians in all types of libraries work to ensure that their organizations provide high quality service to library users in support of the goals of the library’s parent institution. It would be rare indeed to discover an academic library, for example, that did not include the concept of service quality in its mission to support the teaching, learning, and research missions of the college or university in which it operates.

Librarians have sought improved methods of assessment to reflect the changes in libraries and provide pertinent information to library administrators for planning and management purposes. Consequently, library organizations such as ARL have engaged in a number of measurement projects including LibQUAL+TM, which measures expectations and perceptions of library service quality.

The project has grown from 12 participating libraries in 2000, to 199 libraries in 2005. Libraries use the LibQUAL+TM survey instrument to solicit and track users’ opinions of service quality. The survey has its origins in SERVQUAL, a service quality instrument that has been used widely in the commercial sector. Grounded in the gap theory of service quality, the instrument was developed for the for-profit sector in the 1980s.The gap theory of service quality measures three things.

    1. Minimum level of service, or, the lowest level users will put up with,

    2. Desired level of service, or, what they want,

    3. Perceived level of service, or, how they think the agency or company is doing.

     


"Given the increasing investment in electronic information resources in libraries, assessing such services has become critical. A major goal of DigiQUAL is to identify best practices for digital library services."


Once the measures are collected, the gap between perceived and desired levels of service are defined and may be used as the basis for planning and implementing service improvements. LibQUAL+TM survey data are analyzed and each participating library receives reports of the aggregate response data as well as its own institutional data so that they may compare local library performance with that of peer institutions. Librarians report that they have used the survey data to inform practical decision-making, identify best practices, analyze service deficits, and allocate resources.
 

Digital Library Assessment Project

The success of LibQUAL+TM has spawned a related project that is working to modify the LibQUAL+TM survey for assessing services provided by the National Science, Math, Engineering and Technology Education Digital Library (NSDL) program. Applying the LibQUAL+TM protocol to NSDL has been dubbed, “DigiQUAL.” Given the increasing investment in electronic information resources in libraries, assessing such services has become critical. A major goal of DigiQUAL is to identify best practices for digital library services. Eventually, this project may result in the ability to assess service quality for digital libraries and generate data for librarians to use in improving the retrieval of relevant information and learner preparation for an information-rich society.
 

E-Metrics

Another effort working toward determining the effectiveness of digital collections is dependent upon establishing measures. The E-Metrics project began in 2000 and was intended to explore defining and collecting data on the use of electronic resources. It became clear in the course of the E-Metrics project, however, that consistent standards for data did not exist. Without usable definitions, measurement and analysis is difficult and complex. Project COUNTER (Counting Online Usage of Networked Electronic Resources) began in 2002 to address the problem.

Project COUNTER is working to establish practices for consistently and accurately determining the usage of digital resources. Publishers have been invited to participate in the project, as they must agree to use the standards and protocols for defining and recording online usage. The current COUNTER Codes of Practice are published on the ARL website.

With standards in place, library administrators will be able to compare the cost per use of digital resources, and that is a powerful tool in the management of digital assets. Whether publishers accept and adhere to the COUNTER Code remains to be seen.
 

MINES: Measuring the Impact of Networked Electronic Services

Knowing which digital resources are used, how they are used, and by whom becomes a very complex question in a networked environment. Based on an established method for determining the indirect costs of using digital resources for grant-funded activities, MINES differs from E-Metrics and COUNTER which quantify total usage. MINES for Libraries™ collects demographic information about those using electronic resources and the purpose for which they are using them, through a transaction-based survey. Data collected over a two-year period from 14 research libraries revealed, for example, that of the overall use of digital resources for all of the libraries 70 percent occurred outside the library and 30 percent inside the library.

Added to the ARL New Measures program in May 2003, MINES for Libraries TM was implemented in 16 Canadian institutions last year through a contract between ARL and the Ontario Council of University Libraries (OCUL).


 

The Time is Now

This is an exciting and challenging time for academic libraries. The rapid development of information technologies has presented myriad opportunities and challenges. New technology-based tools have spurred librarians to reconsider and redefine collections, services, the skill sets required of library staff, and the attributes of library facilities. In some cases, old planning assumptions must be turned upside down—for instance, the assumptions that all users are in a building, and that the building contains all of the library’s collections. Clearly, contemporary libraries operate with different assumptions about the ways in which they might best carry out their responsibilities.

Assessment is evolving rapidly in education; from K-12 school report cards, to new accreditation requirements for colleges and universities. Librarians are committed to meeting the assessment challenge. We are finding new ways of measuring the quality of collections, services, and programs, and better ways of demonstrating the library’s contribution to student learning. For those libraries that have not yet moved away from merely calculating inputs and outputs to assessing their role and impact on teaching and learning—the time is now.
—Kathleen Hoeth can be reached at khoeth@fgcu.edu
 

References

Cook, C., Heath, F., & Thompson, B. (2003). “Zones of tolerance in perceptions of library service quality: A LibQUAL+.” Portal: Libraries and the Academy, 3(1), 113-123.

Gedeon, J., Radcliff, C. & O’Connor, L. Project SAILS: Facing the Challenges of Information Literacy Assessment, presented at the American Association for Higher Education 2004 Assessment Conference, June 14, 2004, Denver, Colorado.

Hernon, P. (2002). “Quality: New directions in the research.” The Journal of Academic Librarianship 28(4), 224-231.

Kyrillidou, M., Franklin, B. & Plum, T. Assessing the Value of Networked Electronic Services: The MINES survey. Panel Session at the ACRL Annual Conference, Minneapolis, MN, April 8, 2005. http://www.libqual.org/documents/admin/MINES%20Panel%20ACRL.ppt

Mark, A.E. & Boruff-Jones, P. D. (2003). “Information literacy and student engagement: What the National Survey of Student Engagement reveals about your campus.” College & Research Libraries, 64(6), 480-493.

OCUL MINES results for 2004-2005 http://www.arl.org/stats/mines/ocul/MINESFINAL2005.HTML.

Parasuraman, A., Berry, L. L. & Zeithaml, V.A. (1991). “Refinement and reassessment of the SERVQUAL Scale.” Journal of Retailing 67, 420-450.

 

For Further Information

ACRL Information Competency Standards for Higher Education
www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm

ARL New Measures Initiative
www.arl.org/stats/newmeas/index.html

The Evaluation Center
www.wmich.edu/evalctr/

Internet Resources for Higher Education Outcomes Assessment
www2.acs.ncsu.edu/UPA/assmt/resource.htm

National Survey of Student Engagement
www.indiana.edu/~nsse/index.htm

Outcomes Assessment Resources on the Web
www.tamu.edu/marshome/assess/HTMLfiles/oabooks.html

Project COUNTER
www.projectcounter.org

Project SAILS
http://sails.lms.kent.eduindex.php

 

*A review of the regional accreditation requirements reveals a clear shift away from judging quality by describing inputs and outputs, and toward an expectation of routine assessment, with a focus on using assessment data for continual improvement.

 

Middle States (MSCHE) www.msche.org/?Nav1=About&Nav2=FAQ&Nav3=Question07

Standard 2: Planning, Resource Allocation, and Institutional Renewal

An institution conducts ongoing planning and resource allocation based on its mission and uses the results of its assessment activities for institutional renewal.

Standard 7: Institutional Assessment

The institution has developed and implemented an assessment plan and process that evaluates its overall effectiveness in: achieving its mission and goals; implementing planning, resource allocation, and institutional renewal processes; using institutional resources efficiently; providing leadership and governance; providing administrative structures and services; demonstrating institutional integrity; and assuring that institutional processes and resources support appropriate learning and other outcomes for its student and graduates.

New England (NEASC-CIHE)  www.neasc.org/cihe/cihe.htm has issued new standards for accreditation that will be effective January 1, 2006 including the following:

7.12 The institution regularly and systematically evaluates the adequacy, utilization, and impact of its library, information resources and services, and instructional and information technology and uses the findings to improve and increase the effectiveness of these services.

 

North Central (HLC) www.ncahigherlearningcommission.org/

Component 3d The organization’s learning resources support student learning and effective teaching

Example of evidence: The organization regularly assesses the effectiveness of its learning resources to support learning and teaching

Northwest (NWCCU) www.nwccu.org  

Standard 5.E.3 – Planning and Evaluation
The institution regularly and systematically evaluates the quality, adequacy, and utilization of its library and information resources and services, including those provided through cooperative arrangements, and at all locations where courses, programs, or degrees are offered. The institution uses the results of the evaluations to improve the effectiveness of these resources

 

Southern (SACS) www.sacscoc.org/

Core requirement 2.5 The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement, and (b) demonstrates that the institution is effectively accomplishing its mission.

Comprehensive Standard 3.3.1

The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.

 

Western (WASC) www.wascsenior.org/wasc WASC enumerates its values including the following:

promote the development and evaluation by institutions of quantitative and qualitative evidence that is used to improve institutional and educational effectiveness

 

 

 

Library Issues: Briefings for Faculty and Administrators (ISSN 0734-3035) is published bimonthly beginning September 1980 by Mountainside Publishing Co., Inc., 321 S. Main St., #213, Ann Arbor, MI 48104; (734) 662-3925. Library Issues, Vol. 26, no.1  © 2005 by Mountainside Publishing Co., Inc. Subscriptions: $80/one year; $140/two years. Additional subscriptions to same address $25 each/year. Address all correspondence to Library Issues, P.O. Box 8330, Ann Arbor, MI 48107. (Fax: 734-662-4450; E-mail: apdougherty@CompuServe.com) Subscribers have permission to photocopy articles free of charge for distribution on their own campus. Library Issues is available online with a password at http://www.libraryissues.com

 

last modified: September 2005

 

Produced by Mountainside Publishing Co., Inc.