Assessing the Library’s Contribution to Student Learning

Academics have always been prone to compare their current crop of students unfavorably to those they taught a few years ago. To go by anecdotal evidence, students today don’t read, can’t write, and think everything’s on the Internet. But is it as bad as it seems? Just what do our students actually know? Many libraries are addressing this question by developing new methods to better understand what impact their collections and programs have on student learning. In addition to these local efforts, three studies have recently been released that offer benchmark data about how students use libraries and the Internet. The results of these large-scale surveys are surprising—and, at times, apparently at odds.

Three Significant Studies

OCLC. In June of 2002, the Online Computer Library Center (OCLC) published the results of a study of how college students use the Web for research. Carried out by Harris Interactive, this survey of over 1,000 college students found that they are likely to start their research online, that they are concerned about the quality of information found on the “free” Web, and that they recognize a gap between the need for accuracy and what is likely to be available on the Web. If they need help, they prefer to get it face to face rather than online, though they are less likely to ask a librarian for help than they are a fellow student. And, contrary to the complaints often heard from librarians and faculty, 9 out of 10 students claim to use traditional print library resources at least some of the time, including print journals as well as books.

Pew Internet Life.A few months later, a study released by the Pew Internet Life project seemed to contradict some of the OCLC study’s findings. The Internet Goes to College reported that 73 percent of college students used the Internet more than libraries for their research; only 9 percent claimed to use the library more often than the Internet. More dismayingly, 80 percent of students reported using the library less than three hours a week. In addition to surveying students about how they do their work, teams of researchers using ethnographic methods observed students at work and concluded “students use electronic resources more than paper resources.”

Outsell, Inc. A third study, conducted by Outsell, Inc. for the Council on Library and Information Resources and the Digital Library Federation, presented yet another divergent perspective: that just over half of undergraduate students rely on print resources for their research all or most of the time; that the percentage was far higher for graduate students and faculty than for undergraduates. Almost two-thirds use the library as much now as they did two years ago, and expect printed books and journals to remain important sources of information for their research for at least the next five years.

Reconciling the Differences. How can we reconcile these different messages? As with many surveys, much depends on how the questions were asked and how the results are interpreted. In the case of the Pew Internet Life study, the survey questions did not distinguish between library resources delivered through the Web and the “free” Web. Students reporting that they used the Internet for research may have been using that portion of the library’s collection that is online. Had the question been asked differently, the results might have been closer to the findings in the other studies.

Even more problematically, the observers who concluded that students used more Web than print resources were making those observations in computer labs in the library and dormitories, and at campus Internet cafés, where students were naturally more likely to be using Web-based resources than print ones. Had observations been made throughout the library—or, better yet, as students were composing papers or preparing presentations at their own computers —they might have revealed a different result.

Nevertheless, thanks to these three significant studies, we have an unprecedented amount of information about how students use libraries and the Web for their information needs. At the same time, librarians are growing more interested in finding out how their libraries serve their student constituents—and, most importantly, how their efforts contribute to student learning.

 Libraries and Learning in an Age of Assessment

The focus on student learning as an important outcome of library programs is a relatively new concept, but many librarians are embracing the challenge of finding ways to better understand what impact they are having on students and their education. In part, this change in focus is driven by external trends in accreditation and an imperative to develop a “culture of evidence” through formal assessment programs. Fewer words arouse more passion on college campuses than “assessment.” Because of a growing perception that the public demands accountability, the influence of market models on higher education that position education as a product and students as consumers, and (some faculty would allege) an administrative urge to regulate the work of the professoriate, assessment has become a hotly contested and often bitterly resented movement that doesn’t so much move as proceed in fits and starts.

For libraries, the new emphasis on assessing student learning outcomes parallels another change in focus—from the library as an information warehouse to the library as a critical site for learning. The shift in emphasis from measuring inputs and outputs to assessing student learning converges with the library’s emerging role in fostering critical inquiry skills in collaboration with faculty across the curriculum. As these skills—sometimes called “information literacy”—become a part of the curriculum, libraries are compelled to go beyond counting objects and start asking “so what?“ We know how many books and journals we have because they are fairly easy to count, and we can even tell how many times they leave the shelves. But how do our students use them? Does having a strong collection mean students learn more?

We check out so many books or reserve readings, send and receive so many interlibrary loans, answer so many reference questions, which are all easy to count. But how exactly do those activities contribute to student learning? Are there ways they could make a greater contribution?

We know that we teach a certain number of students in library research courses or in course-related instruction sessions, and we may even know whether students think we do it well or not. But do we know whether they can apply the skills they learned? Do we know, in short, if the library contributes something significant to our students’ education?

Measurable Outcomes

It’s not hard to find out how many books are checked out; it’s much harder to devise measures that will tell a library whether or not it is having an effect on student learning. What evidence could we gather that would reveal whether students are learning research skills or not? And given the complexity of finding and using information for academic purposes, how can we attribute success or failure to the library? It could just as well be the work of faculty in the classroom—a well-honed assignment, conversations during office hours, modeling of effective research processes by the teacher—that determine a student’s success. Even so, the more libraries attend to what happens to students as they attempt to become confident and effective inquirers, the better they will be able to devise collections and services to support their learning.

Testing Information Literacy Skills. One approach to assessing a library’s impact on student learning is to test students on their information literacy skills. One notable project to develop a standardized assessment instrument has been under development by librarians at Kent State University. Based on the Information Literacy Competency Standards for Higher Education developed by the Association of College and Research Libraries in 2000,5 this test will be administered at dozens of colleges and universities in the next two years thanks to state and federal grant funding. Not only will this testing help the researchers develop a robust assessment tool, the data gathered in the process will provide comparative information on what our students know and where they are having difficulty. However, the assessment tool is based on multiple choice tests andthesedon’t necessarily work well for assessing outcomes. While they are relatively easy to administer, they do have limitations. A student could, conceivably test well on the specifics of finding and using information, but still have difficulty applying those skills in actual research; conversely, a student who did poorly on a test might be able to compose an excellent research paper. Librarians know this from experience—they are often bemused by faculty members who don’t know a Boolean operator from a handsaw, yet perversely publish brilliant scholarly work. They must be doing something right. Still, a well-designed test could tell us which areas are particularly challenging for students and might help libraries focus their instruction efforts more effectively.

Examining Finished Products. Another approach to assessing students’ competence is to examine finished products. In some cases, librarians simply examine bibliographies of student papers, scoring them by the variety and quality of sources used. Frustratingly, several researchers have found no correlation between what the librarians consider a good bibliography and what faculty consider a well-written research paper. Some librarians have addressed that issue by developing rubrics with faculty in the classroom that score how sources are used in papers—querying, for example, whether sources used provide credible evidence for a claim or whether students engage with their sources rather than simply providing a gloss of sources. If information literacy embraces not just being able to find sources, but the ability to put them to work effectively, the ways in which sources are used in composing an argument becomes an important factor in assessing the students’ skills. Further, creating and applying such rubrics can enable useful conversations across the curriculum.

Qualitative Means. In addition to tests and rubrics, many libraries are using a variety of qualitative means to get at student research behavior. Focus groups with students can reveal not only what they know, but what attitudes and beliefs underlie their behavior.⁸ Libraries have also used interviews with students to explore their research processes, have collected research journals, and have conducted observational studies as students complete research tasks. Some projects combine a variety of methods—for example, giving students tests on their use of online catalogs and databases while capturing their searches—to gain an in-depth understanding of not only what students can do, but the processes they use from start to finish. These insights can be put to use in a variety of ways, from redesigning Web portals to devising documentation or course materials. They also can be informative to faculty in the disciplines who often find their students’ behavior baffling, particularly since the multiple avenues available for finding information have changed so greatly from when they were undergraduates.

Other Measures. Whatever measures libraries may choose to understand what students know—and don’t know—there still is a need to combine that information with the kinds of numbers libraries have always compiled. If a library has less money for acquisitions than its peer institutions, or too small a staff, or a building that is outdated and inhospitable, that will doubtless have an impact on students’ ability to develop strong research skills for lifelong learning. If, on the other hand, a library has a generous budget and a high staff-to-student ratio and the students are not learning how to use those resources, then something isn’t working. The new Standards for College Libraries adopted by the Association of College and Research Libraries in 2000 spell out a number of ways a library can assess its contributions using a combination of inputs, outputs, and outcomes.

Learning How Students Learn

Libraries are complex places these days, and they compete as never before with alternative sources of information that are, for most of our incoming students, far more familiar than catalogs and stacks and call numbers. The Pew study points out that essentially all of our college students have used computers. One fifth began using them when they were between 5 and 8 years old. Very few have used an academic or research library. Almost everything about libraries is new and baffling for our incoming students, even the language we use to explain it. It’s not surprising that students tend to start their research online. Google is familiar and easy to use. The catalog—why click there to find information? The only catalog many students are familiar with is the college’s list of courses. Almost all students are familiar with Web browsers and don’t have to think twice about hyperlinks; very few of them have ever read a scholarly article and have little familiarity with citations as a link from one text to another. Faculty are surprised to discover that students often don’t know how to skim texts for meaning or that indexes in books can pinpoint specific information. It simply hasn’t been part of students’ previous experience.

In response, some would say the best practice is to accept that students are non-linear, multitasking visual learners and restructure education accordingly. Yet, if we make too many assumptions about how students learn without testing them, we’ll make mistakes. In the past few years, libraries realized students were often getting answers by using Ask Jeeves or another search engine. Many took that to mean that students would prefer to ask reference questions online and rushed to establish electronic reference services, only to find that few students are interested. The OCLC study found students rarely use Web-based “ask a librarian“ services. Why? Because they prefer face-to-face help when Jeeves fails them.

A press release announcing results of a survey commissioned by a textbook publisher claimed that 83 percent of faculty believe Web-based technology is more important to student success than libraries. Apart from the fact that much of the library, from its catalog to many of its collections is now “Web-based,“ the survey asked the wrong people. Reading too much into students’ apparent preferences can lead to costly mistakes. The same poll said 42 percent of faculty expected to be using e-textbooks within two years. These high expectations for e-books contrasts interestingly with an e-book experiment conducted at Ball State University, in which students confounded researchers by strongly preferring print to electronic textbooks. Students found traditional texts not only less likely to cause eyestrain, but it was easier to find what they needed in the print versions. Other online book efforts have not been embraced by students with the fervor expected of a hardwired generation. Both Questia and NetLibrary, marketing to the assumed preferences of Generation Y, burned through huge amounts of venture capital without securing a solid base of student acceptance. Why? As it happens, students may like to start research online, but they much prefer to use paper versions of texts when reading, taking notes, and using them in their own research. In an era of belt-tightening, universities can’t afford to make the same mistakes. Assessing library effectiveness by including measures of student learning outcomes is a good way to make sure libraries are making good use of their resources. Libraries can avoid costly lapses of judgment and have a better chance of actually helping their students become information literate if they pay attention to what kinds of experiences students have as they struggle to grasp the complexities of research in a hybrid print-electronic library.