Libraries are for Use

Demonstrating the value of librarianship

The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance: A Propensity Score Matching Design


The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance: A Propensity Score Matching Design.

This article in the July 2015 issue of College & Research Libraries shows a “small, but also meaningful” increase in grades of first year students who use library services and facilities (computers & study rooms).  The value of this study is the extent to which the authors went to reduce the influence self-selection bias (that is, those who would use the library would also get good grades, perhaps regardless of this use).  It’s an interesting method, one that I would like to review more thoroughly.  However, that is not the focus of this posting.

What is interesting is that there are more and more of these studies that are finally being conducted, completed and analyzed.  Each are like a flashlight shining into a dark corridor, each revealing a little more shape, color or detail.  Admittedly, some lights are brighter than others; some lights are tinted, and some add their own shading.  But altogether, the lights are enabling us librarians and educators to see more clearly the actual impact of our work and our resources on the education of our students.

What is also interesting is how small that impact is…at least, according to the studies.  We as a profession may have thought our impact was strong – that students wouldn’t be able to do get a good grade without us.  What few studies that have attempted to measure our impact directly on student outcomes (grades, primarily) have usually resulted in statistically significant but demonstrably small connections.  And actually, connections are all that we’ve been able to establish.

Folks at University of Michigan found that “first-time, first-year undergraduate students who use the library have a higher GPA for their first semester and higher retention from fall to spring than non-library users” (read article).  The results included a difference of 1.4 percentage points in retention, 0.23 percentage points in GPA, and correlations that, while statistically significant (as in, not likely due to chance), were rather weak (r around .34-.45).  Finally, their regression model indicates that library usage explained about 12.4% of the variance in GPA between those who did and did not use the library.

DeeAnn Allison measured similar impacts of library usage (circulation & off-campus proxy logins) with GPA & retention (article).  She found (among other things) similarly significant but rather small connections, with usage explaining between 3-5% of variance of GPAs between those who did and did not use library resources.

There are two troubling aspects of these kinds of studies – they do not establish the direction of these connections or associations, and they do not correct for confounding.  Confounding is a hard concept to understand, but generally, it is the potential for a factor (e.g. age, sex, or program) to be related to both the outcome of interest (e.g. grades) and the factor of interest (e.g. using the library).  This makes it hard to know exactly how much the association of the factor & outcome of interest is.

This study in C&RL attempts to address the latter by adjusting for the propensity of library usage in the first place.

We used propensity score matching as a data preprocessing step…. For each student we generated the propensity score: in other words, the probability of using a particular library resource. We used these propensity scores to match individuals who used a particular library resource with those who did not use that recourse, such that the two groups had similar or almost identical background characteristics. As an example, we matched students who used workstations at least once with students who had a similar propensity score but who never used any workstations.

Results indicate that the probability of using library resources was a function of student
characteristics: clearly, some students were more likely to use library resources
than other students.

In summary, logistic regression results in table 3 suggest that students who used a particular library resource at the level specified differed from those who did not use that resource at the level specified. Therefore, we used logistic regression results to generate predicted probabilities of using library resources at each of the levels specified. We then used these predicted probabilities, which were estimated propensity scores, to match students in the treatment groups (students who used a given library resource) with those in the control groups (students who did not use that library resource).

Results from these analyses indicated that, before matching, the treatment and
control groups differed significantly along many of the observed covariates. However,
matching substantially decreased bias and made treatment and control subjects
similar along the observed covariates. This approach allowed us to eliminate, or at
least substantially decrease, the relationship between library resource utilization and
student characteristics, which in turn made it possible to estimate the impact of library
resource usage more accurately.

As results indicate, all the ATEs (after-treatment effects) were statistically significant, implying that using library resources at the levels specified was associated with higher first-term GPA….

Mean GPA after adjusting for student characteristics

Mean GPA after adjusting for student characteristics

 

The authors reported “gain in terms of GPA” as the usage increased, which was greater for those who overall used more resources (12-15 percentage points higher).  In the end, though, their evaluation of the difference with Cohen’s (which adjusts for variance) ranged from 0.15-0.26, indicating that the “corresponding treatment effect was small but not negligible.”

“Small but not negligible”…these are important words.  Should we, as librarians, be concerned about this?  I think not…indeed, we should celebrate.  This is the effect we have…small, but not negligible.  And these small but not negligible effects could add up over the course a student’s career.  That, I think, should be the focus of our research.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Information

This entry was posted on July 12, 2015 by in Academic Libraries, Assessment, LIS Research, Statistical Analysis.
The Scholarly Kitchen

What’s Hot and Cooking In Scholarly Publishing

Libraries are for Use

Demonstrating the value of librarianship

Scholarly Communication | Scoop.it

Demonstrating the value of librarianship

Library & Information Science Research | Scoop.it

Demonstrating the value of librarianship

Library Collections | Scoop.it

Demonstrating the value of librarianship

Lib(rary) Performance

About library statistics & measurement - by Ray Lyons

Walt at Random

Demonstrating the value of librarianship

ULS Universe

University Libraries Section of ACRL

The Scholarly Kitchen

Demonstrating the value of librarianship

The Quarterly Journal of Economics Current Issue

Demonstrating the value of librarianship

Texas Library Association blogs

Demonstrating the value of librarianship

Demonstrating the value of librarianship

Stephen's Lighthouse

Demonstrating the value of librarianship

ResourceShelf

Demonstrating the value of librarianship

Reference Notes

Demonstrating the value of librarianship

Politifact.com Truth-O-Meter rulings from National

Demonstrating the value of librarianship

Open and Shut?

Demonstrating the value of librarianship

N S R

Demonstrating the value of librarianship

Musings about librarianship

Demonstrating the value of librarianship

LISNews:

Demonstrating the value of librarianship

%d bloggers like this: