STEP 3. Analyzing Data

Introduction

Earlier we introduced the contextual factors - the library and its service model and activities, professional contributions, and the clientele. These contextual factors (along with traditional outputs) give rise to outcomes and point to data collection methods with which to investigate each factor and their synergies. At this point, you have harvested a potent heap of data, from interview and focus group feedback to observations and secondary sources such as library and program documentation, all of which requires analysis. How best to proceed?

In Step 3: Analyzing Data (3.), we present a step-by-step process that you can use as a guide through the very detailed process of qualitative data analysis. This step in determining outcomes may take some time, but patient and thorough sifting of the data will yield a set of outcomes resulting from your library's unique service configuration, and with them, an informed picture of the benefits that the library brings to its community. A one size fits all approach does not work with outcomes; the good news is that the outcomes are out there in the data, waiting to be found.

You may, in addition, have quantitative data such as relevant outputs of the service (programs and the numbers of people who attended them, etc.) to incorporate into your findings. Quantitative output data help to substantiate qualitative findings. If you already know what your library's outcomes are, then they can be represented quantitatively, i.e., the number or percentage of participants in a particular program achieved a particular outcome. The professional literature offers a wide range of resources to assist with quantitative data analysis such as Babbie (2004) and Blaikie (2003).

Analyzing Qualitative Data

The overarching goal of data analysis remains to identify and to organize patterns in the data in order to produce a tailored outcomes set that reflects the local service. At the beginning stages of analysis of qualitative data there is often a sense of overload. At this stage, however, you can also begin to recognize patterns in the feedback from interview, focus group, and/or survey participants regarding "benefits [and drawbacks] to people as a result of your programs and services: specifically, achievements or changes in skill, knowledge, attitude, behavior, condition, or life status" (IMLS). Such recognition is a very positive development as similar responses across users point to outcome patterns.

The following steps can be used both to analyze the qualitative data that have been collected into an outcomes set and to capture the results.

1. Become very familiar with the data.

This introductory step provides you with an early opportunity to reacquaint yourself with the full universe of data that you collected. Read through all of the data and seek to understand the opinions and perspectives of the participants and to look for insights regarding the factors of context; for example:

• The library, service model, and activities: Where is it? What is its mission? What set of activities has the library developed to respond to the clientele?
• Professional contributions: What unique skills, talents, attitudes, etc. do staff bring to the mix?
• The clientele: Who are they? What do they need or want and why? How have they been affected (both positively and negatively)?

When reviewing focus group transcripts, be sure to analyze the data sequentially (i.e., "who says what when") to follow the chain of thought and the reactions among participants. As you proceed through the data analysis process, and begin to scan and flip through pages, the familiarity you gain in this initial step will prove invaluable.

2. Peruse the data and begin to identify patterns.

Patterns show a range of outcomes that are specific to the program but not necessarily unique to it. As you read and reread the data, they will see major themes (i.e., main categories of outcome) emerge. These main headings can be used to organize related sub-themes. Consider, for example, the theme and sub-themes we found regarding the contribution of the Peninsula Library System's Community Information Program (CIP) to capacity building among service providers in San Mateo County, California.

Main themes and sub-themes need to be substantiated with evidence. Librarians can illustrate the patterns they have identified with quotations and other support drawn from the data, as demonstrated in the California example:

"I think [CIP] is a great program. You guys save us a lot of time…[CIP is] coming up with updated printed lists of phone numbers; if they didn't do it we wouldn't have it. We would have to sit there and start from scratch."

"They [CIP] will hold the mailing labels for us and produce lists of providers based on a certain subject area. It's really a great service. I mean if we had to develop that list ourselves it would take hours to print out and type up and we can just call them and they produce it for us."

Reduces duplication of effort

"[CIP is] one-stop shopping…you can go there and have multiple needs filled…" Regarding CIP services, "They will get me the demographic information for the City...so that I will have a profile, they will do a map for me, so we use them for administrative purposes a lot…and they make it all pretty for you. We do not have the capacity to do that."

"CIP helps assure that we don't reinvent the wheel. CIP knows the information about the community; we don't need to know it too. We can go to CIP."

Enhanced decision-making

"We get questioned all the time on the reliability of our information related to disability. And the more we can make that believable and have a clearer picture of it, or at least be able to define the problems, I think the better justification we will have for pursuing particular programs and finally funding them."

"And we have done some significant program development based on a lot of the information that CIP helps up with. They put [our data] in those wonderful charts for us and we get them just automatically once a month."

Enhanced grant development

"We have done some significant program development based on a lot of the information that CIP helps us with. We now have a Client's Rights Advocacy Program that is county general funded, a Kids in Crisis program, all not funded from but anything but general funds because we were able to demonstrate those were needed services."

"We have done some significant program development based on a lot of the information that CIP developed… the maps clearly demonstrated to the county of San Mateo, the county welfare department Human Services Agency and to our agency where we needed to work on developing childcare homes and centers."

Interpretations, of course, can vary. We *strongly* recommend that you ask someone else, such as a co-worker (or two) to do the favor of reading and identifying themes in the data to see if they correspond with yours. This process is referred to as "intercoder reliabililty testing" and is a key element in ensuring the "trustworthiness" of your analytic approach and findings (see Pettigrew, 2000 for more detail).

3. Analyze and organize data.

The steps above enable you to interrogate the data that you collected from observation, interviews, focus groups, user feedback data, etc. Now you need a way to organize the patterns and quotes they have identified into a comprehensive outcomes set. To this end, librarians can use software packages (such as N5 (formerly Nudist), Ethnograph, etc.) or they can place each heading and quote on a 4 X 6 card and sort the cards by hand on a table.

The aim at this stage is not only to identify and substantiate the outcomes but also to arrange them in a logical manner. The analysis always involves shifting data from one category to another. Remember that the goal of the process is to structure the findings -- the outcomes of the service -- as well as to provide support (quotations from users, etc.) for the outcomes that have been identified. When librarians have completed the process they will have assembled a core outcomes set and the evidence to back them up.

4. Document findings.

This is an exciting stage. The qualitative approach you used to analyze the data and determine the outcomes of the service has helped you identify outcomes from the perspective of those who use them. Now what? What will these findings look like?

What they look like will vary with how you plan to use them, discussed in Step 4: Maximizing the Results of Your Outcomes Study, but in order to be most useful, broadly speaking, we recommend considering variations on the reporting formats that we have used in our case studies. Ways to capture the data so that they can be manipulated for broad array of applications (marketing, internal assessments, etc.) include:

Outcomes Tables

Outcome tables are a shorthand way of keeping track of library services' outcomes. Our tables of outcome include the outcome itself as well as the activities and inputs that library staff have designed and used to help to generate the outcome. We cannot overemphasize that the activities in which the staff engage are key to the library's outcomes. The analysis may cause you to rethink the way you and other staff deliver some services. Note that the space limitations of the human eye mean the quotations do not appear in the table format.

Full Reports

Full reports provide a record of the complete study. Several reports included in this book include: 1) the rationale for and need for outcome measurement; 2) the methods used to collect the data and determine the outcomes; 3) the model(s) used by the library to develop the service; and 4) the candidate outcomes, and the qualitative evidence in the form of quotes, stories, etc. identified in our study. Since selecting the outcomes of a particular service is the responsibility of the library staff (and not this, or any, set of researchers), we identify the outcomes in the case studies as 'candidate' outcomes.

Your full report need not look like the ones included in this book, but because it is the chief record of the work you conducted, it should clearly present the following:

1) The purpose and scope of the study including defining any terms or concepts that librarians might be misunderstood. (Note that simple concepts that you think are self-explanatory may not be understandable to other readers, even if they are library staff.)

3) The methodology used to design the study as well as collect and analyze the data.

4) The outcomes set, arranged in a logical way and including substantiating data based on the study (as shown in the outcomes sections of our case study reports).

Carefully developed reports establish the credibility of the data and its interpretation, especially for such audiences as city managers and auditors. In Step 4 (4.), we discuss the uses of the complete report.

Partial Reports

Partial reports such as those designed for particular audiences or those that focus on a single outcome or group of outcomes can be developed.

5. Develop a database of outcomes and activities.

You will want to maintain a file or database of substantiated outcome data, both positive and negative, by adding new comments, testimonials, letters of thanks, etc. as you receive them. These data can be arranged by outcome category, and will allow staff to better manage and share the outcome collection and analysis process over time, especially as you begin to incorporate outcome data collection as a natural element of library programs.

6. Determine the outcomes that staff will continue to monitor.

In all likelihood you will not be surprised at what they find. The outcomes, as we indicated in the introduction, are, of course, based on the activities librarians have developed to carry out the library's services and on the resources (inputs) that they put into the service. You may now want to regularly monitor some of these outcomes to determine the percentage of users who are affected by each outcome, as well as program improvements made as a result of your discovery of negative outcomes. Alternatively they may find that there are certain outcomes that they expected that are not present.

In the final HLLH outcomes study step, Step 4: Maximizing the Results of Your Outcomes Study (4.), we provide suggestions both for internal and external use of outcome findings.

References

Babbie, E. (2004). The practice of social research. Belmont, CA: Wadsworth Thomson Learning.

Blaikie, N. (2003). Analyzing quantitative data: From description to explanation. Thousand Oaks: Sage.

Pettigrew, K. E. (2000). Lay information provision in community settings: How community health nurses disseminate human services information to the elderly. Library Quarterly, 70.1, 47-85.