Analyzing Qualitative Data
Become very familiar with the data
Peruse the data and begin to identify patterns
Analyze and organize data
Develop a database of outcomes and activities
Determine the outcomes that staff will continue to monitor
Earlier we introduced the contextual factors - the library and its service
model and activities, professional contributions, and the clientele. These contextual
factors (along with traditional outputs) give rise to outcomes and point to
data collection methods with which to investigate each factor and their synergies.
At this point, you have harvested a potent heap of data, from interview and
focus group feedback to observations and secondary sources such as library and
program documentation, all of which requires analysis. How best to proceed?
In Step 3: Analyzing Data (3.), we present a step-by-step process that you
can use as a guide through the very detailed process of qualitative data analysis.
This step in determining outcomes may take some time, but patient and thorough
sifting of the data will yield a set of outcomes resulting from your library's
unique service configuration, and with them, an informed picture of the benefits
that the library brings to its community. A one size fits all approach does
not work with outcomes; the good news is that the outcomes are out there in
the data, waiting to be found.
You may, in addition, have quantitative data such as relevant outputs of the
service (programs and the numbers of people who attended them, etc.) to incorporate
into your findings. Quantitative output data help to substantiate qualitative
findings. If you already know what your library's outcomes are, then they can
be represented quantitatively, i.e., the number or percentage of participants
in a particular program achieved a particular outcome. The professional literature
offers a wide range of resources to assist with quantitative data analysis such
as Babbie (2004) and Blaikie (2003).
The overarching goal of data analysis remains to identify and to organize patterns
in the data in order to produce a tailored outcomes set that reflects the local
service. At the beginning stages of analysis of qualitative data there is often
a sense of overload. At this stage, however, you can also begin to recognize
patterns in the feedback from interview, focus group, and/or survey participants
regarding "benefits [and drawbacks] to people as a result of your programs
and services: specifically, achievements or changes in skill, knowledge, attitude,
behavior, condition, or life status" (IMLS). Such recognition is a very
positive development as similar responses across users point to outcome patterns.
The following steps can be used both to analyze the qualitative data that have
been collected into an outcomes set and to capture the results.
This introductory step
provides you with an early opportunity to reacquaint yourself with the full
universe of data that you collected. Read through all of the data and seek to
understand the opinions and perspectives of the participants and to look for
insights regarding the factors of context; for example:
- The library, service model, and activities: Where is it? What is its mission?
What set of activities has the library developed to respond to the clientele?
- Professional contributions: What unique skills, talents, attitudes, etc.
do staff bring to the mix?
- The clientele: Who are they? What do they need or want and why? How have
they been affected (both positively and negatively)?
When reviewing focus group transcripts, be sure to analyze the data sequentially
(i.e., "who says what when") to follow the chain of thought and the
reactions among participants. As you proceed through the data analysis process,
and begin to scan and flip through pages, the familiarity you gain in this initial
step will prove invaluable.
show a range of outcomes that are specific to the program but not necessarily
unique to it. As you read and reread the data, they will see major themes (i.e.,
main categories of outcome) emerge. These main headings can be used to organize
related sub-themes. Consider, for example, the theme and sub-themes we found
regarding the contribution of the Peninsula Library System's Community Information
Program (CIP) to capacity building among service providers in San Mateo County,
Reduces duplication of effort
Enhanced grant development
Main themes and sub-themes need to be substantiated with evidence. Librarians
can illustrate the patterns they have identified with quotations and other support
drawn from the data, as demonstrated in the California example:
"I think [CIP] is a great program. You guys save us a lot of time…[CIP
is] coming up with updated printed lists of phone numbers; if they didn't
do it we wouldn't have it. We would have to sit there and start from scratch."
"They [CIP] will hold the mailing labels for us and produce lists of
providers based on a certain subject area. It's really a great service. I
mean if we had to develop that list ourselves it would take hours to print
out and type up and we can just call them and they produce it for us."
Reduces duplication of effort
"[CIP is] one-stop shopping…you can go there and have multiple
needs filled…" Regarding CIP services, "They will get me the
demographic information for the City...so that I will have a profile, they
will do a map for me, so we use them for administrative purposes a lot…and
they make it all pretty for you. We do not have the capacity to do that."
"CIP helps assure that we don't reinvent the wheel. CIP knows the information
about the community; we don't need to know it too. We can go to CIP."
"We get questioned all the time on the reliability of our information
related to disability. And the more we can make that believable and have a
clearer picture of it, or at least be able to define the problems, I think
the better justification we will have for pursuing particular programs and
finally funding them."
"And we have done some significant program development based on a lot
of the information that CIP helps up with. They put [our data] in those wonderful
charts for us and we get them just automatically once a month."
Enhanced grant development
"We have done some significant program development based on a lot of
the information that CIP helps us with. We now have a Client's Rights Advocacy
Program that is county general funded, a Kids in Crisis program, all not funded
from but anything but general funds because we were able to demonstrate those
were needed services."
"We have done some significant program development based on a lot of
the information that CIP developed… the maps clearly demonstrated to
the county of San Mateo, the county welfare department Human Services Agency
and to our agency where we needed to work on developing childcare homes and
Interpretations, of course, can vary. We *strongly* recommend that you ask
someone else, such as a co-worker (or two) to do the favor of reading and
identifying themes in the data to see if they correspond with yours. This
process is referred to as "intercoder reliabililty testing" and
is a key element in ensuring the "trustworthiness" of your analytic
approach and findings (see Pettigrew, 2000 for more detail).
The steps above enable you to
interrogate the data that you collected from observation, interviews, focus
groups, user feedback data, etc. Now you need a way to organize the patterns
and quotes they have identified into a comprehensive outcomes set. To this end,
librarians can use software packages (such as N5 (formerly Nudist), Ethnograph,
etc.) or they can place each heading and quote on a 4 X 6 card and sort the
cards by hand on a table.
The aim at this stage is not only to identify and substantiate the outcomes
but also to arrange them in a logical manner. The analysis always involves shifting
data from one category to another. Remember that the goal of the process is
to structure the findings -- the outcomes of the service -- as well as to provide
support (quotations from users, etc.) for the outcomes that have been identified.
When librarians have completed the process they will have assembled a core outcomes
set and the evidence to back them up.
This is an exciting stage. The qualitative
approach you used to analyze the data and determine the outcomes of the service
has helped you identify outcomes from the perspective of those who use them.
Now what? What will these findings look like?
What they look like will vary with how you plan to use them, discussed in Step
4: Maximizing the Results of Your Outcomes Study, but in order to be most useful,
broadly speaking, we recommend considering variations on the reporting formats
that we have used in our case studies. Ways to capture the data so that they
can be manipulated for broad array of applications (marketing, internal assessments,
Outcome tables are a shorthand way of keeping track
of library services' outcomes. Our tables of outcome include the outcome itself
as well as the activities and inputs that library staff have designed and used
to help to generate the outcome. We cannot overemphasize that the activities
in which the staff engage are key to the library's outcomes. The analysis may
cause you to rethink the way you and other staff deliver some services. Note
that the space limitations of the human eye mean the quotations do not appear
in the table format.
|Outcome: Capacity Building
||Activities that Foster Capacity Building
County agencies and organizations save time
Reduced duplication of efforts
Decision-making is enhanced, supported
Grant development activities are supported
CIP disseminates information to the providers and the providers disseminate
information to the community
CIP produces mailing labels which agencies request for specialized audiences
Provides one-stop shopping
Custom-developed statistics as well as monthly charts and graphs
Produce Maps that visualize community data
Full reports provide a record of the complete study. Several
reports included in this book include: 1) the rationale for and need for outcome
measurement; 2) the methods used to collect the data and determine the outcomes;
3) the model(s) used by the library to develop the service; and 4) the candidate
outcomes, and the qualitative evidence in the form of quotes, stories, etc.
identified in our study. Since selecting the outcomes of a particular service
is the responsibility of the library staff (and not this, or any, set of researchers),
we identify the outcomes in the case studies as 'candidate' outcomes.
Your full report need not look like the ones included in this book, but because
it is the chief record of the work you conducted, it should clearly present
1) The purpose and scope of the study including defining any terms or concepts
that librarians might be misunderstood. (Note that simple concepts that you
think are self-explanatory may not be understandable to other readers, even
if they are library staff.)
3) The methodology used to design the study as well as collect and analyze
4) The outcomes set, arranged in a logical way and including substantiating
data based on the study (as shown in the outcomes sections of our case study
Carefully developed reports establish the credibility of the data and its interpretation,
especially for such audiences as city managers and auditors. In Step 4 (4.),
we discuss the uses of the complete report.
Partial reports such as those designed
for particular audiences or those that focus on a single outcome or group of
outcomes can be developed.
You will want to maintain
a file or database of substantiated outcome data, both positive and negative,
by adding new comments, testimonials, letters of thanks, etc. as you receive
them. These data can be arranged by outcome category, and will allow staff to
better manage and share the outcome collection and analysis process over time,
especially as you begin to incorporate outcome data collection as a natural
element of library programs.
In all likelihood
you will not be surprised at what they find. The outcomes, as we indicated in
the introduction, are, of course, based on the activities librarians have developed
to carry out the library's services and on the resources (inputs) that they
put into the service. You may now want to regularly monitor some of these outcomes
to determine the percentage of users who are affected by each outcome, as well
as program improvements made as a result of your discovery of negative outcomes.
Alternatively they may find that there are certain outcomes that they expected
that are not present.
In the final HLLH outcomes study step, Step 4: Maximizing the Results of Your
Outcomes Study (4.), we provide suggestions both for internal and external use
of outcome findings.
Babbie, E. (2004). The practice of social research. Belmont, CA: Wadsworth
Blaikie, N. (2003). Analyzing quantitative data: From description to explanation.
Thousand Oaks: Sage.
Pettigrew, K. E. (2000). Lay information provision in community settings: How
community health nurses disseminate human services information to the elderly.
Library Quarterly, 70.1, 47-85.