Measurement And Evaluation Concepts
rating: 0+x
MeasurementEvalMindMap
From Measurement, Assessment and Evaluation General Principles CMAP

From Robert E. Dugan, Peter Hernon, Danuta A. Nitecki Viewing Library Metrics from Different Perspectives: Inputs, Outputs, Outcomes Santa Barbara : Libraries Unlimited, 2009, p. 48.

“Assessment is a process-oriented, cyclical activity involving the collection of information and appraisal data to measure or otherwise gauge progress toward an identified standard or other benchmark.”

“Evaluation is a product-oriented process of interpreting the information and data that were collected and analyzed during assessment; estimating the overall value or worth of a policy, project, program, or service; and/or determining if established standards or benchmarks have been met.”

Three types of evaluation are:

diagnostic—locate or identify the problem
formative – improve ongoing programs
summative – review a completed program

AssessmentParadigm
From Assessment Paradigm CMAP

Library Assessment & Evaluation Paradigm

Why Assessment?

  • Is our library accomplishing its goals and how well are we accomplishing them?
  • Are we a good library or a bad library?
  • How can we improve?
  • What is needed to improve?
  • Can we justify the expenditure of current resources?
  • How can we justify additional resources?

The Assessment Paradigm

  • Resources > Capability > Utilization > Impact
  • Resources: input measures
  • Capability: processing measures
  • Utilization: output measures
  • Impact: outcome measures
  • Input measures: resources allocated (annual budget, size of collection, number of staff)
  • Process measures: transform resources into activities (quantify time or cost to perform specific task or activity such as ordering, receiving, cataloging)
  • Output measures: how inputs and processes utilized (annual circulation, number of reference questions answered, number of users served)
  • Outcome measures: difficult to assess, relates to both individual served and to community

Methods of Assessment

Types of Assessment

  • Satisfaction of the customer.
  • Satisfaction of the worker.
  • Workflow effectiveness and speed.
  • Service delivery effectiveness and speed.
  • Cost benefit analysis.
  • Legality of actions performed.
  • Social justice and ethics.

Assessment Instruments

  • Surveys
  • Focus groups
  • Web usability studies
  • Secret shopper
  • Specific testing
  • Best practices
  • Service level agreements
  • Balanced Scorecard (an array of assessment tools in four areas)
  • Financial – How does the library look to stakeholders?
  • Customer – How do customers see the library?
  • Internal – In what areas must the library excel?
  • Innovation and learning – Can the library continue to improve and create value?

ARL New Measures Program

  • User satisfaction
  • Market penetration
  • Ease and breadth of access
  • Library impact on teaching and learning
  • Library impact on research
  • Cost effectiveness of library operations and services
  • Library facilities and space
  • Organizational capacity
  • LibQUAL+™ is a rigorously tested Web-based survey that libraries use to solicit, track, understand, and act upon users' opinions of service quality.
  • The DigiQUAL™ project is modifying and repurposing the existing LibQUAL+™ protocol to assess the services provided by digital libraries.
    • MINES for Libraries™ is an online transaction- based survey that collects data on the purpose of use of electronic resources and the demographics of users.
  • Climatequal
  • Collectively all three are under the name StatsQual
  • See http://www.arl.org/stats/initiatives/index.shtm
  • Assessment conference proceedings: http://www.libraryassessment.org/archive/

Building an Assessment Program

  • Leadership
  • Who does it (individual, committee)
  • Infrastructure (measurement and evaluation skills, data collection methods, sampling, analysis software)
  • Library culture (political process, positive deviance)
  • Environment and constituencies
  • Rewards and incentives

Planning – Seven Key Questions

  • What do we do?
  • Who are we here for?
  • What do they want and why?
  • How can we better improve their satisfaction and the library’s performance?
  • What is the strategy and process for delivering library services?
  • What needs to be done? Who will do it? When?
  • Do we know or can we determine the library’s contribution to the quality of life of a particular community?

Operational Excellence

  • What economies of scale exist?
  • What level of quality can be attained?
  • How can technology increase efficiencies?
  • What changes in procedures and processes will lead to improved productivity?

Customer Intimacy

  • What is library’s competitive advantage?
  • Who is the competition?
  • How can we better serve our existing customers?
  • What new segments of the population can be targeted?
  • What share of the market can we expect given our resources
  • Do we really know what different types of customers want?

Innovative Services, Reallocation Matrix

  • If there were no constraints, what could we do?
  • How can we take our existing service to a new level?
  • What service can we discontinue to reallocate resources?
  • What has never been tried before?
  • What are the options, new alternatives and potential synergies that should be considered?
  • Would upgrading our technology base provide the library with the potential for introducing new services or improving existing services?
  • Does the library have the talent and skills among existing staff to meet the challenge of innovation?
High
Priority Improvement Opportunity Maintain
Discontinue Reduce
Low High
Performance

UsingMetrics
From Using Metrics CMAP

Benchmarking and Best Practices

From Dugan et al.

“Benchmarking enables a library to focus on 2 questions:

1. In the context of a standard, goal, and objective, how good do we want to be with the services we offer?
2. Is our performance meeting, exceeding, or failing to meet standards?

Best practices, on the other hand, addresses three questions:

1. How do others do it?
2. Are others doing it better than we are?
3. How can we adapt our library to what others do?

Add a New Comment