Moore, Green & Gallis provided a paradigm for evaluating medical education that has become the standard cited reference when discussing outcomes. There are 7 levels:
Level 1: Participation
Level 2: Satisfaction
Level 3: Learning
A. Declarative
B. Procedural
Level 4: Competence
Level 5: Performance
Level 6: Patient Health
Level 7: Community Health
Do these levels cover everything? Paradigms are useful to the extent they allow our processes to be thorough, to the extent they stimulate, rather than restrict our thinking, and when they provide us with quality results. They are harmful when they restrict our creativity, when they rule out different options that may prove useful to us. Moore’s levels are a really good paradigm, but sometimes it’s best to step outside of the paradigm and think what other options we might be missing out on.
In corporate learning, the “Kirkpatrick levels” occupy an equivalent position to Moore’s level in medicine (Kirkpatrick, 1977). There were four in the original formulation, and those have been augmented over time. The full version might look like this:
Level 0: Usage
Level 1: Satisfaction
Level 2: Learning
Level 3: Transfer
Level 4: Business Impact
Level 5: Return on Investment
Level 6: Intangibles
There are more similarities than differences, of course. There are interesting points of departure, of course, for example, the separation of learning into “procedural” and “declarative”. “Transfer” at Level 3, in Kirkpatrick, presents interesting distinctions from “Competence”, Level 4 in the Moore paradigm. “Transfer” simply notes whether desired behaviors have made the leap from a learning environment into a healthcare practice, separating out the issue of whether it works or not. For example, a program might encourage hand-washing behaviors, but whether they will actually show a reduction in post-operative infections is a slightly different matter. Most people in corporate learning and development use the “transfer” measurement to consider what barriers and enablers exist in the workplace. That consideration enhances the responsibility of an instructor: they are not just providing learning, but taking a broader mission of performance improvement. This is an example where a simple reliance on one paradigm may restrict our options.
‘Business Impact’ and performance are much the same thing in the two paradigms. They refer to a metric that is being affected in a measurable way: something that can be counted. That might be a reduction in blood sugar levels in a patient, reduced diagnosis time, or a reduction in hospital stay length. Level 6, Patient Health, and Level 5 in the Kirkpatrick paradigm, are more similar than they first appear. The underlying question in both paradigms is “Was this program worthwhile?” Return on investment is a simple calculation that tallies up dollarized benefits and dividing them by the cost of the intervention program. When a program benefit can be given an accurate value in the medical environment, this is a good idea for several reasons. First, it allows disparate programs to be valued head-to-head using a common yardstick. Secondly, it allows communication of benefits to stakeholders with primarily financial responsibilities to occur, such as administrators or insurers. The final levels, in both cases, serve as areas for expansion, when there are areas that aren’t easily captured by the lower levels of measurement.
Both paradigms work well, but there are others that are useful as well. Dean Spitzer split evaluation up based on when the evaluation was done: predictive, formative, baseline, in-process, and retrospective. Almost all of what the medical community calls “outcomes” is based on the last: what do I think a program changed? It could also be argued that, as far and away the most common evaluation strategy being a survey given at the time of a learning event, that “in process” might be closer to the truth. Measurement and evaluation often has some ambiguity associated with it. Spitzer’s paradigm is enlightening in that it brings together different parts of the timeline: gap analysis and outcomes become part of a consistent effort. It is these sort of insights that you want from paradigms. All too often, gap analysis is simply an expert’s opinion of what a gap is, rather than any more carefully measured strategic measurement.
There are numerous other concepts that the Moore Paradigm doesn’t suggest at all. In my practice, the most useful concepts that I refer to as “optimization” cover some of the important aspects that need to be measured. Here are the concepts I think need to be part of any important measurement effort that aren’t explicitly covered in Moore’s levels. Measurement needs to be about improving the delivery of interventions, and ultimately about improving health care. I use the word “optimization” to refer to those improvements. I’ll quickly list some of the main areas
Segmentation is the breaking down of the impact on a group-by-group basis to understand where the greatest and least impacts are achieved. Do nurses report more knowledge gain than do pharmacists? Do physician assistants have more barriers in transferring their learning to the workplace? Segmentation offers two great opportunities: selection of populations who will benefit the most, and improvement of the interventions to address the needs of those who are underserved. Some caution needs to be exercised into segmentation to avoid making conclusions about very small groups.
Metric Interaction is another important concept. The idea is that the relationship between the different metrics needs to be considered as well. For example, a training program that emphasized how to eliminate barriers to diabetes treatment might produce data where those who had more barriers also perceived more impact. Regression analysis is one of the more helpful tools in this case.
There are many other concepts that a good measurement professional can use to investigate and bring to life a training or other program in healthcare: timeline studies, synergistic or diminishing returns effects, saturation, predictive analytics for future programs, and mixture of programs, to name a few. An experienced analyst can add so much to the process, and advanced statistics and visualization software can add a lot to the mixture.