Improving Response Rates: The Third Post in the Series

This three part post will examine survey response rates from a CME perspective, helping you estimate how many responses you need, what the response rate will likely be, and how to improve the rate.   Data is drawn from personal projects, as well as the CME, healthcare, and broader literature.

Today’s post, the last of three, discusses how to improve response rates, and summarizes the three posts.

Improving Response Rates

There are several reasons to improve response rates.  Higher response rates improve the certainly of conclusions reached, particularly when drilling down into segments of the population.  For example, it may be very useful to distinguish between the various impacts on physicians, nurses, and pharmacists, and higher response number help you slice the data finer.   The higher the proportion of the population who respond, the less the likelihood of a ‘selection bias’ that may skew results.   (There are “modest differences” shown between responders and nonresponders, and between early and late respondents on demographic and/or practice-related characteristics.  (Cull, O'Connor, Sharp, & Tang, 2005) (Olmsted, Murphy, McFarlane, & Hill, 2006))   The more, and better, the textual responses, the richer and the more nuanced the text used in the write-up can be.   Even if the outcomes reporting is not improved by increased response rates, the data collected and analyzed can be smaller, saving time and money.

As made clear elsewhere in the article, the literature focused on CME is not large, and is occupied with other topics than testing techniques for improving response rates.  To discuss different strategies adequately, we are discussing ideas that may not have CME literature or even literature from elsewhere in healthcare.    

Communication: 

Clear communication is essential in any learning and information-based activities.  Here are our suggestions on strategies for communication.

1. Provide advance communication about the questionnaire. This seems intuitively important; research on the US Census indicate that notifying people that the survey was going to arrive improved response rates by six percentage points (Dillman, Clark, & Treat, 1994).  However, not all research agrees on this point; one study of physicians showed no differences between the group that received a premailing and the control group (Shiono & Klebanoff, 1991).   Lockyer et al did promotions, however, and their response rates are lower than those of any other informational surveys listed in Table 1. (Lockyer, Horsley, Zeiter, & Campbell, 2015)
2. Use one or two follow-up reminders. Written and telephone reminders are each associated with about 13 percentage point improvements.  (Asch, Jedrziewski, & Christakis, 1997)
3. Review the questionnaire at the end of the educational session. This helps explain the survey, and gives a chance to motivate the respondents better.
4. Communicate the time limit for submitting responses. If people can put a specific time on their calendar, it is more likely to get done.

Motivation:

1.       Give something away with the questionnaire.   This has been shown to be a very good way to improve response rates, with reservations.   What does help?  Cash has been shown to help, when given without delay, and in a form that can be used.  One study with physicians showed that, while both were effective, checks were more effective than gift cards for a $25 amount (Hogan & LaForce, 2008).  One meta-study establishes that even a dollar provided adequate incentive to improve response rates in physicians, and effects leveled off sharply after $1, even when going up to $20.   (VanGeest, Johnson, & Welch, 2007) Token items such as pencils (Kellerman & Herold, 2001)  or candy (Burt & Woodwell, 2005) don’t seem to help response rates with physicians, nor do “lottery” chances to win some item in the future (Tamayo-Sarver & Baker, 2004).   For internet-based surveys, the author has had good luck with giving away movie tickets in return for survey participation regarding technology use.  (Capital Analytics, 2010).  Comparative studies indicate that cash payments are more effective compared with charity inducements (Deehan, Templeton, Taylor, Drummond, & Strang, 1997) (Olson, Schneiderman, & Armstrong, 1993) monetary donations to their alma mater (Gattellari & Ward, 2001), non-monetary incentives (Easton, Price, Telljohann, & Boehm, 1997), or opportunities to win a cash prize through a lottery. (Tamayo-Sarver & Baker, 2004)    

In the case of commercially supported CME-activities, there’s an obstacle to using incentives for physician learners who participate in these activities:  the Sunshine Act (Health Policy Briefs, 2014).  The Sunshine Act legislation requires reporting of any payments or other transfers of value made to physicians or teaching hospitals from August of 2013 forwards.  

2.       Have an introductory message signed by an executive or other important person in the organization.  One study compared surveys sent with the letterhead of an AMA Vice President, and the other sent by a marketing research firm, and the surveys sent under the more prestigious letterhead had a 11.2 percentage point advantage.  (Olson, Schneiderman, & Armstrong, 1993)

3. Clearly communicate why the questionnaire is important. Show how the data will be integrated with other data.

4. Let participants know what actions will be taken with the data, and who will see it. If appropriate, let the target audience know that they are part of a carefully selected sample.  Add emotional appeal. The world is full of surveys that never get acted on, or whose results aren’t interpreted. Anecdotal evidence in the area of employee engagement suggests that giving an engagement survey, then not acting on the results, actually reduces employee engagement.

Design and Execution:

1. Distribute questionnaire to a captive audience.  This is far and away the best idea.   For CME applications, from the Academy of CME data, if the person has to fill out information to get credit, and the survey information is in the same form, the chance that they will fill out the minimal information to get credit and walk away is low.    In reviewing data from several thousand learners in the last year, all respondents who filled in information sufficient to obtain credit also answered .

2.       Keep the questionnaire simple and as brief as possible. It is believed among those who create and design surveys that the response rate falls dramatically as the length increases; this is borne out by research in the medical community. (Hing, Schappert, Burt, & Shimizu, 2005) (Jepson, Asch, Hershey, & Ubel, 2005)  (Cartwright, 1978)  While exactly what the response curve looks like is unknown, but every extra minute is thought to count.  One study showed that closed ended questionnaires resulted in a 22% higher answer rates.  (Griffith, Cook, Guyatt, & Charles, 1999)

3. Keep questionnaire responses anonymous – or at least confidential. While CME responses do not seem likely to make participants feel uneasy about their identity compared to other possible surveys, this is always good practice.  Keep in mind that confidential may be better than anonymous when analysis would be improved by being able to link together multiple datasets.  For example, if a survey is given at a conference, being able to link that information to demographic information contained in the registration data by a name or email address can make the outcomes reporting richer.  Using a third party to collect and analyze the data can increase the participant’s feeling of privacy.

4. Make it easy to respond; consider an alternative distribution channels, such as both regular mail and e-mail.  If regular mail is used, include a self-addressed, stamped envelope/e-mail; if an internet survey is used, test it rigorously using several different types of browsers and computers.

5. Allow completion of the survey during work or course hours, rather than requiring the participant to use personal time.

6. Design questionnaire to attract attention, with a professional format. A census redesign featuring user-friendly graphic design, with color coding, prominent question numbers,  and categoricallevels improved response rates by 4 percentage points.   (Dillman, Clark, & Treat, 1994)

Summary

Surveys are a keystone in CME evaluation processes, despite a new emphasis on operational data, such as charts.   Surveys are easy to do, and provide information across a variety of levels, including practice change, knowledge gain, transfer of behaviors, enablers and barriers for new techniques, and outcomes estimation.   The number of responses needs to be adequate to test the efficacy of the learning event.   How many responses are required?  That will depend on which question is the most crucial.  That answer to that question should be evaluated based on the complexity of the answer, the mean and variance of the response, the required effect size, and other factors.  You won't have the exact answer, of course, but using similar studies gets you close enough for an intelligent estimate.

The number of expected responses is based on similar work done elsewhere, with the huge caveat that no two studies are exactly alike.   Factors that may need to be considered in comparing studies are the timing of the survey (intelligence gather prior to other contact, event-time surveys, and post-event follow-up surveys), the populations surveyed, and the healthcare domain.   Due to the unpredictability of the response rate, the possibility of underestimating the effect size, and the ever-present desire to slice the data in finer  and finer segments  to improve the ability to optimize by measuring sub-populations, improving the response rate is always a good idea.  Even if everything goes exactly as planned, and no further analysis is desired, higher response rates allow the evaluator, and the participants, the ability to save time and money on distributing surveys.

Numerous ways to improve response rate have been considered over the years; whether there is empirical evidence to confirm the method, and how applicable it is to your particular area can be open questions.    Methods for improving response rates with the strongest support are:

1.       Distribute your survey to a captive audience – those sitting in a room at a presentation, or those filling out information to obtain CME credits.

2.       Communicate clearly the need for the survey, how it will help improve the activity for everyone in the future, and the confidentiality of the activity.  Communications should be from a person who is well-regarded by the audience.  Follow-up with non-responders promptly.

3.       Compensate respondents, in advance, and in cash or some other generally useful good, if possible.  Unfortunately, the Sunshine act eliminates this option for many purposes.

4.       Keep the survey short and relevant.  Use close-ended responses where an open-ended response would not add value.

Dillman (Dillman D. A., 2000) provides the following general advice that we like:

”One of the most common mistakes made in the design of mail surveys is to assume the existence of a “magic bullet”, that is, one technique that will assure a high response rate regardless of how other aspects of the survey are designed”.    

A hybrid of these techniques, mixed judiciously with common sense, will likely produce improvement in response rates.  Keep in mind that, where we have noted percentage point increases, those are examples based on different situations, and it is highly unlikely you will be able to combine a number of different ones without diminishing returns. 

Surveys will continue to be an important tool in measuring and improving medical education.   Response rates are an important aspect of surveys – good response rates allow more higher statistical certainty, deeper drill-downs into population segments, reduce chances of a selective response bias, and provide more textual material that enriches reporting.

Works Cited

Asch, D. A., Jedrziewski, M. K., & Christakis, N. A. (1997, Oct). Response rates to mail surveys published in medical journals. Journal of Clinical Epidemiology, 50(10), 1129-1136.

Buriak, S. E., Potter, J., & Bleckley, M. K. (2015). Using a Predictive Model of Clinician Intention to Improve Continuing Health Professional Education on Cancer Survivorship. Journal of Continuing Education in the Health Professions, 35(1), 57-74.

Burt, C., & Woodwell, D. (2005). Tests of methods to improve response to physicans surveys. Arlington, VA: Paper presented at the 2005 Federal Committe on Statistical Methodology.

Capital Analytics. (2010, June 20). Sun Microsystems University Case Study. Retrieved April 18, 2012, from Capital Analytics: http://www.capanalytics.com/wp-content/uploads/2012/01/BRANDED-SunMentoring-Case-Study-2012.pdf

Cartwright, A. (1978). Professionals as responders: variations in and effects of response rates to quesitionnaires, 1966-77. Br Med J, 2, 1419-1421.

Cook, C. F., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web or internet-based surveys. Educational and Psychological Measurement, 60(7), 821-836.

Cull, W. L., O'Connor, K. G., Sharp, S., & Tang, S. S. (2005). Response rates adn response bias for 50 surveys of pediatricians. Health Services Research, 40, 213-226.

Deehan, A., Templeton, L., Taylor, C., Drummond, C., & Strang, J. (1997). The effect of cash and other financial inducements on the response rate of general practitioners in a national postal survey. British Journal of General Practice, 47, 87-90.

Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method. New York, New York: Wiley and Sons.

Dillman, D. A., Clark, J. R., & Treat, J. (1994). The Influence of 13 Design Factors on Response Rates to Census Surveys. Annual Research Conference Proceedings, U.S. Bureau of the the Census, (pp. 137-159). Washington, D.C.

Easton, A. N., Price, J. H., Telljohann, S. K., & Boehm, K. (1997). An informational versus monetary incentive in increasing physicans response rates. Psychological Reports, 81, 968-970.

Evans, J. A., Mazmanian, P. E., Dow, A. W., Lockeman, K. S., & Yanchick, V. A. (2014). Commitment to Change and Assessment of Confidence: Tools to Inform the Design and Evaluation of Interprofessional Education. Journal of Continuing Education in the Health Professions, 34(3), 155-163.

Field, T. S., Cadoret, C. A., Brown, M. L., Ford, M., Greene, S. M., & Hill, D. (2002). Surveying physcians: Do compoents of the "Total Design Approach" to optimizing survey response rates apply to physicians? Medical Care, 40, 596-606.

Garber, M. (2012, May 30). The Future Growth of the Internet in One Chart. Retrieved from The Atlantic: http://www.theatlantic.com/technology/archive/2012/05/the-future-growth-of-the-internet-in-one-chart-and-one-graph/257811/

Gattellari, M., & Ward, J. E. (2001). Will donations to their learned college increase surgeons' participation in surveys? A randomized trial. Journal of Clincial Epidemoiology, 54, 645-650.

Grava-Gubins, I., & Scott, S. (2008, Oct). Effects of various methodologic strategies: Survey response rates among Canadian physicians and physicians-in-training. Can Fam Physician, 54(10), 1424-1430.

Graves, R. M. (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opin Q, 70(5), 646-675.

Griffith, L. E., Cook, D. J., Guyatt, G. H., & Charles, C. A. (1999). Comparison of open and closed questionnaire formats in obtaining demographic informatoin from Candian general internists. Journal of Clinical Epidemiology, 52, 977-1005.

Grzeskowiak, L. E., Thomas, A. E., To, J., Reeve, E., & Phillips, A. J. (2015). Enhancing Continuing Eduction Activitys Using Audience Response Systems: A Single-Blind Controlled Trial. Jounral of Continuing Education in the Health Professions, 35(1), 38-45.

Harris, W. A., Spencer, P., Winthrop, K., & Kravitz, J. (2014). Training Mid- to Late-Career Health Professionals for Clincial Work in Low-Income Regions Abroad. Journal of Continuing Education in the Healthy Professions, 34(3), 179-184.

Health Policy Briefs. (2014, Oct 2). Retrieved from Health Affairs.org: http://www.healthaffairs.org/healthpolicybriefs/brief.php?brief_id=127

Hing, E., Schappert, S. M., Burt, C. W., & Shimizu, I. M. (2005). Effects of form length and item format on response patterns and estimates of physician office and hospital outpatient department visits. National Ambulatory Medical Care Survey and National Hospital Ambulatory Medical Care Survey. Vital Health Statistics, 2(139), 1-32.

Hogan, S. O., & LaForce, M. (2008). Incentives in Physician Surveys: An Experiment Using Gift Cards and Checks. American Statistical Association Online Proceedings, (pp. 4179-84). Retrieved from http://www.amstat.org/sections/srms/proceedings/y2008/files/hogan.pdf

Jepson, C., Asch, D., Hershey, J. C., & Ubel, P. A. (2005). In a mailed phsycian survey, questionnaire length had a threshold effect effect on response rate. Journal of Clinical Epidemiology, 58, 103-105.

Kellerman, S. E., & Herold, J. (2001). Physician Response to Surveys: A Review of the Literature. Am J Prev Med, 20(1), 61-67.

Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M. F., Schemitsch, E. H., & Tornetta, P. (2004). Internet versus mail questionnaires: A controlled questionnaire. Journal of medical internet research, 39, e39.

Lockyer, J., Horsley, T., Zeiter, J., & Campbell, C. (2015). Role for Assessment in Maintenance of Certification: Physician Perceptions of Assessment. Journal of Continuing Education in the Health Professions, 35(1), 11-17.

Mazmania, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (1998, August). Information about Barriers to Planned Change: A Randomized Controllled Trial Involving Continuing Medical Education Lectures and Commitment to Change. Academic Medicine, 73(8), 882-886.

McConnell, M. H., Azzam, K., Xenodemetropoulos, T., & Panju, A. (2015). Effectiveness of Test-Enhanced Learning in Continuing Health Sciences Education: A Randomized Controlled Trial. Journal of Continuing Education in the Healthy Professions, 35(2), 119-122.

Nulty, D. (2008). The adequacy of response rates to online and paper surveys: what can be done? Assessment & Evaluation in Higher Education, 33(3), 301-314. doi:10.1080/02602930701293231

Olivieri, J. (2012, June 25). Rule of Thumb: Number of Participants per Survey Item. Retrieved December 17, 2015, from assessCME: https://assesscme.wordpress.com/2012/06/25/rule-of-thumb-number-of-participants-per-survey-item/

Olmsted, M. G., Murphy, J., McFarlane, E., & Hill, C. (2006). Evaluating methods for increasing physician survey cooperation. Miami Beach, FL: Annual Conference of the American Association for Public Opinion Research.

Olson, C. A. (2014, Spring). Survey Burden, Response Rates, and the Tragedy of the Commons. Journal of Continuing education in the Health Professions, 34(2), 93-95.

Olson, L., Schneiderman, M., & Armstrong, R. V. (1993). Increasing Physician Survey Response Rates Without Biasing Survey Rates. Proceedings of the section on survey research methods of the American Statistical Association (pp. 1036-1041). Alexandria, VA: American Statistical Association. Retrieved from http://www.amstat.org/sections/srms/Proceedings/papers/1993_177.pdf

Phillips, J. J. (1997). Return on Investment in Training and Performance Improvement Programs. Houston, Texas: Gulf Publishing Company.

Pololi, L. H., Evans, A. T., Civian, J. T., Vasiliou, V., Coplit, L. D., Gillum, L. H., . . . Robert, T. B. (2015). Mentoring Faculty: A US National Survey of Its Adequacy and Linkage to Culture in Academic Health Centers. Journal of Continuing Education in the Health Professsions, 35(3), 176-184.

Sarayani, A., Naderi-Behdani, F., Hadavand, N., Javadi, M., Farsad, F., Hadjibabaie, M., & Gholami, K. (2015). A 3-Armed Randomized Trial of Nurses' Continuing Education Meetins on Adverse Drug Reactions. Journal of Continuing Education in the Health Professions, 35(2), 123-130.

Shiono, P. H., & Klebanoff, M. A. (1991). The effect of two mailing strategies on the response to a survey of physicians. Am J Epidemiol, 134, 539-542.

Singer, E. (2006). Introduction: Nonresponse Bias in Household Surveys. Public Opinion Q, 70(5), 637-645. doi:10.1093/poq/nfl034

Tamayo-Sarver, J. H., & Baker, D. W. (2004). Comparison of responses to a $2 bill versus a chance to win $250 in a mail survey of emergency physcians. Academic Emergency Medicine, 11, 888-892.

VanGeest, J. B., Johnson, T. P., & Welch, V. L. (2007). Methodologies for Improving Response Rates in Surveys of Physicians: A Systematic Review. Evaluations & the Health Professions, 30(4), 303-321. doi:10.1177/0163278707307899

Williams, B. W., Kessler, H. A., & Williams, M. V. (2015). Relationship Among Knowledge Acquisition, Motivation to Change, and Self-Efficacy in CME Participants. Journal of Continuing Education in the Health Professions, 35(S1), S13-S21.

Estimating and Improving Survey Response Rates in CME: What Response Rates Should You Expect?

This three part post will examine survey response rates from a CME perspective, helping you estimate how many responses you need, what the response rate will likely be, and how to improve the rate.   Data is drawn from personal projects, as well as the CME, healthcare, and broader literature.

Today’s post, the second of three, discusses what response rates you might expect from CME work, reviewing recent applicable literature.  The next post will discuss how to improve response rates, and summarize the discussion.

What Response Rates to Expect

Response rates are never as good as they could be, and when surveys are sent out well after an event, rates are even worse.  For this article, we discuss surveys from three categories that cover most CME needs: 

·         Informational Surveys, which are defined as those sent to a broad population who have not had prior contact with the writers, such as participating in a CME event.  Surveying a broad population to ascertain skill gaps during the instructional design phase is an example.

·         Event Questionnaires, which are defined as surveys given immediately upon completion of an educational event.

·         Follow-ups, which are defined as surveys sent after an event within a 30 to 90 day window.

Informationalsurveys seem to have the lowest completion rates.   They are often used for needs and skill gap assessments, to understand the current state of care and knowledge levels.  As previously noted, calculating how many responses are appropriate requires you to estimate the total population of interest (e.g., nurses specializing in rheumatology).

Surveys given out at the time of a CME event, usually immediately afterwards, are the most common type of survey.   Response rates are usually quite high, due to the ‘captive audience’ phenomena.  Where the survey is on the same form as the CME credit request, response rates approach 100%. 

Post-event survey is much more of a challenge.  The data in post-event surveys is particularly meaningful, however; participants are reporting on practice changes they made, compared to those they intend to make, which is a stronger argument.   In a meta-evaluation of ten recent  projects done by Level 6 Analytics, where 30- to 90-day follow-ups were used after a learning event, only 12% of those contacted provided post-event data (nevents= 10; nlearners = 2,707).    That percentage uses as a baseline learners who filled out surveys at the event.  When all learners attending an event are used as the denominator, the percentage drops to only 9%.   Even given that a survey has been filled out, participation on that survey may be lackluster.   Simple items like multiple choice questions are not all answered, and response rates on free-form text inputs, which often provide interesting color and background to results, are almost always below 50%.  With technology changes, changes in privacy laws, and an increasingly saturated audience population, it seems prudent to review only the most recent literature.  Table 1 contains a review of articles in recent issues of Journal of Continuing Education in the Health Professions that used surveys and contained adequate information to calculate response rates.   The first column cites the author and year; the second describes the participant pool; the third gives the response rates, first in percentage and second as a ratio of completed/invited; the fourth gives the type of survey. 

The fifth column notes any extraordinary circumstances which were intended to improve the participation rate, and the last column describes the general topic area of the article reporting the survey.   It should be noted that the few of these articles were simply about a CME program, but most are focused on other topics, such as audience response systems in a CME program (Grzeskowiak, Thomas, To, Reeve, & Phillips, 2015). 

Table 1: Selective Review of Survey Rates in Recent Literatures Studies in the literature

Study	Pool	Response Rates	Survey Type	Special measures used to affect participation	Topic
(Lockyer, Horsley, Zeiter, & Campbell, 2015)	Canadian Physicians	17% (5259 out of 31158)	Informational Questionnaire	Promotion through email; promotional videos; announcements; gifts and drawings.	Physicians perceptions of assessment techniques.
(Harris, Spencer, Winthrop, & Kravitz, 2014)	US Physicians	30% (624/2099)	Informational Questionnaire	None reported	Physician retraining for overseas work
(Pololi, et al., 2015)	US Medical Faculty	52% (2381/4578)	Informational Questionnaire	None reported	Survey of Culture among Medical Faculty.
(Buriak, Potter, & Bleckley, 2015)	US HCP’s	90% (8997/10,000)	Event Questionnaire	None reported	Study of cancer survivor support.
(Evans, Mazmanian, Dow, Lockeman, & Yanchick, 2014)	US HCP’s	87% (120/138) for Event;  55%  (34/62) for post-event survey	Event Questionnaire and Follow-Up	None reported	Interprofessional Education
(McConnell, Azzam, Xenodemetropoulos, & Panju, 2015)	US Physicians	67% (56/83)	Follow-Up	$50 gift card	Study of Test Enhanced Learning in Constipation Management
(Sarayani, et al., 2015)	Iranian Nurses in Single Site	79% (198/250)	Follow-Up	None reported	Pharmacovigilence; study of effects of education type
(Grzeskowiak, Thomas, To, Reeve, & Phillips, 2015)	Australian Pharmacists	78% (62/79)	Follow-Up	None reported	Study tested efficacy of Audience Response Systems in CME
(Williams, Kessler, & Williams, 2015)	US Medical Faculty	(41%) 51/123	Follow-Up	None reported	Study on Self-Efficacy in a HIV/AIDS CME

On Monday:  The thrilling follow-up:  How to improve your response rates.

Estimating and Improving Survey Response Rates in CME: Part 1 of 3

Estimating and Improving Survey Response Rates in CME

Part 1 of 3:   This three part post will examine survey response rates from a CME perspective, helping you estimate how many responses you need, what the response rate will likely be, and how to improve the rate.   Data is drawn from personal projects, as well as the CME, healthcare, and broader literature.

Today’s post introduces the topic and discusses how many responses a given project will need to answer questions.  The next post will cover what we know about response rates currently.  The third and final post will describe how we improve response rates, and summarize the findings.

Surveys, in one form or another, are a cornerstone of data collection in continuing medical education.  Why, with a new focus on collecting ‘hard data’ such as that pulled from patient charts, does this remain the case?  Asking the person involved is the only way to gather information on a variety of  levels in such a straight-forward way.  Data can be obtained on what participants thought of the teaching method and faculty, whether they intend to apply the information upon return to practice, real and perceived changes in knowledge level and competence, as well as information about the environment in which care occurs, and barriers and enablers to transferring that knowledge into practice.   Operational outcomes from original sources, such as chart data, are infeasible in practical terms for many types of events, for example, when participants come from numerous health systems, each with their own data collection standards, privacy rules, and patient populations.  Asking someone who attended a three-day conference to help work through the challenges inherent in collecting patient data from their site is not a realistic request.  A standard rule-of-thumb in corporate L&D settings is that measurement and evaluation should not exceed 5% of the time and effort required for the entire educational effort (Phillips, 1997); it is easy to imagine, from the participant side especially, that such a limit would be exceeded.  Surveys remain an inexpensive, well-understood and accepted data source.  

Using surveys requires learner participation, and getting adequate response rates is not always a given.   Even if response rates are adequate, higher rates have benefits.  This post will consider three important aspects of the issue of survey response rates:

•       What should your number of responses be?
•        What can you expect in terms of a response rate?
•        How can you improve your response rate?

Writing about CME response rates presents difficulties, as there isn’t much research focused on that particular topic.  There is certainly a lot on surveys in general:  census surveys, consumer surveys, student surveys, and even many studies on response rates with physicians and other health care professionals.  The literature on response rates is extensive; however, the amount of truly applicable literature is much, much smaller, due to several factors.  First, when the literature is restricted to populations of health care professions, it becomes much smaller; that subset becomes tinier still when limiting it to CME activities.     The surveys in most research is not sent to subjects who have already participated in a learning event, nor is it specifically about measuring learning.   Another problem in locating applicable literature is the speed of technology change; the size of the internet has more than quadrupled in the last five years (Garber, 2012).  Much of the literature discusses paper-based forms, with extensive discussion of obsolete aspects such as the type of stamp used, whether a booklet or loose pages was the binding mechanism, and how the envelope was addressed, because that was the dominant technology as little as ten years ago.   Worryingly, internet-based surveys have lower response rates than did paper-based ones, although this research is somewhat dated now.  (Leece, et al., 2004)    Unfortunately, the ease of sending multiple surveys via email means that there are many others who are competing for the time and attention of your participants to get data as well; Olson used the metaphor of “Tragedy of the Commons” to refer to the depletion of shared resources by well-meaning individuals (Olson C. A., 2014).

The lack of a literature that is very specific to our issues allows some freedom in writing about the topic.  Rather than simply reviewing the literature, we’ll provide the anecdotes and advice we’ve received over the years about improving response rates, and note where there is relevant research.  Not all the points have research behind them, but we include those that pass the common sense test. We’ll organize our advice into Communication, Motivation,  and Design.  Within each category, the items are listed in order beginning with the one with the strongest support.

How many Responses Do You Need?

Questions about survey response rates begin with how many data points you need to achieve your goals.    Olivieri says, perhaps in moment of flippancy, “ten” (Olivieri, 2012), an answer which resonates. With Level 6, I am very hesitant to report any optimizations in the analysis where a cell has less than ten participants, regardless of what the p value may be.   The total number of participants depends on how finely we slice the data, of course, and how those slices are partitioned.  If we want to say that we are certain that participants reported an impact on competence that would mean ten participants. If, supposing our CME event had attendees from different professions, and we want to say that nurses report less impact than physicians, that would be ten of each; and, so on, up until the point where we want to talk about the impact for nurses with less than five years of experience working in public health facilities, which might require hundreds of participants in total, depending upon the distribution of the data.

“Ten” is of course a rule of thumb, and isn’t theoretically pleasing.   The correct answer requires some math, and unfortunately, several numbers that aren’t accurately available until the data comes in, which isn’t the best news.  Further complicating the matter is the fact that, unlike in textbook examples which use political race outcomes and soda preferences, there’s seldom one single question that determines the usefulness of a CME project.   When the choice is between an estimate that tells you more than you knew before, and doing nothing because the answer isn’t theoretically pure, the decision should be clear:  make the best estimate you can, and proceed.   Estimates can be made using similar projects and some guess work.   This article will avoid presenting one or more equations that produce an answer, as these are best left to someone with a statistics background.  Instead, we’ll suffice with presenting the major factors, to help you gather the facts you need before dusting off your statistics texts or speaking to a professional. 

The key factors are these:

·         Estimated mean.

·         Estimated difference.  In other words, what do you think the difference between the trained and untrained groups, or the “before and after” conditions will be?  Are you content to show that the impact differs significantly from zero, or do you have a more ambitious goal in mind?

·         Standard deviation on the main variable.

·         Statistical certainty required.    The standard 95% certainty (p< .05) isadequate for almost all reporting purposes, although we all love it when we get 99% or higher. 

The variables are unknown until the data comes in, but you can probably make a guess by looking at other, similar, studies.  This is not a complete list, and how to calculate the sample size will depend on exactly the parameters of your study.  For example, if you are doing needs assessment and you want to make a statement about the needs of pediatric oncologists in the US, for example, you will need to know size of that population. 

When you know how many responses you need, dividing that number by the response rate will produce the number of survey invitations needed.  The response rate is the next big question to tackle.  We'll talk about that in the next post.

Moore's Levels: When do they Work, When Do They Fail?

Moore, Green & Gallis  provided a paradigm for evaluating medical education that has become the standard cited reference when discussing outcomes.  There are 7 levels:

Level 1:  Participation

Level 2: Satisfaction

Level 3: Learning

A.      Declarative

B.      Procedural

Level 4:  Competence

Level 5: Performance

Level 6: Patient Health

Level 7: Community Health

Do these levels cover everything?   Paradigms are useful to the extent they allow our processes to be thorough, to the extent they stimulate, rather than restrict our thinking, and when they provide us with quality results.   They are harmful when they restrict our creativity, when they rule out different options that may prove useful to us.   Moore’s levels are a really good paradigm, but sometimes it’s best to step outside of the paradigm and think what other options we might be missing out on.

In corporate learning, the “Kirkpatrick levels” occupy an equivalent position to Moore’s level in medicine (Kirkpatrick, 1977).  There were four in the original formulation, and those have been augmented over time.  The full version might look like this:

Level 0: Usage

Level 1:  Satisfaction

Level 2:  Learning

Level 3:  Transfer

Level 4:  Business Impact

Level 5:  Return on Investment

Level 6:  Intangibles

There are more similarities than differences, of course.   There are interesting points of departure, of course, for example, the separation of learning into “procedural” and “declarative”.     “Transfer” at Level 3, in Kirkpatrick, presents interesting distinctions from “Competence”, Level 4 in the Moore paradigm.   “Transfer” simply notes whether desired behaviors have made the leap from a learning environment into a healthcare practice, separating out the issue of whether it works or not.  For example, a program might encourage hand-washing behaviors, but whether they will actually show a reduction in post-operative infections is a slightly different matter.   Most people in corporate learning and development use the “transfer” measurement to consider what barriers and enablers exist in the workplace.  That consideration enhances the responsibility of an instructor:  they are not just providing learning, but taking a broader mission of performance improvement.  This is an example where a simple reliance on one paradigm may restrict our options.  

‘Business Impact’ and performance are much the same thing in the two paradigms.   They refer to a metric that is being affected in a measurable way:  something that can be counted.   That might be a reduction in blood sugar levels in a patient, reduced diagnosis time, or a reduction in hospital stay length.  Level 6, Patient Health, and Level 5 in the Kirkpatrick paradigm, are more similar than they first appear.   The underlying question in both paradigms is “Was this program worthwhile?”   Return on investment is a simple calculation that tallies up dollarized benefits and dividing them by the cost of the intervention program.   When a program benefit can be given an accurate value in the medical environment, this is a good idea for several reasons.  First, it allows disparate programs to be valued head-to-head using a common yardstick.   Secondly, it allows communication of benefits to stakeholders with primarily financial responsibilities to occur, such as administrators or insurers.  The final levels, in both cases, serve as areas for expansion, when there are areas that aren’t easily captured by the lower levels of measurement.

Both paradigms work well, but there are others that are useful as well.   Dean Spitzer split evaluation up based on when  the evaluation was done:  predictive, formative, baseline, in-process, and retrospective.  Almost all of what the medical community calls “outcomes” is based on the last:  what do I think a program changed?   It could also be argued that, as far and away the most common evaluation strategy being a survey given at the time of a learning event, that “in process” might be closer to the truth.  Measurement and evaluation often has some ambiguity associated with it.  Spitzer’s paradigm is enlightening in that it brings together different parts of the timeline:   gap analysis and outcomes become part of a consistent effort.  It is these sort of insights that you want from paradigms.  All too often, gap analysis is simply an expert’s opinion of what a gap is, rather than any more carefully measured strategic measurement.

There are numerous other concepts that the Moore Paradigm doesn’t suggest at all.  In my practice, the most useful concepts that I refer to as “optimization” cover some of the important aspects that need to be measured.   Here are the concepts I think need to be part of any important measurement effort that aren’t explicitly covered in Moore’s levels.  Measurement needs to be about improving the delivery of interventions, and ultimately about improving health care.  I use the word “optimization” to refer to those improvements.  I’ll quickly list some of the main areas

Segmentation is the breaking down of the impact on a group-by-group basis to understand where the greatest and least impacts are achieved.  Do nurses report more knowledge gain than do pharmacists?  Do physician assistants have more barriers in transferring their learning to the workplace?   Segmentation offers two great opportunities:  selection of populations who will benefit the most, and improvement of the interventions to address the needs of those who are underserved.  Some caution needs to be exercised into segmentation to avoid making conclusions about very small groups. 

Metric Interaction is another important concept.  The idea is that the relationship between the different metrics needs to be considered as well.  For example, a training program that emphasized how to eliminate barriers to diabetes treatment might produce data where those who had more barriers also perceived more impact.  Regression analysis is one of the more helpful tools in this case.

There are many other concepts that a good measurement professional can use to investigate and bring to life a training or other program in healthcare:  timeline studies, synergistic or diminishing returns effects, saturation, predictive analytics for future programs, and mixture of programs, to name a few.   An experienced analyst can add so much to the process, and advanced statistics and visualization software can add a lot to the mixture.

The Visualization IS the Analysis

Graphics used to be the last step in an analysis, a way of presenting findings to the intended stakeholders.  Now, they are much, much more.