Estimating and Improving Survey Response Rates in CME
Part 1 of 3: This three part post will examine survey response rates from a CME perspective, helping you estimate how many responses you need, what the response rate will likely be, and how to improve the rate. Data is drawn from personal projects, as well as the CME, healthcare, and broader literature.
Today’s post introduces the topic and discusses how many responses a given project will need to answer questions. The next post will cover what we know about response rates currently. The third and final post will describe how we improve response rates, and summarize the findings.
Surveys, in one form or another, are a cornerstone of data collection in continuing medical education. Why, with a new focus on collecting ‘hard data’ such as that pulled from patient charts, does this remain the case? Asking the person involved is the only way to gather information on a variety of levels in such a straight-forward way. Data can be obtained on what participants thought of the teaching method and faculty, whether they intend to apply the information upon return to practice, real and perceived changes in knowledge level and competence, as well as information about the environment in which care occurs, and barriers and enablers to transferring that knowledge into practice. Operational outcomes from original sources, such as chart data, are infeasible in practical terms for many types of events, for example, when participants come from numerous health systems, each with their own data collection standards, privacy rules, and patient populations. Asking someone who attended a three-day conference to help work through the challenges inherent in collecting patient data from their site is not a realistic request. A standard rule-of-thumb in corporate L&D settings is that measurement and evaluation should not exceed 5% of the time and effort required for the entire educational effort (Phillips, 1997); it is easy to imagine, from the participant side especially, that such a limit would be exceeded. Surveys remain an inexpensive, well-understood and accepted data source.
Using surveys requires learner participation, and getting adequate response rates is not always a given. Even if response rates are adequate, higher rates have benefits. This post will consider three important aspects of the issue of survey response rates:
- What should your number of responses be?
- What can you expect in terms of a response rate?
- How can you improve your response rate?
Writing about CME response rates presents difficulties, as there isn’t much research focused on that particular topic. There is certainly a lot on surveys in general: census surveys, consumer surveys, student surveys, and even many studies on response rates with physicians and other health care professionals. The literature on response rates is extensive; however, the amount of truly applicable literature is much, much smaller, due to several factors. First, when the literature is restricted to populations of health care professions, it becomes much smaller; that subset becomes tinier still when limiting it to CME activities. The surveys in most research is not sent to subjects who have already participated in a learning event, nor is it specifically about measuring learning. Another problem in locating applicable literature is the speed of technology change; the size of the internet has more than quadrupled in the last five years (Garber, 2012). Much of the literature discusses paper-based forms, with extensive discussion of obsolete aspects such as the type of stamp used, whether a booklet or loose pages was the binding mechanism, and how the envelope was addressed, because that was the dominant technology as little as ten years ago. Worryingly, internet-based surveys have lower response rates than did paper-based ones, although this research is somewhat dated now. (Leece, et al., 2004) Unfortunately, the ease of sending multiple surveys via email means that there are many others who are competing for the time and attention of your participants to get data as well; Olson used the metaphor of “Tragedy of the Commons” to refer to the depletion of shared resources by well-meaning individuals (Olson C. A., 2014).
The lack of a literature that is very specific to our issues allows some freedom in writing about the topic. Rather than simply reviewing the literature, we’ll provide the anecdotes and advice we’ve received over the years about improving response rates, and note where there is relevant research. Not all the points have research behind them, but we include those that pass the common sense test. We’ll organize our advice into Communication, Motivation, and Design. Within each category, the items are listed in order beginning with the one with the strongest support.
How many Responses Do You Need?
Questions about survey response rates begin with how many data points you need to achieve your goals. Olivieri says, perhaps in moment of flippancy, “ten” (Olivieri, 2012), an answer which resonates. With Level 6, I am very hesitant to report any optimizations in the analysis where a cell has less than ten participants, regardless of what the p value may be. The total number of participants depends on how finely we slice the data, of course, and how those slices are partitioned. If we want to say that we are certain that participants reported an impact on competence that would mean ten participants. If, supposing our CME event had attendees from different professions, and we want to say that nurses report less impact than physicians, that would be ten of each; and, so on, up until the point where we want to talk about the impact for nurses with less than five years of experience working in public health facilities, which might require hundreds of participants in total, depending upon the distribution of the data.
“Ten” is of course a rule of thumb, and isn’t theoretically pleasing. The correct answer requires some math, and unfortunately, several numbers that aren’t accurately available until the data comes in, which isn’t the best news. Further complicating the matter is the fact that, unlike in textbook examples which use political race outcomes and soda preferences, there’s seldom one single question that determines the usefulness of a CME project. When the choice is between an estimate that tells you more than you knew before, and doing nothing because the answer isn’t theoretically pure, the decision should be clear: make the best estimate you can, and proceed. Estimates can be made using similar projects and some guess work. This article will avoid presenting one or more equations that produce an answer, as these are best left to someone with a statistics background. Instead, we’ll suffice with presenting the major factors, to help you gather the facts you need before dusting off your statistics texts or speaking to a professional.
The key factors are these:
· Estimated mean.
· Estimated difference. In other words, what do you think the difference between the trained and untrained groups, or the “before and after” conditions will be? Are you content to show that the impact differs significantly from zero, or do you have a more ambitious goal in mind?
· Standard deviation on the main variable.
· Statistical certainty required. The standard 95% certainty (p< .05) isadequate for almost all reporting purposes, although we all love it when we get 99% or higher.
The variables are unknown until the data comes in, but you can probably make a guess by looking at other, similar, studies. This is not a complete list, and how to calculate the sample size will depend on exactly the parameters of your study. For example, if you are doing needs assessment and you want to make a statement about the needs of pediatric oncologists in the US, for example, you will need to know size of that population.
When you know how many responses you need, dividing that number by the response rate will produce the number of survey invitations needed. The response rate is the next big question to tackle. We'll talk about that in the next post.