The Analytics Team at Mal Warwick | Donordigital is pleased to re-release the Response Significance Calculator, a quick and handy way to test the statistical significance in response rate between two test panels when working on a direct response marketing project. When using this Calculator make sure that the two test panels were randomly split from the same population of donors, and that there were no differences between the two panels except for the test. For example, if you were testing an email subject line, both emails should have launched at the same time and had the same content. To use the calculator, enter the number of contacts and number of gifts for each of the two test panels, and then choose the calculate button. Click here to use the Response Significance Calculator. Peter Schoewe is a Vice President and Director of Analytics at Mal Warwick | Donordigital, a full-service, integrated fundraising and advocacy agency serving leading charitable organizations.
One of the great joys of direct response fundraising, at least to strange people like me (and maybe you), is the ability to test your assumptions and to strengthen the performance of a program one step at a time. But testing can be one of the most frustrating aspects of fundraising, too—especially when you go to the trouble to set up a test, track the results, and end up with no actionable conclusions when all the returns are in. I’ve experienced both the joy and agony of direct response testing, and I would like to share examples of both with you. Of course, it’s important to remember that no test results can be universalized—results vary by organization, donor segments, and time of year. A classic example of this is the test between Courier and Times New Roman fonts on a letter. I’ve seen this test performed on many occasions and often with highly significant results. The only problem is that, on those occasions when the test yields statistically significant results, half the time Courier wins, and half the time it’s Times New Roman—although there’s usually no statistically significant difference at all. In the end, the only value of this particular test may be showing that donors like something different. Or, even more likely, that it doesn’t matter at all. One of the tests I’ve been most excited about recently was the test with a premium offer expanded from a one-panel buckslip to a full 8-1/2 x 11” sheet. The control package contained a standard buckslip offering prospective donors a free plush toy with their initial membership gift. Expanding the buckslip to a full sheet allowed the premium to be represented in life size, and the response rate to the package increased by 27%. However, at the same time, the more prominent premium offer lowered the average gift by $1. Both of these results were statistically significant, meaning it’s highly unlikely the observed differences in response between the two packages were due to chance. So, in many ways this test did exactly what we wanted it to do. We modified the package in a way that caused a marked difference in donor behavior. We were able to inspire more donors to send a gift, but by doing that, we caused the value of the gift they offered to decline. Because this is an acquisition package, we cannot yet declare a winner. We need to monitor the long-term results to determine whether more donors giving a lower gift amount are more valuable than fewer donors giving a greater amount. But, because of this test, we now have a clearer idea of what influences response to the package, and we’ll be able to track the impact of that difference long term. A test I loved—but one that failed—used faux stamps on an outer envelope. This is a package treatment I’ve seen used on many occasions, so it must have tested well for somebody. But my test was a complete flop. I tested placing two bright and colorful stickers on the outside of a nonprofit envelope as close to the indicia as the post office would allow. The idea was that the stamps would catch the donor’s eye and help overcome the junk mail stigma that can be attached to letters mailed at the nonprofit rate. Other than the faux stamps, I didn’t modify the control envelope at all. It was a plain, cream outer with no teasers or images, other than a name, logo, and address in the upper left-hand corner. When I reviewed the returns, however, my affection for the fake stamps began to cool. There was less than 2/10ths of a percent difference in response rate between the version with the stamps and without—and no statistical difference between the average gifts. In all likelihood, the faux stamps didn’t change a single individual’s mind one way or the other about sending a gift. And, of course, the bright and colorful stamps added an extra measure of expense to the package that wasn’t counteracted by any increase in giving. It’s okay to have a loser test every now and then, because it’s important to test your assumptions of what will work and what won’t. What’s most critical is that you create tests with a well thought-out rationale behind them and that the hypothesized result of the test is aligned with your overall goals. For example, it doesn’t make sense to test a reduced Ask amount to increase response, if your goal is to acquire higher value donors who will upgrade quickly. In other words, each test you perform—and you should be testing as much as you can—should have the goal of moving your program to the next level. Just remember, it won’t be quick and easy, and you may receive as many muddy results as you do clear winners.
Peter Schoewe is a Vice President at Mal Warwick | Donordigital.
When you segment your membership or donor file, you select some individuals to include while leaving others out. Those you include may be further divided into subgroups that receive packages distinguished by variations in one or more of the Three P’s: Packaging, Postage, and Personalization. By intelligent segmentation of your donor file, you can cut mailing costs, upgrade donation levels, and minimize complaints from your supporters. Here are 5 simple guidelines to follow:
- Don’t reinvent the wheel. Three criteria stand out above all others as predictors of direct mail donor behavior: recency, frequency, and gift amount. Don’t assume you can beat the system by ignoring these three criteria and arbitrarily substituting others. [As with every rule, of course, there are exceptions. For example, two factors that might weigh as heavily as recency, frequency, or gift amount in some circumstances are the original source of the name (non-direct-mail sources generally produce much less responsive donors) or whether a premium has been used as an incentive to obtain the first gift.]
- Remember: some people give more money than other people. Mail most often to those donors who give most frequently, or who have contributed most recently, or whose individual gifts have been the biggest. These are your prime prospects for gifts right now, and they’ve demonstrated their interest in your organization. They’re worth extra attention. By contrast, those who don’t fit these criteria are worth less attention – and less investment in the Three P’s.
- Remember, too: if some people are more equal than others, a smaller number are the most equal of all. If your donor file consists of 10,000 names or more, the top 10% to 30% – as defined by Recency, Frequency and Highest Previous Contribution – are worth extra-special attention. Many nonprofit organizations benefit greatly from extra mailings targeted exclusively on this core group. For example, if you’re accustomed to using bulk postage in your resolicitations, a core group mailing could be sent via first class mail; it’ll probably work even better if you use “live” stamps. And if you normally use mailing labels on your donor appeals, these same people are likely to be worth an investment in personalization, which is likely to increase your mailing’s impact by getting more donors to pay attention.
- Stop beating dead horses. If members or donors have failed to respond during the past 36 months, stop asking them for money. They’re a lost cause. [But here are two things you can do to break this rule: if you’re actively prospecting by mail for new members, include these former members in your acquisition campaign. You could also try calling them by phone in an effort to persuade them to reactivate their support for your organization. Most groups find that both these types of activity work well.]
- Don’t try to squeeze blood from turnips. If the average contribution in your direct mail fundraising program is more than $15, you may be wasting money with repeated mailings to donors of less than $10 gifts. Many big mailers profit from appeals mailed to donors of gifts from $5 to $9.99. But even they rarely mail to donors of less than $5 gifts.