Donor Insight. Fundraising Action.

The Creative Testing Fallacy

Posted by Ben Miller on 12/2017
Find me on:

Image result for Imbalance Fallacy

Roger wrote a piece (The Same Old Santa) on the Agitator about an appeal that has not changed year over year for the last 22 years and was applauding this.  The reason for the praise is that this creative was a masterpiece and did not need to be tinkered with.  He was suggesting that many do not abide by the “If it ain’t broke, don’t fix it” adage, because of their desire to continually make changes.   

I wanted to expand on this notion, because I do not think Roger was suggesting that we should leave well enough alone, which could be misconstrued here.  Instead, I believe he was tapping into a problem in the sector where consultants usually feel this need to make changes to something that is working.  I believe it stems from the idea that we all want to feel like we matter.  If you just put out the same appeal without making any changes year after year, well what are they paying you for right?

I think we can all relate to this sentiment, and I think the point Roger was making was not about “not improving”, but rather prioritizing what you improve.  The appeal he referenced was their top performer.  So instead of changing that, they would be better served to change something that was not working as well.   

We see this often when a nonprofit or their agency believes it is too risky to test a predictive model, but does not even consider the risk when contemplating changing the creative.   I am calling this the Creative Testing Fallacy.  

Let’s take a fictious example, that I think most of us can relate to.  Organization A has employed Agency B to provide the best fundraising counsel they can.  Agency B will make suggestions on what appeals should be on the upcoming year end schedule.  If during this fall campaign pitch of new ideas everyone in the room feels like the Year Ahead is a good alternative to the Same Old Santa appeal, then testing this across a Nth of the file would quickly be agreed upon. 

Let me try to break this down and compare this change in the appeal to using a predictive model to select your donors.  To write the new appeal Agency B charges Organization A $2,500 for the copy and $1,000 for the art.  They test this on 20% of the file.  Here are the fictitious results to this test.



Response Rate

Average Gift


Year Ahead





Same Old Santa










 That means this test cost Organization A -$7,549 or -$4,049 from the quantity that was sent the Year Ahead appeal, and the additional -$3,500 they spent on the creative.  In contrast an organization this size the cost of using the DonorTrends Optimization tool would be $899, and on average we have found that we can identify 30% of the audience that is mailing at a loss.  The best part is that Organization A would have complete transparency into what risk the model may pose, because we calculate this upfront. 

Unfortunately, because we as a sector do not look at these two test in the same way, this comparison is rarely made.   Too often as Roger points out the creative test rarely brings in data to support the decision.   I have seen these same "data driven" fundraisers that do not blink an eye at a creative test, ring the alarm when any sort of audience test is suggested.  If only the model skeptics in the sector could get more comfortable with this science, they would find it actually is a complement to the art instead of a threat.  Have you ever witnessed this type of Creative Testing Fallacy, what was the context,  and do you have any results you could share?  

Want to see what our models can do for your organization?    

Free Trial

Topics: Revenue, testing

Join the Movement! Subscribe to receive must-have fundraising insight.

Fundraising Facts, Stats, Tips, and Tactics for brillant marketers [like you!]

Fundraising Superhero,

We know how many capes you have to wear. Fighting the good fight, trying to raise money to fund critical programs. We're here to help. We're using this space to collect tips and trends that will help strengthen your fundraising superpowers.  Together, we'll fight Fundraising Foes like donor attrition, downgrading, and rising costs. 

Recent Posts