Anyone who has read my “69 Free or Low Cost Tools to Improve Your Website” post knows that I love tools. This past week I was excited to see a new list of tools with the just published “Which Multivariate?” a multivariate testing tool comparison guide. I’m happy they beat me to the punch in putting this list of tools together, because many people have asked me about the various testing platforms and tools, and I haven’t had a chance to get to it.
The Critical Parts of the Testing Equation
When it comes to a choice of testing tools, you can see there is a variety of testing software available other than Google’s Website Optimizer and Omniture’s Test and Target. When you finally decide to start testing, the tool should not be the first part of the equation. It’s like being an emergency room doctor and having lots of crutches around and trying to use a crutch to solve every one of the issues you face. It just won’t solve the problem.
- You first need to understand the business goals and problems you are trying to achieve.
- Then you must choose the right tools for that job. To select the right one, you have to weigh your needs and your company’s level of sophistication when it comes to running experiments against price, and keep in mind that the tool you start out with today may not be a good fit to solve all the challenges. So flexibility is key.
- The last critical part of the equation is: do you have the knowledge internally to decide what to test (things that really will move the needle) and do you have the resources to get the tests executed?
As you can see steps one and three are the most important to your success, the tool is just a tool.
What to Do Is More Important Than How You Do It
A MarketMotive conversion optimization certification student of mine, Philip Anderson, and I were chatting about this when he reminded me of this classic story. Ever heard the story of the giant ship engine that failed? The ship’s owners tried one expert after another, but none of them could figure out how to fix the engine. Then they brought in an old man who had been fixing ships since he was a youngster. He carried a large bag of tools with him, and when he arrived, he immediately went to work. He inspected the engine very carefully, top to bottom.
Two of the ship’s owners were there, watching this man, hoping he would know what to do. When he went down to the engine room he felt the pipes with his hands, took out a stethoscope, listened to the pumps, and finally placed one of his hands on one of the gauges with his eyes closed. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away. The engine was fixed! A week later, the owners received a bill from the old man for $10,000.
“What?!” the owners exclaimed. “He hardly did anything!”
So they wrote the old man a note saying, “Please send us an itemized bill.”
The man sent a bill that read:
Tapping with a hammer: $2.00
Knowing where to tap: $9,998.00
Effort is important, but knowing where to make an effort makes all the difference.
Can You Get It Done?
Once again, to mitigate your risk and achieve the best conversion rate lift, you want to work with someone who has a demonstrated track record of using a variety of tools: reputable, on-demand testing tools that don’t rely heavily on software license revenues and will allow you flexibility to change the software as your needs evolve and your testing needs become more complex. As important as knowing what to do is, it does no good if you can’t do anything about it. It was obvious to me and anyone who saw me that I needed to lose weight, but if I didn’t put the resources and effort in place to do what needed to be done I wouldn’t have lost 70+ pounds since April of last year (you can see a recent picture of me here).
You must have resources in house or else hire an independent firm or contractors to either plan, set up, or execute the variations and creative for your tests. They should be able to fill the gaps you have internally in order for you to continue improving your business goals continuously. Testing is not a one-time event you check off the list. It needs to be part of your everyday culture to succeed.
I heard Guy Kawasaki say, “Execution is not an event – a one-time push toward achieving goals. Rather, it is a way of life.” The reason testing is so critical today is because the other business constant – change – is occurring at a pace much faster than ever before. If you can’t develop the corporate metabolism to test, change, and execute rapidly you are going to be left behind like the horse and buggy makers were by Ford. In the academic world they say “publish or perish,” but in today’s marketing world it is “test or die!”
P.S. Don’t miss my webinar on April 1st sponsored by testing services firm WiderFunnel: Don’t Be April’s Fool: Proven Techniques To Maximize Your Advertising ROI
{ 3 comments… read them below or add one }
There was this biologist who was doing some experiments with frogs. He was measuring just how far frogs could jump. So he puts a frog on a line and says “Jump frog, jump!”. The frog jumps 2 feet. He writes in his lab book: ‘Frog with 4 legs – jumps 2 feet’.
Next he chops off one of the legs and repeats the experiment. “Jump frog jump!” he says. The frog manages to jump 1.5 feet. So he writes in his lab book: ‘Frog with 3 legs – jumps 1.5 feet’.
He chops off another and the frog only jumps 1 foot. He writes in his book: ‘Frog with 2 legs jumps 1 foot’.
He continues and removes yet another leg. ” Jump frog jump!” and the frog somehow jumps a half of a foot. So he writes in his lab book again: ‘Frog with one leg – jumps 0.5 feet’.
Finally he chops off the last leg. He puts the frog on the line and tells it to jump. “Jump frog, jump!”. The frog doesn’t move. “Jump frog, jump!!!”. Again the frog stays on the line. “Come on frog, jump!”. But to no avail.
The biologist finally writes in his book: ‘Frog with no legs – goes deaf’
The point is test all you want but be careful you aren’t making erroneous conclusions from the data!!
All to often I find web analytic’s folks working off of too small sample sets and trying to fit their preconceived notions of what the answer should be versus what it actually is.
This post really resonated with me, and not for the obvious reasons. Sure, you need to test, test and test. And sure, you need to figure out what to test first with your business goals in mind. However, to do those two things very effectively takes 1) experience and 2) unwavering dedication but most importantly it takes 3) a passion for excellence.
Let me go back in time. I was a competitive gymnast growing up. And, I spent hours watching videos of my routines, or listening to coaches, telling me to point my toes, or straighten my legs, or do whatever it takes to stick the landing. But, once all the mechanics were fixed and the execution was greatly improved, I didn’t score a 10. (In fact, I never scored a 10). But I did raise my scores significantly. The more experienced I got, the more I realized that all the collective pieces needed to work together, harmoniously. It wasn’t enough to do everything well in isolation. It was more important to make sure everything flowed and there was constant improvement. Did the video camera provide that? No, did the coaching provide that? No. It was only by a steadfast determination and passion to excel did the scores increase. So, to your point, it’s not the tool. It’s a steadfast commitment and passion for excellence that can move the needle in the right direction.
Hi Bryan,
alot to agree with here, and a bit to debate.
First, I fully concur that a key challenge for marketers and analysts should be to identify and agree upon objectives for an optimization effort. This is the first nut that should be cracked. Without these key objectives (goals) in place, it’s just random 1-off’s with no real direction, and no momentum will be achieved.
I also agree completely that knowing what to test (e.g. which areas of the site, which creative elements, etc.) will save you alot of time and ultimately lead to superior results vs. pure trial-and-error. Drawing on historical web analytics data, voice-of-customer, usability feedback / reviews, and a dose of best practices will help with all of this. And of course you don’t get anything for free, so actual time & effort must be spent to move the needle in a meaningful way.
Where I’d offer some constructive feedback is in the ‘get over the tools’ message. (Full disclosure: as you know, but your readers may not, I’m the CEO of SiteSpect, a provider of multivariate testing & behavioral targeting tools and services.) I think it’s fair to say that each of the numerous tools out there each have strengths and weaknesses. Some tools are more suited to certain types of tests, certain types of sites, certain types of workflow, integration/connectivity with other tools, and so on. It certainly depends on what your optimization goals are (your first point), but selection of the “right” tool (your second point) then enables thorough, well-informed testing to take place (your third point.)
Ultimately, I think I’d paraphrase “get over the tools” as “work smartly to select the right tools so you can move ahead with the important and ongoing work of optimization.” Cheating the tool selection process is likely to lead to aggravation and/or additional time/effort spent (wasted) re-selecting and re-implementing the right tool. Of course, humans often learn best through the process of making mistakes, plus it isn’t always obvious which tool is best. So research is key – sites like whichmvt.com are certainly helpful – and buyers should always speak with other users, check references, and so on.
Thanks for the informative post!
cheers,
Eric