Testing Like an Olympian: An Interview With Janis Lanka, Elastic Path

Janis Lanka (@janislanka, manages front-end development for Elastic Path Software and was responsible for testing on the Official Vancouver 2010 Olympic Store.

1.    What were the precipitating events that caused your company to start testing?

1)      We knew there were areas of the site that could use optimization: our homepage could be less busy; our product details page could better display key information; our checkout process could be shorter (too many people were abandoning the process mid-way). However, since these were “opinions,” and nobody had answers based on facts, we decided to quickly validate the various theories, just in time for the Winter 2010 Games buying season (this deadline was a serious one!).

2.    When deciding to test you had the option of doing it in-house or finding an agency (you do both). What led you to your decision to hire an agency and what was the criteria you used?

2)      We didn’t have sufficient experience with testing to do it in-house. Thus, we looked up a local agency that would help to guide us through this process and help us understand it better. There was no official selection method and we selected Wider Funnel because of personal relationship with Chris Goward. After couple of meetings with them we gained confidence in their knowledge and agreed that they can be great fit for what we needed.

3.    What were some of the greatest surprises learned from your testing?

3)      Issue with reaching statistical significance. We didn’t have much time to test (b/c of major deadlines), and thus had to make decisions on various additional data points like variation conversion rates, bounce rates, etc. Starting this, our assumption was that each test would quickly show which route we have to take, but in reality it required much more analysis.

4.    What kind of impact has your successes in testing had on your organization?

4)      Since we began engaging in testing and optimization, we’ve noticed that opinions expressed in meetings were now being complemented with facts, new feature implementation processes have better structure, and team members had a renewed trust in each other because decisions were now data driven.

5.    How can others in your position help convince management they should be testing? If you were speaking directly to managements what would you say?

5)      Data focused decision making is key to be communicated to management. Get permission to try a low-budget pilot project that would show actual benefits in terms of generated revenue and management will support you in any further tests. Very clear monetary benefit shown through testing low-hanging fruit helped us to pilot this and get all necessary buy-in.

P.S. Don’t miss my free webinar with WiderFunnel on June 1st, “Activate the 10 Steps to a Higher Conversion Rate.” Please register now.

Please share if you think others would benefit.