In my recent column, “E-Commerce Moneyball: Chasing the Market Leader,” I briefly discussed how Amazon built tools for its own use that leverage the big data available to it. Let’s explore the types of tools that are available for you to leverage big data today. There are many reasons one can point to as to why Amazon.com has become such a dominant e-commerce player. Here are a few to highlight:

  • It leveraged social commerce and the voice of the customer for reviews long before anyone else realized their value.
  • It optimizes everything religiously.
  • At any given time it can have upwards of 200 A/B or MVT tests campaigns running.
  • It built a culture that is agile and responsive with small teams of smart people who are given authority to take action.
  • It uses its data! It uses it to improve what products to highlight, how to deliver products at a better price than most of its competitors, how to ship more effectively, and how to merchandize more persuasively.

The old direct marketing mantra of “the money is in the list” should be modified today to say “the money is in the data.”

My brother and I have been advocating a data-centered approach to improving marketing and conversion rates since the mid-1990s. Yet relatively few companies have succeeded at successfully running their companies this way for a number of reasons. We’ve joined many digital analytics evangelists including the likes of Avinash Kaushik, Eric Peterson, and Jim Sterne who have said for years that success is all about the appropriate combination of investment in people, process, and tools. Each and every one of these require the appropriate culture to nurture success.

In 2004, with Jim Sterne and Andrew Edwards and a handful of other volunteers we joined together to deal with what was then, and still is now, one of the most critical issues. We saw that a lack of data analysts in the market as we founded the Web Analytics Association, recently renamed the Digital Analytics Association, would hamper growth. The Digital Analytics Association’s main mission is to advocate and educate for digital analytics. It has certainly accomplished a great deal since then but nowhere close to what consulting firm McKinsey projects: a need for 1.5 million additional managers and analysts in the United States who can ask the right questions and consume the results of the analysis of big data effectively.

Experience will tell you that turning data into valuable insights and information to act on is the toughest part of data analytics. Gathering data and putting it into charts and distributing reports is not what you should be shelling out any significant amount of money for, even though that’s still way too common. I think it is fair to say the people part of the equation is a big issue for the near future.

As I pointed out in my column “Data Rich, Optimization Poor,” only 22 percent of companies have a strategy that ties data collection and analysis to business objectives; down from 25 percent last year. (Source: Econsultancy Online Measurement & Strategy report.) When you can find and develop talent that is capable and intelligent enough to optimize business processes based on data, do everything in your power to keep them.

The good news is that there are now tools that are helping to level the field significantly. We can break out the tools into four types:

  • Analytics tools. They are meant for people to analyze large and small data sets and to manually and increasingly automatically discover any patterns or data anomalies.
  • Predictive analytics tools. They have been created to surface those patterns that matter and notify you of any future trends based on past and current trends.
  • Data-driven automation tools. These tools tend to drive widgets on sites, and trigger events based on actions to personalize marketing and customer experiences based on data-driven events.
  • Adaptive learning and optimization tools. These tools leverage big data, machine learning, and advanced statistical systems such as game theory to develop predictive analytics that then drive some sort of marketing automation and continuously optimize, refining their algorithms without the need for nearly any human intervention.

It is this last category that will allow the “bots” to eventually claim their unfair advantage and deal with the lack of trained people in the marketplace. This is the category that 10 years ago my brother and I would joke around about how people just wanted a magical “black box” connected to their server that would pump out revenue.

As A.C. Clarke said, “Any sufficiently advanced technology is indistinguishable from magic.” The magic is arriving now. How soon will you claim your unfair advantage?