You Don't Need More Data, You Need Better Questions
Do you have lots of data from your business website or app? Are you using the data to make better business decisions?
With tools like Google Analytics or Adobe Analytics, it’s easy to get caught up in descriptive statistics. How many visitors came to our site? What pages did they visit? How many converted? And so on.
At some point, you’ll need to ask yourself, “What decisions are these figures informing?” More than likely, they are informative measures but do not actually inform a decision or set of decisions. Too often, we find ourselves building reports for the sake of building reports.
This post invites you to rethink measurement — what it means, what you measure, and how you measure — for user behavior on your website or app. By asking better questions, you'll go beyond data collection and start leveraging data science.
You Have More Data Than You Think
When I ask clients why they don’t measure the things that they need to inform decisions, I will get responses like, “We don’t have the data for that” or “That’s an intangible, you can’t measure that.”
My response is, “Science was never about having data, it’s about getting data.” My old colleague and mentor1 often said, “You have more data than you think, you need less data than you think, and assume it has been measured before.”
There are three reasons why anyone believes something can't be measured. All three are illusions:
- Concept illusion: Measurements must be exact
- Object illusion: Some things are too difficult to measure
- Method illusion: There's not enough data
We can debunk each of these illusions by asking better questions.
Concept Illusion: Measurements Must Be Exact
The concept of measurement is sometimes the most confusing of the three illusions of measurement. We are trained to think of units of measure as exact point estimates. Think of the last time you asked your front-end designer how long a task would take; they might say three weeks.
But that is not how science and scientists have understood the concept of measurement. Historically, science has defined measurement as a reduction in uncertainty based on observations. Moreover, estimates would never be given as exact points; instead, they would always be given in ranges.
So instead of three weeks, a better and more useful answer might be, “I am 90 percent certain the task will take between 80 and 150 hours.” Ask the questions that will get you this answer!
Object Illusion: Some Things Are Too Difficult to Measure
The object of measurement is a little trickier. A good deal of my background is in brand and brand management. I can’t tell you how many times I’ve heard the following: “You can’t really measure brand because it’s an intangible!”
My response is, “Then why do you care about it?” Imagine two firms, A and B. They are alike in every way possible, except firm B has better brand recognition. When you ask, “What would you expect to measure differently?” you get answers like, “Firm B would have more visitors. Firm A would have fewer sales and bottom-line revenue, etc.”
The point is, once you clarify the object of measurement, then making observation-based measurements is quite easy. It’s important to ask questions that get to the things you can measure.
Method Illusion: There's Not Enough data
The method of measurement often eludes people because they remember things from freshman year stats class like, “That is not a statistically significant sample size.”
There is no such thing as a “statistically significant sample size.” That’s like saying, “50 MPH is too fast.” Too fast for what? Perhaps too fast to drive your car in a school zone, but definitely not fast enough for the expressway, depending on conditions.
Even a sample of one can tell us something and there are a variety of empirical methods to help us fill in the blanks. Asking the right questions leads us to the appropriate methods for analysis.
Go Beyond Data Collection
Our goal is to inform decisions and improve business performance; we aren’t interested in measurement for the sake of measurement. More data isn't always the answer. Go beyond mere data collection, and learn to ask the questions that lead to better measurement.
In our approach to measurement, we apply a variety of quantitative and statistical methods to the most difficult digital problems. We develop and employ models that show a measurable improvement in decision-making over time.
1Douglas Hubbard author How to Measure Anything Willey. 2007.
