The tech industry prides itself on being “data-driven”. We’re so data-driven, in fact, that there are hundreds of startups building analytics tools (Segment alone lists over 100) and A/B testing tools (~30). We both laugh at but also secretly admire stories like Google A/B testing 40 shades of blue for its links. A typical consumer-product tech company might be running anywhere from dozens to thousands of A/B tests concurrently, and analyzing thousands of metrics to decide what to launch.
At the surface, it makes sense. “Data doesn’t lie”, we are told. Data, we are promised, will help us overcome our own numerous cognitive biases. It will cut through team and company politics. Depending on who you ask, data is the new oil, data is the new gold, or data is the new religion. Every product decision you make, every OKR you set—must be grounded in data. “You can’t argue with the data!”
I’ve worked at multiple consumer internet companies, and I’ve seen it first hand. I joined the cult and drank the Kool Aid. And, I love data. I’m an engineer at heart. Data works. But like any religion, data can be taken to the extreme, with dangerous consequences.
So I’m not saying we should throw the baby-data-Messiah out with the bathwater. All I’m saying is that data is a tool, and you should use it as such.
Imagine you’re a product manager at a consumer internet company. Your task is to build a landing page to get users to sign up for your product. So you put a lot of valuable information on that page. The conversion rate is low. You run an A/B test with a bunch of variations, and you realize that withholding critical information boosts the sign-up rate. You run more A/B tests. The relationship holds. Less valuable information, more signups. Before your know it, you’re a full-fledged landing page click-bait artist. Your page is shit but you nailed the conversion rate!
“Wait a minute,” you’re saying. This is a problem that can be solved with more data. And yes, you can start measuring downstream metrics like retention, etc, and maybe you learn that tricking your customers into signing up by withholding information results in lower retention. But now you’ve shifted the problem downstream, and what will likely happen is that you (or another product manager) will now be tasked with increasing the downstream retention, and again, the data guides you towards more dark patterns. Because your entire funnel is now grounded in dark patterns. And now any time you actually try to deliver real value to users, your metrics drop.
If this example sounds cartoonish and hard to believe, I assure you I’ve seen it (or something similar) happen multiple times at very respectable companies. We need to understand that data is not a substitute for anything. It’s not a substitute for understanding your customers and their problems. Data is not a substitute for good judgment. Data can actually become a crutch that gets in the way of problem-solving. More data can lead to data hoarding and decisions to the detriment of your customers, your product, and your company.
Data also leads to large, monopolistic consumer internet companies that have lost sight of the problem they’re trying to solve and instead just want to boost their metrics. It also leads to disenchanted employees. You go out and hire the smartest, most passionate people you can find, and turn them into A/B testing monkeys. Initially, they love it—they make changes, they see numbers go up. They get promoted. But they turn off the part of their brain that cares or thinks critically. Data is not a substitute for purpose. Like any shallow gamification, the effect eventually wears off.
Use data as a tool. It is powerful. Don’t use it as a religion. Work with people and companies who understand that. Work with people who are truly focused on solving a problem. Use data to validate the problem and the solutions, but don’t let it lead you blindly.