Customer Obsession Means Listening Harder—Not Measuring Harder
Large businesses use metrics to handle millions of customers. Yet a single customer story can reveal more than a mountain of metrics—if you’re willing to invest some time to learn more.
Welcome to the Scarlet Ink newsletter. I'm Dave Anderson, an ex-Amazon Tech Director and GM. Each week I write a newsletter article on tech industry careers and tactical leadership advice.
Free members can read some amount of each article, while paid members can read the full article. For some, part of the article is plenty! But if you'd like to read more, I'd love you to consider becoming a paid member!
When I have vacations planned, I need to pre-write articles before I go. Then I schedule them to be released at the appropriate times. It’s a bit of a strange feeling to be sitting in a mountain hut with sketchy cell signal and receive a random email that says something like, “Dave, in your most recent article, why was Charles such a jerk? I hate when people act that way!”
I immediately wonder which article just released and what the heck Charles did to prompt such ire. So that’s a moment in the life of Dave.
A couple of weeks ago my wife and I hiked the TMB (Tour du Mont Blanc) over a 7 day period. I’ll write up an article on the trip soon and share a link with all you folks. I don’t intend to release it on this newsletter because you didn’t sign up for trip reports. But I’ll say that it was a wonderful experience. And my photos will be heavy TMB for at least a few weeks. For now they’re from our phones. Perhaps next week we’ll get the camera photos processed.
Getting back on topic, I had a couple of questions recently that got me thinking. The questions revolved around ownership and scaling companies. Scaling requires changing how you work with customers. With 10 customers, you can meet with them in person. With 10 million customers, you learn about them in aggregate.
Metrics and customers at scale
In fact, if you listened to Lenny’s Podcast with Nick Turley, you’d get a picture of what I’m talking about.
I went through that scaling challenge more than once while I was at Amazon. Amazon became a metrics-driven company by necessity. Each leader learned, as their business grew, how to craft their metrics dashboards to look at their customer base from many angles.
For example, let’s imagine you run the Amazon retail app. There are many aspects of your product you’re worried about. One of thousands of aspects you care about might be the speed of loading the homepage.
But how do you look at literal hundreds of millions of page loads and say if it’s fast or not?
Average? That is bloated by outliers.
Slowest? That’ll almost certainly be a weird edge case with someone in Madagascar.
Fastest? That’ll also be an edge case of an employee testing the app while inside an Amazon datacenter.
What you end up doing is creating a wide variety of views into the data. By pivoting your data in a variety of ways, certain things are emphasized.
So you might measure the 10% fastest loads, 10% slowest. The average and median. Latency segmented by region, segmented for logged in vs not, by device type, etc.
What are we doing? We’re creating a scaling proxy for individual customers.
If you had 100 customers, you’d talk to customer 66 in Tahiti, and they’d explain that their page is slow because their internet sucks.
With 100 million customers, you find these subtle problems by carefully pivoting your metrics.
However, that method has limitations. There are individual issues that metrics will never uncover.
Customers can experience issues that fall beneath the radar of these broad scans. You can uncover issues impacting a million customers, but it’s challenging to discover issues impacting 100 customers.
Taken by themselves, a reasonable response might be that 100 customers (at this scale) aren’t worth your time. Yet they’re often a sign of more serious concerns or an area with inadequate monitoring.