Twitter | Search | |
Niels Hoven May 24
1/ Here is a linegraph of my faith in data vs my time as a product manager ---->
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
2/ When I first got into product, "analytics" basically meant counting pageviews. Social gaming powerhouses like Zynga and Playdom were the first to really showcase the potential of metrics based around user actions
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
3/ Take a moment to appreciate how revolutionary this was. Games are some of the most complex software products in existence, and for most of their history, development decisions were made purely on gut instinct
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
4/ This wasn't specific to just games, though. As pageview-based analytics gave way to event-based analytics, new companies emerged that deeply understood their users' needs and built products to fulfill them, rather than simply taking their best guess
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
5/ In this brave new world, metrics were king. Why would you need a designer? Everything could be tracked, measured, tested. A good PM was one who could "move the metrics". MBAs and management consultants were hired by the boatload
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
6/ And then things started to go wrong. Products began to die as quickly as they had grown. Zynga lost 80% of its market cap. "Shark fin graphs" were everywhere as it became clear that virality and hyper-growth couldn't overcome the drag of weak long-term retention
Reply Retweet Like
Niels Hoven
7/ It turned out that metrics weren't king. Retention was king. Metrics were just heuristics, actionable proxies that could be measured on a short time scale as stand-ins for long-term product health that was critical but couldn't be measured
Reply Retweet Like More
Niels Hoven May 24
Replying to @NielsHoven
8/ "Moving the metrics" frequently manifested as increasing short-term, easily measured metrics (like revenue) at the expense of long-term, hard-to-measure metrics (like user lifetimes)
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
9/ "Throw another sale popup in there, the metrics say it's fine!" Generally, this trade-off was done unintentionally. But not always.
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
10/ The bandaid fix that the industry came up with was to say "PMs own metrics" (i.e. the short term) and "designers own the user experience" (i.e. the long term"). Obviously, this was a terrible idea
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
11/ In case it's not obvious why this is a terrible idea, it's because it makes PMs and designers generate lots of bad ideas and then fight about them
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
12/ The industry needed a more fundamental shift in perspective. Good teams now design for the long term, guided by intuition but informed by data.
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
13/ I like to emphasize the difference between data-driven design (relying on data to make decisions because we have no user empathy) and data-informed design (use data to understand our users, then build features to delight them)
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
14/ And oh man I'm just getting started because those are just the problems that happen when all the metrics are accurate. Spoiler: the metrics are never accurate
Reply Retweet Like
Niels Hoven May 24
Replying to @NielsHoven
15/ Making good decisions based on bad data is one of my favorite topics. More on that tomorrow
Reply Retweet Like
Trevin Chow May 24
Replying to @NielsHoven
If retention is king, which I agree with, why isn’t data on retention important?
Reply Retweet Like
Niels Hoven May 24
Replying to @trevin
I probably should have said something like “user lifetimes are king”. Retention data is important but even so, still just as a proxy for the app’s long term health
Reply Retweet Like
Niels Hoven May 24
Replying to @trevin
Retention is usually strongly correlated with product health, so hurting your product too boost retention is a lot less common than hurting your product to boost revenue. But still happens
Reply Retweet Like
Trevin Chow May 24
Replying to @NielsHoven
I’ve never personally seen a bad product/business decision that boosts retention over a legitimate amount of time. You can’t fake it. I have seen it for median session time though which is why I always use the latter as a diagnostic
Reply Retweet Like
Niels Hoven May 24
Replying to @trevin
Saying “over a legit amount of time” is a cop-out because optimizing for any short-term metric always fails over the long term.
Reply Retweet Like