I posted a question to Twitter to see what people would do in a more complicated situation than some of the B2C examples you normally see out there. There were some fantastic responses, many of which were thought-provoking and some of which I disagreed with. I posted a follow up with some
thoughts of my own. There’s also an entertaining,
if slightly punchy, write up by
Ibrahim Bashir of Amplitude.
To summarise my position, I don’t believe that performance related metrics are good North Star Metrics for API / platform type products. Whilst it’s fair to say that only an Evil Product Manager™ would deliberately game metrics, you want to avoid unintended effects. There are plenty of dark patterns out there that we might reasonably blame bad metric goals for (e.g. if your priority is to make a number go up, you may employ subterfuge to make it… go up). This is worse still if (god forbid) your North Star Metric is tied to compensation or bonus somehow.
There’s a simple example of this for API products. It’s pretty common for API products to mirror the structure of the database that sits behind them, and this can lead to relatively simple requests being split across numerous endpoints. This is bad UX, but if you’re judging yourself on the number of API calls you won’t want to streamline that experience. If you’re benchmarking yourself on “number go up” then you have no motivation to improve the API UX. If you’re an Evil Product Manager™ then you might even consider splitting perfectly good endpoints up to make your numbers go up.
That’s perhaps a dumb example but I can’t say it’s never happened anywhere! And it speaks to a greater truth… any metric can be abused, we just need to (a) be thoughtful about our metrics (b) change tack if we realise they’re driving bad behaviours.
Leaving aside discussions about whether one North Star Metric is a good idea (it probably isn’t), there’s probably not one answer to the API metric conundrum. For me, I always like to try to boil it down to the Fundamental Unit of Value. I’ve almost certainly stolen that phrase from somewhere, but the basic idea is to try to work out what a successful “transaction” looks like.
This could be an actual transaction, or it could be a lookup of some type of entity that you hold, or some other unit of value that can be delivered to a user. Remember, in many cases your user may not be at all visible to you; we’re talking about a platform product after all. You’re providing value to another platform which is providing it to the user, and you may never see what happens or whether the transaction completes from their end. So you need to think about proxy metrics, and find something good enough.
As usual, the most important thing isn’t to find a perfect anything, but to do your best, keep an eye on it and change it if it’s not working for you. Agile AF!