One element of the United Airlines story intrigues me: how firms start to rely on process so heavily that it becomes an excuse for eliminating employee discretion, common sense or kindness. We know that in the recent case, United didn’t follow its processes exactly but the incident demonstrates the fundamental issue of processes debilitating decency. It’s a small example of the risks of algorithmic management.
The entire process was inevitable. It’s also not a unique situation. We’re going to see much more of this in the future as algorithms and authoritarianism grow
To which I’ll add that while the passenger’s treatment resonates viscerally, we are all, as consumers and citizens, at the mercy of black-box processes (aka algorithms). When we engage with business and government and are fed through a decision tree. When an insurance claim is processed. When we are triaged by a telenurse. When we make a complaint. These are often rigid black-box systems.
The humans operating these processes in many cases have limited discretion. This discretion has been whittled away over years in the name of optimisation or standardisation. And in many cases, that process has led to efficiency, fewer errors and higher quality for us.
But in some instances (especially with quasi-monopolists) a black-box culture that is irredeemably inflexible and opaque. In many industries, the human is already out-of-the-loop, simply following a script. We’ve all heard an agent say “the system doesn’t allow me to do that.”
Will automated systems make this more or less common? The silver lining is that they could be less brittle than current processes. A machine learning system could learn rapidly from experience, ultimately optimising more efficiently and with greater efficacy that an ‘architect-once, implement forever’ process manual. Human overseers could have the power to override such genuinely automated systems if the machines suggest macabre outcomes.
The risk is the opposite will happen because firms will expediently implement poorly designed ones with little consumer redress or business impact.